azaion/gps-denied-desktop
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-desktop.git synced 2026-04-25 12:26:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

remove the current solution, add skills

Browse Source
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-03-14 18:37:48 +02:00
parent fd75243a84
commit 767874cb90
363 changed files with 6057 additions and 36380 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/00_problem/1.2_research_prompt.md
-64
View File
@@ -1,64 +0,0 @@
Research this problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
The system should process data samples in the attached files (if any). They are for reference only.
- We have the next restrictions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168
- Altitude is predefined and no more than 1km
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
- Output of our system should meet these acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%
- The number of outliers during the satellite provider images ground check should be less than 10%
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
- Find out all the state-of-the-art solutions for this problem and produce the resulting solution draft in the next format:
- Short Product solution description. Brief component interaction diagram.
- Architecture approach that meets restrictions and acceptance criteria. For each component, analyze the best possible approaches to solve, and form a table comprising all approaches. Each new approach would be a row, and has the next columns:
- Tools (library, platform) to solve component tasks
- Advantages of this approach
- Limitations of this approach
- Requirements for this approach
- How does it fit for the problem component that has to be solved, and the whole solution
- Testing strategy. Research the best approaches to cover all the acceptance criteria. Form a list of integration functional tests and non-functional tests.
Be concise in formulating. The fewer words, the better, but do not miss any important details.
_docs/00_problem/1.3_01_assesment_prompt.md
-329
View File
@@ -1,329 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%
- The number of outliers during the satellite provider images ground check should be less than 10%
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
Here is a solution draft:
# **GEo-Referenced Trajectory and Object Localization System (GEORTOLS): A Hybrid SLAM Architecture**
## **1. Executive Summary**
This report outlines the technical design for a robust, real-time geolocalization system. The objective is to determine the precise GPS coordinates for a sequence of high-resolution images (up to 6252x4168) captured by a fixed-wing, non-stabilized Unmanned Aerial Vehicle (UAV) [User Query]. The system must operate under severe constraints, including the absence of any IMU data, a predefined altitude of no more than 1km, and knowledge of only the starting GPS coordinate [User Query]. The system is required to handle significant in-flight challenges, such as sharp turns with minimal image overlap (<5%), frame-to-frame outliers of up to 350 meters, and operation over low-texture terrain as seen in the provided sample images [User Query, Image 1, Image 7].
The proposed solution is a **Hybrid Visual-Geolocalization SLAM (VG-SLAM)** architecture. This system is designed to meet the demanding acceptance criteria, including a sub-5-second initial processing time per image, streaming output with asynchronous refinement, and high-accuracy GPS localization (60% of photos within 20m error, 80% within 50m error) [User Query].
This hybrid architecture is necessitated by the problem's core constraints. The lack of an IMU makes a purely monocular Visual Odometry (VO) system susceptible to catastrophic scale drift.1 Therefore, the system integrates two cooperative sub-systems:
1. A **Visual Odometry (VO) Front-End:** This component uses state-of-the-art deep-learning feature matchers (SuperPoint + SuperGlue/LightGlue) to provide fast, real-time *relative* pose estimates. This approach is selected for its proven robustness in low-texture environments where traditional features fail.4 This component delivers the initial, sub-5-second pose estimate.
2. A **Cross-View Geolocalization (CVGL) Module:** This component provides *absolute*, drift-free GPS pose estimates by matching UAV images against the available satellite provider (Google Maps).7 It functions as the system's "global loop closure" mechanism, correcting the VO's scale drift and, critically, relocalizing the UAV after tracking is lost during sharp turns or outlier frames [User Query].
These two systems run in parallel. A **Back-End Pose-Graph Optimizer** fuses their respective measurements—high-frequency relative poses from VO and high-confidence absolute poses from CVGL—into a single, globally consistent, and incrementally refined trajectory. This architecture directly satisfies the requirements for immediate, streaming results and subsequent asynchronous refinement [User Query].
## **2. Product Solution Description and Component Interaction**
### **Product Solution Description**
The proposed system, "GEo-Referenced Trajectory and Object Localization System (GEORTOLS)," is a real-time, streaming-capable software solution. It is designed for deployment on a stationary computer or laptop equipped with an NVIDIA GPU (RTX 2060 or better) [User Query].
* **Inputs:**
1. A sequence of consecutively named monocular images (FullHD to 6252x4168).
2. The absolute GPS coordinate (Latitude, Longitude) of the *first* image in the sequence.
3. A pre-calibrated camera intrinsic matrix.
4. Access to the Google Maps satellite imagery API.
* **Outputs:**
1. A real-time, streaming feed of estimated GPS coordinates (Latitude, Longitude, Altitude) and 6-DoF poses (including Roll, Pitch, Yaw) for the center of each image.
2. Asynchronous refinement messages for previously computed poses as the back-end optimizer improves the global trajectory.
3. A service to provide the absolute GPS coordinate for any user-selected pixel coordinate (u,v) within any geolocated image.
### **Component Interaction Diagram**
The system is architected as four asynchronous, parallel-processing components to meet the stringent real-time and refinement requirements.
1. **Image Ingestion & Pre-processing:** This module acts as the entry point. It receives the new, high-resolution image (Image N). It immediately creates scaled-down, lower-resolution (e.g., 1024x768) copies of the image for real-time processing by the VO and CVGL modules, while retaining the full-resolution original for object-level GPS lookups.
2. **Visual Odometry (VO) Front-End:** This module's sole task is high-speed, frame-to-frame relative pose estimation. It maintains a short-term "sliding window" of features, matching Image N to Image N-1. It uses GPU-accelerated deep-learning models (SuperPoint + SuperGlue) to find feature matches and calculates the 6-DoF relative transform. This result is immediately sent to the Back-End.
3. **Cross-View Geolocalization (CVGL) Module:** This is a heavier, slower, asynchronous module. It takes the pre-processed Image N and queries the Google Maps database to find an *absolute* GPS pose. This involves a two-stage retrieval-and-match process. When a high-confidence match is found, its absolute pose is sent to the Back-End as a "global-pose constraint."
4. **Trajectory Optimization Back-End:** This is the system's central "brain," managing the complete pose graph.10 It receives two types of data:
* *High-frequency, low-confidence relative poses* from the VO Front-End.
* Low-frequency, high-confidence absolute poses from the CVGL Module.
It continuously fuses these constraints in a pose-graph optimization framework (e.g., g2o or Ceres Solver). When the VO Front-End provides a new relative pose, it is quickly added to the graph to produce the "Initial Pose" (<5s). When the CVGL Module provides a new absolute pose, it triggers a more comprehensive re-optimization of the entire graph, correcting drift and broadcasting "Refined Poses" to the user.11
## **3. Core Architectural Framework: Hybrid Visual-Geolocalization SLAM (VG-SLAM)**
### **Rationale for the Hybrid Approach**
The core constraints of this problem—monocular, IMU-less flight over potentially long distances (up to 3000 images at \~100m intervals equates to a 300km flight) [User Query]—render simple solutions unviable.
A **VO-Only** system is guaranteed to fail. Monocular Visual Odometry (and SLAM) suffers from an inherent, unobservable ambiguity: the *scale* of the world.1 Because there is no IMU to provide an accelerometer-based scale reference or a gravity vector 12, the system has no way to know if it moved 1 meter or 10 meters. This leads to compounding scale drift, where the entire trajectory will grow or shrink over time.3 Over a 300km flight, the resulting positional error would be measured in kilometers, not the 20-50 meters required [User Query].
A **CVGL-Only** system is also unviable. Cross-View Geolocalization (CVGL) matches the UAV image to a satellite map to find an absolute pose.7 While this is drift-free, it is a large-scale image retrieval problem. Querying the entire map of Ukraine for a match for every single frame is computationally impossible within the <5 second time limit.13 Furthermore, this approach is brittle; if the Google Maps data is outdated (a specific user restriction) [User Query], the CVGL match will fail, and the system would have no pose estimate at all.
Therefore, the **Hybrid VG-SLAM** architecture is the only robust solution.
* The **VO Front-End** provides the fast, high-frequency relative motion. It works even if the satellite map is outdated, as it tracks features in the *real*, current world.
* The **CVGL Module** acts as the *only* mechanism for scale correction and absolute georeferencing. It provides periodic, drift-free "anchors" to the real-world GPS coordinates.
* The **Back-End Optimizer** fuses these two data streams. The CVGL poses function as "global loop closures" in the SLAM pose graph. They correct the scale drift accumulated by the VO and, critically, serve to relocalize the system after a "kidnapping" event, such as the specified sharp turns or 350m outliers [User Query].
### **Data Flow for Streaming and Refinement**
This architecture is explicitly designed to meet the <5s initial output and asynchronous refinement criteria [User Query]. The data flow for a single image (Image N) is as follows:
* **T \= 0.0s:** Image N (6200x4100) is received by the **Ingestion Module**.
* **T \= 0.2s:** Image N is pre-processed (scaled to 1024px) and passed to the VO and CVGL modules.
* **T \= 1.0s:** The **VO Front-End** completes GPU-accelerated matching (SuperPoint+SuperGlue) of Image N -> Image N-1. It computes the Relative_Pose(N-1 -> N).
* **T \= 1.1s:** The **Back-End Optimizer** receives this Relative_Pose. It appends this pose to the graph relative to the last known pose of N-1.
* **T \= 1.2s:** The Back-End broadcasts the **Initial Pose_N_Est** to the user interface. (**<5s criterion met**).
* **(Parallel Thread) T \= 1.5s:** The **CVGL Module** (on a separate thread) begins its two-stage search for Image N against the Google Maps database.
* **(Parallel Thread) T \= 6.0s:** The CVGL Module successfully finds a high-confidence Absolute_Pose_N_Abs from the satellite match.
* **T \= 6.1s:** The **Back-End Optimizer** receives this new, high-confidence absolute constraint for Image N.
* **T \= 6.2s:** The Back-End triggers a graph re-optimization. This new "anchor" corrects any scale or positional drift for Image N and all surrounding poses in the graph.
* **T \= 6.3s:** The Back-End broadcasts a **Pose_N_Refined** (and Pose_N-1_Refined, Pose_N-2_Refined, etc.) to the user interface. (**Refinement criterion met**).
## **4. Component Analysis: Front-End (Visual Odometry and Relocalization)**
The task of the VO Front-End is to rapidly and robustly estimate the 6-DoF relative motion between consecutive frames. This component's success is paramount for the high-frequency tracking required to meet the <5s criterion.
The primary challenge is the nature of the imagery. The specified operational area and sample images (e.g., Image 1, Image 7) show vast, low-texture agricultural fields [User Query]. These environments are a known failure case for traditional, gradient-based feature extractors like SIFT or ORB, which rely on high-gradient corners and cannot find stable features in "weak texture areas".5 Furthermore, the non-stabilized camera [User Query] will introduce significant rotational motion and viewpoint change, breaking the assumptions of many simple trackers.16
Deep-learning (DL) based feature extractors and matchers have been developed specifically to overcome these "challenging visual conditions".5 Models like SuperPoint, SuperGlue, and LoFTR are trained to find more robust and repeatable features, even in low-texture scenes.4
### **Table 1: Analysis of State-of-the-Art Feature Extraction and Matching Techniques**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SIFT + BFMatcher/FLANN** (OpenCV) | - Scale and rotation invariant. - High-quality, robust matches. - Well-studied and mature.15 | - Computationally slow (CPU-based). - Poor performance in low-texture or weakly-textured areas.14 - Patented (though expired). | - High-contrast, well-defined features. | **Poor.** Too slow for the <5s target and will fail to find features in the low-texture agricultural landscapes shown in sample images. |
| **ORB + BFMatcher** (OpenCV) | - Extremely fast and lightweight. - Standard for real-time SLAM (e.g., ORB-SLAM).21 - Rotation invariant. | - *Not* scale invariant (uses a pyramid). - Performs very poorly in low-texture scenes.5 - Unstable in high-blur scenarios. | - CPU, lightweight. - High-gradient corners. | **Very Poor.** While fast, it fails on the *robustness* requirement. It is designed for textured, indoor/urban scenes, not sparse, natural terrain. |
| **SuperPoint + SuperGlue** (PyTorch, C++/TensorRT) | - SOTA robustness in low-texture, high-blur, and challenging conditions.4 - End-to-end learning for detection and matching.24 - Multiple open-source SLAM integrations exist (e.g., SuperSLAM).25 | - Requires a powerful GPU for real-time performance. - Sparse feature-based (not dense). | - NVIDIA GPU (RTX 2060+). - PyTorch (research) or TensorRT (deployment).26 | **Excellent.** This approach is *designed* for the exact "challenging conditions" of this problem. It provides SOTA robustness in low-texture scenes.4 The user's hardware (RTX 2060+) meets the requirements. |
| **LoFTR** (PyTorch) | - Detector-free dense matching.14 - Extremely robust to viewpoint and texture challenges.14 - Excellent performance on natural terrain and low-overlap images.19 | - High computational and VRAM cost. - Can cause CUDA Out-of-Memory (OOM) errors on very high-resolution images.30 - Slower than sparse-feature methods. | - High-end NVIDIA GPU. - PyTorch. | **Good, but Risky.** While its robustness is excellent, its dense, Transformer-based nature makes it vulnerable to OOM errors on the 6252x4168 images.30 The sparse SuperPoint approach is a safer, more-scalable choice for the VO front-end. |
### **Selected Approach (VO Front-End): SuperPoint + SuperGlue/LightGlue**
The selected approach is a VO front-end based on **SuperPoint** for feature extraction and **SuperGlue** (or its faster successor, **LightGlue**) for matching.18
* **Robustness:** This combination is proven to provide superior robustness and accuracy in sparse-texture scenes, extracting more and higher-quality matches than ORB.4
* **Performance:** It is designed for GPU acceleration and is used in SOTA real-time SLAM systems, demonstrating its feasibility within the <5s target on an RTX 2060.25
* **Scalability:** As a sparse-feature method, it avoids the memory-scaling issues of dense matchers like LoFTR when faced with the user's maximum 6252x4168 resolution.30 The image can be downscaled for real-time VO, and SuperPoint will still find stable features.
## **5. Component Analysis: Back-End (Trajectory Optimization and Refinement)**
The task of the Back-End is to fuse all incoming measurements (high-frequency/low-accuracy relative VO poses, low-frequency/high-accuracy absolute CVGL poses) into a single, globally consistent trajectory. This component's design is dictated by the user's real-time streaming and refinement requirements [User Query].
A critical architectural choice must be made between a traditional, batch **Structure from Motion (SfM)** pipeline and a real-time **SLAM (Simultaneous Localization and Mapping)** pipeline.
* **Batch SfM:** (e.g., COLMAP).32 This approach is an offline process. It collects all 1500-3000 images, performs feature matching, and then runs a large, non-real-time "Bundle Adjustment" (BA) to solve for all camera poses and 3D points simultaneously.35 While this produces the most accurate possible result, it can take hours to compute. It *cannot* meet the <5s/image or "immediate results" criteria.
* **Real-time SLAM:** (e.g., ORB-SLAM3).28 This approach is *online* and *incremental*. It maintains a "pose graph" of the trajectory.10 It provides an immediate pose estimate based on the VO front-end. When a new, high-quality measurement arrives (like a loop closure 37, or in our case, a CVGL fix), it triggers a fast re-optimization of the graph, publishing a *refined* result.11
The user's requirements for "results...appear immediately" and "system could refine existing calculated results" [User Query] are a textbook description of a real-time SLAM back-end.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (g2o, Ceres Solver, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates. - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (CVGL) data arrives.10 - Meets the <5s and streaming criteria. | - Initial estimate is less accurate than a full batch process. - Susceptible to drift *until* a loop closure (CVGL fix) is made. | - A graph optimization library (g2o, Ceres). - A robust cost function to reject outliers. | **Excellent.** This is the *only* architecture that satisfies the user's real-time streaming and asynchronous refinement constraints. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory.35 - Can import custom DL matches.38 | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails all timing and streaming criteria. | - All images must be available before processing starts. - High RAM and CPU. | **Unsuitable (for the *online* system).** This approach is ideal for an *optional, post-flight, high-accuracy* refinement, but it cannot be the primary system. |
### **Selected Approach (Back-End): Incremental Pose-Graph Optimization (g2o/Ceres)**
The system's back-end will be built as an **Incremental Pose-Graph Optimizer** using a library like **g2o** or **Ceres Solver**. This is the only way to meet the real-time streaming and refinement constraints [User Query].
The graph will contain:
* **Nodes:** The 6-DoF pose of each camera frame.
* **Edges (Constraints):**
1. **Odometry Edges:** Relative 6-DoF transforms from the VO Front-End (SuperPoint+SuperGlue). These are high-frequency but have accumulating drift/scale error.
2. **Georeferencing Edges:** Absolute 6-DoF poses from the CVGL Module. These are low-frequency but are drift-free and provide the absolute scale.
3. **Start-Point Edge:** A high-confidence absolute pose for Image 1, fixed to the user-provided start GPS.
This architecture allows the system to provide an immediate estimate (from odometry) and then drastically improve its accuracy (correcting scale and drift) whenever a new georeferencing edge is added.
## **6. Component Analysis: Global-Pose Correction (Georeferencing Module)**
This module is the most critical component for meeting the accuracy requirements. Its task is to provide absolute GPS pose estimates by matching the UAV's nadir-pointing-but-non-stabilized images to the Google Maps satellite provider [User Query]. This is the only component that can correct the monocular scale drift.
This task is known as **Cross-View Geolocalization (CVGL)**.7 It is extremely challenging due to the "domain gap" 44 between the two image sources:
1. **Viewpoint:** The UAV is at low altitude (<1km) and non-nadir (due to fixed-wing tilt) 45, while the satellite is at a very high altitude and is perfectly nadir.
2. **Appearance:** The images come from different sensors, with different lighting (shadows), and at different times. The Google Maps data may be "outdated" [User Query], showing different seasons, vegetation, or man-made structures.47
A simple, brute-force feature match is computationally impossible. The solution is a **hierarchical, two-stage approach** that mimics SOTA research 7:
* **Stage 1: Coarse Retrieval.** We cannot run expensive matching against the entire map. Instead, we treat this as an image retrieval problem. We use a Deep Learning model (e.g., a Siamese or Dual CNN trained on this task 50) to generate a compact "embedding vector" (a digital signature) for the UAV image. In an offline step, we pre-compute embeddings for *all* satellite map tiles in the operational area. The UAV image's embedding is then used to perform a very fast (e.g., FAISS library) similarity search against the satellite database, returning the Top-K most likely-matching satellite tiles.
* **Stage 2: Fine-Grained Pose.** *Only* for these Top-K candidates do we perform the heavy-duty feature matching. We use our selected **SuperPoint+SuperGlue** matcher 53 to find precise correspondences between the UAV image and the K satellite tiles. If a high-confidence geometric match (e.g., >50 inliers) is found, we can compute the precise 6-DoF pose of the UAV relative to that tile, thus yielding an absolute GPS coordinate.
### **Table 3: Analysis of State-of-the-Art Cross-View Geolocalization (CVGL) Techniques**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Coarse Retrieval (Siamese/Dual CNNs)** (PyTorch, ResNet18) | - Extremely fast for retrieval (database lookup). - Learns features robust to seasonal and appearance changes.50 - Narrows search space from millions to a few. | - Does *not* provide a precise 6-DoF pose, only a "best match" tile. - Requires training on a dataset of matched UAV-satellite pairs. | - Pre-trained model (e.g., on ResNet18).52 - Pre-computed satellite embedding database. | **Essential (as Stage 1).** This is the only computationally feasible way to "find" the UAV on the map. |
| **Fine-Grained Feature Matching** (SuperPoint + SuperGlue) | - Provides a highly-accurate 6-Dof pose estimate.53 - Re-uses the same robust matcher from the VO Front-End.54 | - Too slow to run on the entire map. - *Requires* a good initial guess (from Stage 1) to be effective. | - NVIDIA GPU. - Top-K candidate tiles from Stage 1. | **Essential (as Stage 2).** This is the component that actually computes the precise GPS pose from the coarse candidates. |
| **End-to-End DL Models (Transformers)** (PFED, ReCOT, etc.) | - SOTA accuracy in recent benchmarks.13 - Can be highly efficient (e.g., PFED).13 - Can perform retrieval and pose estimation in one model. | - Often research-grade, not robustly open-sourced. - May be complex to train and deploy. - Less modular and harder to debug than the two-stage approach. | - Specific, complex model architectures.13 - Large-scale training datasets. | **Not Recommended (for initial build).** While powerful, these are less practical for a version 1 build. The two-stage approach is more modular, debuggable, and uses components already required by the VO system. |
### **Selected Approach (CVGL Module): Hierarchical Retrieval + Matching**
The CVGL module will be implemented as a two-stage hierarchical system:
1. **Stage 1 (Coarse):** A **Siamese CNN** 52 (or similar model) generates an embedding for the UAV image. This embedding is used to retrieve the Top-5 most similar satellite tiles from a pre-computed database.
2. **Stage 2 (Fine):** The **SuperPoint+SuperGlue** matcher 53 is run between the UAV image and these 5 tiles. The match with the highest inlier count and lowest reprojection error is used to calculate the absolute 6-DoF pose, which is then sent to the Back-End optimizer.
## **7. Addressing Critical Acceptance Criteria and Failure Modes**
This hybrid architecture's logic is designed to handle the most difficult acceptance criteria [User Query] through a robust, multi-stage escalation process.
### **Stage 1: Initial State (Normal Operation)**
* **Condition:** VO(N-1 -> N) succeeds.
* **System Logic:** The **VO Front-End** provides the high-frequency relative pose. This is added to the graph, and the **Initial Pose** is sent to the user (<5s).
* **Resolution:** The **CVGL Module** runs asynchronously to provide a Refined Pose later, which corrects for scale drift.
### **Stage 2: Transient Failure / Outlier Handling (AC-3)**
* **Condition:** VO(N-1 -> N) fails (e.g., >350m jump, severe motion blur, low overlap) [User Query]. This triggers an immediate, high-priority CVGL(N) query.
* **System Logic:**
1. If CVGL(N) *succeeds*, the system has conflicting data: a failed VO link and a successful CVGL pose. The **Back-End Optimizer** uses a robust kernel to reject the high-error VO link as an outlier and accepts the CVGL pose.56 The trajectory "jumps" to the correct location, and VO resumes from Image N+1.
2. If CVGL(N) *also fails* (e.g., due to cloud cover or outdated map), the system assumes Image N is a single bad frame (an outlier).
* **Resolution (Frame Skipping):** The system buffers Image N and, upon receiving Image N+1, the **VO Front-End** attempts to "bridge the gap" by matching VO(N-1 -> N+1).
* **If successful,** a pose for N+1 is found. Image N is marked as a rejected outlier, and the system continues.
* **If VO(N-1 -> N+1) fails,** it repeats for VO(N-1 -> N+2).
* If this "bridging" fails for 3 consecutive frames, the system concludes it is not a transient outlier but a persistent tracking loss. This escalates to Stage 3.
### **Stage 3: Persistent Tracking Loss / Sharp Turn Handling (AC-4)**
* **Condition:** VO tracking is lost, and the "frame-skipping" in Stage 2 fails (e.g., a "sharp turn" with no overlap) [User Query].
* **System Logic (Multi-Map "Chunking"):** The **Back-End Optimizer** declares a "Tracking Lost" state and creates a *new, independent map* ("Chunk 2").
* The **VO Front-End** is re-initialized and begins populating this new chunk, tracking VO(N+3 -> N+4), VO(N+4 -> N+5), etc. This new chunk is internally consistent but has no absolute GPS position (it is "floating").
* **Resolution (Asynchronous Relocalization):**
1. The **CVGL Module** now runs asynchronously on all frames in this new "Chunk 2".
2. Crucially, it uses the last known GPS coordinate from "Chunk 1" as a *search prior*, narrowing the satellite map search area to the vicinity.
3. The system continues to build Chunk 2 until the CVGL module successfully finds a high-confidence Absolute_Pose for *any* frame in that chunk (e.g., for Image N+20).
4. Once this single GPS "anchor" is found, the **Back-End Optimizer** performs a full graph optimization. It calculates the 7-DoF transformation (3D position, 3D rotation, and **scale**) to align all of Chunk 2 and merge it with Chunk 1.
5. This "chunking" method robustly handles the "correctly continue the work" criterion by allowing the system to keep tracking locally even while globally lost, confident it can merge the maps later.
### **Stage 4: Catastrophic Failure / User Intervention (AC-6)**
* **Condition:** The system has entered Stage 3 and is building "Chunk 2," but the **CVGL Module** has *also* failed for a prolonged period (e.g., 20% of the route, or 50+ consecutive frames) [User Query]. This is a "worst-case" scenario where the UAV is in an area with no VO features (e.g., over a lake) *and* no CVGL features (e.g., heavy clouds or outdated maps).
* **System Logic:** The system is "absolutely incapable" of determining its pose.
* **Resolution (User Input):** The system triggers the "ask the user for input" event. A UI prompt will show the last known good image (from Chunk 1) on the map and the new, "lost" image (e.g., N+50). It will ask the user to "Click on the map to provide a coarse location." This user-provided GPS point is then fed to the CVGL module as a *strong prior*, drastically narrowing the search space and enabling it to re-acquire a lock.
## **8. Implementation and Output Generation**
### **Real-time Workflow (<5s Initial, Async Refinement)**
A concrete implementation plan for processing Image N:
1. **T=0.0s:** Image[N] (6200px) received.
2. **T=0.1s:** Image pre-processed: Scaled to 1024px for VO/CVGL. Full-res original stored.
3. **T=0.5s:** **VO Front-End** (GPU): SuperPoint features extracted for 1024px image.
4. **T=1.0s:** **VO Front-End** (GPU): SuperGlue matches 1024px Image[N] -> 1024px Image[N-1]. Relative_Pose (6-DoF) estimated via RANSAC/PnP.
5. **T=1.1s:** **Back-End:** Relative_Pose added to graph. Optimizer updates trajectory.
6. **T=1.2s:** **OUTPUT:** Initial Pose_N_Est (GPS) sent to user. **(<5s criterion met)**.
7. **T=1.3s:** **CVGL Module (Async Task)** (GPU): Siamese/Dual CNN generates embedding for 1024px Image[N].
8. **T=1.5s:** **CVGL Module (Async Task):** Coarse retrieval (FAISS lookup) returns Top-5 satellite tile candidates.
9. **T=4.0s:** **CVGL Module (Async Task)** (GPU): Fine-grained matching. SuperPoint+SuperGlue runs 5 times (Image[N] vs. 5 satellite tiles).
10. **T=4.5s:** **CVGL Module (Async Task):** A high-confidence match is found. Absolute_Pose_N_Abs (6-DoF) is computed.
11. **T=4.6s:** **Back-End:** High-confidence Absolute_Pose_N_Abs added to pose graph. Graph re-optimization is triggered.
12. **T=4.8s:** **OUTPUT:** Pose_N_Refined (GPS) sent to user. **(Refinement criterion met)**.
### **Determining Object-Level GPS (from Pixel Coordinate)**
The requirement to find the "coordinates of the center of any object in these photos" [User Query] is met by projecting a pixel to its 3D world coordinate. This requires the (u,v) pixel, the camera's 6-DoF pose, and the camera's intrinsic matrix (K).
Two methods will be implemented to support the streaming/refinement architecture:
1. **Method 1 (Immediate, <5s): Flat-Earth Projection.**
* When the user clicks pixel (u,v) on Image[N], the system uses the *Initial Pose_N_Est*.
* It assumes the ground is a flat plane at the predefined altitude (e.g., 900m altitude if flying at 1km and ground is at 100m) [User Query].
* It computes the 3D ray from the camera center through (u,v) using the intrinsic matrix (K).
* It calculates the 3D intersection point of this ray with the flat ground plane.
* This 3D world point is converted to a GPS coordinate and sent to the user. This is very fast but less accurate in non-flat terrain.
2. **Method 2 (Refined, Post-BA): Structure-from-Motion Projection.**
* The Back-End's pose-graph optimization, as a byproduct, will create a sparse 3D point cloud of the world (i.e., the "SfM" part of SLAM).35
* When the user clicks (u,v), the system uses the *Pose_N_Refined*.
* It raycasts from the camera center through (u,v) and finds the 3D intersection point with the *actual 3D point cloud* generated by the system.
* This 3D point's coordinate (X,Y,Z) is converted to GPS. This is far more accurate as it accounts for real-world topography (hills, ditches) captured in the 3D map.
## **9. Testing and Validation Strategy**
A rigorous testing strategy is required to validate all 10 acceptance criteria. The foundation of this strategy is the creation of a **Ground-Truth Test Dataset**. This will involve flying several test routes and manually creating a "checkpoint" (CP) file, similar to the provided coordinates.csv 58, using a high-precision RTK/PPK GPS. This provides the "real GPS" for validation.59
### **Accuracy Validation Methodology (AC-1, AC-2, AC-5, AC-8, AC-9)**
These tests validate the system's accuracy and completion metrics.59
1. A test flight of 1000 images with high-precision ground-truth CPs is prepared.
2. The system is run given only the first GPS coordinate.
3. A test script compares the system's *final refined GPS output* for each image against its *ground-truth CP*. The Haversine distance (error in meters) is calculated for all 1000 images.
4. This yields a list of 1000 error values.
5. **Test_Accuracy_50m (AC-1):** ASSERT (count(errors < 50m) / 1000) >= 0.80
6. **Test_Accuracy_20m (AC-2):** ASSERT (count(errors < 20m) / 1000) >= 0.60
7. **Test_Outlier_Rate (AC-5):** ASSERT (count(un-localized_images) / 1000) < 0.10
8. **Test_Image_Registration_Rate (AC-8):** ASSERT (count(localized_images) / 1000) > 0.95
9. **Test_Mean_Reprojection_Error (AC-9):** ASSERT (Back-End.final_MRE) < 1.0
10. **Test_RMSE:** The overall Root Mean Square Error (RMSE) of the entire trajectory will be calculated as a primary performance benchmark.59
### **Integration and Functional Tests (AC-3, AC-4, AC-6)**
These tests validate the system's logic and robustness to failure modes.62
* Test_Low_Overlap_Relocalization (AC-4):
* **Setup:** Create a test sequence of 50 images. From this, manually delete images 20-24 (simulating 5 lost frames during a sharp turn).63
* **Test:** Run the system on this "broken" sequence.
* **Pass/Fail:** The system must report "Tracking Lost" at frame 20, initiate a new "chunk," and then "Tracking Re-acquired" and "Maps Merged" when the CVGL module successfully localizes frame 25 (or a subsequent frame). The final trajectory error for frame 25 must be < 50m.
* Test_350m_Outlier_Rejection (AC-3):
* **Setup:** Create a test sequence. At image 30, insert a "rogue" image (Image 30b) known to be 350m away.
* **Test:** Run the system on this sequence (..., 29, 30, 30b, 31,...).
* **Pass/Fail:** The system must correctly identify Image 30b as an outlier (RANSAC failure 56), reject it (or jump to its CVGL-verified pose), and "correctly continue the work" by successfully tracking Image 31 from Image 30 (using the frame-skipping logic). The trajectory must not be corrupted.
* Test_User_Intervention_Prompt (AC-6):
* **Setup:** Create a test sequence with 50 consecutive "bad" frames (e.g., pure sky, lens cap) to ensure the transient and chunking logics are bypassed.
* **Test:** Run the system.
* **Pass/Fail:** The system must enter a "LOST" state, attempt and fail to relocalize via CVGL for 50 frames, and then correctly trigger the "ask for user input" event.
### **Non-Functional Tests (AC-7, AC-8, Hardware)**
These tests validate performance and resource requirements.66
* Test_Performance_Per_Image (AC-7):
* **Setup:** Run the 1000-image test set on the minimum-spec RTX 2060.
* **Test:** Measure the time from "Image In" to "Initial Pose Out" for every frame.
* **Pass/Fail:** ASSERT average_time < 5.0s.
* Test_Streaming_Refinement (AC-8):
* **Setup:** Run the 1000-image test set.
* **Test:** A logger must verify that *two* poses are received for >80% of images: an "Initial" pose (T < 5s) and a "Refined" pose (T > 5s, after CVGL).
* **Pass/Fail:** The refinement mechanism is functioning correctly.
* Test_Scalability_Large_Route (Constraints):
* **Setup:** Run the system on a full 3000-image dataset.
* **Test:** Monitor system RAM, VRAM, and processing time per frame over the entire run.
* **Pass/Fail:** The system must complete the run without memory leaks, and the processing time per image must not degrade significantly as the pose graph grows.
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report. Put here all new findings, what was updated, replaced, or removed from the previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
_docs/00_problem/1.3_02_assesment_prompt.md
-325
View File
@@ -1,325 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%
- The number of outliers during the satellite provider images ground check should be less than 10%
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
Here is a solution draft:
# **GEORTOLS-SA UAV Image Geolocalization in IMU-Denied Environments**
The GEORTOLS-SA system is an asynchronous, four-component software solution designed for deployment on an NVIDIA RTX 2060+ GPU. It is architected from the ground up to handle the specific challenges of IMU-denied, scale-aware localization and real-time streaming output.
### **Product Solution Description**
* **Inputs:**
1. A sequence of consecutively named images (FullHD to 6252x4168).
2. The absolute GPS coordinate (Latitude, Longitude) for the first image (Image 0).
3. A pre-calibrated camera intrinsic matrix ($K$).
4. The predefined, absolute metric altitude of the UAV ($H$, e.g., 900 meters).
5. API access to the Google Maps satellite provider.
* **Outputs (Streaming):**
1. **Initial Pose (T \< 5s):** A high-confidence, *metric-scale* estimate ($Pose\_N\_Est$) of the image's 6-DoF pose and GPS coordinate. This is sent to the user immediately upon calculation (AC-7, AC-8).
2. **Refined Pose (T > 5s):** A globally-optimized pose ($Pose\_N\_Refined$) sent asynchronously as the back-end optimizer fuses data from the CVGL module (AC-8).
### **Component Interaction Diagram and Data Flow**
The system is architected as four parallel-processing components to meet the stringent real-time and refinement requirements.
1. **Image Ingestion & Pre-processing:** This module receives the new, high-resolution Image_N. It immediately creates two copies:
* Image_N_LR (Low-Resolution, e.g., 1536x1024): This copy is immediately dispatched to the SA-VO Front-End for real-time processing.
* Image_N_HR (High-Resolution, 6.2K): This copy is stored and made available to the CVGL Module for its asynchronous, high-accuracy matching pipeline.
2. **Scale-Aware VO (SA-VO) Front-End (High-Frequency Thread):** This component's sole task is high-speed, *metric-scale* relative pose estimation. It matches Image_N_LR to Image_N-1_LR, computes the 6-DoF relative transform, and critically, uses the "known altitude" ($H$) constraint to recover the absolute scale (detailed in Section 3.0). It sends this high-confidence Relative_Metric_Pose to the Back-End.
3. **Cross-View Geolocalization (CVGL) Module (Low-Frequency, Asynchronous Thread):** This is a heavier, slower module. It takes Image_N (both LR and HR) and queries the Google Maps database to find an *absolute GPS pose*. When a high-confidence match is found, its Absolute_GPS_Pose is sent to the Back-End as a global "anchor" constraint.
4. **Trajectory Optimization Back-End (Central Hub):** This component manages the complete flight trajectory as a pose graph.10 It continuously fuses two distinct, high-quality data streams:
* **On receiving Relative_Metric_Pose (T \< 5s):** It appends this pose to the graph, calculates the Pose_N_Est, and **sends this initial result to the user (AC-7, AC-8 met)**.
* **On receiving Absolute_GPS_Pose (T > 5s):** It adds this as a high-confidence "global anchor" constraint 12, triggers a full graph re-optimization to correct any minor biases, and **sends the Pose_N_Refined to the user (AC-8 refinement met)**.
###
### **VO "Trust Model" of GEORTOLS-SA**
In GEORTOLS-SA, the trust model:
* The **SA-VO Front-End** is now *highly trusted* for its local, frame-to-frame *metric* accuracy.
* The **CVGL Module** remains *highly trusted* for its *global* (GPS) accuracy.
Both components are operating in the same scale-aware, metric space. The Back-End's job is no longer to fix a broken, drifting VO. Instead, it performs a robust fusion of two independent, high-quality metric measurements.12
This model is self-correcting. If the user's predefined altitude $H$ is slightly incorrect (e.g., entered as 900m but is truly 880m), the SA-VO front-end will be *consistently* off by a small percentage. The periodic, high-confidence CVGL "anchors" will create a consistent, low-level "tension" in the pose graph. The graph optimizer (e.g., Ceres Solver) 3 will resolve this tension by slightly "pulling" the SA-VO poses to fit the global anchors, effectively *learning* and correcting for the altitude bias. This robust fusion is the key to meeting the 20-meter and 50-meter accuracy targets (AC-1, AC-2).
## **3.0 Core Component: The Scale-Aware Visual Odometry (SA-VO) Front-End**
This component is the new, critical engine of the system. Its sole task is to compute the *metric-scale* 6-DoF relative motion between consecutive frames, thereby eliminating scale drift at its source.
### **3.1 Rationale and Mechanism for Per-Frame Scale Recovery**
The SA-VO front-end implements a geometric algorithm to recover the absolute scale $s$ for *every* frame-to-frame transition. This algorithm directly leverages the query's "known altitude" ($H$) and "planar ground" constraints.5
The SA-VO algorithm for processing Image_N (relative to Image_N-1) is as follows:
1. **Feature Matching:** Extract and match robust features between Image_N and Image_N-1 using the selected feature matcher (see Section 3.2). This yields a set of corresponding 2D pixel coordinates.
2. **Essential Matrix:** Use RANSAC (Random Sample Consensus) and the camera intrinsic matrix $K$ to compute the Essential Matrix $E$ from the "inlier" correspondences.2
3. **Pose Decomposition:** Decompose $E$ to find the relative Rotation $R$ and the *unscaled* translation vector $t$, where the magnitude $||t||$ is fixed to 1.2
4. **Triangulation:** Triangulate the 3D-world points $X$ for all inlier features using the unscaled pose $$.15 These 3D points ($X_i$) are now in a local, *unscaled* coordinate system (i.e., we know the *shape* of the point cloud, but not its *size*).
5. **Ground Plane Fitting:** The query states "terrain height can be neglected," meaning we assume a planar ground. A *second* RANSAC pass is performed, this time fitting a 3D plane to the set of triangulated 3D points $X$. The inliers to this RANSAC are identified as the ground points $X_g$.5 This method is highly robust as it does not rely on a single point, but on the consensus of all visible ground features.16
6. **Unscaled Height ($h$):** From the fitted plane equation n^T X + d = 0, the parameter $d$ represents the perpendicular distance from the camera (at the coordinate system's origin) to the computed ground plane. This is our *unscaled* height $h$.
7. **Scale Computation:** We now have two values: the *real, metric* altitude $h$ (e.g., 900m) provided by the user, and our *computed, unscaled* altitude $h$. The absolute scale $s$ for this frame is the ratio of these two values: s = h / h.
8. **Metric Pose:** The final, metric-scale relative pose is $$, where the metric translation $T = s * t$. This high-confidence, scale-aware pose is sent to the Back-End.
### **3.2 Feature Matching Sub-System Analysis**
The success of the SA-VO algorithm depends *entirely* on the quality of the initial feature matches, especially in the low-texture agricultural terrain specified in the query. The system requires a matcher that is both robust (for sparse textures) and extremely fast (for AC-7).
The initial draft's choice of SuperGlue 17 is a strong, proven baseline. However, its successor, LightGlue 18, offers a critical, non-obvious advantage: **adaptivity**.
The UAV flight is specified as *mostly* straight, with high overlap. Sharp turns (AC-4) are "rather an exception." This means \~95% of our image pairs are "easy" to match, while 5% are "hard."
* SuperGlue uses a fixed-depth Graph Neural Network (GNN), spending the *same* (large) amount of compute on an "easy" pair as a "hard" pair.19 This is inefficient.
* LightGlue is *adaptive*.19 For an easy, high-overlap pair, it can exit early (e.g., at layer 3/9), returning a high-confidence match in a fraction of the time. For a "hard" low-overlap pair, it will use its full depth to get the best possible result.19
By using LightGlue, the system saves *enormous* amounts of computational budget on the 95% of "easy" frames, ensuring it *always* meets the \<5s budget (AC-7) and reserving that compute for the harder CVGL tasks. LightGlue is a "plug-and-play replacement" 19 that is faster, more accurate, and easier to train.19
### **Table 1: Analysis of State-of-the-Art Feature Matchers (For SA-VO Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** 17 | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems.22 | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue.19 - Training is complex.19 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.25 | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the \<5s time budget (AC-7). |
| **SuperPoint + LightGlue** 18 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.19 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy.19 - **Easier to Train:** Simpler architecture and loss.19 - Direct plug-and-play replacement for SuperGlue. | - Newer, less long-term-SLAM-proven than SuperGlue (though rapidly being adopted). | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.28 | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, preserving the budget for the 5% of hard (turn) frames, maximizing our ability to meet AC-7. |
### **3.3 Selected Approach (SA-VO): SuperPoint + LightGlue**
The SA-VO front-end will be built using:
* **Detector:** **SuperPoint** 24 to detect sparse, robust features on the Image_N_LR.
* **Matcher:** **LightGlue** 18 to match features from Image_N_LR to Image_N-1_LR.
This combination provides the SOTA robustness required for low-texture fields, while LightGlue's adaptive performance 19 is the key to meeting the \<5s (AC-7) real-time requirement.
## **4.0 Global Anchoring: The Cross-View Geolocalization (CVGL) Module**
With the SA-VO front-end handling metric scale, the CVGL module's task is refined. Its purpose is no longer to *correct scale*, but to provide *absolute global "anchor" poses*. This corrects for any accumulated bias (e.g., if the $h$ prior is off by 5m) and, critically, *relocalizes* the system after a persistent tracking loss (AC-4).
### **4.1 Hierarchical Retrieval-and-Match Pipeline**
This module runs asynchronously and is computationally heavy. A brute-force search against the entire Google Maps database is impossible. A two-stage hierarchical pipeline is required:
1. **Stage 1: Coarse Retrieval.** This is treated as an image retrieval problem.29
* A **Siamese CNN** 30 (or similar Dual-CNN architecture) is used to generate a compact "embedding vector" (a digital signature) for the Image_N_LR.
* An embedding database will be pre-computed for *all* Google Maps satellite tiles in the specified Eastern Ukraine operational area.
* The UAV image's embedding is then used to perform a very fast (e.g., FAISS library) similarity search against the satellite database, returning the *Top-K* (e.g., K=5) most likely-matching satellite tiles.
2. **Stage 2: Fine-Grained Pose.**
* *Only* for these Top-5 candidates, the system performs the heavy-duty **SuperPoint + LightGlue** matching.
* This match is *not* Image_N -> Image_N-1. It is Image_N -> Satellite_Tile_K.
* The match with the highest inlier count and lowest reprojection error (MRE \< 1.0, AC-10) is used to compute the precise 6-DoF pose of the UAV relative to that georeferenced satellite tile. This yields the final Absolute_GPS_Pose.
### **4.2 Critical Insight: Solving the Oblique-to-Nadir "Domain Gap"**
A critical, unaddressed failure mode exists. The query states the camera is **"not autostabilized"** [User Query]. On a fixed-wing UAV, this guarantees that during a bank or sharp turn (AC-4), the camera will *not* be nadir (top-down). It will be *oblique*, capturing the ground from an angle. The Google Maps reference, however, is *perfectly nadir*.32
This creates a severe "domain gap".33 A CVGL system trained *only* to match nadir-to-nadir images will *fail* when presented with an oblique UAV image.34 This means the CVGL module will fail *precisely* when it is needed most: during the sharp turns (AC-4) when SA-VO tracking is also lost.
The solution is to *close this domain gap* during training. Since the real-world UAV images will be oblique, the network must be taught to match oblique views to nadir ones.
Solution: Synthetic Data Generation for Robust Training
The Stage 1 Siamese CNN 30 must be trained on a custom, synthetically-generated dataset.37 The process is as follows:
1. Acquire nadir satellite imagery and a corresponding Digital Elevation Model (DEM) for the operational area.
2. Use this data to *synthetically render* the nadir satellite imagery from a wide variety of *oblique* viewpoints, simulating the UAV's roll and pitch.38
3. Create thousands of training pairs, each consisting of (Nadir_Satellite_Tile, Synthetically_Oblique_Tile_Angle_30_Deg).
4. Train the Siamese network 29 to learn that these two images—despite their *vastly* different appearances—are a *match*.
This process teaches the retrieval network to be *viewpoint-invariant*.35 It learns to ignore perspective distortion and match the true underlying ground features (road intersections, field boundaries). This is the *only* way to ensure the CVGL module can robustly relocalize the UAV during a sharp turn (AC-4).
## **5.0 Trajectory Fusion: The Robust Optimization Back-End**
This component is the system's central "brain." It runs continuously, fusing all incoming measurements (high-frequency/metric-scale SA-VO poses, low-frequency/globally-absolute CVGL poses) into a single, globally consistent trajectory. This component's design is dictated by the requirements for streaming (AC-8), refinement (AC-8), and outlier-rejection (AC-3).
### **5.1 Selected Strategy: Incremental Pose-Graph Optimization**
The user's requirements for "results...appear immediately" and "system could refine existing calculated results" [User Query] are a textbook description of a real-time SLAM back-end.11 A batch Structure from Motion (SfM) process, which requires all images upfront and can take hours, is unsuitable for the primary system.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (g2o 13, Ceres Solver 10, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (CVGL) data arrives (AC-8).11 - **Robust:** Can handle outliers via robust kernels.39 | - Initial estimate is less accurate than a full batch process. - Can drift *if* not anchored (though our SA-VO minimizes this). | - A graph optimization library (g2o, Ceres). - A robust cost function.41 | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming and asynchronous refinement. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process after the flight. |
The system's back-end will be built as an **Incremental Pose-Graph Optimizer** using **Ceres Solver**.10 Ceres is selected due to its large user community, robust documentation, excellent support for robust loss functions 10, and proven scalability for large-scale nonlinear least-squares problems.42
### **5.2 Mechanism for Automatic Outlier Rejection (AC-3, AC-5)**
The system must "correctly continue the work even in the presence of up to 350 meters of an outlier" (AC-3). A standard least-squares optimizer would be catastrophically corrupted by this event, as it would try to *average* this 350m error, pulling the *entire* 300km trajectory out of alignment.
A modern optimizer does not need to use brittle, hand-coded if-then logic to reject outliers. It can *mathematically* and *automatically* down-weight them using **Robust Loss Functions (Kernels)**.41
The mechanism is as follows:
1. The Ceres Back-End 10 maintains a graph of nodes (poses) and edges (constraints, or measurements).
2. A 350m outlier (AC-3) will create an edge with a *massive* error (residual).
3. A standard (quadratic) loss function $cost(error) = error^2$ would create a *catastrophic* cost, forcing the optimizer to ruin the entire graph to accommodate it.
4. Instead, the system will wrap its cost functions in a **Robust Loss Function**, such as **CauchyLoss** or **HuberLoss**.10
5. A robust loss function behaves quadratically for small errors (which it tries hard to fix) but becomes *sub-linear* for large errors. When it "sees" the 350m error, it mathematically *down-weights its influence*.43
6. The optimizer effectively *acknowledges* the 350m error but *refuses* to pull the entire graph to fix this one "insane" measurement. It automatically, and gracefully, treats the outlier as a "lost cause" and optimizes the 99.9% of "sane" measurements. This is the modern, robust solution to AC-3 and AC-5.
## **6.0 High-Resolution (6.2K) and Performance Optimization**
The system must simultaneously handle massive 6252x4168 (26-Megapixel) images and run on a modest RTX 2060 GPU [User Query] with a \<5s time limit (AC-7). These are opposing constraints.
### **6.1 The Multi-Scale Patch-Based Processing Pipeline**
Running *any* deep learning model (SuperPoint, LightGlue) on a full 6.2K image will be impossibly slow and will *immediately* cause a CUDA Out-of-Memory (OOM) error on a 6GB RTX 2060.45
The solution is not to process the full 6.2K image in real-time. Instead, a **multi-scale, patch-based pipeline** is required, where different components use the resolution best suited to their task.46
1. **For SA-VO (Real-time, \<5s):** The SA-VO front-end is concerned with *motion*, not fine-grained detail. The 6.2K Image_N_HR is *immediately* downscaled to a manageable 1536x1024 (Image_N_LR). The entire SA-VO (SuperPoint + LightGlue) pipeline runs *only* on this low-resolution, fast-to-process image. This is how the \<5s (AC-7) budget is met.
2. **For CVGL (High-Accuracy, Async):** The CVGL module, which runs asynchronously, is where the 6.2K detail is *selectively* used to meet the 20m (AC-2) accuracy target. It uses a "coarse-to-fine" 48 approach:
* **Step A (Coarse):** The Siamese CNN 30 runs on the *downscaled* 1536px Image_N_LR to get a coarse [Lat, Lon] guess.
* **Step B (Fine):** The system uses this coarse guess to fetch the corresponding *high-resolution* satellite tile.
* **Step C (Patching):** The system runs the SuperPoint detector on the *full 6.2K* Image_N_HR to find the Top 100 *most confident* feature keypoints. It then extracts 100 small (e.g., 256x256) *patches* from the full-resolution image, centered on these keypoints.49
* **Step D (Matching):** The system then matches *these small, full-resolution patches* against the high-res satellite tile.
This hybrid method provides the best of both worlds: the fine-grained matching accuracy 50 of the 6.2K image, but without the catastrophic OOM errors or performance penalties.45
### **6.2 Real-Time Deployment with TensorRT**
PyTorch is a research and training framework. Its default inference speed, even on an RTX 2060, is often insufficient to meet a \<5s production requirement.23
For the final production system, the key neural networks (SuperPoint, LightGlue, Siamese CNN) *must* be converted from their PyTorch-native format into a highly-optimized **NVIDIA TensorRT engine**.
* **Benefits:** TensorRT is an inference optimizer that applies graph optimizations, layer fusion, and precision reduction (e.g., to FP16).52 This can achieve a 2x-4x (or more) speedup over native PyTorch.28
* **Deployment:** The resulting TensorRT engine can be deployed via a C++ API 25, which is far more suitable for a robust, high-performance production system.
This conversion is a *mandatory* deployment step. It is what makes a 2-second inference (well within the 5-second AC-7 budget) *achievable* on the specified RTX 2060 hardware.
## **7.0 System Robustness: Failure Mode and Logic Escalation**
The system's logic is designed as a multi-stage escalation process to handle the specific failure modes in the acceptance criteria (AC-3, AC-4, AC-6), ensuring the >95% registration rate (AC-9).
### **Stage 1: Normal Operation (Tracking)**
* **Condition:** SA-VO(N-1 -> N) succeeds. The LightGlue match is high-confidence, and the computed scale $s$ is reasonable.
* **Logic:**
1. The Relative_Metric_Pose is sent to the Back-End.
2. The Pose_N_Est is calculated and sent to the user (\<5s).
3. The CVGL module is queued to run asynchronously to provide a Pose_N_Refined at a later time.
### **Stage 2: Transient SA-VO Failure (AC-3 Outlier Handling)**
* **Condition:** SA-VO(N-1 -> N) fails. This could be a 350m outlier (AC-3), a severely blurred image, or an image with no features (e.g., over a cloud). The LightGlue match fails, or the computed scale $s$ is nonsensical.
* **Logic (Frame Skipping):**
1. The system *buffers* Image_N and marks it as "tentatively lost."
2. When Image_N+1 arrives, the SA-VO front-end attempts to "bridge the gap" by matching SA-VO(N-1 -> N+1).
3. **If successful:** A Relative_Metric_Pose for N+1 is found. Image_N is officially marked as a rejected outlier (AC-5). The system "correctly continues the work" (AC-3 met).
4. **If fails:** The system repeats for SA-VO(N-1 -> N+2).
5. If this "bridging" fails for 3 consecutive frames, the system concludes it is not a transient outlier but a persistent tracking loss, and escalates to Stage 3.
### **Stage 3: Persistent Tracking Loss (AC-4 Sharp Turn Handling)**
* **Condition:** The "frame-skipping" in Stage 2 fails. This is the "sharp turn" scenario [AC-4] where there is \<5% overlap between Image_N-1 and Image_N+k.
* **Logic (Multi-Map "Chunking"):**
1. The Back-End declares a "Tracking Lost" state at Image_N and creates a *new, independent map chunk* ("Chunk 2").
2. The SA-VO Front-End is re-initialized at Image_N and begins populating this new chunk, tracking SA-VO(N -> N+1), SA-VO(N+1 -> N+2), etc.
3. Because the front-end is **Scale-Aware**, this new "Chunk 2" is *already in metric scale*. It is a "floating island" of *known size and shape*; it just is not anchored to the global GPS map.
* **Resolution (Asynchronous Relocalization):**
1. The **CVGL Module** is now tasked, high-priority, to find a *single* Absolute_GPS_Pose for *any* frame in this new "Chunk 2".
2. Once the CVGL module (which is robust to oblique views, per Section 4.2) finds one (e.g., for Image_N+20), the Back-End has all the information it needs.
3. **Merging:** The Back-End calculates the simple 6-DoF transformation (3D translation and rotation, scale=1) to align all of "Chunk 2" and merge it with "Chunk 1". This robustly handles the "correctly continue the work" criterion (AC-4).
### **Stage 4: Catastrophic Failure (AC-6 User Intervention)**
* **Condition:** The system has entered Stage 3 and is building "Chunk 2," but the **CVGL Module** has *also* failed for a prolonged period (e.g., 20% of the route, or 50+ consecutive frames). This is the "worst-case" scenario (e.g., heavy clouds *and* over a large, featureless lake). The system is "absolutely incapable" [User Query].
* **Logic:**
1. The system has a metric-scale "Chunk 2" but zero idea where it is in the world.
2. The Back-End triggers the AC-6 flag.
* **Resolution (User Input):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The UI displays the last known good image (from Chunk 1) and the new, "lost" image (e.g., Image_N+50).
3. The user clicks *one point* on the satellite map.
4. This user-provided [Lat, Lon] is *not* taken as ground truth. It is fed to the CVGL module as a *strong prior*, drastically narrowing its search area from "all of Ukraine" to "a 10km-radius circle."
5. This allows the CVGL module to re-acquire a lock, which triggers the Stage 3 merge, and the system continues.
## **8.0 Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated and how the system's compliance with all 10 acceptance criteria will be validated.
### **8.1 Generating Object-Level GPS (from Pixel Coordinate)**
This meets the requirement to find the "coordinates of the center of any object in these photos" [User Query]. The system provides this via a **Ray-Plane Intersection** method.
* **Inputs:**
1. The user clicks pixel coordinate $(u,v)$ on Image_N.
2. The system retrieves the refined, global 6-DoF pose $$ for Image_N from the Back-End.
3. The system uses the known camera intrinsic matrix $K$.
4. The system uses the known *global ground-plane equation* (e.g., $Z=150m$, based on the predefined altitude and start coordinate).
* **Method:**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} \= K^{-1} \cdot [u, v, 1]^T$.
2. **Transform Ray:** This ray direction is transformed into the *global* coordinate system using the pose's rotation matrix: $d_{global} \= R \cdot d_{cam}$.
3. **Define Ray:** A 3D ray is now defined, originating at the camera's global position $T$ (from the pose) and traveling in the direction $d_{global}$.
4. **Intersect:** The system solves the 3D line-plane intersection equation for this ray and the known global ground plane (e.g., find the intersection with $Z=150m$).
5. **Result:** The 3D intersection point $(X, Y, Z)$ is the *metric* world coordinate of the object on the ground.
6. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate. This process is immediate and can be performed for any pixel on any geolocated image.
### **8.2 Rigorous Validation Methodology**
A comprehensive test plan is required to validate all 10 acceptance criteria. The foundation of this is the creation of a **Ground-Truth Test Harness**.
* **Test Harness:**
1. **Ground-Truth Data:** Several test flights will be conducted in the operational area using a UAV equipped with a high-precision RTK/PPK GPS. This provides the "real GPS" (ground truth) for every image.
2. **Test Datasets:** Multiple test datasets will be curated from this ground-truth data:
* Test_Baseline_1000: A standard 1000-image flight.
* Test_Outlier_350m (AC-3): Test_Baseline_1000 with a single image from 350m away manually inserted at frame 30.
* Test_Sharp_Turn_5pct (AC-4): A sequence where frames 20-24 are manually deleted, simulating a \<5% overlap jump.
* Test_Catastrophic_Fail_20pct (AC-6): A sequence with 200 (20%) consecutive "bad" frames (e.g., pure sky, lens cap) inserted.
* Test_Full_3000: A full 3000-image sequence to test scalability and memory usage.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* Run Test_Baseline_1000. A test script will compare the system's *final refined GPS output* for each image against its *ground-truth GPS*.
* ASSERT (count(errors \< 50m) / 1000) \geq 0.80 (AC-1)
* ASSERT (count(errors \< 20m) / 1000) \geq 0.60 (AC-2)
* ASSERT (count(un-localized_images) / 1000) \< 0.10 (AC-5)
* ASSERT (count(localized_images) / 1000) > 0.95 (AC-9)
* **Test_MRE (AC-10):**
* ASSERT (BackEnd.final_MRE) \< 1.0 (AC-10)
* **Test_Performance (AC-7, AC-8):**
* Run Test_Full_3000 on the minimum-spec RTX 2060.
* Log timestamps for "Image In" -> "Initial Pose Out". ASSERT average_time \< 5.0s (AC-7).
* Log the output stream. ASSERT that >80% of images receive *two* poses: an "Initial" and a "Refined" (AC-8).
* **Test_Robustness (AC-3, AC-4, AC-6):**
* Run Test_Outlier_350m. ASSERT the system correctly continues and the final trajectory error for Image_31 is \< 50m (AC-3).
* Run Test_Sharp_Turn_5pct. ASSERT the system logs "Tracking Lost" and "Maps Merged," and the final trajectory is complete and accurate (AC-4).
* Run Test_Catastrophic_Fail_20pct. ASSERT the system correctly triggers the "ask for user input" event (AC-6).
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report. Put here all new findings, what was updated, replaced, or removed from the previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
_docs/00_problem/1.3_03_assesment_prompt.md
-301
View File
@@ -1,301 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%
- The number of outliers during the satellite provider images ground check should be less than 10%
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
Here is a solution draft:
**GEORTEX-R: A Geospatial-Temporal Robust Extraction System for IMU-Denied UAV Geolocalization**
## **1.0 GEORTEX-R: System Architecture and Data Flow**
The GEORTEX-R system is an asynchronous, three-component software solution designed for deployment on an NVIDIA RTX 2060+ GPU. It is architected from the ground up to handle the specific, demonstrated challenges of IMU-denied localization in *non-planar terrain* (as seen in Images 1-9) and *temporally-divergent* (outdated) reference maps (AC-5).
The system's core design principle is the *decoupling of unscaled relative motion from global metric scale*. The front-end estimates high-frequency, robust, but *unscaled* motion. The back-end asynchronously provides sparse, high-confidence *metric* and *geospatial* anchors. The central hub fuses these two data streams into a single, globally-optimized, metric-scale trajectory.
### **1.1 Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate (Latitude, Longitude) for the first image.
3. **Camera Intrinsics ($K$):** A pre-calibrated camera intrinsic matrix.
4. **Altitude Prior ($H_{prior}$):** The *approximate* predefined metric altitude (e.g., 900 meters). This is used as a *prior* (a hint) for optimization, *not* a hard constraint.
5. **Geospatial API Access:** Credentials for an on-demand satellite and DEM provider (e.g., Copernicus, EOSDA).
### **1.2 Streaming Outputs**
1. **Initial Pose ($Pose\\_N\\_Est$):** An *unscaled* pose estimate. This is sent immediately to the UI for real-time visualization of the UAV's *path shape* (AC-7, AC-8).
2. **Refined Pose ($Pose\\_N\\_Refined$) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose (X, Y, Z, Qx, Qy, Qz, Qw) and its corresponding [Lat, Lon, Alt] coordinate. This is sent to the user whenever the Trajectory Optimization Hub re-converges, updating all past poses (AC-1, AC-2, AC-8).
### **1.3 Component Interaction and Data Flow**
The system is architected as three parallel-processing components:
1. **Image Ingestion & Pre-processing:** This module receives the new Image_N (up to 6.2K). It creates two copies:
* Image_N_LR (Low-Resolution, e.g., 1536x1024): Dispatched *immediately* to the V-SLAM Front-End for real-time processing.
* Image_N_HR (High-Resolution, 6.2K): Stored for asynchronous use by the Geospatial Anchoring Back-End (GAB).
2. **V-SLAM Front-End (High-Frequency Thread):** This component's sole task is high-speed, *unscaled* relative pose estimation. It tracks Image_N_LR against a *local map of keyframes*. It performs local bundle adjustment to minimize drift 12 and maintains a co-visibility graph of all keyframes. It sends Relative_Unscaled_Pose estimates to the Trajectory Optimization Hub (TOH).
3. **Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread):** This is the system's "anchor." When triggered by the TOH, it fetches *on-demand* geospatial data (satellite imagery and DEMs) from an external API.3 It then performs a robust *hybrid semantic-visual* search 5 to find an *absolute, metric, global pose* for a given keyframe, robust to outdated maps (AC-5) 5 and oblique views (AC-4).14 This Absolute_Metric_Anchor is sent to the TOH.
4. **Trajectory Optimization Hub (TOH) (Central Hub):** This component manages the complete flight trajectory as a **Sim(3) pose graph** (7-DoF). It continuously fuses two distinct data streams:
* **On receiving Relative_Unscaled_Pose (T \< 5s):** It appends this pose to the graph, calculates the Pose_N_Est, and sends this *unscaled* initial result to the user (AC-7, AC-8 met).
* **On receiving Absolute_Metric_Anchor (T > 5s):** This is the critical event. It adds this as a high-confidence *global metric constraint*. This anchor creates "tension" in the graph, which the optimizer (Ceres Solver 15) resolves by finding the *single global scale factor* that best fits all V-SLAM and CVGL measurements. It then triggers a full graph re-optimization, "stretching" the entire trajectory to the correct metric scale, and sends the new Pose_N_Refined stream to the user for all affected poses (AC-1, AC-2, AC-8 refinement met).
## **2.0 Core Component: The High-Frequency V-SLAM Front-End**
This component's sole task is to robustly and accurately compute the *unscaled* 6-DoF relative motion of the UAV and build a geometrically-consistent map of keyframes. It is explicitly designed to be more robust to drift than simple frame-to-frame odometry.
### **2.1 Rationale: Keyframe-Based Monocular SLAM**
The choice of a keyframe-based V-SLAM front-end over a frame-to-frame VO is deliberate and critical for system robustness.
* **Drift Mitigation:** Frame-to-frame VO is "prone to drift accumulation due to errors introduced by each frame-to-frame motion estimation".13 A single poor match permanently corrupts all future poses.
* **Robustness:** A keyframe-based system tracks new images against a *local map* of *multiple* previous keyframes, not just Image_N-1. This provides resilience to transient failures (e.g., motion blur, occlusion).
* **Optimization:** This architecture enables "local bundle adjustment" 12, a process where a sliding window of recent keyframes is continuously re-optimized, actively minimizing error and drift *before* it can accumulate.
* **Relocalization:** This architecture possesses *innate relocalization capabilities* (see Section 6.3), which is the correct, robust solution to the "sharp turn" (AC-4) requirement.
### **2.2 Feature Matching Sub-System**
The success of the V-SLAM front-end depends entirely on high-quality feature matches, especially in the sparse, low-texture agricultural terrain seen in the provided images (e.g., Image 6, Image 7). The system requires a matcher that is robust (for sparse textures 17) and extremely fast (for AC-7).
The selected approach is **SuperPoint + LightGlue**.
* **SuperPoint:** A SOTA (State-of-the-Art) feature detector proven to find robust, repeatable keypoints in challenging, low-texture conditions 17
* **LightGlue:** A highly optimized GNN-based matcher that is the successor to SuperGlue 19
The key advantage of selecting LightGlue 19 over SuperGlue 20 is its *adaptive nature*. The query states sharp turns (AC-4) are "rather an exception." This implies \~95% of image pairs are "easy" (high-overlap, straight flight) and 5% are "hard" (low-overlap, turns). SuperGlue uses a fixed-depth GNN, spending the *same* large amount of compute on an "easy" pair as a "hard" one. LightGlue is *adaptive*.19 For an "easy" pair, it can exit its GNN early, returning a high-confidence match in a fraction of the time. This saves *enormous* computational budget on the 95% of "easy" frames, ensuring the system *always* meets the \<5s budget (AC-7) and reserving that compute for the GAB.
#### **Table 1: Analysis of State-of-the-Art Feature Matchers (For V-SLAM Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** 20 | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems. | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue.19 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.21 | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the \<5s time budget (AC-7). |
| **SuperPoint + LightGlue** 17 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.19 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy. - SOTA "in practice" choice for large-scale matching.17 | - Newer, but rapidly being adopted and proven.21 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.22 | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, maximizing our ability to meet AC-7. |
## **3.0 Core Component: The Geospatial Anchoring Back-End (GAB)**
This component is the system's "anchor to reality." It runs asynchronously to provide the *absolute, metric-scale* constraints needed to solve the trajectory. It is an *on-demand* system that solves three distinct "domain gaps": the hardware/scale gap, the temporal gap, and the viewpoint gap.
### **3.1 On-Demand Geospatial Data Retrieval**
A "pre-computed database" for all of Eastern Ukraine is operationally unfeasible on laptop-grade hardware.1 This design is replaced by an on-demand, API-driven workflow.
* **Mechanism:** When the TOH requests a global anchor, the GAB receives a *coarse* [Lat, Lon] estimate. The GAB then performs API calls to a geospatial data provider (e.g., EOSDA 3, Copernicus 8).
* **Dual-Retrieval:** The API query requests *two* distinct products for the specified Area of Interest (AOI):
1. **Visual Tile:** A high-resolution (e.g., 30-50cm) satellite ortho-image.26
2. **Terrain Tile:** The corresponding **Digital Elevation Model (DEM)**, such as the Copernicus GLO-30 (30m resolution) or SRTM (30m).7
This "Dual-Retrieval" mechanism is the central, enabling synergy of the new architecture. The **Visual Tile** is used by the CVGL (Section 3.2) to find the *geospatial pose*. The **DEM Tile** is used by the *output module* (Section 7.1) to perform high-accuracy **Ray-DEM Intersection**, solving the final output accuracy problem.
### **3.2 Hybrid Semantic-Visual Localization**
The "temporal gap" (evidenced by burn scars in Images 1-9) and "outdated maps" (AC-5) makes a purely visual CVGL system unreliable.5 The GAB solves this using a robust, two-stage *hybrid* matching pipeline.
1. **Stage 1: Coarse Visual Retrieval (Siamese CNN).** A lightweight Siamese CNN 14 is used to find the *approximate* location of the Image_N_LR *within* the large, newly-fetched satellite tile. This acts as a "candidate generator."
2. **Stage 2: Fine-Grained Semantic-Visual Fusion.** For the top candidates, the GAB performs a *dual-channel alignment*.
* **Visual Channel (Unreliable):** It runs SuperPoint+LightGlue on high-resolution *patches* (from Image_N_HR) against the satellite tile. This match may be *weak* due to temporal gaps.5
* **Semantic Channel (Reliable):** It extracts *temporally-invariant* semantic features (e.g., road-vectors, field-boundaries, tree-cluster-polygons, lake shorelines) from *both* the UAV image (using a segmentation model) and the satellite/OpenStreetMap data.5
* **Fusion:** A RANSAC-based optimizer finds the 6-DoF pose that *best aligns* this *hybrid* set of features.
This hybrid approach is robust to the exact failure mode seen in the images. When matching Image 3 (burn scars), the *visual* LightGlue match will be poor. However, the *semantic* features (the dirt road, the tree line) are *unchanged*. The optimizer will find a high-confidence pose by *trusting the semantic alignment* over the poor visual alignment, thereby succeeding despite the "outdated map" (AC-5).
### **3.3 Solution to Viewpoint Gap: Synthetic Oblique View Training**
This component is critical for handling "sharp turns" (AC-4). The camera *will* be oblique, not nadir, during turns.
* **Problem:** The GAB's Stage 1 Siamese CNN 14 will be matching an *oblique* UAV view to a *nadir* satellite tile. This "viewpoint gap" will cause a match failure.14
* **Mechanism (Synthetic Data Generation):** The network must be trained for *viewpoint invariance*.28
1. Using the on-demand DEMs (fetched in 3.1) and satellite tiles, the system can *synthetically render* the satellite imagery from *any* roll, pitch, and altitude.
2. The Siamese network is trained on (Nadir_Tile, Synthetic_Oblique_Tile) pairs.14
* **Result:** This process teaches the network to match the *underlying ground features*, not the *perspective distortion*. It ensures the GAB can relocalize the UAV *precisely* when it is needed most: during a sharp, banking turn (AC-4) when VO tracking has been lost.
## **4.0 Core Component: The Trajectory Optimization Hub (TOH)**
This component is the system's central "brain." It runs continuously, fusing all measurements (high-frequency/unscaled V-SLAM, low-frequency/metric-scale GAB anchors) into a single, globally consistent trajectory.
### **4.1 Incremental Sim(3) Pose-Graph Optimization**
The "planar ground" SA-VO (Finding 1) is removed. This component is its replacement. The system must *discover* the global scale, not *assume* it.
* **Selected Strategy:** An incremental pose-graph optimizer using **Ceres Solver**.15
* **The Sim(3) Insight:** The V-SLAM front-end produces *unscaled* 6-DoF ($SE(3)$) relative poses. The GAB produces *metric-scale* 6-DoF ($SE(3)$) *absolute* poses. These cannot be directly combined. The graph must be optimized in **Sim(3) (7-DoF)**, which adds a *single global scale factor $s$* as an optimizable variable.
* **Mechanism (Ceres Solver):**
1. **Nodes:** Each keyframe pose (7-DoF: $X, Y, Z, Qx, Qy, Qz, s$).
2. **Edge 1 (V-SLAM):** A relative pose constraint between Keyframe_i and Keyframe_j. The error is computed in Sim(3).
3. **Edge 2 (GAB):** An *absolute* pose constraint on Keyframe_k. This constraint *fixes* Keyframe_k's pose to the *metric* GPS coordinate and *fixes its scale $s$ to 1.0*.
* **Bootstrapping Scale:** The TOH graph "bootstraps" the scale.32 The GAB's $s=1.0$ anchor creates "tension" in the graph. The Ceres optimizer 15 resolves this tension by finding the *one* global scale $s$ for all V-SLAM nodes that minimizes the total error, effectively "stretching" the entire unscaled trajectory to fit the metric anchors. This is robust to *any* terrain.34
#### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (Ceres Solver 15, g2o 35, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (GAB) data arrives (AC-8).13 - **Robust:** Can handle outliers via robust kernels.15 | - Initial estimate is *unscaled* until a GAB anchor arrives. - Can drift *if* not anchored (though V-SLAM minimizes this). | - A graph optimization library (Ceres). - A robust cost function. | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming and asynchronous refinement. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process. |
### **4.2 Automatic Outlier Rejection (AC-3, AC-5)**
The system must handle 350m outliers (AC-3) and \<10% bad GAB matches (AC-5).
* **Mechanism (Robust Loss Functions):** A standard least-squares optimizer (like Ceres 15) would be catastrophically corrupted by a 350m error. The solution is to wrap *all* constraints in a **Robust Loss Function (e.g., HuberLoss, CauchyLoss)**.15
* **Result:** A robust loss function mathematically *down-weights* the influence of constraints with large errors. When it "sees" the 350m error (AC-3), it effectively acknowledges the measurement but *refuses* to pull the entire 3000-image trajectory to fit this one "insane" data point. It automatically and gracefully *ignores* the outlier, optimizing the 99.9% of "sane" measurements. This is the modern, robust solution to AC-3 and AC-5.
## **5.0 High-Performance Compute & Deployment**
The system must run on an RTX 2060 (AC-7) and process 6.2K images. These are opposing constraints.
### **5.1 Multi-Scale, Patch-Based Processing Pipeline**
Running deep learning models (SuperPoint, LightGlue) on a full 6.2K (26-Megapixel) image will cause a CUDA Out-of-Memory (OOM) error and be impossibly slow.
* **Mechanism (Coarse-to-Fine):**
1. **For V-SLAM (Real-time, \<5s):** The V-SLAM front-end (Section 2.0) runs *only* on the Image_N_LR (e.g., 1536x1024) copy. This is fast enough to meet the AC-7 budget.
2. **For GAB (High-Accuracy, Async):** The GAB (Section 3.0) uses the full-resolution Image_N_HR *selectively* to meet the 20m accuracy (AC-2).
* It first runs its coarse Siamese CNN 27 on the Image_N_LR.
* It then runs the SuperPoint detector on the *full 6.2K* image to find the *most confident* feature keypoints.
* It then extracts small, 256x256 *patches* from the *full-resolution* image, centered on these keypoints.
* It matches *these small, full-resolution patches* against the high-res satellite tile.
* **Result:** This hybrid method provides the fine-grained matching accuracy of the 6.2K image (needed for AC-2) without the catastrophic OOM errors or performance penalties.
### **5.2 Mandatory Deployment: NVIDIA TensorRT Acceleration**
PyTorch is a research framework. For production, its inference speed is insufficient.
* **Requirement:** The key neural networks (SuperPoint, LightGlue, Siamese CNN) *must* be converted from PyTorch into a highly-optimized **NVIDIA TensorRT engine**.
* **Research Validation:** 23 demonstrates this process for LightGlue, achieving "2x-4x speed gains over compiled PyTorch." 22 and 21 provide open-source repositories for SuperPoint+LightGlue conversion to ONNX and TensorRT.
* **Result:** This is not an "optional" optimization. It is a *mandatory* deployment step. This conversion (which applies layer fusion, graph optimization, and FP16 precision) is what makes achieving the \<5s (AC-7) performance *possible* on the specified RTX 2060 hardware.36
## **6.0 System Robustness: Failure Mode Escalation Logic**
This logic defines the system's behavior during real-world failures, ensuring it meets criteria AC-3, AC-4, AC-6, and AC-9.
### **6.1 Stage 1: Normal Operation (Tracking)**
* **Condition:** V-SLAM front-end (Section 2.0) is healthy.
* **Logic:**
1. V-SLAM successfully tracks Image_N_LR against its local keyframe map.
2. A new Relative_Unscaled_Pose is sent to the TOH.
3. TOH sends Pose_N_Est (unscaled) to the user (\<5s).
4. If Image_N is selected as a new keyframe, the GAB (Section 3.0) is *queued* to find an Absolute_Metric_Anchor for it, which will trigger a Pose_N_Refined update later.
### **6.2 Stage 2: Transient VO Failure (Outlier Rejection)**
* **Condition:** Image_N is unusable (e.g., severe blur, sun-glare, 350m outlier per AC-3).
* **Logic (Frame Skipping):**
1. V-SLAM front-end fails to track Image_N_LR against the local map.
2. The system *discards* Image_N (marking it as a rejected outlier, AC-5).
3. When Image_N+1 arrives, the V-SLAM front-end attempts to track it against the *same* local keyframe map (from Image_N-1).
4. **If successful:** Tracking resumes. Image_N is officially an outlier. The system "correctly continues the work" (AC-3 met).
5. **If fails:** The system repeats for Image_N+2, N+3. If this fails for \~5 consecutive frames, it escalates to Stage 3.
### **6.3 Stage 3: Persistent VO Failure (Relocalization)**
* **Condition:** Tracking is lost for multiple frames. This is the "sharp turn" (AC-4) or "low overlap" (AC-4) scenario.
* **Logic (Keyframe-Based Relocalization):**
1. The V-SLAM front-end declares "Tracking Lost."
2. **Critically:** It does *not* create a "new map chunk."
3. Instead, it enters **Relocalization Mode**. For every new Image_N+k, it extracts features (SuperPoint) and queries the *entire* existing database of past keyframes for a match.
* **Resolution:** The UAV completes its sharp turn. Image_N+5 now has high overlap with Image_N-10 (from *before* the turn).
1. The relocalization query finds a strong match.
2. The V-SLAM front-end computes the 6-DoF pose of Image_N+5 relative to the *existing map*.
3. Tracking is *resumed* seamlessly. The system "correctly continues the work" (AC-4 met). This is vastly more robust than the previous "map-merging" logic.
### **6.4 Stage 4: Catastrophic Failure (User Intervention)**
* **Condition:** The system is in Stage 3 (Lost), but *also*, the **GAB (Section 3.0) has failed** to find *any* global anchors for a prolonged period (e.g., 20% of the route). This is the "absolutely incapable" scenario (AC-6), (e.g., heavy fog *and* over a featureless ocean).
* **Logic:**
1. The system has an *unscaled* trajectory, and *zero* idea where it is in the world.
2. The TOH triggers the AC-6 flag.
* **Resolution (User-Aided Prior):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The user clicks *one point* on a map.
3. This [Lat, Lon] is *not* taken as ground truth. It is fed to the **GAB (Section 3.1)** as a *strong prior* for its on-demand API query.
4. This narrows the GAB's search area from "all of Ukraine" to "a 5km radius." This *guarantees* the GAB's Dual-Retrieval (Section 3.1) will fetch the *correct* satellite and DEM tiles, allowing the Hybrid Matcher (Section 3.2) to find a high-confidence Absolute_Metric_Anchor, which in turn re-scales (Section 4.1) and relocalizes the entire trajectory.
## **7.0 Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated, specifically solving the "planar ground" output flaw, and how the system's compliance with all 10 ACs will be validated.
### **7.1 High-Accuracy Object Geolocalization via Ray-DEM Intersection**
The "Ray-Plane Intersection" method is inaccurate for non-planar terrain 37 and is replaced with a high-accuracy ray-tracing method. This is the correct method for geolocating an object on the *non-planar* terrain visible in Images 1-9.
* **Inputs:**
1. User clicks pixel coordinate $(u,v)$ on Image_N.
2. System retrieves the *final, refined, metric* 7-DoF pose $P = (R, T, s)$ for Image_N from the TOH.
3. The system uses the known camera intrinsic matrix $K$.
4. System retrieves the specific **30m DEM tile** 8 that was fetched by the GAB (Section 3.1) for this region of the map. This DEM is a 3D terrain mesh.
* **Algorithm (Ray-DEM Intersection):**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} = K^{-1} \\cdot [u, v, 1]^T$.
2. **Transform Ray:** This ray direction $d_{cam}$ and origin (0,0,0) are transformed into the *global, metric* coordinate system using the pose $P$. This yields a ray originating at $T$ and traveling in direction $R \\cdot d_{cam}$.
3. **Intersect:** The system performs a numerical *ray-mesh intersection* 39 to find the 3D point $(X, Y, Z)$ where this global ray *intersects the 3D terrain mesh* of the DEM.
4. **Result:** This 3D intersection point $(X, Y, Z)$ is the *metric* world coordinate of the object *on the actual terrain*.
5. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate.
This method correctly accounts for terrain. A pixel aimed at the top of a hill will intersect the DEM at a high Z-value. A pixel aimed at the ravine (Image 1) will intersect at a low Z-value. This is the *only* method that can reliably meet the 20m accuracy (AC-2) for object localization.
### **7.2 Rigorous Validation Methodology**
A comprehensive test plan is required. The foundation is a **Ground-Truth Test Harness** using the provided coordinates.csv.42
* **Test Harness:**
1. **Ground-Truth Data:** The file coordinates.csv 42 provides ground-truth [Lat, Lon] for 60 images (e.g., AD000001.jpg...AD000060.jpg).
2. **Test Datasets:**
* Test_Baseline_60 42: The 60 images and their coordinates.
* Test_Outlier_350m (AC-3): Test_Baseline_60 with a single, unrelated image inserted at frame 30.
* Test_Sharp_Turn_5pct (AC-4): A sequence where frames 20-24 are manually deleted, simulating a \<5% overlap jump.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* **Run:** Execute GEORTEX-R on Test_Baseline_60, providing AD000001.jpg's coordinate (48.275292, 37.385220) as the Start Coordinate 42
* **Script:** A validation script will compute the Haversine distance error between the *system's refined GPS output* for each image (2-60) and the *ground-truth GPS* from coordinates.csv.
* **ASSERT** (count(errors \< 50m) / 60) >= 0.80 **(AC-1 Met)**
* **ASSERT** (count(errors \< 20m) / 60) >= 0.60 **(AC-2 Met)**
* **ASSERT** (count(un-localized_images) / 60) \< 0.10 **(AC-5 Met)**
* **ASSERT** (count(localized_images) / 60) > 0.95 **(AC-9 Met)**
* **Test_MRE (AC-10):**
* **Run:** After Test_Baseline_60 completes.
* **ASSERT** TOH.final_Mean_Reprojection_Error \< 1.0 **(AC-10 Met)**
* **Test_Performance (AC-7, AC-8):**
* **Run:** Execute on a 1500-image sequence on the minimum-spec RTX 2060.
* **Log:** Log timestamps for "Image In" -> "Initial Pose Out".
* **ASSERT** average_time \< 5.0s **(AC-7 Met)**
* **Log:** Log the output stream.
* **ASSERT** >80% of images receive *two* poses: an "Initial" and a "Refined" **(AC-8 Met)**
* **Test_Robustness (AC-3, AC-4):**
* **Run:** Execute Test_Outlier_350m.
* **ASSERT** System logs "Stage 2: Discarding Outlier" and the final trajectory error for Image_31 is \< 50m **(AC-3 Met)**.
* **Run:** Execute Test_Sharp_Turn_5pct.
* **ASSERT** System logs "Stage 3: Tracking Lost" and "Relocalization Succeeded," and the final trajectory is complete and accurate **(AC-4 Met)**.
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report. Put here all new findings, what was updated, replaced, or removed from the previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
_docs/00_problem/1.3_04_assesment_prompt.md
-370
View File
@@ -1,370 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 150m drift and at an angle of less than 50%
- The number of outliers during the satellite provider images ground check should be less than 10%
- System should try to operate when UAV made a sharp turn, and all the next photos has no common points with previous route. In that situation system should try to figure out location of the new piece of the route and connect it to the previous route. Also this separate chunks could be more than 2, so this strategy should be in the core of the system
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
Here is a solution draft:
## **The ATLAS-GEOFUSE System Architecture**
Multi-component architecture designed for high-performance, real-time geolocalization in IMU-denied, high-drift environments. Its architecture is explicitly designed around **pre-flight data caching** and **multi-map robustness**.
### **2.1 Core Design Principles**
1. **Pre-Flight Caching:** To meet the <5s (AC-7) real-time requirement, all network latency must be eliminated. The system mandates a "Pre-Flight" step (Section 3.0) where all geospatial data (satellite tiles, DEMs, vector data) for the Area of Interest (AOI) is downloaded from a viable open-source provider (e.g., Copernicus 6) and stored in a local database on the processing laptop. All real-time queries are made against this local cache.
2. **Decoupled Multi-Map SLAM:** The system separates *relative* motion from *absolute* scale. A Visual SLAM (V-SLAM) "Atlas" Front-End (Section 4.0) computes high-frequency, robust, but *unscaled* relative motion. A Local Geospatial Anchoring Back-End (GAB) (Section 5.0) provides sparse, high-confidence, *absolute metric* anchors by querying the local cache. A Trajectory Optimization Hub (TOH) (Section 6.0) fuses these two streams in a Sim(3) pose-graph to solve for the global 7-DoF trajectory (pose + scale).
3. **Multi-Map Robustness (Atlas):** To solve the "sharp turn" (AC-4) and "tracking loss" (AC-6) requirements, the V-SLAM front-end is based on an "Atlas" architecture.14 Tracking loss initiates a *new, independent map fragment*.13 The TOH is responsible for anchoring and merging *all* fragments geodetically 19 into a single, globally-consistent trajectory.
### **2.2 Component Interaction and Data Flow**
* **Component 1: Pre-Flight Caching Module (PCM) (Offline)**
* *Input:* User-defined Area of Interest (AOI) (e.g., a KML polygon).
* *Action:* Queries Copernicus 6 and OpenStreetMap APIs. Downloads and builds a local geospatial database (GeoPackage/SpatiaLite) containing satellite tiles, DEM tiles, and road/river vectors for the AOI.
* *Output:* A single, self-contained **Local Geo-Database file**.
* **Component 2: Image Ingestion & Pre-processing (Real-time)**
* *Input:* Image_N (up to 6.2K), Camera Intrinsics ($K$).
* *Action:* Creates two copies:
* **Image_N_LR** (Low-Resolution, e.g., 1536x1024): Dispatched *immediately* to the V-SLAM Front-End.
* **Image_N_HR** (High-Resolution, 6.2K): Stored for asynchronous use by the GAB.
* **Component 3: V-SLAM "Atlas" Front-End (High-Frequency Thread)**
* *Input:* Image_N_LR.
* *Action:* Tracks Image_N_LR against its *active map fragment*. Manages keyframes, local bundle adjustment 38, and the co-visibility graph. If tracking is lost (e.g., AC-4 sharp turn), it initializes a *new map fragment* 14 and continues tracking.
* *Output:* **Relative_Unscaled_Pose** and **Local_Point_Cloud** data, sent to the TOH.
* **Component 4: Local Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread)**
* *Input:* A keyframe (Image_N_HR) and its *unscaled* pose, triggered by the TOH.
* *Action:* Performs a visual-only, coarse-to-fine search 34 against the *Local Geo-Database*.
* *Output:* An **Absolute_Metric_Anchor** (a high-confidence [Lat, Lon, Alt] pose) for that keyframe, sent to the TOH.
* **Component 5: Trajectory Optimization Hub (TOH) (Central Hub Thread)**
* *Input:* (1) High-frequency Relative_Unscaled_Pose stream. (2) Low-frequency Absolute_Metric_Anchor stream.
* *Action:* Manages the complete flight trajectory as a **Sim(3) pose graph** 39 using Ceres Solver.19 Continuously fuses all data.
* *Output 1 (Real-time):* **Pose_N_Est** (unscaled) sent to UI (meets AC-7, AC-8).
* *Output 2 (Refined):* **Pose_N_Refined** (metric-scale, globally-optimized) sent to UI (meets AC-1, AC-2, AC-8).
### **2.3 System Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate (Latitude, Longitude).
3. **Camera Intrinsics ($K$):** Pre-calibrated camera intrinsic matrix.
4. **Local Geo-Database File:** The single file generated by the Pre-Flight Caching Module (Section 3.0).
### **2.4 Streaming Outputs (Meets AC-7, AC-8)**
1. **Initial Pose ($Pose_N^{Est}$):** An *unscaled* pose estimate. This is sent immediately (<5s, AC-7) to the UI for real-time visualization of the UAV's *path shape*.
2. **Refined Pose ($Pose_N^{Refined}$) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose (X, Y, Z, Qx, Qy, Qz, Qw) and its corresponding [Lat, Lon, Alt] coordinate. This is sent to the user whenever the TOH re-converges (e.g., after a new GAB anchor or map-merge), updating all past poses (AC-1, AC-2, AC-8 refinement met).
## **3.0 Pre-Flight Component: The Geospatial Caching Module (PCM)**
This component is a new, mandatory, pre-flight utility that solves the fatal flaws (Section 1.1, 1.2) of the GEORTEX-R design. It eliminates all real-time network latency (AC-7) and all ToS violations (AC-5), ensuring the project is both performant and legally viable.
### **3.1 Defining the Area of Interest (AOI)**
The system is designed for long-range flights. Given 3000 photos at 100m intervals, the maximum linear track is 300km. The user must provide a coarse "bounding box" or polygon (e.g., KML/GeoJSON format) of the intended flight area. The PCM will automatically add a generous buffer (e.g., 20km) to this AOI to account for navigational drift and ensure all necessary reference data is captured.
### **3.2 Legal & Viable Data Sources (Copernicus & OpenStreetMap)**
As established in 1.1, the system *must* use open-data providers. The PCM is architected to use the following:
1. **Visual/Terrain Data (Primary):** The **Copernicus Data Space Ecosystem** 6 is the primary source. The PCM will use the Copernicus Processing and Catalogue APIs 6 to query, process, and download two key products for the buffered AOI:
* **Sentinel-2 Satellite Imagery:** High-resolution (10m) visual tiles.
* **Copernicus GLO-30 DEM:** A 30m-resolution Digital Elevation Model.7 This DEM is *not* used for high-accuracy object localization (see 1.4), but as a coarse altitude *prior* for the TOH and for the critical dynamic-warping step (Section 5.3).
2. **Semantic Data (Secondary):** OpenStreetMap (OSM) data 40 for the AOI will be downloaded. This provides temporally-invariant vector data (roads, rivers, building footprints) which can be used as a secondary, optional verification layer for the GAB, especially in cases of extreme temporal divergence (e.g., new construction).42
### **3.3 Building the Local Geo-Database**
The PCM utility will process all downloaded data into a single, efficient, compressed file. A modern GeoPackage or SpatiaLite database is the ideal format. This database will contain the satellite tiles, DEM tiles, and vector features, all indexed by a common spatial grid (e.g., UTM).
This single file is then loaded by the main ATLAS-GEOFUSE application at runtime. The GAB's (Section 5.0) "API calls" are thus transformed from high-latency, unreliable HTTP requests 9 into high-speed, zero-latency local SQL queries, guaranteeing that data I/O is never the bottleneck for meeting the AC-7 performance requirement.
## **4.0 Core Component: The Multi-Map V-SLAM "Atlas" Front-End**
This component's sole task is to robustly and accurately compute the *unscaled* 6-DoF relative motion of the UAV and build a geometrically-consistent map of keyframes. It is explicitly designed to be more robust than simple frame-to-frame odometry and to handle catastrophic tracking loss (AC-4) gracefully.
### **4.1 Rationale: ORB-SLAM3 "Atlas" Architecture**
The system will implement a V-SLAM front-end based on the "Atlas" multi-map paradigm, as seen in SOTA systems like ORB-SLAM3.14 This is the industry-standard solution for robust, long-term navigation in environments where tracking loss is possible.13
The mechanism is as follows:
1. The system initializes and begins tracking on **Map_Fragment_0**, using the known start GPS as a metadata tag.
2. It tracks all new frames (Image_N_LR) against this active map.
3. **If tracking is lost** (e.g., a sharp turn (AC-4) or a persistent 350m outlier (AC-3)):
* The "Atlas" architecture does not fail. It declares Map_Fragment_0 "inactive," stores it, and *immediately initializes* **Map_Fragment_1** from the current frame.14
* Tracking *resumes instantly* on this new map fragment, ensuring the system "correctly continues the work" (AC-4).
This architecture converts the "sharp turn" failure case into a *standard operating procedure*. The system never "fails"; it simply fragments. The burden of stitching these fragments together is correctly moved from the V-SLAM front-end (which has no global context) to the TOH (Section 6.0), which *can* solve it using global-metric anchors.
### **4.2 Feature Matching Sub-System: SuperPoint + LightGlue**
The V-SLAM front-end's success depends entirely on high-quality feature matches, especially in the sparse, low-texture agricultural terrain seen in the user's images. The selected approach is **SuperPoint + LightGlue**.
* **SuperPoint:** A SOTA feature detector proven to find robust, repeatable keypoints in challenging, low-texture conditions.43
* **LightGlue:** A highly optimized GNN-based matcher that is the successor to SuperGlue.44
The choice of LightGlue over SuperGlue is a deliberate performance optimization. LightGlue is *adaptive*.46 The user query states sharp turns (AC-4) are "rather an exception." This implies \~95% of image pairs are "easy" (high-overlap, straight flight) and 5% are "hard" (low-overlap, turns). LightGlue's adaptive-depth GNN exits early on "easy" pairs, returning a high-confidence match in a fraction of the time. This saves *enormous* computational budget on the 95% of normal frames, ensuring the system *always* meets the <5s budget (AC-7) and reserving that compute for the GAB and TOH. This component will run on **Image_N_LR** (low-res) to guarantee performance, and will be accelerated via TensorRT (Section 7.0).
### **4.3 Keyframe Management and Local 3D Cloud**
The front-end will maintain a co-visibility graph of keyframes for its *active map fragment*. It will perform local Bundle Adjustment 38 continuously over a sliding window of recent keyframes to minimize drift *within* that fragment.
Crucially, it will triangulate features to create a **local, high-density 3D point cloud** for its map fragment.28 This point cloud is essential for two reasons:
1. It provides robust tracking (tracking against a 3D map, not just a 2D frame).
2. It serves as the **high-accuracy source** for the object localization output (Section 9.1), as established in 1.4, allowing the system to bypass the high-error external DEM.
#### **Table 1: Analysis of State-of-the-Art Feature Matchers (For V-SLAM Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems. | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue. | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT. | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the <5s time budget (AC-7). |
| **SuperPoint + LightGlue** 44 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.46 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy. - SOTA "in practice" choice for large-scale matching. | - Newer, but rapidly being adopted and proven.48 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT. | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, maximizing our ability to meet AC-7. |
## **5.0 Core Component: The Local Geospatial Anchoring Back-End (GAB)**
This asynchronous component is the system's "anchor to reality." Its sole purpose is to find a high-confidence, *absolute-metric* pose for a given V-SLAM keyframe by matching it against the **local, pre-cached geo-database** (from Section 3.0). This component is a full replacement for the high-risk, high-latency GAB from the GEORTEX-R draft (see 1.2, 1.5).
### **5.1 Rationale: Local-First Query vs. On-Demand API**
As established in 1.2, all queries are made to the local SSD. This guarantees zero-latency I/O, which is a hard requirement for a real-time system, as external network latency is unacceptably high and variable.9 The GAB itself runs asynchronously and can take longer than 5s (e.g., 10-15s), but it must not be *blocked* by network I/O, which would stall the entire processing pipeline.
### **5.2 SOTA Visual-Only Coarse-to-Fine Localization**
This component implements a state-of-the-art, two-stage *visual-only* pipeline, which is lower-risk and more performant (see 1.5) than the GEORTEX-R's semantic-hybrid model. This approach is well-supported by SOTA research in aerial localization.34
1. **Stage 1 (Coarse): Global Descriptor Retrieval.**
* *Action:* When the TOH requests an anchor for Keyframe_k, the GAB first computes a *global descriptor* (a compact vector representation) for the *nadir-warped* (see 5.3) low-resolution Image_k_LR.
* *Technology:* A SOTA Visual Place Recognition (VPR) model like **SALAD** 49, **TransVLAD** 50, or **NetVLAD** 33 will be used. These are designed for this "image retrieval" task.45
* *Result:* This descriptor is used to perform a fast FAISS/vector search against the descriptors of the *local satellite tiles* (which were pre-computed and stored in the Geo-Database). This returns the Top-K (e.g., K=5) most likely satellite tiles in milliseconds.
2. **Stage 2 (Fine): Local Feature Matching.**
* *Action:* The system runs **SuperPoint+LightGlue** 43 to find pixel-level correspondences.
* *Performance:* This is *not* run on the *full* UAV image against the *full* satellite map. It is run *only* between high-resolution patches (from **Image_k_HR**) and the **Top-K satellite tiles** identified in Stage 1.
* *Result:* This produces a set of 2D-2D (image-to-map) feature matches. A PnP/RANSAC solver then computes a high-confidence 6-DoF pose. This pose is the **Absolute_Metric_Anchor** that is sent to the TOH.
### **5.3 Solving the Viewpoint Gap: Dynamic Feature Warping**
The GAB must solve the "viewpoint gap" 33: the UAV image is oblique (due to roll/pitch), while the satellite tiles are nadir (top-down).
The GEORTEX-R draft proposed a complex, high-risk deep learning solution. The ATLAS-GEOFUSE solution is far more elegant and requires zero R\&D:
1. The V-SLAM Front-End (Section 4.0) already *knows* the camera's *relative* 6-DoF pose, including its **roll and pitch** orientation relative to the *local map's ground plane*.
2. The *Local Geo-Database* (Section 3.0) contains a 30m-resolution DEM for the AOI.
3. When the GAB processes Keyframe_k, it *first* performs a **dynamic homography warp**. It projects the V-SLAM ground plane onto the coarse DEM, and then uses the known camera roll/pitch to calculate the perspective transform (homography) needed to *un-distort* the oblique UAV image into a synthetic *nadir-view*.
This *nadir-warped* UAV image is then used in the Coarse-to-Fine pipeline (5.2). It will now match the *nadir* satellite tiles with extremely high-fidelity. This method *eliminates* the viewpoint gap *without* training any new neural networks, leveraging the inherent synergy between the V-SLAM component and the GAB's pre-cached DEM.
## **6.0 Core Component: The Multi-Map Trajectory Optimization Hub (TOH)**
This component is the system's central "brain." It runs continuously, fusing all measurements (high-frequency/unscaled V-SLAM, low-frequency/metric-scale GAB anchors) from *all map fragments* into a single, globally consistent trajectory.
### **6.1 Incremental Sim(3) Pose-Graph Optimization**
The central challenge of monocular, IMU-denied SLAM is scale-drift. The V-SLAM front-end produces *unscaled* 6-DoF ($SE(3)$) relative poses.37 The GAB produces *metric-scale* 6-DoF ($SE(3)$) *absolute* poses. These cannot be directly combined.
The solution is that the graph *must* be optimized in **Sim(3) (7-DoF)**.39 This adds a *single global scale factor $s$* as an optimizable variable to each V-SLAM map fragment. The TOH will maintain a pose-graph using **Ceres Solver** 19, a SOTA optimization library.
The graph is constructed as follows:
1. **Nodes:** Each keyframe pose (7-DoF: $X, Y, Z, Qx, Qy, Qz, s$).
2. **Edge 1 (V-SLAM):** A relative pose constraint between Keyframe_i and Keyframe_j *within the same map fragment*. The error is computed in Sim(3).29
3. **Edge 2 (GAB):** An *absolute* pose constraint on Keyframe_k. This constraint *fixes* Keyframe_k's pose to the *metric* GPS coordinate from the GAB anchor and *fixes its scale $s$ to 1.0*.
The GAB's $s=1.0$ anchor creates "tension" in the graph. The Ceres optimizer 20 resolves this tension by finding the *one* global scale $s$ for all *other* V-SLAM nodes in that fragment that minimizes the total error. This effectively "stretches" or "shrinks" the entire unscaled V-SLAM fragment to fit the metric anchors, which is the core of monocular SLAM scale-drift correction.29
### **6.2 Geodetic Map-Merging via Absolute Anchors**
This is the robust solution to the "sharp turn" (AC-4) problem, replacing the flawed "relocalization" model from the original draft.
* **Scenario:** The UAV makes a sharp turn (AC-4). The V-SLAM front-end *loses tracking* on Map_Fragment_0 and *creates* Map_Fragment_1 (per Section 4.1). The TOH's pose graph now contains *two disconnected components*.
* **Mechanism (Geodetic Merging):**
1. The GAB (Section 5.0) is *queued* to find anchors for keyframes in *both* fragments.
2. The GAB returns Anchor_A for Keyframe_10 (in Map_Fragment_0) with GPS [Lat_A, Lon_A].
3. The GAB returns Anchor_B for Keyframe_50 (in Map_Fragment_1) with GPS ``.
4. The TOH adds *both* of these as absolute, metric constraints (Edge 2) to the global pose-graph.
* The graph optimizer 20 now has all the information it needs. It will solve for the 7-DoF pose of *both fragments*, placing them in their correct, globally-consistent metric positions. The two fragments are *merged geodetically* (i.e., by their global coordinates) even if they *never* visually overlap. This is a vastly more robust and modern solution than simple visual loop closure.19
### **6.3 Automatic Outlier Rejection (AC-3, AC-5)**
The system must be robust to 350m outliers (AC-3) and <10% bad GAB matches (AC-5). A standard least-squares optimizer (like Ceres 20) would be catastrophically corrupted by a 350m error.
This is a solved problem in modern graph optimization.19 The solution is to wrap *all* constraints (V-SLAM and GAB) in a **Robust Loss Function (e.g., HuberLoss, CauchyLoss)** within Ceres Solver.
A robust loss function mathematically *down-weights* the influence of constraints with large errors (high residuals). When the TOH "sees" the 350m error from a V-SLAM relative pose (AC-3) or a bad GAB anchor (AC-5), the robust loss function effectively acknowledges the measurement but *refuses* to pull the entire 3000-image trajectory to fit this one "insane" data point. It automatically and gracefully *ignores* the outlier, optimizing the 99.9% of "sane" measurements, thus meeting AC-3 and AC-5.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (Ceres Solver 19, g2o, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (GAB) data arrives (AC-8). - **Robust:** Can handle outliers via robust kernels.19 | - Initial estimate is *unscaled* until a GAB anchor arrives. - Can drift *if* not anchored. | - A graph optimization library (Ceres). - A robust cost function (Huber). | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming (AC-7) and asynchronous refinement (AC-8). |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process. |
## **7.0 High-Performance Compute & Deployment**
The system must run on an RTX 2060 (AC-7) while processing 6.2K images. These are opposing constraints that require a deliberate compute strategy to balance speed and accuracy.
### **7.1 Multi-Scale, Coarse-to-Fine Processing Pipeline**
The system must balance the conflicting demands of real-time speed (AC-7) and high accuracy (AC-2). This is achieved by running different components at different resolutions.
* **V-SLAM Front-End (Real-time, <5s):** This component (Section 4.0) runs *only* on the **Image_N_LR** (e.g., 1536x1024) copy. This is fast enough to meet the AC-7 budget.46
* **GAB (Asynchronous, High-Accuracy):** This component (Section 5.0) uses the full-resolution **Image_N_HR** *selectively* to meet the 20m accuracy (AC-2).
1. Stage 1 (Coarse) runs on the low-res, nadir-warped image.
2. Stage 2 (Fine) runs SuperPoint on the *full 6.2K* image to find the *most confident* keypoints. It then extracts small, 256x256 *patches* from the *full-resolution* image, centered on these keypoints.
3. It matches *these small, full-resolution patches* against the high-res satellite tile.
This hybrid, multi-scale method provides the fine-grained matching accuracy of the 6.2K image (needed for AC-2) without the catastrophic CUDA Out-of-Memory errors (an RTX 2060 has only 6GB VRAM 30) or performance penalties that full-resolution processing would entail.
### **7.2 Mandatory Deployment: NVIDIA TensorRT Acceleration**
The deep learning models (SuperPoint, LightGlue, NetVLAD) will be too slow in their native PyTorch framework to meet AC-7 on an RTX 2060.
This is not an "optional" optimization; it is a *mandatory* deployment step. The key neural networks *must* be converted from PyTorch into a highly-optimized **NVIDIA TensorRT engine**.
Research *specifically* on accelerating LightGlue with TensorRT shows **"2x-4x speed gains over compiled PyTorch"**.48 Other benchmarks confirm TensorRT provides 30-70% speedups for deep learning inference.52 This conversion (which applies layer fusion, graph optimization, and FP16/INT8 precision) is what makes achieving the <5s (AC-7) performance *possible* on the specified RTX 2060 hardware.
## **8.0 System Robustness: Failure Mode Escalation Logic**
This logic defines the system's behavior during real-world failures, ensuring it meets criteria AC-3, AC-4, AC-6, and AC-9, and is built upon the new "Atlas" multi-map architecture.
### **8.1 Stage 1: Normal Operation (Tracking)**
* **Condition:** V-SLAM front-end (Section 4.0) is healthy.
* **Logic:**
1. V-SLAM successfully tracks Image_N_LR against its *active map fragment*.
2. A new **Relative_Unscaled_Pose** is sent to the TOH (Section 6.0).
3. TOH sends **Pose_N_Est** (unscaled) to the user (AC-7, AC-8 met).
4. If Image_N is selected as a keyframe, the GAB (Section 5.0) is *queued* to find an anchor for it, which will trigger a **Pose_N_Refined** update later.
### **8.2 Stage 2: Transient VO Failure (Outlier Rejection)**
* **Condition:** Image_N is unusable (e.g., severe blur, sun-glare, or the 350m outlier from AC-3).
* **Logic (Frame Skipping):**
1. V-SLAM front-end fails to track Image_N_LR against the active map.
2. The system *discards* Image_N (marking it as a rejected outlier, AC-5).
3. When Image_N+1 arrives, the V-SLAM front-end attempts to track it against the *same* local keyframe map (from Image_N-1).
4. **If successful:** Tracking resumes. Image_N is officially an outlier. The system "correctly continues the work" (AC-3 met).
5. **If fails:** The system repeats for Image_N+2, N+3. If this fails for \~5 consecutive frames, it escalates to Stage 3.
### **8.3 Stage 3: Persistent VO Failure (New Map Initialization)**
* **Condition:** Tracking is lost for multiple frames. This is the **"sharp turn" (AC-4)** or "low overlap" (AC-4) scenario.
* **Logic (Atlas Multi-Map):**
1. The V-SLAM front-end (Section 4.0) declares "Tracking Lost."
2. It marks the current Map_Fragment_k as "inactive".13
3. It *immediately* initializes a **new** Map_Fragment_k+1 using the current frame (Image_N+5).
4. **Tracking resumes instantly** on this new, unscaled, un-anchored map fragment.
5. This "registering" of a new map ensures the system "correctly continues the work" (AC-4 met) and maintains the >95% registration rate (AC-9) by not counting this as a failure.
### **8.4 Stage 4: Map-Merging & Global Relocalization (GAB-Assisted)**
* **Condition:** The system is now tracking on Map_Fragment_k+1, while Map_Fragment_k is inactive. The TOH pose-graph (Section 6.0) is disconnected.
* **Logic (Geodetic Merging):**
1. The TOH queues the GAB (Section 5.0) to find anchors for *both* map fragments.
2. The GAB finds anchors for keyframes in *both* fragments.
3. The TOH (Section 6.2) receives these metric anchors, adds them to the graph, and the Ceres optimizer 20 *finds the global 7-DoF pose for both fragments*, merging them into a single, metrically-consistent trajectory.
### **8.5 Stage 5: Catastrophic Failure (User Intervention)**
* **Condition:** The system is in Stage 3 (Lost), *and* the GAB (Section 5.0) has *also* failed to find *any* global anchors for a new Map_Fragment_k+1 for a prolonged period (e.g., 20% of the route). This is the "absolutely incapable" scenario (AC-6), (e.g., flying over a large, featureless body of water or dense, uniform fog).
* **Logic:**
1. The system has an *unscaled, un-anchored* map fragment (Map_Fragment_k+1) and *zero* idea where it is in the world.
2. The TOH triggers the AC-6 flag.
* **Resolution (User-Aided Prior):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The user clicks *one point* on a map.
3. This [Lat, Lon] is *not* taken as ground truth. It is fed to the **GAB (Section 5.0)** as a *strong spatial prior* for its *local database query* (Section 5.2).
4. This narrows the GAB's Stage 1 search area from "the entire AOI" to "a 5km radius around the user's click." This *guarantees* the GAB will find the correct satellite tile, find a high-confidence **Absolute_Metric_Anchor**, and allow the TOH (Stage 4) to re-scale 29 and geodetically-merge 20 this lost fragment, re-localizing the entire trajectory.
## **9.0 High-Accuracy Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated, specifically replacing the flawed "Ray-DEM" method (see 1.4) with a high-accuracy "Ray-Cloud" method to meet the 20m accuracy (AC-2).
### **9.1 High-Accuracy Object Geolocalization via Ray-Cloud Intersection**
As established in 1.4, using an external 30m DEM 21 for object localization introduces uncontrollable errors (up to 4m+22) that make meeting the 20m (AC-2) accuracy goal impossible. The system *must* use its *own*, internally-generated 3D map, which is locally far more accurate.25
* **Inputs:**
1. User clicks pixel coordinate $(u,v)$ on Image_N.
2. The system retrieves the **final, refined, metric 7-DoF Sim(3) pose** $P_{sim(3)} = (s, R, T)$ for the *map fragment* that Image_N belongs to. This transform $P_{sim(3)}$ maps the *local V-SLAM coordinate system* to the *global metric coordinate system*.
3. The system retrieves the *local, unscaled* **V-SLAM 3D point cloud** ($P_{local_cloud}$) generated by the Front-End (Section 4.3).
4. The known camera intrinsic matrix $K$.
* **Algorithm (Ray-Cloud Intersection):**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} = K^{-1} \\cdot [u, v, 1]^T$.
2. **Transform Ray (Local):** This ray is transformed using the *local V-SLAM pose* of Image_N to get a ray in the *local map fragment's* coordinate system.
3. **Intersect (Local):** The system performs a numerical *ray-mesh intersection* (or nearest-neighbor search) to find the 3D point $P_{local}$ where this local ray *intersects the local V-SLAM point cloud* ($P_{local_cloud}$).25 This $P_{local}$ is *highly accurate* relative to the V-SLAM map.26
4. **Transform (Global):** This local 3D point $P_{local}$ is now transformed to the global, metric coordinate system using the 7-DoF Sim(3) transform from the TOH: $P_{metric} = s \\cdot (R \\cdot P_{local}) + T$.
5. **Result:** This 3D intersection point $P_{metric}$ is the *metric* world coordinate of the object.
6. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate.55
This method correctly isolates the error. The object's accuracy is now *only* dependent on the V-SLAM's geometric fidelity (AC-10 MRE < 1.0px) and the GAB's global anchoring (AC-1, AC-2). It *completely eliminates* the external 30m DEM error 22 from this critical, high-accuracy calculation.
### **9.2 Rigorous Validation Methodology**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** (e.g., using the provided coordinates.csv data).
* **Test Harness:**
1. **Ground-Truth Data:** coordinates.csv provides ground-truth [Lat, Lon] for a set of images.
2. **Test Datasets:**
* Test_Baseline: The ground-truth images and coordinates.
* Test_Outlier_350m (AC-3): Test_Baseline with a single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence where several frames are manually deleted to simulate <5% overlap.
* Test_Long_Route (AC-9): A 1500-image sequence.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* **Run:** Execute ATLAS-GEOFUSE on Test_Baseline, providing the first image's coordinate as the Start Coordinate.
* **Script:** A validation script will compute the Haversine distance error between the *system's refined GPS output* ($Pose_N^{Refined}$) for each image and the *ground-truth GPS*.
* **ASSERT** (count(errors < 50m) / total_images) >= 0.80 **(AC-1 Met)**
* **ASSERT** (count(errors < 20m) / total_images) >= 0.60 **(AC-2 Met)**
* **ASSERT** (count(un-localized_images) / total_images) < 0.10 **(AC-5 Met)**
* **ASSERT** (count(localized_images) / total_images) > 0.95 **(AC-9 Met)**
* **Test_MRE (AC-10):**
* **Run:** After Test_Baseline completes.
* **ASSERT** TOH.final_Mean_Reprojection_Error < 1.0 **(AC-10 Met)**
* **Test_Performance (AC-7, AC-8):**
* **Run:** Execute on Test_Long_Route on the minimum-spec RTX 2060.
* **Log:** Log timestamps for "Image In" -> "Initial Pose Out" ($Pose_N^{Est}$).
* **ASSERT** average_time < 5.0s **(AC-7 Met)**
* **Log:** Log the output stream.
* **ASSERT** >80% of images receive *two* poses: an "Initial" and a "Refined" **(AC-8 Met)**
* **Test_Robustness (AC-3, AC-4, AC-6):**
* **Run:** Execute Test_Outlier_350m.
* **ASSERT** System logs "Stage 2: Discarding Outlier" or "Stage 3: New Map" *and* the final trajectory error for the *next* frame is < 50m **(AC-3 Met)**.
* **Run:** Execute Test_Sharp_Turn_5pct.
* **ASSERT** System logs "Stage 3: New Map Initialization" and "Stage 4: Geodetic Map-Merge," and the final trajectory is complete and accurate **(AC-4 Met)**.
* **Run:** Execute on a sequence with no GAB anchors possible for 20% of the route.
* **ASSERT** System logs "Stage 5: User Intervention Requested" **(AC-6 Met)**.
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report.
At the very beginning of the report list most profound changes you've made to previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
_docs/00_problem/1.3_05.1_assesment_prompt.md
-379
View File
@@ -1,379 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 200m drift and at an angle of less than 70%
- System should try to operate when UAV made a sharp turn, and all the next photos has no common points with previous route. In that situation system should try to figure out location of the new piece of the route and connect it to the previous route. Also this separate chunks could be more than 2, so this strategy should be in the core of the system
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
- The whole system should work as a background service. The interaction should be done by zeromq. Sevice should be up and running and awaiting for the initial input message. On the input message processing should started, and immediately after the first results system should provide them to the client
Here is a solution draft:
# **ASTRAL System Architecture: A High-Fidelity Geopositioning Framework for IMU-Denied Aerial Operations**
## **2.0 The ASTRAL (Advanced Scale-Aware Trajectory-Refinement and Localization) System Architecture**
The ASTRAL architecture is a multi-map, decoupled, loosely-coupled system designed to solve the flaws identified in Section 1.0 and meet all 10 Acceptance Criteria.
### **2.1 Core Principles**
The ASTRAL architecture is built on three principles:
1. **Tiered Geospatial Database:** The system *cannot* rely on a single data source. It is architected around a *tiered* local database.
* **Tier-1 (Baseline):** Google Maps data. This is used to meet the 50m (AC-1) requirement and provide geolocalization.
* **Tier-2 (High-Accuracy):** A framework for ingesting *commercial, sub-meter* data (visual 4; and DEM 5). This tier is *required* to meet the 20m (AC-2) accuracy. The system will *run* on Tier-1 but *achieve* AC-2 when "fueled" with Tier-2 data.
2. **Viewpoint-Invariant Anchoring:** The system *rejects* geometric warping. The GAB (Section 5.0) is built on SOTA Visual Place Recognition (VPR) models that are *inherently* invariant to the oblique-to-nadir viewpoint change, decoupling it from the V-SLAM's unstable orientation.
3. **Continuously-Scaled Trajectory:** The system *rejects* the "single-scale-per-fragment" model. The TOH (Section 6.0) is a Sim(3) pose-graph optimizer 11 that models scale as a *per-keyframe optimizable parameter*.15 This allows the trajectory to "stretch" and "shrink" elastically to absorb continuous monocular scale drift.12
### **2.2 Component Interaction and Data Flow**
The system is multi-threaded and asynchronous, designed for real-time streaming (AC-7) and refinement (AC-8).
* **Component 1: Tiered GDB (Pre-Flight):**
* *Input:* User-defined Area of Interest (AOI).
* *Action:* Downloads and builds a local SpatiaLite/GeoPackage.
* *Output:* A single **Local-Geo-Database file** containing:
* Tier-1 (Google Maps) + GLO-30 DSM
* Tier-2 (Commercial) satellite tiles + WorldDEM DTM elevation tiles.
* A *pre-computed FAISS vector index* of global descriptors (e.g., SALAD 8) for *all* satellite tiles (see 3.4).
* **Component 2: Image Ingestion (Real-time):**
* *Input:* Image_N (up to 6.2K), Camera Intrinsics ($K$).
* *Action:* Creates Image_N_LR (Low-Res, e.g., 1536x1024) and Image_N_HR (High-Res, 6.2K).
* *Dispatch:* Image_N_LR -> V-SLAM. Image_N_HR -> GAB (for patches).
* **Component 3: "Atlas" V-SLAM Front-End (High-Frequency Thread):**
* *Input:* Image_N_LR.
* *Action:* Tracks Image_N_LR against the *active map fragment*. Manages keyframes and local BA. If tracking lost (AC-4, AC-6), it *initializes a new map fragment*.
* *Output:* Relative_Unscaled_Pose, Local_Point_Cloud, and Map_Fragment_ID -> TOH.
* **Component 4: VPR Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread):**
* *Input:* A keyframe (Image_N_LR, Image_N_HR) and its Map_Fragment_ID.
* *Action:* Performs SOTA two-stage VPR (Section 5.0) against the **Local-Geo-Database file**.
* *Output:* Absolute_Metric_Anchor ([Lat, Lon, Alt] pose) and its Map_Fragment_ID -> TOH.
* **Component 5: Scale-Aware Trajectory Optimization Hub (TOH) (Central Hub Thread):**
* *Input 1:* High-frequency Relative_Unscaled_Pose stream.
* *Input 2:* Low-frequency Absolute_Metric_Anchor stream.
* *Action:* Manages the *global Sim(3) pose-graph* 13 with *per-keyframe scale*.15
* *Output 1 (Real-time):* Pose_N_Est (unscaled) -> UI (Meets AC-7).
* *Output 2 (Refined):* Pose_N_Refined (metric-scale) -> UI (Meets AC-1, AC-2, AC-8).
### **2.3 System Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate [Lat, Lon].
3. **Camera Intrinsics (K):** Pre-calibrated camera intrinsic matrix.
4. **Local-Geo-Database File:** The single file generated by Component 1.
### **2.4 Streaming Outputs (Meets AC-7, AC-8)**
1. **Initial Pose (Pose_N^{Est}):** An *unscaled* pose. This is the raw output from the V-SLAM Front-End, transformed by the *current best estimate* of the trajectory. It is sent immediately (<5s, AC-7) to the UI for real-time visualization of the UAV's *path shape*.
2. **Refined Pose (Pose_N^{Refined}) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose. This is sent to the user *whenever the TOH re-converges* (e.g., after a new GAB anchor or a map-merge). This *re-writes* the history of poses (e.g., Pose_{N-100} to Pose_N), meeting the refinement (AC-8) and accuracy (AC-1, AC-2) requirements.
## **3.0 Component 1: The Tiered Pre-Flight Geospatial Database (GDB)**
This component is the implementation of the "Tiered Geospatial" principle. It is a mandatory pre-flight utility that solves both the *legal* problem (Flaw 1.4) and the *accuracy* problem (Flaw 1.1).
### **3.2 Tier-1 (Baseline): Google Maps and GLO-30 DEM**
This tier provides the baseline capability and satisfies AC-1.
* **Visual Data:** Google Maps (coarse Maxar)
* *Resolution:* 10m.
* *Geodetic Accuracy:* \~1 m to 20m
* *Purpose:* Meets AC-1 (80% < 50m error). Provides a robust baseline for coarse geolocalization.
* **Elevation Data:** Copernicus GLO-30 DEM
* *Resolution:* 30m.
* *Type:* DSM (Digital Surface Model).2 This is a *weakness*, as it includes buildings/trees.
* *Purpose:* Provides a coarse altitude prior for the TOH and the initial GAB search.
### **3.3 Tier-2 (High-Accuracy): Ingestion Framework for Commercial Data**
This is the *procurement and integration framework* required to meet AC-2.
* **Visual Data:** Commercial providers, e.g., Maxar (30-50cm) or Satellogic (70cm)
* *Resolution:* < 1m.
* *Geodetic Accuracy:* Typically < 5m.
* *Purpose:* Provides the high-resolution, high-accuracy reference needed for the GAB to achieve a sub-20m total error.
* **Elevation Data:** Commercial providers, e.g., WorldDEM Neo 5 or Elevation10.32
* *Resolution:* 5m-12m.
* *Vertical Accuracy:* < 4m.32
* *Type:* DTM (Digital Terrain Model).32
The use of a DTM (bare-earth) in Tier-2 is a critical advantage over the Tier-1 DSM (surface). The V-SLAM Front-End (Section 4.0) will triangulate a 3D point cloud of what it *sees*, which is the *ground* in fields or *tree-tops* in forests. The Tier-1 GLO-30 DSM 2 represents the *top* of the canopy/buildings. If the V-SLAM maps the *ground* (e.g., altitude 100m) and the GAB tries to anchor it to a DSM *prior* that shows a forest (e.g., altitude 120m), the 20m altitude discrepancy will introduce significant error into the TOH. The Tier-2 DTM (bare-earth) 5 provides a *vastly* superior altitude anchor, as it represents the same ground plane the V-SLAM is tracking, significantly improving the entire 7-DoF pose solution.
### **3.4 Local Database Generation: Pre-computing Global Descriptors**
This is the key performance optimization for the GAB. During the pre-flight caching step, the GDB utility does not just *store* tiles; it *processes* them.
For *every* satellite tile (e.g., 256x256m) in the AOI, the utility will load the tile into the VPR model (e.g., SALAD 8), compute its global descriptor (a compact feature vector), and store this vector in a high-speed vector index (e.g., FAISS).
This step moves 99% of the GAB's "Stage 1" (Coarse Retrieval) workload into an offline, pre-flight step. The *real-time* GAB query (Section 5.2) is now reduced to: (1) Compute *one* vector for the UAV image, and (2) Perform a very fast K-Nearest-Neighbor search on the pre-computed FAISS index. This is what makes a SOTA deep-learning GAB 6 fast enough to support the real-time refinement loop.
#### **Table 1: Geospatial Reference Data Analysis (Decision Matrix)**
| Data Product | Type | Resolution | Geodetic Accuracy (Horiz.) | Type | Cost | AC-2 (20m) Compliant? |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| Google Maps | Visual | 1m | 1m - 10m | N/A | Free | **Depending on the location** |
| Copernicus GLO-30 | Elevation | 30m | \~10-30m | **DSM** (Surface) | Free | **No (Fails Error Budget)** |
| **Tier-2: Maxar/Satellogic** | Visual | 0.3m - 0.7m | < 5 m (Est.) | N/A | Commercial | **Yes** |
| **Tier-2: WorldDEM Neo** | Elevation | 5m | < 4m | **DTM** (Bare-Earth) | Commercial | **Yes** |
## **4.0 Component 2: The "Atlas" Relative Motion Front-End**
This component's sole task is to robustly compute *unscaled* 6-DoF relative motion and handle tracking failures (AC-3, AC-4).
### **4.1 Feature Matching Sub-System: SuperPoint + LightGlue**
The system will use **SuperPoint** for feature detection and **LightGlue** for matching. This choice is driven by the project's specific constraints:
* **Rationale (Robustness):** The UAV flies over "eastern and southern parts of Ukraine," which includes large, low-texture agricultural areas. SuperPoint is a SOTA deep-learning detector renowned for its robustness and repeatability in these challenging, low-texture environments.
* **Rationale (Performance):** The RTX 2060 (AC-7) is a *hard* constraint with only 6GB VRAM.34 Performance is paramount. LightGlue is an SOTA matcher that provides a 4-10x speedup over its predecessor, SuperGlue. Its "adaptive" nature is a key optimization: it exits early on "easy" pairs (high-overlap, straight-flight) and spends more compute only on "hard" pairs (turns). This saves critical GPU budget on 95% of normal frames, ensuring the <5s (AC-7) budget is met.
This subsystem will run on the Image_N_LR (low-res) copy to guarantee it fits in VRAM and meets the real-time budget.
#### **Table 2: Analysis of State-of-the-Art Feature Matchers (V-SLAM Front-End)**
| Approach (Tools/Library) | Robustness (Low-Texture) | Speed (RTX 2060) | Fitness for Problem |
| :---- | :---- | :---- | :---- |
| ORB 33 (e.g., ORB-SLAM3) | Poor. Fails on low-texture. | Excellent (CPU/GPU) | **Good.** Fails robustness in target environment. |
| SuperPoint + SuperGlue | Excellent. | Good, but heavy. Fixed-depth GNN. 4-10x Slower than LightGlue.35 | **Good.** Robust, but risks AC-7 budget. |
| **SuperPoint + LightGlue** 35 | Excellent. | **Excellent.** Adaptive depth 35 saves budget. 4-10x faster. | **Excellent (Selected).** Balances robustness and performance. |
### **4.2 The "Atlas" Multi-Map Paradigm (Solution for AC-3, AC-4, AC-6)**
This architecture is the industry-standard solution for IMU-denied, long-term SLAM and is critical for robustness.
* **Mechanism (AC-4, Sharp Turn):**
1. The system is tracking on $Map_Fragment_0$.
2. The UAV makes a sharp turn (AC-4, <5% overlap). The V-SLAM *loses tracking*.
3. Instead of failing, the Atlas architecture *initializes a new map*: $Map_Fragment_1$.
4. Tracking *resumes instantly* on this new, unanchored map.
* **Mechanism (AC-3, 350m Outlier):**
1. The system is tracking. A 350m outlier $Image_N$ arrives.
2. The V-SLAM fails to match $Image_N$ (a "Transient VO Failure," see 7.3). It is *discarded*.
3. $Image_N+1$ arrives (back on track). V-SLAM re-acquires its location on $Map_Fragment_0$.
4. The system "correctly continues the work" (AC-3) by simply rejecting the outlier.
This design turns "catastrophic failure" (AC-3, AC-4) into a *standard operating procedure*. The "problem" of stitching the fragments ($Map_0$, $Map_1$) together is moved from the V-SLAM (which has no global context) to the TOH (which *can* solve it using GAB anchors, see 6.4).
### **4.3 Local Bundle Adjustment and High-Fidelity 3D Cloud**
The V-SLAM front-end will continuously run Local Bundle Adjustment (BA) over a sliding window of recent keyframes to minimize drift *within* that fragment. It will also triangulate a sparse, but high-fidelity, 3D point cloud for its *local map fragment*.
This 3D cloud serves a critical dual function:
1. It provides a robust 3D map for frame-to-map tracking, which is more stable than frame-to-frame odometry.
2. It serves as the **high-accuracy data source** for the object localization output (Section 7.2). This is the key to decoupling object-pointing accuracy from external DEM accuracy 19, a critical flaw in simpler designs.
## **5.0 Component 3: The Viewpoint-Invariant Geospatial Anchoring Back-End (GAB)**
This component *replaces* the draft's "Dynamic Warping" (Section 5.0) and implements the "Viewpoint-Invariant Anchoring" principle (Section 2.1).
### **5.1 Rationale: Viewpoint-Invariant VPR vs. Geometric Warping (Solves Flaw 1.2)**
As established in 1.2, geometrically warping the image using the V-SLAM's *drifty* roll/pitch estimate creates a *brittle*, high-risk failure spiral. The ASTRAL GAB *decouples* from the V-SLAM's orientation. It uses a SOTA VPR pipeline that *learns* to match oblique UAV images to nadir satellite images *directly*, at the feature level.6
### **5.2 Stage 1 (Coarse Retrieval): SOTA Global Descriptors**
When triggered by the TOH, the GAB takes Image_N_LR. It computes a *global descriptor* (a single feature vector) using a SOTA VPR model like **SALAD** 6 or **MixVPR**.7
This choice is driven by two factors:
1. **Viewpoint Invariance:** These models are SOTA for this exact task.
2. **Inference Speed:** They are extremely fast. SALAD reports < 3ms per image inference 8, and MixVPR is also noted for "fastest inference speed".37 This low overhead is essential for the AC-7 (<5s) budget.
This vector is used to query the *pre-computed FAISS vector index* (from 3.4), which returns the Top-K (e.g., K=5) most likely satellite tiles from the *entire AOI* in milliseconds.
#### **Table 3: Analysis of VPR Global Descriptors (GAB Back-End)**
| Model (Backbone) | Key Feature | Viewpoint Invariance | Inference Speed (ms) | Fitness for GAB |
| :---- | :---- | :---- | :---- | :---- |
| NetVLAD 7 (CNN) | Baseline | Poor. Not designed for oblique-to-nadir. | Moderate (\~20-50ms) | **Poor.** Fails robustness. |
| **SALAD** 8 (DINOv2) | Foundation Model.6 | **Excellent.** Designed for this. | **< 3ms**.8 Extremely fast. | **Excellent (Selected).** |
| **MixVPR** 36 (ResNet) | All-MLP aggregator.36 | **Very Good.**.7 | **Very Fast.**.37 | **Excellent (Selected).** |
### **5.3 Stage 2 (Fine): Local Feature Matching and Pose Refinement**
The system runs **SuperPoint+LightGlue** 35 to find pixel-level matches, but *only* between the UAV image and the **Top-K satellite tiles** identified in Stage 1.
A **Multi-Resolution Strategy** is employed to solve the VRAM bottleneck.
1. Stage 1 (Coarse) runs on the Image_N_LR.
2. Stage 2 (Fine) runs SuperPoint *selectively* on the Image_N_HR (6.2K) to get high-accuracy keypoints.
3. It then matches small, full-resolution *patches* from the full-res image, *not* the full image.
This hybrid approach is the *only* way to meet both AC-7 (speed) and AC-2 (accuracy). The 6.2K image *cannot* be processed in <5s on an RTX 2060 (6GB VRAM 34). But its high-resolution *pixels* are needed for the 20m *accuracy*. Using full-res *patches* provides the pixel-level accuracy without the VRAM/compute cost.
A PnP/RANSAC solver then computes a high-confidence 6-DoF pose. This pose, converted to [Lat, Lon, Alt], is the **$Absolute_Metric_Anchor$** sent to the TOH.
## **6.0 Component 4: The Scale-Aware Trajectory Optimization Hub (TOH)**
This component is the system's "brain" and implements the "Continuously-Scaled Trajectory" principle (Section 2.1). It *replaces* the draft's flawed "Single Scale" optimizer.
### **6.1 The $Sim(3)$ Pose-Graph as the Optimization Backbone**
The central challenge of IMU-denied monocular SLAM is *scale drift*.11 The V-SLAM (Component 3) produces 6-DoF poses, but they are *unscaled* ($SE(3)$). The GAB (Component 4) produces *metric* 6-DoF poses ($SE(3)$).
The solution is to optimize the *entire graph* in the 7-DoF "Similarity" group, **$Sim(3)$**.11 This adds a 7th degree of freedom (scale, $s$) to the poses. The optimization backbone will be **Ceres Solver** 14, a SOTA C++ library for large, complex non-linear least-squares problems.
### **6.2 Advanced Scale-Drift Correction: Modeling Scale as a Per-Keyframe Parameter (Solves Flaw 1.3)**
This is the *core* of the ASTRAL optimizer, solving Flaw 1.3. The draft's flawed model ($Pose_Graph(Fragment_i) = \\{Pose_1...Pose_n, s_i\\}$) is replaced by ASTRAL's correct model: $Pose_Graph = \\{ (Pose_1, s_1), (Pose_2, s_2),..., (Pose_N, s_N) \\}$.
The graph is constructed as follows:
* **Nodes:** Each keyframe pose is a 7-DoF $Sim(3)$ variable $\\{s_k, R_k, t_k\\}$.
* **Edge 1 (V-SLAM):** A *relative* $Sim(3)$ constraint between $Pose_k$ and $Pose_{k+1}$ from the V-SLAM Front-End.
* **Edge 2 (GAB):** An *absolute* $SE(3)$ constraint on $Pose_j$ from a GAB anchor. This constraint *fixes* the 6-DoF pose $(R_j, t_j)$ to the metric GAB value and *fixes its scale* $s_j = 1.0$.
This "per-keyframe scale" model 15 enables "elastic" trajectory refinement. When the graph is a long, unscaled "chain" of V-SLAM constraints, a GAB anchor (Edge 2) arrives at $Pose_{100}$, "nailing" it to the metric map and setting $s_{100} = 1.0$. As the V-SLAM continues, scale drifts. When a second anchor arrives at $Pose_{200}$ (setting $s_{200} = 1.0$), the Ceres optimizer 14 has a problem: the V-SLAM data *between* them has drifted.
The ASTRAL model *allows* the optimizer to solve for all intermediate scales (s_{101}, s_{102},..., s_{199}) as variables. The optimizer will find a *smooth, continuous* scale correction 15 that "elastically" stretches/shrinks the 100-frame sub-segment to *perfectly* fit both metric anchors. This *correctly* models the physics of scale drift 12 and is the *only* way to achieve the 20m accuracy (AC-2) and 1.0px MRE (AC-10).
### **6.3 Robust M-Estimation (Solution for AC-3, AC-5)**
A 350m outlier (AC-3) or a bad GAB match (AC-5) will add a constraint with a *massive* error. A standard least-squares optimizer 14 would be *catastrophically* corrupted, pulling the *entire* 3000-image trajectory to try and fit this one bad point.
This is a solved problem. All constraints (V-SLAM and GAB) *must* be wrapped in a **Robust Loss Function** (e.g., HuberLoss, CauchyLoss) within Ceres Solver. This function mathematically *down-weights* the influence of constraints with large errors (high residuals). It effectively tells the optimizer: "This measurement is insane. Ignore it." This provides automatic, graceful outlier rejection, meeting AC-3 and AC-5.
### **6.4 Geodetic Map-Merging (Solution for AC-4, AC-6)**
This mechanism is the robust solution to the "sharp turn" (AC-4) problem.
* **Scenario:** The UAV makes a sharp turn (AC-4). The V-SLAM (4.2) creates Map_Fragment_0 and Map_Fragment_1. The TOH's graph now has two *disconnected* components.
* **Mechanism (Geodetic Merging):**
1. The TOH queues the GAB (Section 5.0) to find anchors for *both* fragments.
2. GAB returns Anchor_A for Map_Fragment_0 and Anchor_B for Map_Fragment_1.
3. The TOH adds *both* of these as absolute, metric constraints (Edge 2) to the *single global pose-graph*.
4. The Ceres optimizer 14 now has all the information it needs. It solves for the 7-Dof pose of *both fragments*, placing them in their correct, globally-consistent metric positions.
The two fragments are *merged geodetically* (by their global coordinates 11) even if they *never* visually overlap. This is a vastly more robust solution to AC-4 and AC-6 than simple visual loop closure.
## **7.0 Performance, Deployment, and High-Accuracy Outputs**
### **7.1 Meeting the <5s Budget (AC-7): Mandatory Acceleration with NVIDIA TensorRT**
The system must run on an RTX 2060 (AC-7). This is a low-end, 6GB VRAM card 34, which is a *severe* constraint. Running three deep-learning models (SuperPoint, LightGlue, SALAD/MixVPR) plus a Ceres optimizer 38 will saturate this hardware.
* **Solution 1: Multi-Scale Pipeline.** As defined in 5.3, the system *never* processes a full 6.2K image on the GPU. It uses low-res for V-SLAM/GAB-Coarse and high-res *patches* for GAB-Fine.
* **Solution 2: Mandatory TensorRT Deployment.** Running these models in their native PyTorch framework will be too slow. All neural networks (SuperPoint, LightGlue, SALAD/MixVPR) *must* be converted from PyTorch into optimized **NVIDIA TensorRT engines**. Research *specifically* on accelerating LightGlue shows this provides **"2x-4x speed gains over compiled PyTorch"**.35 This 200-400% speedup is *not* an optimization; it is a *mandatory deployment step* to make the <5s (AC-7) budget *possible* on an RTX 2060.
### **7.2 High-Accuracy Object Geolocalization via Ray-Cloud Intersection (Solves AC-2/AC-10)**
The user must be able to find the GPS of an *object* in a photo. A simple approach of ray-casting from the camera and intersecting with the 30m GLO-30 DEM 2 is fatally flawed. The DEM error itself can be up to 30m 19, making AC-2 impossible.
The ASTRAL system uses a **Ray-Cloud Intersection** method that *decouples* object accuracy from external DEM accuracy.
* **Algorithm:**
1. The user clicks pixel (u,v) on Image_N.
2. The system retrieves the *final, refined, metric 7-DoF pose* P_{sim(3)} = (s, R, T) for Image_N from the TOH.
3. It also retrieves the V-SLAM's *local, high-fidelity 3D point cloud* (P_{local_cloud}) from Component 3 (Section 4.3).
4. **Step 1 (Local):** The pixel (u,v) is un-projected into a ray. This ray is intersected with the *local* P_{local_cloud}. This finds the 3D point $P_{local} *relative to the V-SLAM map*. The accuracy of this step is defined by AC-10 (MRE < 1.0px).
5. **Step 2 (Global):** This *highly-accurate* local point P_{local} is transformed into the global metric coordinate system using the *highly-accurate* refined pose from the TOH: P_{metric} = s * (R * P_{local}) + T.
6. **Step 3 (Convert):** P_{metric} (an X,Y,Z world coordinate) is converted to [Latitude, Longitude, Altitude].
This method correctly isolates error. The object's accuracy is now *only* dependent on the V-SLAM's internal geometry (AC-10) and the TOH's global pose accuracy (AC-1, AC-2). It *completely eliminates* the external 30m DEM error 2 from this critical, high-accuracy calculation.
### **7.3 Failure Mode Escalation Logic (Meets AC-3, AC-4, AC-6, AC-9)**
The system is built on a robust state machine to handle real-world failures.
* **Stage 1: Normal Operation (Tracking):** V-SLAM tracks, TOH optimizes.
* **Stage 2: Transient VO Failure (Outlier Rejection):**
* *Condition:* Image_N is a 350m outlier (AC-3) or severe blur.
* *Logic:* V-SLAM fails to track Image_N. System *discards* it (AC-5). Image_N+1 arrives, V-SLAM re-tracks.
* *Result:* **AC-3 Met.**
* **Stage 3: Persistent VO Failure (New Map Initialization):**
* *Condition:* "Sharp turn" (AC-4) or >5 frames of tracking loss.
* *Logic:* V-SLAM (Section 4.2) declares "Tracking Lost." Initializes *new* Map_Fragment_k+1. Tracking *resumes instantly*.
* *Result:* **AC-4 Met.** System "correctly continues the work." The >95% registration rate (AC-9) is met because this is *not* a failure, it's a *new registration*.
* **Stage 4: Map-Merging & Global Relocalization (GAB-Assisted):**
* *Condition:* System is on Map_Fragment_k+1, Map_Fragment_k is "lost."
* *Logic:* TOH (Section 6.4) receives GAB anchors for *both* fragments and *geodetically merges* them in the global optimizer.14
* *Result:* **AC-6 Met** (strategy to connect separate chunks).
* **Stage 5: Catastrophic Failure (User Intervention):**
* *Condition:* System is in Stage 3 (Lost) *and* the GAB has failed for 20% of the route. The "absolutely incapable" scenario (AC-6).
* *Logic:* TOH triggers the AC-6 flag. UI prompts user: "Please provide a coarse location for the *current* image."
* *Action:* This user-click is *not* taken as ground-truth. It is fed to the **GAB (Section 5.0)** as a *strong spatial prior*, narrowing its Stage 1 8 search from "the entire AOI" to "a 5km radius." This *guarantees* the GAB finds a match, which triggers Stage 4, re-localizing the system.
* *Result:* **AC-6 Met** (user input).
## **8.0 ASTRAL Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report.
At the very beginning of the report, list the most profound changes you've made to the previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks; do not compare to the previous solution draft, just make a new solution as if from scratch.
Also, find out more ideas, like a
- A Cross-View Geo-Localization Algorithm Using UAV Image
https://www.mdpi.com/1424-8220/24/12/3719
- Exploring the best way for UAV visual localization under Low-altitude Multi-view Observation condition
https://arxiv.org/pdf/2503.10692
Assess them and try to either integrate or replace some of the components in the current solution draft
_docs/00_problem/1.3_05_assesment_prompt.md
-373
View File
@@ -1,373 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 200m drift and at an angle of less than 70%
- System should try to operate when UAV made a sharp turn, and all the next photos has no common points with previous route. In that situation system should try to figure out location of the new piece of the route and connect it to the previous route. Also this separate chunks could be more than 2, so this strategy should be in the core of the system
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
- The whole system should work as a background service. The interaction should be done by zeromq. Sevice should be up and running and awaiting for the initial input message. On the input message processing should started, and immediately after the first results system should provide them to the client
Here is a solution draft:
# **ASTRAL System Architecture: A High-Fidelity Geopositioning Framework for IMU-Denied Aerial Operations**
## **2.0 The ASTRAL (Advanced Scale-Aware Trajectory-Refinement and Localization) System Architecture**
The ASTRAL architecture is a multi-map, decoupled, loosely-coupled system designed to solve the flaws identified in Section 1.0 and meet all 10 Acceptance Criteria.
### **2.1 Core Principles**
The ASTRAL architecture is built on three principles:
1. **Tiered Geospatial Database:** The system *cannot* rely on a single data source. It is architected around a *tiered* local database.
* **Tier-1 (Baseline):** Google Maps data. This is used to meet the 50m (AC-1) requirement and provide geolocalization.
* **Tier-2 (High-Accuracy):** A framework for ingesting *commercial, sub-meter* data (visual 4; and DEM 5). This tier is *required* to meet the 20m (AC-2) accuracy. The system will *run* on Tier-1 but *achieve* AC-2 when "fueled" with Tier-2 data.
2. **Viewpoint-Invariant Anchoring:** The system *rejects* geometric warping. The GAB (Section 5.0) is built on SOTA Visual Place Recognition (VPR) models that are *inherently* invariant to the oblique-to-nadir viewpoint change, decoupling it from the V-SLAM's unstable orientation.
3. **Continuously-Scaled Trajectory:** The system *rejects* the "single-scale-per-fragment" model. The TOH (Section 6.0) is a Sim(3) pose-graph optimizer 11 that models scale as a *per-keyframe optimizable parameter*.15 This allows the trajectory to "stretch" and "shrink" elastically to absorb continuous monocular scale drift.12
### **2.2 Component Interaction and Data Flow**
The system is multi-threaded and asynchronous, designed for real-time streaming (AC-7) and refinement (AC-8).
* **Component 1: Tiered GDB (Pre-Flight):**
* *Input:* User-defined Area of Interest (AOI).
* *Action:* Downloads and builds a local SpatiaLite/GeoPackage.
* *Output:* A single **Local-Geo-Database file** containing:
* Tier-1 (Google Maps) + GLO-30 DSM
* Tier-2 (Commercial) satellite tiles + WorldDEM DTM elevation tiles.
* A *pre-computed FAISS vector index* of global descriptors (e.g., SALAD 8) for *all* satellite tiles (see 3.4).
* **Component 2: Image Ingestion (Real-time):**
* *Input:* Image_N (up to 6.2K), Camera Intrinsics ($K$).
* *Action:* Creates Image_N_LR (Low-Res, e.g., 1536x1024) and Image_N_HR (High-Res, 6.2K).
* *Dispatch:* Image_N_LR -> V-SLAM. Image_N_HR -> GAB (for patches).
* **Component 3: "Atlas" V-SLAM Front-End (High-Frequency Thread):**
* *Input:* Image_N_LR.
* *Action:* Tracks Image_N_LR against the *active map fragment*. Manages keyframes and local BA. If tracking lost (AC-4, AC-6), it *initializes a new map fragment*.
* *Output:* Relative_Unscaled_Pose, Local_Point_Cloud, and Map_Fragment_ID -> TOH.
* **Component 4: VPR Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread):**
* *Input:* A keyframe (Image_N_LR, Image_N_HR) and its Map_Fragment_ID.
* *Action:* Performs SOTA two-stage VPR (Section 5.0) against the **Local-Geo-Database file**.
* *Output:* Absolute_Metric_Anchor ([Lat, Lon, Alt] pose) and its Map_Fragment_ID -> TOH.
* **Component 5: Scale-Aware Trajectory Optimization Hub (TOH) (Central Hub Thread):**
* *Input 1:* High-frequency Relative_Unscaled_Pose stream.
* *Input 2:* Low-frequency Absolute_Metric_Anchor stream.
* *Action:* Manages the *global Sim(3) pose-graph* 13 with *per-keyframe scale*.15
* *Output 1 (Real-time):* Pose_N_Est (unscaled) -> UI (Meets AC-7).
* *Output 2 (Refined):* Pose_N_Refined (metric-scale) -> UI (Meets AC-1, AC-2, AC-8).
### **2.3 System Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate [Lat, Lon].
3. **Camera Intrinsics (K):** Pre-calibrated camera intrinsic matrix.
4. **Local-Geo-Database File:** The single file generated by Component 1.
### **2.4 Streaming Outputs (Meets AC-7, AC-8)**
1. **Initial Pose (Pose_N^{Est}):** An *unscaled* pose. This is the raw output from the V-SLAM Front-End, transformed by the *current best estimate* of the trajectory. It is sent immediately (<5s, AC-7) to the UI for real-time visualization of the UAV's *path shape*.
2. **Refined Pose (Pose_N^{Refined}) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose. This is sent to the user *whenever the TOH re-converges* (e.g., after a new GAB anchor or a map-merge). This *re-writes* the history of poses (e.g., Pose_{N-100} to Pose_N), meeting the refinement (AC-8) and accuracy (AC-1, AC-2) requirements.
## **3.0 Component 1: The Tiered Pre-Flight Geospatial Database (GDB)**
This component is the implementation of the "Tiered Geospatial" principle. It is a mandatory pre-flight utility that solves both the *legal* problem (Flaw 1.4) and the *accuracy* problem (Flaw 1.1).
### **3.2 Tier-1 (Baseline): Google Maps and GLO-30 DEM**
This tier provides the baseline capability and satisfies AC-1.
* **Visual Data:** Google Maps (coarse Maxar)
* *Resolution:* 10m.
* *Geodetic Accuracy:* \~1 m to 20m
* *Purpose:* Meets AC-1 (80% < 50m error). Provides a robust baseline for coarse geolocalization.
* **Elevation Data:** Copernicus GLO-30 DEM
* *Resolution:* 30m.
* *Type:* DSM (Digital Surface Model).2 This is a *weakness*, as it includes buildings/trees.
* *Purpose:* Provides a coarse altitude prior for the TOH and the initial GAB search.
### **3.3 Tier-2 (High-Accuracy): Ingestion Framework for Commercial Data**
This is the *procurement and integration framework* required to meet AC-2.
* **Visual Data:** Commercial providers, e.g., Maxar (30-50cm) or Satellogic (70cm)
* *Resolution:* < 1m.
* *Geodetic Accuracy:* Typically < 5m.
* *Purpose:* Provides the high-resolution, high-accuracy reference needed for the GAB to achieve a sub-20m total error.
* **Elevation Data:** Commercial providers, e.g., WorldDEM Neo 5 or Elevation10.32
* *Resolution:* 5m-12m.
* *Vertical Accuracy:* < 4m.32
* *Type:* DTM (Digital Terrain Model).32
The use of a DTM (bare-earth) in Tier-2 is a critical advantage over the Tier-1 DSM (surface). The V-SLAM Front-End (Section 4.0) will triangulate a 3D point cloud of what it *sees*, which is the *ground* in fields or *tree-tops* in forests. The Tier-1 GLO-30 DSM 2 represents the *top* of the canopy/buildings. If the V-SLAM maps the *ground* (e.g., altitude 100m) and the GAB tries to anchor it to a DSM *prior* that shows a forest (e.g., altitude 120m), the 20m altitude discrepancy will introduce significant error into the TOH. The Tier-2 DTM (bare-earth) 5 provides a *vastly* superior altitude anchor, as it represents the same ground plane the V-SLAM is tracking, significantly improving the entire 7-DoF pose solution.
### **3.4 Local Database Generation: Pre-computing Global Descriptors**
This is the key performance optimization for the GAB. During the pre-flight caching step, the GDB utility does not just *store* tiles; it *processes* them.
For *every* satellite tile (e.g., 256x256m) in the AOI, the utility will load the tile into the VPR model (e.g., SALAD 8), compute its global descriptor (a compact feature vector), and store this vector in a high-speed vector index (e.g., FAISS).
This step moves 99% of the GAB's "Stage 1" (Coarse Retrieval) workload into an offline, pre-flight step. The *real-time* GAB query (Section 5.2) is now reduced to: (1) Compute *one* vector for the UAV image, and (2) Perform a very fast K-Nearest-Neighbor search on the pre-computed FAISS index. This is what makes a SOTA deep-learning GAB 6 fast enough to support the real-time refinement loop.
#### **Table 1: Geospatial Reference Data Analysis (Decision Matrix)**
| Data Product | Type | Resolution | Geodetic Accuracy (Horiz.) | Type | Cost | AC-2 (20m) Compliant? |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| Google Maps | Visual | 1m | 1m - 10m | N/A | Free | **Depending on the location** |
| Copernicus GLO-30 | Elevation | 30m | \~10-30m | **DSM** (Surface) | Free | **No (Fails Error Budget)** |
| **Tier-2: Maxar/Satellogic** | Visual | 0.3m - 0.7m | < 5 m (Est.) | N/A | Commercial | **Yes** |
| **Tier-2: WorldDEM Neo** | Elevation | 5m | < 4m | **DTM** (Bare-Earth) | Commercial | **Yes** |
## **4.0 Component 2: The "Atlas" Relative Motion Front-End**
This component's sole task is to robustly compute *unscaled* 6-DoF relative motion and handle tracking failures (AC-3, AC-4).
### **4.1 Feature Matching Sub-System: SuperPoint + LightGlue**
The system will use **SuperPoint** for feature detection and **LightGlue** for matching. This choice is driven by the project's specific constraints:
* **Rationale (Robustness):** The UAV flies over "eastern and southern parts of Ukraine," which includes large, low-texture agricultural areas. SuperPoint is a SOTA deep-learning detector renowned for its robustness and repeatability in these challenging, low-texture environments.
* **Rationale (Performance):** The RTX 2060 (AC-7) is a *hard* constraint with only 6GB VRAM.34 Performance is paramount. LightGlue is an SOTA matcher that provides a 4-10x speedup over its predecessor, SuperGlue. Its "adaptive" nature is a key optimization: it exits early on "easy" pairs (high-overlap, straight-flight) and spends more compute only on "hard" pairs (turns). This saves critical GPU budget on 95% of normal frames, ensuring the <5s (AC-7) budget is met.
This subsystem will run on the Image_N_LR (low-res) copy to guarantee it fits in VRAM and meets the real-time budget.
#### **Table 2: Analysis of State-of-the-Art Feature Matchers (V-SLAM Front-End)**
| Approach (Tools/Library) | Robustness (Low-Texture) | Speed (RTX 2060) | Fitness for Problem |
| :---- | :---- | :---- | :---- |
| ORB 33 (e.g., ORB-SLAM3) | Poor. Fails on low-texture. | Excellent (CPU/GPU) | **Good.** Fails robustness in target environment. |
| SuperPoint + SuperGlue | Excellent. | Good, but heavy. Fixed-depth GNN. 4-10x Slower than LightGlue.35 | **Good.** Robust, but risks AC-7 budget. |
| **SuperPoint + LightGlue** 35 | Excellent. | **Excellent.** Adaptive depth 35 saves budget. 4-10x faster. | **Excellent (Selected).** Balances robustness and performance. |
### **4.2 The "Atlas" Multi-Map Paradigm (Solution for AC-3, AC-4, AC-6)**
This architecture is the industry-standard solution for IMU-denied, long-term SLAM and is critical for robustness.
* **Mechanism (AC-4, Sharp Turn):**
1. The system is tracking on $Map_Fragment_0$.
2. The UAV makes a sharp turn (AC-4, <5% overlap). The V-SLAM *loses tracking*.
3. Instead of failing, the Atlas architecture *initializes a new map*: $Map_Fragment_1$.
4. Tracking *resumes instantly* on this new, unanchored map.
* **Mechanism (AC-3, 350m Outlier):**
1. The system is tracking. A 350m outlier $Image_N$ arrives.
2. The V-SLAM fails to match $Image_N$ (a "Transient VO Failure," see 7.3). It is *discarded*.
3. $Image_N+1$ arrives (back on track). V-SLAM re-acquires its location on $Map_Fragment_0$.
4. The system "correctly continues the work" (AC-3) by simply rejecting the outlier.
This design turns "catastrophic failure" (AC-3, AC-4) into a *standard operating procedure*. The "problem" of stitching the fragments ($Map_0$, $Map_1$) together is moved from the V-SLAM (which has no global context) to the TOH (which *can* solve it using GAB anchors, see 6.4).
### **4.3 Local Bundle Adjustment and High-Fidelity 3D Cloud**
The V-SLAM front-end will continuously run Local Bundle Adjustment (BA) over a sliding window of recent keyframes to minimize drift *within* that fragment. It will also triangulate a sparse, but high-fidelity, 3D point cloud for its *local map fragment*.
This 3D cloud serves a critical dual function:
1. It provides a robust 3D map for frame-to-map tracking, which is more stable than frame-to-frame odometry.
2. It serves as the **high-accuracy data source** for the object localization output (Section 7.2). This is the key to decoupling object-pointing accuracy from external DEM accuracy 19, a critical flaw in simpler designs.
## **5.0 Component 3: The Viewpoint-Invariant Geospatial Anchoring Back-End (GAB)**
This component *replaces* the draft's "Dynamic Warping" (Section 5.0) and implements the "Viewpoint-Invariant Anchoring" principle (Section 2.1).
### **5.1 Rationale: Viewpoint-Invariant VPR vs. Geometric Warping (Solves Flaw 1.2)**
As established in 1.2, geometrically warping the image using the V-SLAM's *drifty* roll/pitch estimate creates a *brittle*, high-risk failure spiral. The ASTRAL GAB *decouples* from the V-SLAM's orientation. It uses a SOTA VPR pipeline that *learns* to match oblique UAV images to nadir satellite images *directly*, at the feature level.6
### **5.2 Stage 1 (Coarse Retrieval): SOTA Global Descriptors**
When triggered by the TOH, the GAB takes Image_N_LR. It computes a *global descriptor* (a single feature vector) using a SOTA VPR model like **SALAD** 6 or **MixVPR**.7
This choice is driven by two factors:
1. **Viewpoint Invariance:** These models are SOTA for this exact task.
2. **Inference Speed:** They are extremely fast. SALAD reports < 3ms per image inference 8, and MixVPR is also noted for "fastest inference speed".37 This low overhead is essential for the AC-7 (<5s) budget.
This vector is used to query the *pre-computed FAISS vector index* (from 3.4), which returns the Top-K (e.g., K=5) most likely satellite tiles from the *entire AOI* in milliseconds.
#### **Table 3: Analysis of VPR Global Descriptors (GAB Back-End)**
| Model (Backbone) | Key Feature | Viewpoint Invariance | Inference Speed (ms) | Fitness for GAB |
| :---- | :---- | :---- | :---- | :---- |
| NetVLAD 7 (CNN) | Baseline | Poor. Not designed for oblique-to-nadir. | Moderate (\~20-50ms) | **Poor.** Fails robustness. |
| **SALAD** 8 (DINOv2) | Foundation Model.6 | **Excellent.** Designed for this. | **< 3ms**.8 Extremely fast. | **Excellent (Selected).** |
| **MixVPR** 36 (ResNet) | All-MLP aggregator.36 | **Very Good.**.7 | **Very Fast.**.37 | **Excellent (Selected).** |
### **5.3 Stage 2 (Fine): Local Feature Matching and Pose Refinement**
The system runs **SuperPoint+LightGlue** 35 to find pixel-level matches, but *only* between the UAV image and the **Top-K satellite tiles** identified in Stage 1.
A **Multi-Resolution Strategy** is employed to solve the VRAM bottleneck.
1. Stage 1 (Coarse) runs on the Image_N_LR.
2. Stage 2 (Fine) runs SuperPoint *selectively* on the Image_N_HR (6.2K) to get high-accuracy keypoints.
3. It then matches small, full-resolution *patches* from the full-res image, *not* the full image.
This hybrid approach is the *only* way to meet both AC-7 (speed) and AC-2 (accuracy). The 6.2K image *cannot* be processed in <5s on an RTX 2060 (6GB VRAM 34). But its high-resolution *pixels* are needed for the 20m *accuracy*. Using full-res *patches* provides the pixel-level accuracy without the VRAM/compute cost.
A PnP/RANSAC solver then computes a high-confidence 6-DoF pose. This pose, converted to [Lat, Lon, Alt], is the **$Absolute_Metric_Anchor$** sent to the TOH.
## **6.0 Component 4: The Scale-Aware Trajectory Optimization Hub (TOH)**
This component is the system's "brain" and implements the "Continuously-Scaled Trajectory" principle (Section 2.1). It *replaces* the draft's flawed "Single Scale" optimizer.
### **6.1 The $Sim(3)$ Pose-Graph as the Optimization Backbone**
The central challenge of IMU-denied monocular SLAM is *scale drift*.11 The V-SLAM (Component 3) produces 6-DoF poses, but they are *unscaled* ($SE(3)$). The GAB (Component 4) produces *metric* 6-DoF poses ($SE(3)$).
The solution is to optimize the *entire graph* in the 7-DoF "Similarity" group, **$Sim(3)$**.11 This adds a 7th degree of freedom (scale, $s$) to the poses. The optimization backbone will be **Ceres Solver** 14, a SOTA C++ library for large, complex non-linear least-squares problems.
### **6.2 Advanced Scale-Drift Correction: Modeling Scale as a Per-Keyframe Parameter (Solves Flaw 1.3)**
This is the *core* of the ASTRAL optimizer, solving Flaw 1.3. The draft's flawed model ($Pose_Graph(Fragment_i) = \\{Pose_1...Pose_n, s_i\\}$) is replaced by ASTRAL's correct model: $Pose_Graph = \\{ (Pose_1, s_1), (Pose_2, s_2),..., (Pose_N, s_N) \\}$.
The graph is constructed as follows:
* **Nodes:** Each keyframe pose is a 7-DoF $Sim(3)$ variable $\\{s_k, R_k, t_k\\}$.
* **Edge 1 (V-SLAM):** A *relative* $Sim(3)$ constraint between $Pose_k$ and $Pose_{k+1}$ from the V-SLAM Front-End.
* **Edge 2 (GAB):** An *absolute* $SE(3)$ constraint on $Pose_j$ from a GAB anchor. This constraint *fixes* the 6-DoF pose $(R_j, t_j)$ to the metric GAB value and *fixes its scale* $s_j = 1.0$.
This "per-keyframe scale" model 15 enables "elastic" trajectory refinement. When the graph is a long, unscaled "chain" of V-SLAM constraints, a GAB anchor (Edge 2) arrives at $Pose_{100}$, "nailing" it to the metric map and setting $s_{100} = 1.0$. As the V-SLAM continues, scale drifts. When a second anchor arrives at $Pose_{200}$ (setting $s_{200} = 1.0$), the Ceres optimizer 14 has a problem: the V-SLAM data *between* them has drifted.
The ASTRAL model *allows* the optimizer to solve for all intermediate scales (s_{101}, s_{102},..., s_{199}) as variables. The optimizer will find a *smooth, continuous* scale correction 15 that "elastically" stretches/shrinks the 100-frame sub-segment to *perfectly* fit both metric anchors. This *correctly* models the physics of scale drift 12 and is the *only* way to achieve the 20m accuracy (AC-2) and 1.0px MRE (AC-10).
### **6.3 Robust M-Estimation (Solution for AC-3, AC-5)**
A 350m outlier (AC-3) or a bad GAB match (AC-5) will add a constraint with a *massive* error. A standard least-squares optimizer 14 would be *catastrophically* corrupted, pulling the *entire* 3000-image trajectory to try and fit this one bad point.
This is a solved problem. All constraints (V-SLAM and GAB) *must* be wrapped in a **Robust Loss Function** (e.g., HuberLoss, CauchyLoss) within Ceres Solver. This function mathematically *down-weights* the influence of constraints with large errors (high residuals). It effectively tells the optimizer: "This measurement is insane. Ignore it." This provides automatic, graceful outlier rejection, meeting AC-3 and AC-5.
### **6.4 Geodetic Map-Merging (Solution for AC-4, AC-6)**
This mechanism is the robust solution to the "sharp turn" (AC-4) problem.
* **Scenario:** The UAV makes a sharp turn (AC-4). The V-SLAM (4.2) creates Map_Fragment_0 and Map_Fragment_1. The TOH's graph now has two *disconnected* components.
* **Mechanism (Geodetic Merging):**
1. The TOH queues the GAB (Section 5.0) to find anchors for *both* fragments.
2. GAB returns Anchor_A for Map_Fragment_0 and Anchor_B for Map_Fragment_1.
3. The TOH adds *both* of these as absolute, metric constraints (Edge 2) to the *single global pose-graph*.
4. The Ceres optimizer 14 now has all the information it needs. It solves for the 7-Dof pose of *both fragments*, placing them in their correct, globally-consistent metric positions.
The two fragments are *merged geodetically* (by their global coordinates 11) even if they *never* visually overlap. This is a vastly more robust solution to AC-4 and AC-6 than simple visual loop closure.
## **7.0 Performance, Deployment, and High-Accuracy Outputs**
### **7.1 Meeting the <5s Budget (AC-7): Mandatory Acceleration with NVIDIA TensorRT**
The system must run on an RTX 2060 (AC-7). This is a low-end, 6GB VRAM card 34, which is a *severe* constraint. Running three deep-learning models (SuperPoint, LightGlue, SALAD/MixVPR) plus a Ceres optimizer 38 will saturate this hardware.
* **Solution 1: Multi-Scale Pipeline.** As defined in 5.3, the system *never* processes a full 6.2K image on the GPU. It uses low-res for V-SLAM/GAB-Coarse and high-res *patches* for GAB-Fine.
* **Solution 2: Mandatory TensorRT Deployment.** Running these models in their native PyTorch framework will be too slow. All neural networks (SuperPoint, LightGlue, SALAD/MixVPR) *must* be converted from PyTorch into optimized **NVIDIA TensorRT engines**. Research *specifically* on accelerating LightGlue shows this provides **"2x-4x speed gains over compiled PyTorch"**.35 This 200-400% speedup is *not* an optimization; it is a *mandatory deployment step* to make the <5s (AC-7) budget *possible* on an RTX 2060.
### **7.2 High-Accuracy Object Geolocalization via Ray-Cloud Intersection (Solves AC-2/AC-10)**
The user must be able to find the GPS of an *object* in a photo. A simple approach of ray-casting from the camera and intersecting with the 30m GLO-30 DEM 2 is fatally flawed. The DEM error itself can be up to 30m 19, making AC-2 impossible.
The ASTRAL system uses a **Ray-Cloud Intersection** method that *decouples* object accuracy from external DEM accuracy.
* **Algorithm:**
1. The user clicks pixel (u,v) on Image_N.
2. The system retrieves the *final, refined, metric 7-DoF pose* P_{sim(3)} = (s, R, T) for Image_N from the TOH.
3. It also retrieves the V-SLAM's *local, high-fidelity 3D point cloud* (P_{local_cloud}) from Component 3 (Section 4.3).
4. **Step 1 (Local):** The pixel (u,v) is un-projected into a ray. This ray is intersected with the *local* P_{local_cloud}. This finds the 3D point $P_{local} *relative to the V-SLAM map*. The accuracy of this step is defined by AC-10 (MRE < 1.0px).
5. **Step 2 (Global):** This *highly-accurate* local point P_{local} is transformed into the global metric coordinate system using the *highly-accurate* refined pose from the TOH: P_{metric} = s * (R * P_{local}) + T.
6. **Step 3 (Convert):** P_{metric} (an X,Y,Z world coordinate) is converted to [Latitude, Longitude, Altitude].
This method correctly isolates error. The object's accuracy is now *only* dependent on the V-SLAM's internal geometry (AC-10) and the TOH's global pose accuracy (AC-1, AC-2). It *completely eliminates* the external 30m DEM error 2 from this critical, high-accuracy calculation.
### **7.3 Failure Mode Escalation Logic (Meets AC-3, AC-4, AC-6, AC-9)**
The system is built on a robust state machine to handle real-world failures.
* **Stage 1: Normal Operation (Tracking):** V-SLAM tracks, TOH optimizes.
* **Stage 2: Transient VO Failure (Outlier Rejection):**
* *Condition:* Image_N is a 350m outlier (AC-3) or severe blur.
* *Logic:* V-SLAM fails to track Image_N. System *discards* it (AC-5). Image_N+1 arrives, V-SLAM re-tracks.
* *Result:* **AC-3 Met.**
* **Stage 3: Persistent VO Failure (New Map Initialization):**
* *Condition:* "Sharp turn" (AC-4) or >5 frames of tracking loss.
* *Logic:* V-SLAM (Section 4.2) declares "Tracking Lost." Initializes *new* Map_Fragment_k+1. Tracking *resumes instantly*.
* *Result:* **AC-4 Met.** System "correctly continues the work." The >95% registration rate (AC-9) is met because this is *not* a failure, it's a *new registration*.
* **Stage 4: Map-Merging & Global Relocalization (GAB-Assisted):**
* *Condition:* System is on Map_Fragment_k+1, Map_Fragment_k is "lost."
* *Logic:* TOH (Section 6.4) receives GAB anchors for *both* fragments and *geodetically merges* them in the global optimizer.14
* *Result:* **AC-6 Met** (strategy to connect separate chunks).
* **Stage 5: Catastrophic Failure (User Intervention):**
* *Condition:* System is in Stage 3 (Lost) *and* the GAB has failed for 20% of the route. The "absolutely incapable" scenario (AC-6).
* *Logic:* TOH triggers the AC-6 flag. UI prompts user: "Please provide a coarse location for the *current* image."
* *Action:* This user-click is *not* taken as ground-truth. It is fed to the **GAB (Section 5.0)** as a *strong spatial prior*, narrowing its Stage 1 8 search from "the entire AOI" to "a 5km radius." This *guarantees* the GAB finds a match, which triggers Stage 4, re-localizing the system.
* *Result:* **AC-6 Met** (user input).
## **8.0 ASTRAL Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
Identify all potential weak points and problems. Address them and find out ways to solve them. Based on your findings, form a new solution draft in the same format.
If your finding requires a complete reorganization of the flow and different components, state it.
Put all the findings regarding what was weak and poor at the beginning of the report.
At the very beginning of the report list most profound changes you've made to previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
_docs/00_problem/1.3_06_assesment_prompt.md
-375
View File
@@ -1,375 +0,0 @@
Read carefully about the problem:
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
System has next restrictions and conditions:
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
Output of the system should address next acceptance criteria:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 200m drift and at an angle of less than 70%
- System should try to operate when UAV made a sharp turn, and all the next photos has no common points with previous route. In that situation system should try to figure out location of the new piece of the route and connect it to the previous route. Also this separate chunks could be more than 2, so this strategy should be in the core of the system
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
- The whole system should work as a background service. The interaction should be done by zeromq. Sevice should be up and running and awaiting for the initial input message. On the input message processing should started, and immediately after the first results system should provide them to the client
Here is a solution draft:
# **ASTRAL-Next: A Resilient, GNSS-Denied Geo-Localization Architecture for Wing-Type UAVs in Complex Semantic Environments**
## **1. Executive Summary and Operational Context**
The strategic necessity of operating Unmanned Aerial Vehicles (UAVs) in Global Navigation Satellite System (GNSS)-denied environments has precipitated a fundamental shift in autonomous navigation research. The specific operational profile under analysis—high-speed, fixed-wing UAVs operating without Inertial Measurement Units (IMU) over the visually homogenous and texture-repetitive terrain of Eastern and Southern Ukraine—presents a confluence of challenges that render traditional Simultaneous Localization and Mapping (SLAM) approaches insufficient. The target environment, characterized by vast agricultural expanses, seasonal variability, and potential conflict-induced terrain alteration, demands a navigation architecture that moves beyond simple visual odometry to a robust, multi-layered Absolute Visual Localization (AVL) system.
This report articulates the design and theoretical validation of **ASTRAL-Next**, a comprehensive architectural framework engineered to supersede the limitations of preliminary dead-reckoning solutions. By synthesizing state-of-the-art (SOTA) research emerging in 2024 and 2025, specifically leveraging **LiteSAM** for efficient cross-view matching 1, **AnyLoc** for universal place recognition 2, and **SuperPoint+LightGlue** for robust sequential tracking 1, the proposed system addresses the critical failure modes inherent in wing-type UAV flight dynamics. These dynamics include sharp banking maneuvers, significant pitch variations leading to ground sampling distance (GSD) disparities, and the potential for catastrophic track loss (the "kidnapped robot" problem).
The analysis indicates that relying solely on sequential image overlap is viable only for short-term trajectory smoothing. The core innovation of ASTRAL-Next lies in its "Hierarchical + Anchor" topology, which decouples the relative motion estimation from absolute global anchoring. This ensures that even during zero-overlap turns or 350-meter positional outliers caused by airframe tilt, the system can re-localize against a pre-cached satellite reference map within the required 5-second latency window.3 Furthermore, the system accounts for the semantic disconnect between live UAV imagery and potentially outdated satellite reference data (e.g., Google Maps) by prioritizing semantic geometry over pixel-level photometric consistency.
### **1.1 Operational Environment and Constraints Analysis**
The operational theater—specifically the left bank of the Dnipro River in Ukraine—imposes rigorous constraints on computer vision algorithms. The absence of IMU data removes the ability to directly sense acceleration and angular velocity, creating a scale ambiguity in monocular vision systems that must be resolved through external priors (altitude) and absolute reference data.
| Constraint Category | Specific Challenge | Implication for System Design |
| :---- | :---- | :---- |
| **Sensor Limitation** | **No IMU Data** | The system cannot distinguish between pure translation and camera rotation (pitch/roll) without visual references. Scale must be constrained via altitude priors and satellite matching.5 |
| **Flight Dynamics** | **Wing-Type UAV** | Unlike quadcopters, fixed-wing aircraft cannot hover. They bank to turn, causing horizon shifts and perspective distortions. "Sharp turns" result in 0% image overlap.6 |
| **Terrain Texture** | **Agricultural Fields** | Repetitive crop rows create aliasing for standard descriptors (SIFT/ORB). Feature matching requires context-aware deep learning methods (SuperPoint).7 |
| **Reference Data** | **Google Maps (2025)** | Public satellite data may be outdated or lower resolution than restricted military feeds. Matches must rely on invariant features (roads, tree lines) rather than ephemeral textures.9 |
| **Compute Hardware** | **NVIDIA RTX 2060/3070** | Algorithms must be optimized for TensorRT to meet the <5s per frame requirement. Heavy transformers (e.g., ViT-Huge) are prohibitive; efficient architectures (LiteSAM) are required.1 |
The confluence of these factors necessitates a move away from simple "dead reckoning" (accumulating relative movements) which drifts exponentially. Instead, ASTRAL-Next operates as a **Global-Local Hybrid System**, where a high-frequency visual odometry layer handles frame-to-frame continuity, while a parallel global localization layer periodically "resets" the drift by anchoring the UAV to the satellite map.
## **2. Architectural Critique of Legacy Approaches**
The initial draft solution ("ASTRAL") and similar legacy approaches typically rely on a unified SLAM pipeline, often attempting to use the same feature extractors for both sequential tracking and global localization. Recent literature highlights substantial deficiencies in this monolithic approach, particularly when applied to the specific constraints of this project.
### **2.1 The Failure of Classical Descriptors in Agricultural Settings**
Classical feature descriptors like SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) rely on detecting "corners" and "blobs" based on local pixel intensity gradients. In the agricultural landscapes of Eastern Ukraine, this approach faces severe aliasing. A field of sunflowers or wheat presents thousands of identical "blobs," causing the nearest-neighbor matching stage to generate a high ratio of outliers.8
Research demonstrates that deep-learning-based feature extractors, specifically SuperPoint, trained on large datasets of synthetic and real-world imagery, learn to identify interest points that are semantically significant (e.g., the intersection of a tractor path and a crop line) rather than just texturally distinct.1 Consequently, a redesign must replace SIFT/ORB with SuperPoint for the front-end tracking.
### **2.2 The Inadequacy of Dead Reckoning without IMU**
In a standard Visual-Inertial Odometry (VIO) system, the IMU provides a high-frequency prediction of the camera's pose, which the visual system then refines. Without an IMU, the system is purely Visual Odometry (VO). In VO, the scale of the world is unobservable from a single camera (monocular scale ambiguity). A 1-meter movement of a small object looks identical to a 10-meter movement of a large object.5
While the prompt specifies a "predefined altitude," relying on this as a static constant is dangerous due to terrain undulations and barometric drift. ASTRAL-Next must implement a Scale-Constrained Bundle Adjustment, treating the altitude not as a hard fact, but as a strong prior that prevents the scale drift common in monocular systems.5
### **2.3 Vulnerability to "Kidnapped Robot" Scenarios**
The requirement to recover from sharp turns where the "next photo doesn't overlap at all" describes the classic "Kidnapped Robot Problem" in robotics—where a robot is teleported to an unknown location and must relocalize.14
Sequential matching algorithms (optical flow, feature tracking) function on the assumption of overlap. When overlap is zero, these algorithms fail catastrophically. The legacy solution's reliance on continuous tracking makes it fragile to these flight dynamics. The redesigned architecture must incorporate a dedicated Global Place Recognition module that treats every frame as a potential independent query against the satellite database, independent of the previous frame's history.2
## **3. ASTRAL-Next: System Architecture and Methodology**
To meet the acceptance criteria—specifically the 80% success rate within 50m error and the <5 second processing time—ASTRAL-Next utilizes a tri-layer processing topology. These layers operate concurrently, feeding into a central state estimator.
### **3.1 The Tri-Layer Localization Strategy**
The architecture separates the concerns of continuity, recovery, and precision into three distinct algorithmic pathways.
| Layer | Functionality | Algorithm | Latency | Role in Acceptance Criteria |
| :---- | :---- | :---- | :---- | :---- |
| **L1: Sequential Tracking** | Frame-to-Frame Relative Pose | **SuperPoint + LightGlue** | \~50-100ms | Handles continuous flight, bridges small gaps (overlap < 5%), and maintains trajectory smoothness. Essential for the 100m spacing requirement. 1 |
| **L2: Global Re-Localization** | "Kidnapped Robot" Recovery | **AnyLoc (DINOv2 + VLAD)** | \~200ms | Detects location after sharp turns (0% overlap) or track loss. Matches current view to the satellite database tile. Addresses the sharp turn recovery criterion. 2 |
| **L3: Metric Refinement** | Precise GPS Anchoring | **LiteSAM / HLoc** | \~300-500ms | "Stitches" the UAV image to the satellite tile with pixel-level accuracy to reset drift. Ensures the "80% < 50m" and "60% < 20m" accuracy targets. 1 |
### **3.2 Data Flow and State Estimation**
The system utilizes a **Factor Graph Optimization** (using libraries like GTSAM) as the central "brain."
1. **Inputs:**
* **Relative Factors:** Provided by Layer 1 (Change in pose from $t-1$ to $t$).
* **Absolute Factors:** Provided by Layer 3 (Global GPS coordinate at $t$).
* **Priors:** Altitude constraint and Ground Plane assumption.
2. **Processing:** The factor graph optimizes the trajectory by minimizing the error between these conflicting constraints.
3. **Output:** A smoothed, globally consistent trajectory $(x, y, z, \\text{roll}, \\text{pitch}, \\text{yaw})$ for every image timestamp.
### **3.3 ZeroMQ Background Service Architecture**
As per the requirement, the system operates as a background service.
* **Communication Pattern:** The service utilizes a REP-REQ (Reply-Request) pattern for control commands (Start/Stop/Reset) and a PUB-SUB (Publish-Subscribe) pattern for the continuous stream of localization results.
* **Concurrency:** Layer 1 runs on a high-priority thread to ensure immediate feedback. Layers 2 and 3 run asynchronously; when a global match is found, the result is injected into the Factor Graph, which then "back-propagates" the correction to previous frames, refining the entire recent trajectory.
## **4. Layer 1: Robust Sequential Visual Odometry**
The first line of defense against localization loss is robust tracking between consecutive UAV images. Given the challenging agricultural environment, standard feature matching is prone to failure. ASTRAL-Next employs **SuperPoint** and **LightGlue**.
### **4.1 SuperPoint: Semantic Feature Detection**
SuperPoint is a fully convolutional neural network trained to detect interest points and compute their descriptors. Unlike SIFT, which uses handcrafted mathematics to find corners, SuperPoint is trained via self-supervision on millions of images.
* **Relevance to Ukraine:** In a wheat field, SIFT might latch onto hundreds of identical wheat stalks. SuperPoint, however, learns to prioritize more stable features, such as the boundary between the field and a dirt road, or a specific patch of discoloration in the crop canopy.1
* **Performance:** SuperPoint runs efficiently on the RTX 2060/3070, with inference times around 15ms per image when optimized with TensorRT.16
### **4.2 LightGlue: The Attention-Based Matcher**
**LightGlue** represents a paradigm shift from the traditional "Nearest Neighbor + RANSAC" matching pipeline. It is a deep neural network that takes two sets of SuperPoint features and jointly predicts the matches.
* **Mechanism:** LightGlue uses a transformer-based attention mechanism. It allows features in Image A to "look at" all features in Image B (and vice versa) to determine the best correspondence. Crucially, it has a "dustbin" mechanism to explicitly reject points that have no match (occlusion or field of view change).12
* **Addressing the <5% Overlap:** The user specifies handling overlaps of "less than 5%." Traditional RANSAC fails here because the inlier ratio is too low. LightGlue, however, can confidently identify the few remaining matches because its attention mechanism considers the global geometric context of the points. If only a single road intersection is visible in the corner of both images, LightGlue is significantly more likely to match it correctly than SIFT.8
* **Efficiency:** LightGlue is designed to be "light." It features an adaptive depth mechanism—if the images are easy to match, it exits early. If they are hard (low overlap), it uses more layers. This adaptability is perfect for the variable difficulty of the UAV flight path.19
## **5. Layer 2: Global Place Recognition (The "Kidnapped Robot" Solver)**
When the UAV executes a sharp turn, resulting in a completely new view (0% overlap), sequential tracking (Layer 1) is mathematically impossible. The system must recognize the new terrain solely based on its appearance. This is the domain of **AnyLoc**.
### **5.1 Universal Place Recognition with Foundation Models**
**AnyLoc** leverages **DINOv2**, a massive self-supervised vision transformer developed by Meta. DINOv2 is unique because it is not trained with labels; it is trained to understand the geometry and semantic layout of images.
* **Why DINOv2 for Satellite Matching:** Satellite images and UAV images have different "domains." The satellite image might be from summer (green), while the UAV flies in autumn (brown). DINOv2 features are remarkably invariant to these texture changes. It "sees" the shape of the road network or the layout of the field boundaries, rather than the color of the leaves.2
* **VLAD Aggregation:** AnyLoc extracts dense features from the image using DINOv2 and aggregates them using **VLAD** (Vector of Locally Aggregated Descriptors) into a single, compact vector (e.g., 4096 dimensions). This vector represents the "fingerprint" of the location.21
### **5.2 Implementation Strategy**
1. **Database Preparation:** Before the mission, the system downloads the satellite imagery for the operational bounding box (Eastern/Southern Ukraine). These images are tiled (e.g., 512x512 pixels with overlap) and processed through AnyLoc to generate a database of descriptors.
2. **Faiss Indexing:** These descriptors are indexed using **Faiss**, a library for efficient similarity search.
3. **In-Flight Retrieval:** When Layer 1 reports a loss of tracking (or periodically), the current UAV image is processed by AnyLoc. The resulting vector is queried against the Faiss index.
4. **Result:** The system retrieves the top-5 most similar satellite tiles. These tiles represent the coarse global location of the UAV (e.g., "You are in Grid Square B7").2
## **6. Layer 3: Fine-Grained Metric Localization (LiteSAM)**
Retrieving the correct satellite tile (Layer 2) gives a location error of roughly the tile size (e.g., 200 meters). To meet the "60% < 20m" and "80% < 50m" criteria, the system must precisely align the UAV image onto the satellite tile. ASTRAL-Next utilizes **LiteSAM**.
### **6.1 Justification for LiteSAM over TransFG**
While **TransFG** (Transformer for Fine-Grained recognition) is a powerful architecture for cross-view geo-localization, it is computationally heavy.23 **LiteSAM** (Lightweight Satellite-Aerial Matching) is specifically architected for resource-constrained platforms (like UAV onboard computers or efficient ground stations) while maintaining state-of-the-art accuracy.
* **Architecture:** LiteSAM utilizes a **Token Aggregation-Interaction Transformer (TAIFormer)**. It employs a convolutional token mixer (CTM) to model correlations between the UAV and satellite images.
* **Multi-Scale Processing:** LiteSAM processes features at multiple scales. This is critical because the UAV altitude varies (<1km), meaning the scale of objects in the UAV image will not perfectly match the fixed scale of the satellite image (Google Maps Zoom Level 19). LiteSAM's multi-scale approach inherently handles this discrepancy.1
* **Performance Data:** Empirical benchmarks on the **UAV-VisLoc** dataset show LiteSAM achieving an RMSE@30 (Root Mean Square Error within 30 meters) of 17.86 meters, directly supporting the project's accuracy requirements. Its inference time is approximately 61.98ms on standard GPUs, ensuring it fits within the overall 5-second budget.1
### **6.2 The Alignment Process**
1. **Input:** The UAV Image and the Top-1 Satellite Tile from Layer 2.
2. **Processing:** LiteSAM computes the dense correspondence field between the two images.
3. **Homography Estimation:** Using the correspondences, the system computes a homography matrix $H$ that maps pixels in the UAV image to pixels in the georeferenced satellite tile.
4. **Pose Extraction:** The camera's absolute GPS position is derived from this homography, utilizing the known GSD of the satellite tile.18
## **7. Satellite Data Management and Coordinate Systems**
The reliability of the entire system hinges on the quality and handling of the reference map data. The restriction to "Google Maps" necessitates a rigorous approach to coordinate transformation and data freshness management.
### **7.1 Google Maps Static API and Mercator Projection**
The Google Maps Static API delivers images without embedded georeferencing metadata (GeoTIFF tags). The system must mathematically derive the bounding box of each downloaded tile to assign coordinates to the pixels. Google Maps uses the **Web Mercator Projection (EPSG:3857)**.
The system must implement the following derivation to establish the **Ground Sampling Distance (GSD)**, or meters_per_pixel, which varies significantly with latitude:
$$ \\text{meters_per_pixel} = 156543.03392 \\times \\frac{\\cos(\\text{latitude} \\times \\frac{\\pi}{180})}{2^{\\text{zoom}}} $$
For the operational region (Ukraine, approx. Latitude 48N):
* At **Zoom Level 19**, the resolution is approximately 0.30 meters/pixel. This resolution is compatible with the input UAV imagery (Full HD at <1km altitude), providing sufficient detail for the LiteSAM matcher.24
**Bounding Box Calculation Algorithm:**
1. **Input:** Center Coordinate $(lat, lon)$, Zoom Level ($z$), Image Size $(w, h)$.
2. **Project to World Coordinates:** Convert $(lat, lon)$ to world pixel coordinates $(px, py)$ at the given zoom level.
3. **Corner Calculation:**
* px_{NW} = px - (w / 2)
* py_{NW} = py - (h / 2)
4. Inverse Projection: Convert $(px_{NW}, py_{NW})$ back to Latitude/Longitude to get the North-West corner. Repeat for South-East.
This calculation is critical. A precision error here translates directly to a systematic bias in the final GPS output.
### **7.2 Mitigating Data Obsolescence (The 2025 Problem)**
The provided research highlights that satellite imagery access over Ukraine is subject to restrictions and delays (e.g., Maxar restrictions in 2025).10 Google Maps data may be several years old.
* **Semantic Anchoring:** This reinforces the selection of **AnyLoc** (Layer 2) and **LiteSAM** (Layer 3). These algorithms are trained to ignore transient features (cars, temporary structures, vegetation color) and focus on persistent structural features (road geometry, building footprints).
* **Seasonality:** Research indicates that DINOv2 features (used in AnyLoc) exhibit strong robustness to seasonal changes (e.g., winter satellite map vs. summer UAV flight), maintaining high retrieval recall where pixel-based methods fail.17
## **8. Optimization and State Estimation (The "Brain")**
The individual outputs of the visual layers are noisy. Layer 1 drifts over time; Layer 3 may have occasional outliers. The **Factor Graph Optimization** fuses these inputs into a coherent trajectory.
### **8.1 Handling the 350-Meter Outlier (Tilt)**
The prompt specifies that "up to 350 meters of an outlier... could happen due to tilt." This large displacement masquerading as translation is a classic source of divergence in Kalman Filters.
* **Robust Cost Functions:** In the Factor Graph, the error terms for the visual factors are wrapped in a **Robust Kernel** (specifically the **Cauchy** or **Huber** kernel).
* *Mechanism:* Standard least-squares optimization penalizes errors quadratically ($e^2$). If a 350m error occurs, the penalty is massive, dragging the entire trajectory off-course. A robust kernel changes the penalty to be linear ($|e|$) or logarithmic after a certain threshold. This allows the optimizer to effectively "ignore" or down-weight the 350m jump if it contradicts the consensus of other measurements, treating it as a momentary outlier or solving for it as a rotation rather than a translation.19
### **8.2 The Altitude Soft Constraint**
To resolve the monocular scale ambiguity without IMU, the altitude ($h_{prior}$) is added as a **Unary Factor** to the graph.
* $E_{alt} = |
| z_{est} \- h_{prior} ||*{\\Sigma*{alt}}$
* $\\Sigma_{alt}$ (covariance) is set relatively high (soft constraint), allowing the visual odometry to adjust the altitude slightly to maintain consistency, but preventing the scale from collapsing to zero or exploding to infinity. This effectively creates an **Altimeter-Aided Monocular VIO** system, where the altimeter (virtual or barometric) replaces the accelerometer for scale determination.5
## **9. Implementation Specifications**
### **9.1 Hardware Acceleration (TensorRT)**
Meeting the <5 second per frame requirement on an RTX 2060 requires optimizing the deep learning models. Python/PyTorch inference is typically too slow due to overhead.
* **Model Export:** All core models (SuperPoint, LightGlue, LiteSAM) must be exported to **ONNX** (Open Neural Network Exchange) format.
* **TensorRT Compilation:** The ONNX models are then compiled into **TensorRT Engines**. This process performs graph fusion (combining multiple layers into one) and kernel auto-tuning (selecting the fastest GPU instructions for the specific RTX 2060/3070 architecture).26
* **Precision:** The models should be quantized to **FP16** (16-bit floating point). Research shows that FP16 inference on NVIDIA RTX cards offers a 2x-3x speedup with negligible loss in matching accuracy for these specific networks.16
### **9.2 Background Service Architecture (ZeroMQ)**
The system is encapsulated as a headless service.
**ZeroMQ Topology:**
* **Socket 1 (REP - Port 5555):** Command Interface. Accepts JSON messages:
* {"cmd": "START", "config": {"lat": 48.1, "lon": 37.5}}
* {"cmd": "USER_FIX", "lat": 48.22, "lon": 37.66} (Human-in-the-loop input).
* **Socket 2 (PUB - Port 5556):** Data Stream. Publishes JSON results for every frame:
* {"frame_id": 1024, "gps": [48.123, 37.123], "object_centers": [...], "status": "LOCKED", "confidence": 0.98}.
Asynchronous Pipeline:
The system utilizes a Python multiprocessing architecture. One process handles the camera/image ingest and ZeroMQ communication. A second process hosts the TensorRT engines and runs the Factor Graph. This ensures that the heavy computation of Bundle Adjustment does not block the receipt of new images or user commands.
## **10. Human-in-the-Loop Strategy**
The requirement stipulates that for the "20% of the route" where automation fails, the user must intervene. The system must proactively detect its own failure.
### **10.1 Failure Detection with PDM@K**
The system monitors the **PDM@K** (Positioning Distance Measurement) metric continuously.
* **Definition:** PDM@K measures the percentage of queries localized within $K$ meters.3
* **Real-Time Proxy:** In flight, we cannot know the true PDM (as we don't have ground truth). Instead, we use the **Marginal Covariance** from the Factor Graph. If the uncertainty ellipse for the current position grows larger than a radius of 50 meters, or if the **Image Registration Rate** (percentage of inliers in LightGlue/LiteSAM) drops below 10% for 3 consecutive frames, the system triggers a **Critical Failure Mode**.19
### **10.2 The User Interaction Workflow**
1. **Trigger:** Critical Failure Mode activated.
2. **Action:** The Service publishes a status {"status": "REQ_INPUT"} via ZeroMQ.
3. **Data Payload:** It sends the current UAV image and the top-3 retrieved satellite tiles (from Layer 2) to the client UI.
4. **User Input:** The user clicks a distinctive feature (e.g., a specific crossroad) in the UAV image and the corresponding point on the satellite map.
5. **Recovery:** This pair of points is treated as a **Hard Constraint** in the Factor Graph. The optimizer immediately snaps the trajectory to this user-defined anchor, resetting the covariance and effectively "healing" the localized track.19
## **11. Performance Evaluation and Benchmarks**
### **11.1 Accuracy Validation**
Based on the reported performance of the selected components in relevant datasets (UAV-VisLoc, AnyVisLoc):
* **LiteSAM** demonstrates an accuracy of 17.86m (RMSE) for cross-view matching. This aligns with the requirement that 60% of photos be within 20m error.18
* **AnyLoc** achieves high recall rates (Top-1 Recall > 85% on aerial benchmarks), supporting the recovery from sharp turns.2
* **Factor Graph Fusion:** By combining sequential and global measurements, the overall system error is expected to be lower than the individual component errors, satisfying the "80% within 50m" criterion.
### **11.2 Latency Analysis**
The breakdown of processing time per frame on an RTX 3070 is estimated as follows:
* **SuperPoint + LightGlue:** \~50ms.1
* **AnyLoc (Global Retrieval):** \~150ms (run only on keyframes or tracking loss).
* **LiteSAM (Metric Refinement):** \~60ms.1
* **Factor Graph Optimization:** \~100ms (using incremental updates/iSAM2).
* Total: \~360ms per frame (worst case with all layers active).
This is an order of magnitude faster than the 5-second limit, providing ample headroom for higher resolution processing or background tasks.
## **12.0 ASTRAL-Next Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
Also, here are more detailed validation plan:
## **ASTRAL Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
Put all the findings what was weak and poor at the beginning of the report. Put here all new findings, what was updated, replaced, or removed from the previous solution.
Then form a new solution design without referencing the previous system. Remove Poor and Very Poor component choices from the component analysis tables, but leave Good and Excellent ones.
In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch
Also, investigate these ideas:
- A Cross-View Geo-Localization Algorithm Using UAV Image
https://www.mdpi.com/1424-8220/24/12/3719
- Exploring the best way for UAV visual localization under Low-altitude Multi-view Observation condition
https://arxiv.org/pdf/2503.10692
and find out more like this.
Assess them and try to either integrate or replace some of the components in the current solution draft
_docs/00_problem/1.3_07_assesment_prompt.md
-362
View File
@@ -1,362 +0,0 @@
## The problem description
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD. Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
## Data samples
Are in attachments: images and csv
## Restrictions for the input data
- Photos are taken by only airplane type UAVs.
- Photos are taken by the camera pointing downwards and fixed, but it is not autostabilized.
- The flying range is restricted by the eastern and southern parts of Ukraine (To the left of the Dnipro River)
- The image resolution could be from FullHD to 6252*4168. Camera parameters are known: focal length, sensor width, resolution and so on.
- Altitude is predefined and no more than 1km. The height of the terrain can be neglected.
- There is NO data from IMU
- Flights are done mostly in sunny weather
- We can use satellite providers, but we're limited right now to Google Maps, which could be outdated for some regions
- Number of photos could be up to 3000, usually in the 500-1500 range
- During the flight, UAVs can make sharp turns, so that the next photo may be absolutely different from the previous one (no same objects), but it is rather an exception than the rule
- Processing is done on a stationary computer or laptop with NVidia GPU at least RTX2060, better 3070. (For the UAV solution Jetson Orin Nano would be used, but that is out of scope.)
## Acceptance criteria for the output of the system:
- The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS
- The system should find out the GPS of centers of 60% of the photos from the flight within an error of no more than 20 meters in comparison to the real GPS
- The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
- System should correctly continue the work even during sharp turns, where the next photo doesn't overlap at all, or overlaps in less than 5%. The next photo should be in less than 200m drift and at an angle of less than 70%
- System should try to operate when UAV made a sharp turn, and all the next photos has no common points with previous route. In that situation system should try to figure out location of the new piece of the route and connect it to the previous route. Also this separate chunks could be more than 2, so this strategy should be in the core of the system
- In case of being absolutely incapable of determining the system to determine next, second next, and third next images GPS, by any means (these 20% of the route), then it should ask the user for input for the next image, so that the user can specify the location
- Less than 5 seconds for processing one image
- Results of image processing should appear immediately to user, so that user shouldn't wait for the whole route to complete in order to analyze first results. Also, system could refine existing calculated results and send refined results again to user
- Image Registration Rate > 95%. The system can find enough matching features to confidently calculate the camera's 6-DoF pose (position and orientation) and "stitch" that image into the final trajectory
- Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location.
- The whole system should work as a background service. The interaction should be done by zeromq. Sevice should be up and running and awaiting for the initial input message. On the input message processing should started, and immediately after the first results system should provide them to the client
## Existing solution draft:
# **ASTRAL-Next: A Resilient, GNSS-Denied Geo-Localization Architecture for Wing-Type UAVs in Complex Semantic Environments**
## **1. Executive Summary and Operational Context**
The strategic necessity of operating Unmanned Aerial Vehicles (UAVs) in Global Navigation Satellite System (GNSS)-denied environments has precipitated a fundamental shift in autonomous navigation research. The specific operational profile under analysis—high-speed, fixed-wing UAVs operating without Inertial Measurement Units (IMU) over the visually homogenous and texture-repetitive terrain of Eastern and Southern Ukraine—presents a confluence of challenges that render traditional Simultaneous Localization and Mapping (SLAM) approaches insufficient. The target environment, characterized by vast agricultural expanses, seasonal variability, and potential conflict-induced terrain alteration, demands a navigation architecture that moves beyond simple visual odometry to a robust, multi-layered Absolute Visual Localization (AVL) system.
This report articulates the design and theoretical validation of **ASTRAL-Next**, a comprehensive architectural framework engineered to supersede the limitations of preliminary dead-reckoning solutions. By synthesizing state-of-the-art (SOTA) research emerging in 2024 and 2025, specifically leveraging **LiteSAM** for efficient cross-view matching 1, **AnyLoc** for universal place recognition 2, and **SuperPoint+LightGlue** for robust sequential tracking 1, the proposed system addresses the critical failure modes inherent in wing-type UAV flight dynamics. These dynamics include sharp banking maneuvers, significant pitch variations leading to ground sampling distance (GSD) disparities, and the potential for catastrophic track loss (the "kidnapped robot" problem).
The analysis indicates that relying solely on sequential image overlap is viable only for short-term trajectory smoothing. The core innovation of ASTRAL-Next lies in its "Hierarchical + Anchor" topology, which decouples the relative motion estimation from absolute global anchoring. This ensures that even during zero-overlap turns or 350-meter positional outliers caused by airframe tilt, the system can re-localize against a pre-cached satellite reference map within the required 5-second latency window.3 Furthermore, the system accounts for the semantic disconnect between live UAV imagery and potentially outdated satellite reference data (e.g., Google Maps) by prioritizing semantic geometry over pixel-level photometric consistency.
### **1.1 Operational Environment and Constraints Analysis**
The operational theater—specifically the left bank of the Dnipro River in Ukraine—imposes rigorous constraints on computer vision algorithms. The absence of IMU data removes the ability to directly sense acceleration and angular velocity, creating a scale ambiguity in monocular vision systems that must be resolved through external priors (altitude) and absolute reference data.
| Constraint Category | Specific Challenge | Implication for System Design |
| :---- | :---- | :---- |
| **Sensor Limitation** | **No IMU Data** | The system cannot distinguish between pure translation and camera rotation (pitch/roll) without visual references. Scale must be constrained via altitude priors and satellite matching.5 |
| **Flight Dynamics** | **Wing-Type UAV** | Unlike quadcopters, fixed-wing aircraft cannot hover. They bank to turn, causing horizon shifts and perspective distortions. "Sharp turns" result in 0% image overlap.6 |
| **Terrain Texture** | **Agricultural Fields** | Repetitive crop rows create aliasing for standard descriptors (SIFT/ORB). Feature matching requires context-aware deep learning methods (SuperPoint).7 |
| **Reference Data** | **Google Maps (2025)** | Public satellite data may be outdated or lower resolution than restricted military feeds. Matches must rely on invariant features (roads, tree lines) rather than ephemeral textures.9 |
| **Compute Hardware** | **NVIDIA RTX 2060/3070** | Algorithms must be optimized for TensorRT to meet the <5s per frame requirement. Heavy transformers (e.g., ViT-Huge) are prohibitive; efficient architectures (LiteSAM) are required.1 |
The confluence of these factors necessitates a move away from simple "dead reckoning" (accumulating relative movements) which drifts exponentially. Instead, ASTRAL-Next operates as a **Global-Local Hybrid System**, where a high-frequency visual odometry layer handles frame-to-frame continuity, while a parallel global localization layer periodically "resets" the drift by anchoring the UAV to the satellite map.
## **2. Architectural Critique of Legacy Approaches**
The initial draft solution ("ASTRAL") and similar legacy approaches typically rely on a unified SLAM pipeline, often attempting to use the same feature extractors for both sequential tracking and global localization. Recent literature highlights substantial deficiencies in this monolithic approach, particularly when applied to the specific constraints of this project.
### **2.1 The Failure of Classical Descriptors in Agricultural Settings**
Classical feature descriptors like SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) rely on detecting "corners" and "blobs" based on local pixel intensity gradients. In the agricultural landscapes of Eastern Ukraine, this approach faces severe aliasing. A field of sunflowers or wheat presents thousands of identical "blobs," causing the nearest-neighbor matching stage to generate a high ratio of outliers.8
Research demonstrates that deep-learning-based feature extractors, specifically SuperPoint, trained on large datasets of synthetic and real-world imagery, learn to identify interest points that are semantically significant (e.g., the intersection of a tractor path and a crop line) rather than just texturally distinct.1 Consequently, a redesign must replace SIFT/ORB with SuperPoint for the front-end tracking.
### **2.2 The Inadequacy of Dead Reckoning without IMU**
In a standard Visual-Inertial Odometry (VIO) system, the IMU provides a high-frequency prediction of the camera's pose, which the visual system then refines. Without an IMU, the system is purely Visual Odometry (VO). In VO, the scale of the world is unobservable from a single camera (monocular scale ambiguity). A 1-meter movement of a small object looks identical to a 10-meter movement of a large object.5
While the prompt specifies a "predefined altitude," relying on this as a static constant is dangerous due to terrain undulations and barometric drift. ASTRAL-Next must implement a Scale-Constrained Bundle Adjustment, treating the altitude not as a hard fact, but as a strong prior that prevents the scale drift common in monocular systems.5
### **2.3 Vulnerability to "Kidnapped Robot" Scenarios**
The requirement to recover from sharp turns where the "next photo doesn't overlap at all" describes the classic "Kidnapped Robot Problem" in robotics—where a robot is teleported to an unknown location and must relocalize.14
Sequential matching algorithms (optical flow, feature tracking) function on the assumption of overlap. When overlap is zero, these algorithms fail catastrophically. The legacy solution's reliance on continuous tracking makes it fragile to these flight dynamics. The redesigned architecture must incorporate a dedicated Global Place Recognition module that treats every frame as a potential independent query against the satellite database, independent of the previous frame's history.2
## **3. ASTRAL-Next: System Architecture and Methodology**
To meet the acceptance criteria—specifically the 80% success rate within 50m error and the <5 second processing time—ASTRAL-Next utilizes a tri-layer processing topology. These layers operate concurrently, feeding into a central state estimator.
### **3.1 The Tri-Layer Localization Strategy**
The architecture separates the concerns of continuity, recovery, and precision into three distinct algorithmic pathways.
| Layer | Functionality | Algorithm | Latency | Role in Acceptance Criteria |
| :---- | :---- | :---- | :---- | :---- |
| **L1: Sequential Tracking** | Frame-to-Frame Relative Pose | **SuperPoint + LightGlue** | \~50-100ms | Handles continuous flight, bridges small gaps (overlap < 5%), and maintains trajectory smoothness. Essential for the 100m spacing requirement. 1 |
| **L2: Global Re-Localization** | "Kidnapped Robot" Recovery | **AnyLoc (DINOv2 + VLAD)** | \~200ms | Detects location after sharp turns (0% overlap) or track loss. Matches current view to the satellite database tile. Addresses the sharp turn recovery criterion. 2 |
| **L3: Metric Refinement** | Precise GPS Anchoring | **LiteSAM / HLoc** | \~300-500ms | "Stitches" the UAV image to the satellite tile with pixel-level accuracy to reset drift. Ensures the "80% < 50m" and "60% < 20m" accuracy targets. 1 |
### **3.2 Data Flow and State Estimation**
The system utilizes a **Factor Graph Optimization** (using libraries like GTSAM) as the central "brain."
1. **Inputs:**
* **Relative Factors:** Provided by Layer 1 (Change in pose from $t-1$ to $t$).
* **Absolute Factors:** Provided by Layer 3 (Global GPS coordinate at $t$).
* **Priors:** Altitude constraint and Ground Plane assumption.
2. **Processing:** The factor graph optimizes the trajectory by minimizing the error between these conflicting constraints.
3. **Output:** A smoothed, globally consistent trajectory $(x, y, z, \\text{roll}, \\text{pitch}, \\text{yaw})$ for every image timestamp.
### **3.3 ZeroMQ Background Service Architecture**
As per the requirement, the system operates as a background service.
* **Communication Pattern:** The service utilizes a REP-REQ (Reply-Request) pattern for control commands (Start/Stop/Reset) and a PUB-SUB (Publish-Subscribe) pattern for the continuous stream of localization results.
* **Concurrency:** Layer 1 runs on a high-priority thread to ensure immediate feedback. Layers 2 and 3 run asynchronously; when a global match is found, the result is injected into the Factor Graph, which then "back-propagates" the correction to previous frames, refining the entire recent trajectory.
## **4. Layer 1: Robust Sequential Visual Odometry**
The first line of defense against localization loss is robust tracking between consecutive UAV images. Given the challenging agricultural environment, standard feature matching is prone to failure. ASTRAL-Next employs **SuperPoint** and **LightGlue**.
### **4.1 SuperPoint: Semantic Feature Detection**
SuperPoint is a fully convolutional neural network trained to detect interest points and compute their descriptors. Unlike SIFT, which uses handcrafted mathematics to find corners, SuperPoint is trained via self-supervision on millions of images.
* **Relevance to Ukraine:** In a wheat field, SIFT might latch onto hundreds of identical wheat stalks. SuperPoint, however, learns to prioritize more stable features, such as the boundary between the field and a dirt road, or a specific patch of discoloration in the crop canopy.1
* **Performance:** SuperPoint runs efficiently on the RTX 2060/3070, with inference times around 15ms per image when optimized with TensorRT.16
### **4.2 LightGlue: The Attention-Based Matcher**
**LightGlue** represents a paradigm shift from the traditional "Nearest Neighbor + RANSAC" matching pipeline. It is a deep neural network that takes two sets of SuperPoint features and jointly predicts the matches.
* **Mechanism:** LightGlue uses a transformer-based attention mechanism. It allows features in Image A to "look at" all features in Image B (and vice versa) to determine the best correspondence. Crucially, it has a "dustbin" mechanism to explicitly reject points that have no match (occlusion or field of view change).12
* **Addressing the <5% Overlap:** The user specifies handling overlaps of "less than 5%." Traditional RANSAC fails here because the inlier ratio is too low. LightGlue, however, can confidently identify the few remaining matches because its attention mechanism considers the global geometric context of the points. If only a single road intersection is visible in the corner of both images, LightGlue is significantly more likely to match it correctly than SIFT.8
* **Efficiency:** LightGlue is designed to be "light." It features an adaptive depth mechanism—if the images are easy to match, it exits early. If they are hard (low overlap), it uses more layers. This adaptability is perfect for the variable difficulty of the UAV flight path.19
## **5. Layer 2: Global Place Recognition (The "Kidnapped Robot" Solver)**
When the UAV executes a sharp turn, resulting in a completely new view (0% overlap), sequential tracking (Layer 1) is mathematically impossible. The system must recognize the new terrain solely based on its appearance. This is the domain of **AnyLoc**.
### **5.1 Universal Place Recognition with Foundation Models**
**AnyLoc** leverages **DINOv2**, a massive self-supervised vision transformer developed by Meta. DINOv2 is unique because it is not trained with labels; it is trained to understand the geometry and semantic layout of images.
* **Why DINOv2 for Satellite Matching:** Satellite images and UAV images have different "domains." The satellite image might be from summer (green), while the UAV flies in autumn (brown). DINOv2 features are remarkably invariant to these texture changes. It "sees" the shape of the road network or the layout of the field boundaries, rather than the color of the leaves.2
* **VLAD Aggregation:** AnyLoc extracts dense features from the image using DINOv2 and aggregates them using **VLAD** (Vector of Locally Aggregated Descriptors) into a single, compact vector (e.g., 4096 dimensions). This vector represents the "fingerprint" of the location.21
### **5.2 Implementation Strategy**
1. **Database Preparation:** Before the mission, the system downloads the satellite imagery for the operational bounding box (Eastern/Southern Ukraine). These images are tiled (e.g., 512x512 pixels with overlap) and processed through AnyLoc to generate a database of descriptors.
2. **Faiss Indexing:** These descriptors are indexed using **Faiss**, a library for efficient similarity search.
3. **In-Flight Retrieval:** When Layer 1 reports a loss of tracking (or periodically), the current UAV image is processed by AnyLoc. The resulting vector is queried against the Faiss index.
4. **Result:** The system retrieves the top-5 most similar satellite tiles. These tiles represent the coarse global location of the UAV (e.g., "You are in Grid Square B7").2
## **6. Layer 3: Fine-Grained Metric Localization (LiteSAM)**
Retrieving the correct satellite tile (Layer 2) gives a location error of roughly the tile size (e.g., 200 meters). To meet the "60% < 20m" and "80% < 50m" criteria, the system must precisely align the UAV image onto the satellite tile. ASTRAL-Next utilizes **LiteSAM**.
### **6.1 Justification for LiteSAM over TransFG**
While **TransFG** (Transformer for Fine-Grained recognition) is a powerful architecture for cross-view geo-localization, it is computationally heavy.23 **LiteSAM** (Lightweight Satellite-Aerial Matching) is specifically architected for resource-constrained platforms (like UAV onboard computers or efficient ground stations) while maintaining state-of-the-art accuracy.
* **Architecture:** LiteSAM utilizes a **Token Aggregation-Interaction Transformer (TAIFormer)**. It employs a convolutional token mixer (CTM) to model correlations between the UAV and satellite images.
* **Multi-Scale Processing:** LiteSAM processes features at multiple scales. This is critical because the UAV altitude varies (<1km), meaning the scale of objects in the UAV image will not perfectly match the fixed scale of the satellite image (Google Maps Zoom Level 19). LiteSAM's multi-scale approach inherently handles this discrepancy.1
* **Performance Data:** Empirical benchmarks on the **UAV-VisLoc** dataset show LiteSAM achieving an RMSE@30 (Root Mean Square Error within 30 meters) of 17.86 meters, directly supporting the project's accuracy requirements. Its inference time is approximately 61.98ms on standard GPUs, ensuring it fits within the overall 5-second budget.1
### **6.2 The Alignment Process**
1. **Input:** The UAV Image and the Top-1 Satellite Tile from Layer 2.
2. **Processing:** LiteSAM computes the dense correspondence field between the two images.
3. **Homography Estimation:** Using the correspondences, the system computes a homography matrix $H$ that maps pixels in the UAV image to pixels in the georeferenced satellite tile.
4. **Pose Extraction:** The camera's absolute GPS position is derived from this homography, utilizing the known GSD of the satellite tile.18
## **7. Satellite Data Management and Coordinate Systems**
The reliability of the entire system hinges on the quality and handling of the reference map data. The restriction to "Google Maps" necessitates a rigorous approach to coordinate transformation and data freshness management.
### **7.1 Google Maps Static API and Mercator Projection**
The Google Maps Static API delivers images without embedded georeferencing metadata (GeoTIFF tags). The system must mathematically derive the bounding box of each downloaded tile to assign coordinates to the pixels. Google Maps uses the **Web Mercator Projection (EPSG:3857)**.
The system must implement the following derivation to establish the **Ground Sampling Distance (GSD)**, or meters_per_pixel, which varies significantly with latitude:
$$ \\text{meters_per_pixel} = 156543.03392 \\times \\frac{\\cos(\\text{latitude} \\times \\frac{\\pi}{180})}{2^{\\text{zoom}}} $$
For the operational region (Ukraine, approx. Latitude 48N):
* At **Zoom Level 19**, the resolution is approximately 0.30 meters/pixel. This resolution is compatible with the input UAV imagery (Full HD at <1km altitude), providing sufficient detail for the LiteSAM matcher.24
**Bounding Box Calculation Algorithm:**
1. **Input:** Center Coordinate $(lat, lon)$, Zoom Level ($z$), Image Size $(w, h)$.
2. **Project to World Coordinates:** Convert $(lat, lon)$ to world pixel coordinates $(px, py)$ at the given zoom level.
3. **Corner Calculation:**
* px_{NW} = px - (w / 2)
* py_{NW} = py - (h / 2)
4. Inverse Projection: Convert $(px_{NW}, py_{NW})$ back to Latitude/Longitude to get the North-West corner. Repeat for South-East.
This calculation is critical. A precision error here translates directly to a systematic bias in the final GPS output.
### **7.2 Mitigating Data Obsolescence (The 2025 Problem)**
The provided research highlights that satellite imagery access over Ukraine is subject to restrictions and delays (e.g., Maxar restrictions in 2025).10 Google Maps data may be several years old.
* **Semantic Anchoring:** This reinforces the selection of **AnyLoc** (Layer 2) and **LiteSAM** (Layer 3). These algorithms are trained to ignore transient features (cars, temporary structures, vegetation color) and focus on persistent structural features (road geometry, building footprints).
* **Seasonality:** Research indicates that DINOv2 features (used in AnyLoc) exhibit strong robustness to seasonal changes (e.g., winter satellite map vs. summer UAV flight), maintaining high retrieval recall where pixel-based methods fail.17
## **8. Optimization and State Estimation (The "Brain")**
The individual outputs of the visual layers are noisy. Layer 1 drifts over time; Layer 3 may have occasional outliers. The **Factor Graph Optimization** fuses these inputs into a coherent trajectory.
### **8.1 Handling the 350-Meter Outlier (Tilt)**
The prompt specifies that "up to 350 meters of an outlier... could happen due to tilt." This large displacement masquerading as translation is a classic source of divergence in Kalman Filters.
* **Robust Cost Functions:** In the Factor Graph, the error terms for the visual factors are wrapped in a **Robust Kernel** (specifically the **Cauchy** or **Huber** kernel).
* *Mechanism:* Standard least-squares optimization penalizes errors quadratically ($e^2$). If a 350m error occurs, the penalty is massive, dragging the entire trajectory off-course. A robust kernel changes the penalty to be linear ($|e|$) or logarithmic after a certain threshold. This allows the optimizer to effectively "ignore" or down-weight the 350m jump if it contradicts the consensus of other measurements, treating it as a momentary outlier or solving for it as a rotation rather than a translation.19
### **8.2 The Altitude Soft Constraint**
To resolve the monocular scale ambiguity without IMU, the altitude ($h_{prior}$) is added as a **Unary Factor** to the graph.
* $E_{alt} = |
| z_{est} \- h_{prior} ||*{\\Sigma*{alt}}$
* $\\Sigma_{alt}$ (covariance) is set relatively high (soft constraint), allowing the visual odometry to adjust the altitude slightly to maintain consistency, but preventing the scale from collapsing to zero or exploding to infinity. This effectively creates an **Altimeter-Aided Monocular VIO** system, where the altimeter (virtual or barometric) replaces the accelerometer for scale determination.5
## **9. Implementation Specifications**
### **9.1 Hardware Acceleration (TensorRT)**
Meeting the <5 second per frame requirement on an RTX 2060 requires optimizing the deep learning models. Python/PyTorch inference is typically too slow due to overhead.
* **Model Export:** All core models (SuperPoint, LightGlue, LiteSAM) must be exported to **ONNX** (Open Neural Network Exchange) format.
* **TensorRT Compilation:** The ONNX models are then compiled into **TensorRT Engines**. This process performs graph fusion (combining multiple layers into one) and kernel auto-tuning (selecting the fastest GPU instructions for the specific RTX 2060/3070 architecture).26
* **Precision:** The models should be quantized to **FP16** (16-bit floating point). Research shows that FP16 inference on NVIDIA RTX cards offers a 2x-3x speedup with negligible loss in matching accuracy for these specific networks.16
### **9.2 Background Service Architecture (ZeroMQ)**
The system is encapsulated as a headless service.
**ZeroMQ Topology:**
* **Socket 1 (REP - Port 5555):** Command Interface. Accepts JSON messages:
* {"cmd": "START", "config": {"lat": 48.1, "lon": 37.5}}
* {"cmd": "USER_FIX", "lat": 48.22, "lon": 37.66} (Human-in-the-loop input).
* **Socket 2 (PUB - Port 5556):** Data Stream. Publishes JSON results for every frame:
* {"frame_id": 1024, "gps": [48.123, 37.123], "object_centers": [...], "status": "LOCKED", "confidence": 0.98}.
Asynchronous Pipeline:
The system utilizes a Python multiprocessing architecture. One process handles the camera/image ingest and ZeroMQ communication. A second process hosts the TensorRT engines and runs the Factor Graph. This ensures that the heavy computation of Bundle Adjustment does not block the receipt of new images or user commands.
## **10. Human-in-the-Loop Strategy**
The requirement stipulates that for the "20% of the route" where automation fails, the user must intervene. The system must proactively detect its own failure.
### **10.1 Failure Detection with PDM@K**
The system monitors the **PDM@K** (Positioning Distance Measurement) metric continuously.
* **Definition:** PDM@K measures the percentage of queries localized within $K$ meters.3
* **Real-Time Proxy:** In flight, we cannot know the true PDM (as we don't have ground truth). Instead, we use the **Marginal Covariance** from the Factor Graph. If the uncertainty ellipse for the current position grows larger than a radius of 50 meters, or if the **Image Registration Rate** (percentage of inliers in LightGlue/LiteSAM) drops below 10% for 3 consecutive frames, the system triggers a **Critical Failure Mode**.19
### **10.2 The User Interaction Workflow**
1. **Trigger:** Critical Failure Mode activated.
2. **Action:** The Service publishes a status {"status": "REQ_INPUT"} via ZeroMQ.
3. **Data Payload:** It sends the current UAV image and the top-3 retrieved satellite tiles (from Layer 2) to the client UI.
4. **User Input:** The user clicks a distinctive feature (e.g., a specific crossroad) in the UAV image and the corresponding point on the satellite map.
5. **Recovery:** This pair of points is treated as a **Hard Constraint** in the Factor Graph. The optimizer immediately snaps the trajectory to this user-defined anchor, resetting the covariance and effectively "healing" the localized track.19
## **11. Performance Evaluation and Benchmarks**
### **11.1 Accuracy Validation**
Based on the reported performance of the selected components in relevant datasets (UAV-VisLoc, AnyVisLoc):
* **LiteSAM** demonstrates an accuracy of 17.86m (RMSE) for cross-view matching. This aligns with the requirement that 60% of photos be within 20m error.18
* **AnyLoc** achieves high recall rates (Top-1 Recall > 85% on aerial benchmarks), supporting the recovery from sharp turns.2
* **Factor Graph Fusion:** By combining sequential and global measurements, the overall system error is expected to be lower than the individual component errors, satisfying the "80% within 50m" criterion.
### **11.2 Latency Analysis**
The breakdown of processing time per frame on an RTX 3070 is estimated as follows:
* **SuperPoint + LightGlue:** \~50ms.1
* **AnyLoc (Global Retrieval):** \~150ms (run only on keyframes or tracking loss).
* **LiteSAM (Metric Refinement):** \~60ms.1
* **Factor Graph Optimization:** \~100ms (using incremental updates/iSAM2).
* Total: \~360ms per frame (worst case with all layers active).
This is an order of magnitude faster than the 5-second limit, providing ample headroom for higher resolution processing or background tasks.
## **12.0 ASTRAL-Next Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
## Role
You are a professional software architect
## Task
- Thorougly research in internet about the problem and identify all potential weak points and problems.
- Address these problems and find out ways to solve them.
- Based on your findings, form a new solution draft in the same format.
## Output format
- Put here all new findings, what was updated, replaced, or removed from the previous solution in the next table:
- Old component solution
- Weak point
- Solution (component's new solution)
- Form the new solution draft. In the updated report, do not put "new" marks, do not compare to the previous solution draft, just make a new solution as if from scratch. Put it in the next format:
- Short Product solution description. Brief component interaction diagram.
- Architecture solution that meets restrictions and acceptance criteria.
For each component, analyze the best possible solutions, and form a comparison table.
Each possible component solution would be a row, and has the next columns:
- Tools (library, platform) to solve component tasks
- Advantages of this solution. For example, LiteSAM AI feature is picked for UAV - Satellite matching finding, and it make its job perfectly in milliseconds timeframe.
- Limitations of this solution. For example, LiteSAM AI feature matcher requires to work efficiently on RTX Gpus and since it is sparsed, the quality a bit lower than densed feature matcher.
- Requirements for this solution. For example, LiteSAM AI feature matcher requires that photos it comparing to be aligned by rotation with no more than 45 degree difference. This requires additional preparation step for pre-rotating either UAV either Satellite images in order to be aligned.
- How does it fit for the problem component that has to be solved, and the whole solution
- Testing strategy. Research how to cover system with tests in order to meet all the acceptance criteria. Form a list of integration functional tests and non-functional tests.
## Additional sources
- A Cross-View Geo-Localization Algorithm Using UAV Image
https://www.mdpi.com/1424-8220/24/12/3719
- Exploring the best way for UAV visual localization under Low-altitude Multi-view Observation condition
https://arxiv.org/pdf/2503.10692
- find out more like this.
Assess them and try to either integrate or replace some of the components in the current solution draft
_docs/00_problem/problem_description.md → _docs/00_problem/problem.md
+2 -1
View File
@@ -1,3 +1,4 @@
We have a lot of images taken from a wing-type UAV using a camera with at least Full HD resolution. Resolution of each photo could be up to 6200*4100 for the whole flight, but for other flights, it could be FullHD
Photos are taken and named consecutively within 100 meters of each other.
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos
We know only the starting GPS coordinates. We need to determine the GPS of the centers of each next image. And also the coordinates of the center of any object in these photos. We can use an external satellite provider for ground checks on the existing photos.
The real world examples are in input_data folder
_docs/00_templates/definition_of_done.md
-92
View File
@@ -1,92 +0,0 @@
# Definition of Done (DoD)
A feature/task is considered DONE when all applicable items are completed.
---
## Code Complete
- [ ] All acceptance criteria from the spec are implemented
- [ ] Code compiles/builds without errors
- [ ] No new linting errors or warnings
- [ ] Code follows project coding standards and conventions
- [ ] No hardcoded values (use configuration/environment variables)
- [ ] Error handling implemented per project standards
---
## Testing Complete
- [ ] Unit tests written for new code
- [ ] Unit tests pass locally
- [ ] Integration tests written (if applicable)
- [ ] Integration tests pass
- [ ] Code coverage meets minimum threshold (75%)
- [ ] Manual testing performed for UI changes
---
## Code Review Complete
- [ ] Pull request created with proper description
- [ ] PR linked to Jira ticket
- [ ] At least one approval from reviewer
- [ ] All review comments addressed
- [ ] No merge conflicts
---
## Documentation Complete
- [ ] Code comments for complex logic (if needed)
- [ ] API documentation updated (if endpoints changed)
- [ ] README updated (if setup/usage changed)
- [ ] CHANGELOG updated with changes
---
## CI/CD Complete
- [ ] All CI pipeline stages pass
- [ ] Security scan passes (no critical/high vulnerabilities)
- [ ] Build artifacts generated successfully
---
## Deployment Ready
- [ ] Database migrations tested (if applicable)
- [ ] Configuration changes documented
- [ ] Feature flags configured (if applicable)
- [ ] Rollback plan identified
---
## Communication Complete
- [ ] Jira ticket moved to Done
- [ ] Stakeholders notified of completion (if required)
- [ ] Any blockers or follow-up items documented
---
## Quick Reference
| Category | Must Have | Nice to Have |
|----------|-----------|--------------|
| Code | Builds, No lint errors | Optimized |
| Tests | Unit + Integration pass | E2E tests |
| Coverage | >= 75% | >= 85% |
| Review | 1 approval | 2 approvals |
| Docs | CHANGELOG | Full API docs |
---
## Exceptions
If any DoD item cannot be completed, document:
1. Which item is incomplete
2. Reason for exception
3. Plan to address (with timeline)
4. Approval from tech lead
_docs/00_templates/environment_strategy.md
-139
View File
@@ -1,139 +0,0 @@
# Environment Strategy Template
## Overview
Define the environment strategy for the project, including configuration, access, and deployment procedures for each environment.
---
## Environments
### Development (dev)
**Purpose**: Local development and feature testing
| Aspect | Configuration |
|--------|---------------|
| Branch | `dev`, feature branches |
| Database | Local or shared dev instance |
| External Services | Mock/sandbox endpoints |
| Logging Level | DEBUG |
| Access | All developers |
**Configuration**:
```
# .env.development
ENV=development
DATABASE_URL=<dev_database_url>
API_TIMEOUT=30
LOG_LEVEL=DEBUG
```
### Staging (stage)
**Purpose**: Pre-production testing, QA, UAT
| Aspect | Configuration |
|--------|---------------|
| Branch | `stage` |
| Database | Staging instance (production-like) |
| External Services | Sandbox/test endpoints |
| Logging Level | INFO |
| Access | Development team, QA |
**Configuration**:
```
# .env.staging
ENV=staging
DATABASE_URL=<staging_database_url>
API_TIMEOUT=15
LOG_LEVEL=INFO
```
**Deployment Trigger**: Merge to `stage` branch
### Production (prod)
**Purpose**: Live system serving end users
| Aspect | Configuration |
|--------|---------------|
| Branch | `main` |
| Database | Production instance |
| External Services | Production endpoints |
| Logging Level | WARN |
| Access | Restricted (ops team) |
**Configuration**:
```
# .env.production
ENV=production
DATABASE_URL=<production_database_url>
API_TIMEOUT=10
LOG_LEVEL=WARN
```
**Deployment Trigger**: Manual approval after staging validation
---
## Secrets Management
### Secret Categories
- Database credentials
- API keys (internal and external)
- Encryption keys
- Service account credentials
### Storage
| Environment | Secret Storage |
|-------------|----------------|
| Development | .env.local (gitignored) |
| Staging | CI/CD secrets / Vault |
| Production | CI/CD secrets / Vault |
### Rotation Policy
- Database passwords: Every 90 days
- API keys: Every 180 days or on compromise
- Encryption keys: Annually
---
## Environment Parity
### Required Parity
- Same database engine and version
- Same runtime version
- Same dependency versions
- Same configuration structure
### Allowed Differences
- Resource scaling (CPU, memory)
- External service endpoints (sandbox vs production)
- Logging verbosity
- Feature flags
---
## Access Control
| Role | Dev | Staging | Production |
|------|-----|---------|------------|
| Developer | Full | Read + Deploy | Read logs only |
| QA | Read | Full | Read logs only |
| DevOps | Full | Full | Full |
| Stakeholder | None | Read | Read dashboards |
---
## Backup & Recovery
| Environment | Backup Frequency | Retention | RTO | RPO |
|-------------|------------------|-----------|-----|-----|
| Development | None | N/A | N/A | N/A |
| Staging | Daily | 7 days | 4 hours | 24 hours |
| Production | Hourly | 30 days | 1 hour | 1 hour |
---
## Notes
- Never copy production data to lower environments without anonymization
- All environment-specific values must be externalized (no hardcoding)
- Document any environment-specific behaviors in code comments
_docs/00_templates/feature_dependency_matrix.md
-103
View File
@@ -1,103 +0,0 @@
# Feature Dependency Matrix
Track feature dependencies to ensure proper implementation order.
---
## Active Features
| Feature ID | Feature Name | Status | Dependencies | Blocks |
|------------|--------------|--------|--------------|--------|
| | | Draft/In Progress/Done | List IDs | List IDs |
---
## Dependency Rules
### Status Definitions
- **Draft**: Spec created, not started
- **In Progress**: Development started
- **Done**: Merged to dev, verified
- **Blocked**: Waiting on dependencies
### Dependency Types
- **Hard**: Cannot start without dependency complete
- **Soft**: Can mock dependency, integrate later
- **API**: Depends on API contract (can parallelize with mock)
- **Data**: Depends on data/schema (must be complete)
---
## Current Dependencies
### [Feature A] depends on:
| Dependency | Type | Status | Blocker? |
|------------|------|--------|----------|
| | Hard/Soft/API/Data | Done/In Progress | Yes/No |
### [Feature B] depends on:
| Dependency | Type | Status | Blocker? |
|------------|------|--------|----------|
| | | | |
---
## Dependency Graph
```
Feature A (Done)
└── Feature B (In Progress)
└── Feature D (Draft)
└── Feature C (Draft)
Feature E (Done)
└── Feature F (In Progress)
```
---
## Implementation Order
Based on dependencies, recommended implementation order:
1. **Phase 1** (No dependencies)
- [ ] Feature X
- [ ] Feature Y
2. **Phase 2** (Depends on Phase 1)
- [ ] Feature Z (after X)
- [ ] Feature W (after Y)
3. **Phase 3** (Depends on Phase 2)
- [ ] Feature V (after Z, W)
---
## Handling Blocked Features
When a feature is blocked:
1. **Identify** the blocking dependency
2. **Escalate** if blocker is delayed
3. **Consider** if feature can proceed with mocks
4. **Document** any workarounds used
5. **Schedule** integration when blocker completes
---
## Mock Strategy
When using mocks for dependencies:
| Feature | Mocked Dependency | Mock Type | Integration Task |
|---------|-------------------|-----------|------------------|
| | | Interface/Data/API | Link to task |
---
## Update Log
| Date | Feature | Change | By |
|------|---------|--------|-----|
| | | Added/Updated/Completed | |
_docs/00_templates/feature_parity_checklist.md
-129
View File
@@ -1,129 +0,0 @@
# Feature Parity Checklist
Use this checklist to ensure all functionality is preserved during refactoring.
---
## Project: [Project Name]
## Refactoring Scope: [Brief description]
## Date: [YYYY-MM-DD]
---
## Feature Inventory
### API Endpoints
| Endpoint | Method | Before | After | Verified |
|----------|--------|--------|-------|----------|
| /api/v1/example | GET | Working | | [ ] |
| | | | | [ ] |
### Core Functions
| Function/Module | Purpose | Before | After | Verified |
|-----------------|---------|--------|-------|----------|
| | | Working | | [ ] |
| | | | | [ ] |
### User Workflows
| Workflow | Steps | Before | After | Verified |
|----------|-------|--------|-------|----------|
| User login | 1. Enter credentials 2. Submit | Working | | [ ] |
| | | | | [ ] |
### Integrations
| External System | Integration Type | Before | After | Verified |
|-----------------|------------------|--------|-------|----------|
| | API/Webhook/DB | Working | | [ ] |
| | | | | [ ] |
---
## Behavioral Parity
### Input Handling
- [ ] Same inputs produce same outputs
- [ ] Error messages unchanged (or improved)
- [ ] Validation rules preserved
- [ ] Edge cases handled identically
### Output Format
- [ ] Response structure unchanged
- [ ] Data types preserved
- [ ] Null handling consistent
- [ ] Date/time formats preserved
### Side Effects
- [ ] Database writes produce same results
- [ ] File operations unchanged
- [ ] External API calls preserved
- [ ] Event emissions maintained
---
## Non-Functional Parity
### Performance
- [ ] Response times within baseline +10%
- [ ] Memory usage within baseline +10%
- [ ] CPU usage within baseline +10%
- [ ] No new N+1 queries introduced
### Security
- [ ] Authentication unchanged
- [ ] Authorization rules preserved
- [ ] Input sanitization maintained
- [ ] No new vulnerabilities introduced
### Reliability
- [ ] Error handling preserved
- [ ] Retry logic maintained
- [ ] Timeout behavior unchanged
- [ ] Circuit breakers preserved
---
## Test Coverage
| Test Type | Before | After | Status |
|-----------|--------|-------|--------|
| Unit Tests | X pass | | [ ] Same or better |
| Integration Tests | X pass | | [ ] Same or better |
| E2E Tests | X pass | | [ ] Same or better |
---
## Verification Steps
### Automated Verification
1. [ ] All existing tests pass
2. [ ] No new linting errors
3. [ ] Coverage >= baseline
### Manual Verification
1. [ ] Smoke test critical paths
2. [ ] Verify UI behavior (if applicable)
3. [ ] Test error scenarios
### Stakeholder Sign-off
- [ ] QA approved
- [ ] Product owner approved (if behavior changed)
---
## Discrepancies Found
| Feature | Expected | Actual | Resolution | Status |
|---------|----------|--------|------------|--------|
| | | | | |
---
## Notes
- Any intentional behavior changes must be documented and approved
- Update this checklist as refactoring progresses
- Keep baseline metrics for comparison
_docs/00_templates/incident_playbook.md
-157
View File
@@ -1,157 +0,0 @@
# Incident Playbook Template
## Incident Overview
| Field | Value |
|-------|-------|
| Playbook Name | [Name] |
| Severity | Critical / High / Medium / Low |
| Last Updated | [YYYY-MM-DD] |
| Owner | [Team/Person] |
---
## Detection
### Symptoms
- [How will you know this incident is occurring?]
- Alert: [Alert name that triggers]
- User reports: [Expected user complaints]
### Monitoring
- Dashboard: [Link to relevant dashboard]
- Logs: [Log query to investigate]
- Metrics: [Key metrics to watch]
---
## Assessment
### Impact Analysis
- Users affected: [All / Subset / Internal only]
- Data at risk: [Yes / No]
- Revenue impact: [High / Medium / Low / None]
### Severity Determination
| Condition | Severity |
|-----------|----------|
| Service completely down | Critical |
| Partial degradation | High |
| Intermittent issues | Medium |
| Minor impact | Low |
---
## Response
### Immediate Actions (First 5 minutes)
1. [ ] Acknowledge alert
2. [ ] Verify incident is real (not false positive)
3. [ ] Notify on-call team
4. [ ] Start incident channel/call
### Investigation Steps
1. [ ] Check recent deployments
2. [ ] Review error logs
3. [ ] Check infrastructure metrics
4. [ ] Identify affected components
### Communication
| Audience | Channel | Frequency |
|----------|---------|-----------|
| Engineering | Slack #incidents | Continuous |
| Stakeholders | Email | Every 30 min |
| Users | Status page | Major updates |
---
## Resolution
### Common Fixes
#### Fix 1: [Common issue]
```bash
# Commands to fix
```
Expected outcome: [What should happen]
#### Fix 2: [Another common issue]
```bash
# Commands to fix
```
Expected outcome: [What should happen]
### Rollback Procedure
1. [ ] Identify last known good version
2. [ ] Execute rollback
```bash
# Rollback commands
```
3. [ ] Verify service restored
4. [ ] Monitor for 15 minutes
### Escalation Path
| Time | Action |
|------|--------|
| 0-15 min | On-call engineer |
| 15-30 min | Team lead |
| 30-60 min | Engineering manager |
| 60+ min | Director/VP |
---
## Post-Incident
### Verification
- [ ] Service fully restored
- [ ] All alerts cleared
- [ ] User-facing functionality verified
- [ ] Monitoring back to normal
### Documentation
- [ ] Timeline documented
- [ ] Root cause identified
- [ ] Action items created
- [ ] Post-mortem scheduled
### Post-Mortem Template
```markdown
## Incident Summary
- Date/Time:
- Duration:
- Impact:
- Root Cause:
## Timeline
- [Time] - Event
## What Went Well
-
## What Went Wrong
-
## Action Items
| Action | Owner | Due Date |
|--------|-------|----------|
| | | |
```
---
## Contacts
| Role | Name | Contact |
|------|------|---------|
| On-call | | |
| Team Lead | | |
| Manager | | |
---
## Revision History
| Date | Author | Changes |
|------|--------|---------|
| | | |
_docs/00_templates/pr_template.md
-70
View File
@@ -1,70 +0,0 @@
# Pull Request Template
## Description
Brief description of changes.
## Related Issue
Jira ticket: [AZ-XXX](link)
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Refactoring
- [ ] Documentation
- [ ] Performance improvement
- [ ] Security fix
## Checklist
- [ ] Code follows project conventions
- [ ] Self-review completed
- [ ] Tests added/updated
- [ ] All tests pass
- [ ] Code coverage maintained/improved
- [ ] Documentation updated (if needed)
- [ ] CHANGELOG updated
## Breaking Changes
<!-- List any breaking changes, or write "None" -->
- None
## API Changes
<!-- List any API changes (new endpoints, changed signatures, removed endpoints) -->
- None
## Database Changes
<!-- List any database changes (migrations, schema changes) -->
- [ ] No database changes
- [ ] Migration included and tested
- [ ] Rollback migration included
## Deployment Notes
<!-- Special considerations for deployment -->
- [ ] No special deployment steps required
- [ ] Environment variables added/changed (documented in .env.example)
- [ ] Feature flags configured
- [ ] External service dependencies
## Rollback Plan
<!-- Steps to rollback if issues arise -->
1. Revert this PR commit
2. [Additional steps if needed]
## Testing
How to test these changes:
1.
2.
3.
## Performance Impact
<!-- Note any performance implications -->
- [ ] No performance impact expected
- [ ] Performance tested (attach results if applicable)
## Security Considerations
<!-- Note any security implications -->
- [ ] No security implications
- [ ] Security review completed
- [ ] Sensitive data handling reviewed
## Screenshots (if applicable)
<!-- Add screenshots for UI changes -->
_docs/00_templates/quality_gates.md
-140
View File
@@ -1,140 +0,0 @@
# Quality Gates
Quality gates are checkpoints that must pass before proceeding to the next phase.
---
## Kickstart Tutorial Quality Gates
### Gate 1: Research Complete (after 1.40)
Before proceeding to Planning phase:
- [ ] Problem description is clear and complete
- [ ] Acceptance criteria are measurable and testable
- [ ] Restrictions are documented
- [ ] Security requirements defined
- [ ] Solution draft reviewed and finalized
- [ ] Tech stack evaluated and selected
### Gate 2: Planning Complete (after 2.40)
Before proceeding to Implementation phase:
- [ ] All components defined with clear boundaries
- [ ] Data model designed and reviewed
- [ ] API contracts defined
- [ ] Test specifications created
- [ ] Jira epics/tasks created
- [ ] Effort estimated
- [ ] Risks identified and mitigated
### Gate 3: Implementation Complete (after 3.40)
Before merging to main:
- [ ] All components implemented
- [ ] Code coverage >= 75%
- [ ] All tests pass (unit, integration)
- [ ] Code review approved
- [ ] Security scan passed
- [ ] CI/CD pipeline green
- [ ] Deployment tested on staging
- [ ] Documentation complete
---
## Iterative Tutorial Quality Gates
### Gate 1: Spec Ready (after step 20)
Before creating Jira task:
- [ ] Building block clearly defines problem/goal
- [ ] Feature spec has measurable acceptance criteria
- [ ] Dependencies identified
- [ ] Complexity estimated
### Gate 2: Implementation Ready (after step 50)
Before starting development:
- [ ] Plan reviewed and approved
- [ ] Test strategy defined
- [ ] Dependencies available or mocked
### Gate 3: Merge Ready (after step 70)
Before creating PR:
- [ ] All acceptance criteria met
- [ ] Tests pass locally
- [ ] Definition of Done checklist completed
- [ ] No unresolved TODOs in code
---
## Refactoring Tutorial Quality Gates
### Gate 1: Safety Net Ready (after 4.50)
Before starting refactoring:
- [ ] Baseline metrics captured
- [ ] Current behavior documented
- [ ] Integration tests pass (>= 75% coverage)
- [ ] Feature parity checklist created
### Gate 2: Refactoring Safe (after each 4.70 cycle)
After each refactoring step:
- [ ] All existing tests still pass
- [ ] No functionality lost (feature parity check)
- [ ] Performance not degraded (compare to baseline)
### Gate 3: Refactoring Complete (after 4.95)
Before declaring refactoring done:
- [ ] All tests pass
- [ ] Performance improved or maintained
- [ ] Security review passed
- [ ] Technical debt reduced
- [ ] Documentation updated
---
## Automated Gate Checks
### CI Pipeline Gates
```yaml
gates:
build:
- compilation_success: true
quality:
- lint_errors: 0
- code_coverage: ">= 75%"
- code_smells: "< 10 new"
security:
- critical_vulnerabilities: 0
- high_vulnerabilities: 0
tests:
- unit_tests_pass: true
- integration_tests_pass: true
```
### Manual Gate Checks
Some gates require human verification:
- Architecture review
- Security review
- UX review (for UI changes)
- Stakeholder sign-off
---
## Gate Failure Handling
When a gate fails:
1. **Stop** - Do not proceed to next phase
2. **Identify** - Determine which checks failed
3. **Fix** - Address the failures
4. **Re-verify** - Run gate checks again
5. **Document** - If exception needed, get approval and document reason
---
## Exception Process
If a gate must be bypassed:
1. Document the reason
2. Get tech lead approval
3. Create follow-up task to address
4. Set deadline for resolution
5. Add to risk register
_docs/00_templates/rollback_strategy.md
-173
View File
@@ -1,173 +0,0 @@
# Rollback Strategy Template
## Overview
| Field | Value |
|-------|-------|
| Service/Component | [Name] |
| Last Updated | [YYYY-MM-DD] |
| Owner | [Team/Person] |
| Max Rollback Time | [Target: X minutes] |
---
## Rollback Triggers
### Automatic Rollback Triggers
- [ ] Health check failures > 3 consecutive
- [ ] Error rate > 10% for 5 minutes
- [ ] P99 latency > 2x baseline for 5 minutes
- [ ] Critical alert triggered
### Manual Rollback Triggers
- [ ] User-reported critical bug
- [ ] Data corruption detected
- [ ] Security vulnerability discovered
- [ ] Stakeholder decision
---
## Pre-Rollback Checklist
- [ ] Incident acknowledged and documented
- [ ] Stakeholders notified of rollback decision
- [ ] Current state captured (logs, metrics snapshot)
- [ ] Rollback target version identified
- [ ] Database state assessed (migrations reversible?)
---
## Rollback Procedures
### Application Rollback
#### Option 1: Revert Deployment (Preferred)
```bash
# Using CI/CD
# Trigger previous successful deployment
# Manual (if needed)
git revert <commit-hash>
git push origin main
```
#### Option 2: Blue-Green Switch
```bash
# Switch traffic to previous version
# [Platform-specific commands]
```
#### Option 3: Feature Flag Disable
```bash
# Disable feature flag
# [Feature flag system commands]
```
### Database Rollback
#### If Migration is Reversible
```bash
# Run down migration
# [Migration tool command]
```
#### If Migration is NOT Reversible
1. [ ] Restore from backup
2. [ ] Point-in-time recovery to pre-deployment
3. [ ] **WARNING**: May cause data loss - requires approval
### Configuration Rollback
```bash
# Restore previous configuration
# [Config management commands]
```
---
## Post-Rollback Verification
### Immediate (0-5 minutes)
- [ ] Service responding to health checks
- [ ] No error spikes in logs
- [ ] Basic functionality verified
### Short-term (5-30 minutes)
- [ ] All critical paths functional
- [ ] Error rate returned to baseline
- [ ] Performance metrics normal
### Extended (30-60 minutes)
- [ ] No delayed issues appearing
- [ ] User reports resolved
- [ ] All alerts cleared
---
## Communication Plan
### During Rollback
| Audience | Message | Channel |
|----------|---------|---------|
| Engineering | "Initiating rollback due to [reason]" | Slack |
| Stakeholders | "Service issue detected, rollback in progress" | Email |
| Users | "We're aware of issues and working on a fix" | Status page |
### After Rollback
| Audience | Message | Channel |
|----------|---------|---------|
| Engineering | "Rollback complete, monitoring" | Slack |
| Stakeholders | "Service restored, post-mortem scheduled" | Email |
| Users | "Issue resolved, service fully operational" | Status page |
---
## Known Limitations
### Cannot Rollback If:
- [ ] Database migration deleted columns with data
- [ ] External API contracts changed
- [ ] Third-party integrations updated
### Partial Rollback Scenarios
- [ ] When only specific components affected
- [ ] When data migration is complex
---
## Recovery After Rollback
### Investigation
1. [ ] Collect all relevant logs
2. [ ] Identify root cause
3. [ ] Document findings
### Re-deployment Planning
1. [ ] Fix identified in development
2. [ ] Additional tests added
3. [ ] Staged rollout planned
4. [ ] Monitoring enhanced
---
## Rollback Testing
### Test Schedule
- [ ] Monthly rollback drill
- [ ] After major infrastructure changes
- [ ] Before critical releases
### Test Scenarios
1. Application rollback
2. Database rollback (in staging)
3. Configuration rollback
---
## Contacts
| Role | Name | Contact |
|------|------|---------|
| On-call | | |
| Database Admin | | |
| Platform Team | | |
_docs/01_solution/01_solution_draft.md
-288
View File
@@ -1,288 +0,0 @@
# **GEo-Referenced Trajectory and Object Localization System (GEORTOLS): A Hybrid SLAM Architecture**
## **1. Executive Summary**
This report outlines the technical design for a robust, real-time geolocalization system. The objective is to determine the precise GPS coordinates for a sequence of high-resolution images (up to 6252x4168) captured by a fixed-wing, non-stabilized Unmanned Aerial Vehicle (UAV) [User Query]. The system must operate under severe constraints, including the absence of any IMU data, a predefined altitude of no more than 1km, and knowledge of only the starting GPS coordinate [User Query]. The system is required to handle significant in-flight challenges, such as sharp turns with minimal image overlap (<5%), frame-to-frame outliers of up to 350 meters, and operation over low-texture terrain as seen in the provided sample images [User Query, Image 1, Image 7].
The proposed solution is a **Hybrid Visual-Geolocalization SLAM (VG-SLAM)** architecture. This system is designed to meet the demanding acceptance criteria, including a sub-5-second initial processing time per image, streaming output with asynchronous refinement, and high-accuracy GPS localization (60% of photos within 20m error, 80% within 50m error) [User Query].
This hybrid architecture is necessitated by the problem's core constraints. The lack of an IMU makes a purely monocular Visual Odometry (VO) system susceptible to catastrophic scale drift.1 Therefore, the system integrates two cooperative sub-systems:
1. A **Visual Odometry (VO) Front-End:** This component uses state-of-the-art deep-learning feature matchers (SuperPoint + SuperGlue/LightGlue) to provide fast, real-time *relative* pose estimates. This approach is selected for its proven robustness in low-texture environments where traditional features fail.4 This component delivers the initial, sub-5-second pose estimate.
2. A **Cross-View Geolocalization (CVGL) Module:** This component provides *absolute*, drift-free GPS pose estimates by matching UAV images against the available satellite provider (Google Maps).7 It functions as the system's "global loop closure" mechanism, correcting the VO's scale drift and, critically, relocalizing the UAV after tracking is lost during sharp turns or outlier frames [User Query].
These two systems run in parallel. A **Back-End Pose-Graph Optimizer** fuses their respective measurements—high-frequency relative poses from VO and high-confidence absolute poses from CVGL—into a single, globally consistent, and incrementally refined trajectory. This architecture directly satisfies the requirements for immediate, streaming results and subsequent asynchronous refinement [User Query].
## **2. Product Solution Description and Component Interaction**
### **Product Solution Description**
The proposed system, "GEo-Referenced Trajectory and Object Localization System (GEORTOLS)," is a real-time, streaming-capable software solution. It is designed for deployment on a stationary computer or laptop equipped with an NVIDIA GPU (RTX 2060 or better) [User Query].
* **Inputs:**
1. A sequence of consecutively named monocular images (FullHD to 6252x4168).
2. The absolute GPS coordinate (Latitude, Longitude) of the *first* image in the sequence.
3. A pre-calibrated camera intrinsic matrix.
4. Access to the Google Maps satellite imagery API.
* **Outputs:**
1. A real-time, streaming feed of estimated GPS coordinates (Latitude, Longitude, Altitude) and 6-DoF poses (including Roll, Pitch, Yaw) for the center of each image.
2. Asynchronous refinement messages for previously computed poses as the back-end optimizer improves the global trajectory.
3. A service to provide the absolute GPS coordinate for any user-selected pixel coordinate (u,v) within any geolocated image.
### **Component Interaction Diagram**
The system is architected as four asynchronous, parallel-processing components to meet the stringent real-time and refinement requirements.
1. **Image Ingestion & Pre-processing:** This module acts as the entry point. It receives the new, high-resolution image (Image N). It immediately creates scaled-down, lower-resolution (e.g., 1024x768) copies of the image for real-time processing by the VO and CVGL modules, while retaining the full-resolution original for object-level GPS lookups.
2. **Visual Odometry (VO) Front-End:** This module's sole task is high-speed, frame-to-frame relative pose estimation. It maintains a short-term "sliding window" of features, matching Image N to Image N-1. It uses GPU-accelerated deep-learning models (SuperPoint + SuperGlue) to find feature matches and calculates the 6-DoF relative transform. This result is immediately sent to the Back-End.
3. **Cross-View Geolocalization (CVGL) Module:** This is a heavier, slower, asynchronous module. It takes the pre-processed Image N and queries the Google Maps database to find an *absolute* GPS pose. This involves a two-stage retrieval-and-match process. When a high-confidence match is found, its absolute pose is sent to the Back-End as a "global-pose constraint."
4. **Trajectory Optimization Back-End:** This is the system's central "brain," managing the complete pose graph.10 It receives two types of data:
* *High-frequency, low-confidence relative poses* from the VO Front-End.
* Low-frequency, high-confidence absolute poses from the CVGL Module.
It continuously fuses these constraints in a pose-graph optimization framework (e.g., g2o or Ceres Solver). When the VO Front-End provides a new relative pose, it is quickly added to the graph to produce the "Initial Pose" (<5s). When the CVGL Module provides a new absolute pose, it triggers a more comprehensive re-optimization of the entire graph, correcting drift and broadcasting "Refined Poses" to the user.11
## **3. Core Architectural Framework: Hybrid Visual-Geolocalization SLAM (VG-SLAM)**
### **Rationale for the Hybrid Approach**
The core constraints of this problem—monocular, IMU-less flight over potentially long distances (up to 3000 images at \~100m intervals equates to a 300km flight) [User Query]—render simple solutions unviable.
A **VO-Only** system is guaranteed to fail. Monocular Visual Odometry (and SLAM) suffers from an inherent, unobservable ambiguity: the *scale* of the world.1 Because there is no IMU to provide an accelerometer-based scale reference or a gravity vector 12, the system has no way to know if it moved 1 meter or 10 meters. This leads to compounding scale drift, where the entire trajectory will grow or shrink over time.3 Over a 300km flight, the resulting positional error would be measured in kilometers, not the 20-50 meters required [User Query].
A **CVGL-Only** system is also unviable. Cross-View Geolocalization (CVGL) matches the UAV image to a satellite map to find an absolute pose.7 While this is drift-free, it is a large-scale image retrieval problem. Querying the entire map of Ukraine for a match for every single frame is computationally impossible within the <5 second time limit.13 Furthermore, this approach is brittle; if the Google Maps data is outdated (a specific user restriction) [User Query], the CVGL match will fail, and the system would have no pose estimate at all.
Therefore, the **Hybrid VG-SLAM** architecture is the only robust solution.
* The **VO Front-End** provides the fast, high-frequency relative motion. It works even if the satellite map is outdated, as it tracks features in the *real*, current world.
* The **CVGL Module** acts as the *only* mechanism for scale correction and absolute georeferencing. It provides periodic, drift-free "anchors" to the real-world GPS coordinates.
* The **Back-End Optimizer** fuses these two data streams. The CVGL poses function as "global loop closures" in the SLAM pose graph. They correct the scale drift accumulated by the VO and, critically, serve to relocalize the system after a "kidnapping" event, such as the specified sharp turns or 350m outliers [User Query].
### **Data Flow for Streaming and Refinement**
This architecture is explicitly designed to meet the <5s initial output and asynchronous refinement criteria [User Query]. The data flow for a single image (Image N) is as follows:
* **T \= 0.0s:** Image N (6200x4100) is received by the **Ingestion Module**.
* **T \= 0.2s:** Image N is pre-processed (scaled to 1024px) and passed to the VO and CVGL modules.
* **T \= 1.0s:** The **VO Front-End** completes GPU-accelerated matching (SuperPoint+SuperGlue) of Image N -> Image N-1. It computes the Relative_Pose(N-1 -> N).
* **T \= 1.1s:** The **Back-End Optimizer** receives this Relative_Pose. It appends this pose to the graph relative to the last known pose of N-1.
* **T \= 1.2s:** The Back-End broadcasts the **Initial Pose_N_Est** to the user interface. (**<5s criterion met**).
* **(Parallel Thread) T \= 1.5s:** The **CVGL Module** (on a separate thread) begins its two-stage search for Image N against the Google Maps database.
* **(Parallel Thread) T \= 6.0s:** The CVGL Module successfully finds a high-confidence Absolute_Pose_N_Abs from the satellite match.
* **T \= 6.1s:** The **Back-End Optimizer** receives this new, high-confidence absolute constraint for Image N.
* **T \= 6.2s:** The Back-End triggers a graph re-optimization. This new "anchor" corrects any scale or positional drift for Image N and all surrounding poses in the graph.
* **T \= 6.3s:** The Back-End broadcasts a **Pose_N_Refined** (and Pose_N-1_Refined, Pose_N-2_Refined, etc.) to the user interface. (**Refinement criterion met**).
## **4. Component Analysis: Front-End (Visual Odometry and Relocalization)**
The task of the VO Front-End is to rapidly and robustly estimate the 6-DoF relative motion between consecutive frames. This component's success is paramount for the high-frequency tracking required to meet the <5s criterion.
The primary challenge is the nature of the imagery. The specified operational area and sample images (e.g., Image 1, Image 7) show vast, low-texture agricultural fields [User Query]. These environments are a known failure case for traditional, gradient-based feature extractors like SIFT or ORB, which rely on high-gradient corners and cannot find stable features in "weak texture areas".5 Furthermore, the non-stabilized camera [User Query] will introduce significant rotational motion and viewpoint change, breaking the assumptions of many simple trackers.16
Deep-learning (DL) based feature extractors and matchers have been developed specifically to overcome these "challenging visual conditions".5 Models like SuperPoint, SuperGlue, and LoFTR are trained to find more robust and repeatable features, even in low-texture scenes.4
### **Table 1: Analysis of State-of-the-Art Feature Extraction and Matching Techniques**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SIFT + BFMatcher/FLANN** (OpenCV) | - Scale and rotation invariant. - High-quality, robust matches. - Well-studied and mature.15 | - Computationally slow (CPU-based). - Poor performance in low-texture or weakly-textured areas.14 - Patented (though expired). | - High-contrast, well-defined features. | **Poor.** Too slow for the <5s target and will fail to find features in the low-texture agricultural landscapes shown in sample images. |
| **ORB + BFMatcher** (OpenCV) | - Extremely fast and lightweight. - Standard for real-time SLAM (e.g., ORB-SLAM).21 - Rotation invariant. | - *Not* scale invariant (uses a pyramid). - Performs very poorly in low-texture scenes.5 - Unstable in high-blur scenarios. | - CPU, lightweight. - High-gradient corners. | **Very Poor.** While fast, it fails on the *robustness* requirement. It is designed for textured, indoor/urban scenes, not sparse, natural terrain. |
| **SuperPoint + SuperGlue** (PyTorch, C++/TensorRT) | - SOTA robustness in low-texture, high-blur, and challenging conditions.4 - End-to-end learning for detection and matching.24 - Multiple open-source SLAM integrations exist (e.g., SuperSLAM).25 | - Requires a powerful GPU for real-time performance. - Sparse feature-based (not dense). | - NVIDIA GPU (RTX 2060+). - PyTorch (research) or TensorRT (deployment).26 | **Excellent.** This approach is *designed* for the exact "challenging conditions" of this problem. It provides SOTA robustness in low-texture scenes.4 The user's hardware (RTX 2060+) meets the requirements. |
| **LoFTR** (PyTorch) | - Detector-free dense matching.14 - Extremely robust to viewpoint and texture challenges.14 - Excellent performance on natural terrain and low-overlap images.19 | - High computational and VRAM cost. - Can cause CUDA Out-of-Memory (OOM) errors on very high-resolution images.30 - Slower than sparse-feature methods. | - High-end NVIDIA GPU. - PyTorch. | **Good, but Risky.** While its robustness is excellent, its dense, Transformer-based nature makes it vulnerable to OOM errors on the 6252x4168 images.30 The sparse SuperPoint approach is a safer, more-scalable choice for the VO front-end. |
### **Selected Approach (VO Front-End): SuperPoint + SuperGlue/LightGlue**
The selected approach is a VO front-end based on **SuperPoint** for feature extraction and **SuperGlue** (or its faster successor, **LightGlue**) for matching.18
* **Robustness:** This combination is proven to provide superior robustness and accuracy in sparse-texture scenes, extracting more and higher-quality matches than ORB.4
* **Performance:** It is designed for GPU acceleration and is used in SOTA real-time SLAM systems, demonstrating its feasibility within the <5s target on an RTX 2060.25
* **Scalability:** As a sparse-feature method, it avoids the memory-scaling issues of dense matchers like LoFTR when faced with the user's maximum 6252x4168 resolution.30 The image can be downscaled for real-time VO, and SuperPoint will still find stable features.
## **5. Component Analysis: Back-End (Trajectory Optimization and Refinement)**
The task of the Back-End is to fuse all incoming measurements (high-frequency/low-accuracy relative VO poses, low-frequency/high-accuracy absolute CVGL poses) into a single, globally consistent trajectory. This component's design is dictated by the user's real-time streaming and refinement requirements [User Query].
A critical architectural choice must be made between a traditional, batch **Structure from Motion (SfM)** pipeline and a real-time **SLAM (Simultaneous Localization and Mapping)** pipeline.
* **Batch SfM:** (e.g., COLMAP).32 This approach is an offline process. It collects all 1500-3000 images, performs feature matching, and then runs a large, non-real-time "Bundle Adjustment" (BA) to solve for all camera poses and 3D points simultaneously.35 While this produces the most accurate possible result, it can take hours to compute. It *cannot* meet the <5s/image or "immediate results" criteria.
* **Real-time SLAM:** (e.g., ORB-SLAM3).28 This approach is *online* and *incremental*. It maintains a "pose graph" of the trajectory.10 It provides an immediate pose estimate based on the VO front-end. When a new, high-quality measurement arrives (like a loop closure 37, or in our case, a CVGL fix), it triggers a fast re-optimization of the graph, publishing a *refined* result.11
The user's requirements for "results...appear immediately" and "system could refine existing calculated results" [User Query] are a textbook description of a real-time SLAM back-end.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (g2o, Ceres Solver, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates. - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (CVGL) data arrives.10 - Meets the <5s and streaming criteria. | - Initial estimate is less accurate than a full batch process. - Susceptible to drift *until* a loop closure (CVGL fix) is made. | - A graph optimization library (g2o, Ceres). - A robust cost function to reject outliers. | **Excellent.** This is the *only* architecture that satisfies the user's real-time streaming and asynchronous refinement constraints. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory.35 - Can import custom DL matches.38 | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails all timing and streaming criteria. | - All images must be available before processing starts. - High RAM and CPU. | **Unsuitable (for the *online* system).** This approach is ideal for an *optional, post-flight, high-accuracy* refinement, but it cannot be the primary system. |
### **Selected Approach (Back-End): Incremental Pose-Graph Optimization (g2o/Ceres)**
The system's back-end will be built as an **Incremental Pose-Graph Optimizer** using a library like **g2o** or **Ceres Solver**. This is the only way to meet the real-time streaming and refinement constraints [User Query].
The graph will contain:
* **Nodes:** The 6-DoF pose of each camera frame.
* **Edges (Constraints):**
1. **Odometry Edges:** Relative 6-DoF transforms from the VO Front-End (SuperPoint+SuperGlue). These are high-frequency but have accumulating drift/scale error.
2. **Georeferencing Edges:** Absolute 6-DoF poses from the CVGL Module. These are low-frequency but are drift-free and provide the absolute scale.
3. **Start-Point Edge:** A high-confidence absolute pose for Image 1, fixed to the user-provided start GPS.
This architecture allows the system to provide an immediate estimate (from odometry) and then drastically improve its accuracy (correcting scale and drift) whenever a new georeferencing edge is added.
## **6. Component Analysis: Global-Pose Correction (Georeferencing Module)**
This module is the most critical component for meeting the accuracy requirements. Its task is to provide absolute GPS pose estimates by matching the UAV's nadir-pointing-but-non-stabilized images to the Google Maps satellite provider [User Query]. This is the only component that can correct the monocular scale drift.
This task is known as **Cross-View Geolocalization (CVGL)**.7 It is extremely challenging due to the "domain gap" 44 between the two image sources:
1. **Viewpoint:** The UAV is at low altitude (<1km) and non-nadir (due to fixed-wing tilt) 45, while the satellite is at a very high altitude and is perfectly nadir.
2. **Appearance:** The images come from different sensors, with different lighting (shadows), and at different times. The Google Maps data may be "outdated" [User Query], showing different seasons, vegetation, or man-made structures.47
A simple, brute-force feature match is computationally impossible. The solution is a **hierarchical, two-stage approach** that mimics SOTA research 7:
* **Stage 1: Coarse Retrieval.** We cannot run expensive matching against the entire map. Instead, we treat this as an image retrieval problem. We use a Deep Learning model (e.g., a Siamese or Dual CNN trained on this task 50) to generate a compact "embedding vector" (a digital signature) for the UAV image. In an offline step, we pre-compute embeddings for *all* satellite map tiles in the operational area. The UAV image's embedding is then used to perform a very fast (e.g., FAISS library) similarity search against the satellite database, returning the Top-K most likely-matching satellite tiles.
* **Stage 2: Fine-Grained Pose.** *Only* for these Top-K candidates do we perform the heavy-duty feature matching. We use our selected **SuperPoint+SuperGlue** matcher 53 to find precise correspondences between the UAV image and the K satellite tiles. If a high-confidence geometric match (e.g., >50 inliers) is found, we can compute the precise 6-DoF pose of the UAV relative to that tile, thus yielding an absolute GPS coordinate.
### **Table 3: Analysis of State-of-the-Art Cross-View Geolocalization (CVGL) Techniques**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Coarse Retrieval (Siamese/Dual CNNs)** (PyTorch, ResNet18) | - Extremely fast for retrieval (database lookup). - Learns features robust to seasonal and appearance changes.50 - Narrows search space from millions to a few. | - Does *not* provide a precise 6-DoF pose, only a "best match" tile. - Requires training on a dataset of matched UAV-satellite pairs. | - Pre-trained model (e.g., on ResNet18).52 - Pre-computed satellite embedding database. | **Essential (as Stage 1).** This is the only computationally feasible way to "find" the UAV on the map. |
| **Fine-Grained Feature Matching** (SuperPoint + SuperGlue) | - Provides a highly-accurate 6-Dof pose estimate.53 - Re-uses the same robust matcher from the VO Front-End.54 | - Too slow to run on the entire map. - *Requires* a good initial guess (from Stage 1) to be effective. | - NVIDIA GPU. - Top-K candidate tiles from Stage 1. | **Essential (as Stage 2).** This is the component that actually computes the precise GPS pose from the coarse candidates. |
| **End-to-End DL Models (Transformers)** (PFED, ReCOT, etc.) | - SOTA accuracy in recent benchmarks.13 - Can be highly efficient (e.g., PFED).13 - Can perform retrieval and pose estimation in one model. | - Often research-grade, not robustly open-sourced. - May be complex to train and deploy. - Less modular and harder to debug than the two-stage approach. | - Specific, complex model architectures.13 - Large-scale training datasets. | **Not Recommended (for initial build).** While powerful, these are less practical for a version 1 build. The two-stage approach is more modular, debuggable, and uses components already required by the VO system. |
### **Selected Approach (CVGL Module): Hierarchical Retrieval + Matching**
The CVGL module will be implemented as a two-stage hierarchical system:
1. **Stage 1 (Coarse):** A **Siamese CNN** 52 (or similar model) generates an embedding for the UAV image. This embedding is used to retrieve the Top-5 most similar satellite tiles from a pre-computed database.
2. **Stage 2 (Fine):** The **SuperPoint+SuperGlue** matcher 53 is run between the UAV image and these 5 tiles. The match with the highest inlier count and lowest reprojection error is used to calculate the absolute 6-DoF pose, which is then sent to the Back-End optimizer.
## **7. Addressing Critical Acceptance Criteria and Failure Modes**
This hybrid architecture's logic is designed to handle the most difficult acceptance criteria [User Query] through a robust, multi-stage escalation process.
### **Stage 1: Initial State (Normal Operation)**
* **Condition:** VO(N-1 -> N) succeeds.
* **System Logic:** The **VO Front-End** provides the high-frequency relative pose. This is added to the graph, and the **Initial Pose** is sent to the user (<5s).
* **Resolution:** The **CVGL Module** runs asynchronously to provide a Refined Pose later, which corrects for scale drift.
### **Stage 2: Transient Failure / Outlier Handling (AC-3)**
* **Condition:** VO(N-1 -> N) fails (e.g., >350m jump, severe motion blur, low overlap) [User Query]. This triggers an immediate, high-priority CVGL(N) query.
* **System Logic:**
1. If CVGL(N) *succeeds*, the system has conflicting data: a failed VO link and a successful CVGL pose. The **Back-End Optimizer** uses a robust kernel to reject the high-error VO link as an outlier and accepts the CVGL pose.56 The trajectory "jumps" to the correct location, and VO resumes from Image N+1.
2. If CVGL(N) *also fails* (e.g., due to cloud cover or outdated map), the system assumes Image N is a single bad frame (an outlier).
* **Resolution (Frame Skipping):** The system buffers Image N and, upon receiving Image N+1, the **VO Front-End** attempts to "bridge the gap" by matching VO(N-1 -> N+1).
* **If successful,** a pose for N+1 is found. Image N is marked as a rejected outlier, and the system continues.
* **If VO(N-1 -> N+1) fails,** it repeats for VO(N-1 -> N+2).
* If this "bridging" fails for 3 consecutive frames, the system concludes it is not a transient outlier but a persistent tracking loss. This escalates to Stage 3.
### **Stage 3: Persistent Tracking Loss / Sharp Turn Handling (AC-4)**
* **Condition:** VO tracking is lost, and the "frame-skipping" in Stage 2 fails (e.g., a "sharp turn" with no overlap) [User Query].
* **System Logic (Multi-Map "Chunking"):** The **Back-End Optimizer** declares a "Tracking Lost" state and creates a *new, independent map* ("Chunk 2").
* The **VO Front-End** is re-initialized and begins populating this new chunk, tracking VO(N+3 -> N+4), VO(N+4 -> N+5), etc. This new chunk is internally consistent but has no absolute GPS position (it is "floating").
* **Resolution (Asynchronous Relocalization):**
1. The **CVGL Module** now runs asynchronously on all frames in this new "Chunk 2".
2. Crucially, it uses the last known GPS coordinate from "Chunk 1" as a *search prior*, narrowing the satellite map search area to the vicinity.
3. The system continues to build Chunk 2 until the CVGL module successfully finds a high-confidence Absolute_Pose for *any* frame in that chunk (e.g., for Image N+20).
4. Once this single GPS "anchor" is found, the **Back-End Optimizer** performs a full graph optimization. It calculates the 7-DoF transformation (3D position, 3D rotation, and **scale**) to align all of Chunk 2 and merge it with Chunk 1.
5. This "chunking" method robustly handles the "correctly continue the work" criterion by allowing the system to keep tracking locally even while globally lost, confident it can merge the maps later.
### **Stage 4: Catastrophic Failure / User Intervention (AC-6)**
* **Condition:** The system has entered Stage 3 and is building "Chunk 2," but the **CVGL Module** has *also* failed for a prolonged period (e.g., 20% of the route, or 50+ consecutive frames) [User Query]. This is a "worst-case" scenario where the UAV is in an area with no VO features (e.g., over a lake) *and* no CVGL features (e.g., heavy clouds or outdated maps).
* **System Logic:** The system is "absolutely incapable" of determining its pose.
* **Resolution (User Input):** The system triggers the "ask the user for input" event. A UI prompt will show the last known good image (from Chunk 1) on the map and the new, "lost" image (e.g., N+50). It will ask the user to "Click on the map to provide a coarse location." This user-provided GPS point is then fed to the CVGL module as a *strong prior*, drastically narrowing the search space and enabling it to re-acquire a lock.
## **8. Implementation and Output Generation**
### **Real-time Workflow (<5s Initial, Async Refinement)**
A concrete implementation plan for processing Image N:
1. **T=0.0s:** Image[N] (6200px) received.
2. **T=0.1s:** Image pre-processed: Scaled to 1024px for VO/CVGL. Full-res original stored.
3. **T=0.5s:** **VO Front-End** (GPU): SuperPoint features extracted for 1024px image.
4. **T=1.0s:** **VO Front-End** (GPU): SuperGlue matches 1024px Image[N] -> 1024px Image[N-1]. Relative_Pose (6-DoF) estimated via RANSAC/PnP.
5. **T=1.1s:** **Back-End:** Relative_Pose added to graph. Optimizer updates trajectory.
6. **T=1.2s:** **OUTPUT:** Initial Pose_N_Est (GPS) sent to user. **(<5s criterion met)**.
7. **T=1.3s:** **CVGL Module (Async Task)** (GPU): Siamese/Dual CNN generates embedding for 1024px Image[N].
8. **T=1.5s:** **CVGL Module (Async Task):** Coarse retrieval (FAISS lookup) returns Top-5 satellite tile candidates.
9. **T=4.0s:** **CVGL Module (Async Task)** (GPU): Fine-grained matching. SuperPoint+SuperGlue runs 5 times (Image[N] vs. 5 satellite tiles).
10. **T=4.5s:** **CVGL Module (Async Task):** A high-confidence match is found. Absolute_Pose_N_Abs (6-DoF) is computed.
11. **T=4.6s:** **Back-End:** High-confidence Absolute_Pose_N_Abs added to pose graph. Graph re-optimization is triggered.
12. **T=4.8s:** **OUTPUT:** Pose_N_Refined (GPS) sent to user. **(Refinement criterion met)**.
### **Determining Object-Level GPS (from Pixel Coordinate)**
The requirement to find the "coordinates of the center of any object in these photos" [User Query] is met by projecting a pixel to its 3D world coordinate. This requires the (u,v) pixel, the camera's 6-DoF pose, and the camera's intrinsic matrix (K).
Two methods will be implemented to support the streaming/refinement architecture:
1. **Method 1 (Immediate, <5s): Flat-Earth Projection.**
* When the user clicks pixel (u,v) on Image[N], the system uses the *Initial Pose_N_Est*.
* It assumes the ground is a flat plane at the predefined altitude (e.g., 900m altitude if flying at 1km and ground is at 100m) [User Query].
* It computes the 3D ray from the camera center through (u,v) using the intrinsic matrix (K).
* It calculates the 3D intersection point of this ray with the flat ground plane.
* This 3D world point is converted to a GPS coordinate and sent to the user. This is very fast but less accurate in non-flat terrain.
2. **Method 2 (Refined, Post-BA): Structure-from-Motion Projection.**
* The Back-End's pose-graph optimization, as a byproduct, will create a sparse 3D point cloud of the world (i.e., the "SfM" part of SLAM).35
* When the user clicks (u,v), the system uses the *Pose_N_Refined*.
* It raycasts from the camera center through (u,v) and finds the 3D intersection point with the *actual 3D point cloud* generated by the system.
* This 3D point's coordinate (X,Y,Z) is converted to GPS. This is far more accurate as it accounts for real-world topography (hills, ditches) captured in the 3D map.
## **9. Testing and Validation Strategy**
A rigorous testing strategy is required to validate all 10 acceptance criteria. The foundation of this strategy is the creation of a **Ground-Truth Test Dataset**. This will involve flying several test routes and manually creating a "checkpoint" (CP) file, similar to the provided coordinates.csv 58, using a high-precision RTK/PPK GPS. This provides the "real GPS" for validation.59
### **Accuracy Validation Methodology (AC-1, AC-2, AC-5, AC-8, AC-9)**
These tests validate the system's accuracy and completion metrics.59
1. A test flight of 1000 images with high-precision ground-truth CPs is prepared.
2. The system is run given only the first GPS coordinate.
3. A test script compares the system's *final refined GPS output* for each image against its *ground-truth CP*. The Haversine distance (error in meters) is calculated for all 1000 images.
4. This yields a list of 1000 error values.
5. **Test_Accuracy_50m (AC-1):** ASSERT (count(errors < 50m) / 1000) >= 0.80
6. **Test_Accuracy_20m (AC-2):** ASSERT (count(errors < 20m) / 1000) >= 0.60
7. **Test_Outlier_Rate (AC-5):** ASSERT (count(un-localized_images) / 1000) < 0.10
8. **Test_Image_Registration_Rate (AC-8):** ASSERT (count(localized_images) / 1000) > 0.95
9. **Test_Mean_Reprojection_Error (AC-9):** ASSERT (Back-End.final_MRE) < 1.0
10. **Test_RMSE:** The overall Root Mean Square Error (RMSE) of the entire trajectory will be calculated as a primary performance benchmark.59
### **Integration and Functional Tests (AC-3, AC-4, AC-6)**
These tests validate the system's logic and robustness to failure modes.62
* Test_Low_Overlap_Relocalization (AC-4):
* **Setup:** Create a test sequence of 50 images. From this, manually delete images 20-24 (simulating 5 lost frames during a sharp turn).63
* **Test:** Run the system on this "broken" sequence.
* **Pass/Fail:** The system must report "Tracking Lost" at frame 20, initiate a new "chunk," and then "Tracking Re-acquired" and "Maps Merged" when the CVGL module successfully localizes frame 25 (or a subsequent frame). The final trajectory error for frame 25 must be < 50m.
* Test_350m_Outlier_Rejection (AC-3):
* **Setup:** Create a test sequence. At image 30, insert a "rogue" image (Image 30b) known to be 350m away.
* **Test:** Run the system on this sequence (..., 29, 30, 30b, 31,...).
* **Pass/Fail:** The system must correctly identify Image 30b as an outlier (RANSAC failure 56), reject it (or jump to its CVGL-verified pose), and "correctly continue the work" by successfully tracking Image 31 from Image 30 (using the frame-skipping logic). The trajectory must not be corrupted.
* Test_User_Intervention_Prompt (AC-6):
* **Setup:** Create a test sequence with 50 consecutive "bad" frames (e.g., pure sky, lens cap) to ensure the transient and chunking logics are bypassed.
* **Test:** Run the system.
* **Pass/Fail:** The system must enter a "LOST" state, attempt and fail to relocalize via CVGL for 50 frames, and then correctly trigger the "ask for user input" event.
### **Non-Functional Tests (AC-7, AC-8, Hardware)**
These tests validate performance and resource requirements.66
* Test_Performance_Per_Image (AC-7):
* **Setup:** Run the 1000-image test set on the minimum-spec RTX 2060.
* **Test:** Measure the time from "Image In" to "Initial Pose Out" for every frame.
* **Pass/Fail:** ASSERT average_time < 5.0s.
* Test_Streaming_Refinement (AC-8):
* **Setup:** Run the 1000-image test set.
* **Test:** A logger must verify that *two* poses are received for >80% of images: an "Initial" pose (T < 5s) and a "Refined" pose (T > 5s, after CVGL).
* **Pass/Fail:** The refinement mechanism is functioning correctly.
* Test_Scalability_Large_Route (Constraints):
* **Setup:** Run the system on a full 3000-image dataset.
* **Test:** Monitor system RAM, VRAM, and processing time per frame over the entire run.
* **Pass/Fail:** The system must complete the run without memory leaks, and the processing time per image must not degrade significantly as the pose graph grows.
_docs/01_solution/02_solution_draft.md
-284
View File
@@ -1,284 +0,0 @@
# **GEORTOLS-SA UAV Image Geolocalization in IMU-Denied Environments**
The GEORTOLS-SA system is an asynchronous, four-component software solution designed for deployment on an NVIDIA RTX 2060+ GPU. It is architected from the ground up to handle the specific challenges of IMU-denied, scale-aware localization and real-time streaming output.
### **Product Solution Description**
* **Inputs:**
1. A sequence of consecutively named images (FullHD to 6252x4168).
2. The absolute GPS coordinate (Latitude, Longitude) for the first image (Image 0).
3. A pre-calibrated camera intrinsic matrix ($K$).
4. The predefined, absolute metric altitude of the UAV ($H$, e.g., 900 meters).
5. API access to the Google Maps satellite provider.
* **Outputs (Streaming):**
1. **Initial Pose (T \< 5s):** A high-confidence, *metric-scale* estimate ($Pose\_N\_Est$) of the image's 6-DoF pose and GPS coordinate. This is sent to the user immediately upon calculation (AC-7, AC-8).
2. **Refined Pose (T > 5s):** A globally-optimized pose ($Pose\_N\_Refined$) sent asynchronously as the back-end optimizer fuses data from the CVGL module (AC-8).
### **Component Interaction Diagram and Data Flow**
The system is architected as four parallel-processing components to meet the stringent real-time and refinement requirements.
1. **Image Ingestion & Pre-processing:** This module receives the new, high-resolution Image_N. It immediately creates two copies:
* Image_N_LR (Low-Resolution, e.g., 1536x1024): This copy is immediately dispatched to the SA-VO Front-End for real-time processing.
* Image_N_HR (High-Resolution, 6.2K): This copy is stored and made available to the CVGL Module for its asynchronous, high-accuracy matching pipeline.
2. **Scale-Aware VO (SA-VO) Front-End (High-Frequency Thread):** This component's sole task is high-speed, *metric-scale* relative pose estimation. It matches Image_N_LR to Image_N-1_LR, computes the 6-DoF relative transform, and critically, uses the "known altitude" ($H$) constraint to recover the absolute scale (detailed in Section 3.0). It sends this high-confidence Relative_Metric_Pose to the Back-End.
3. **Cross-View Geolocalization (CVGL) Module (Low-Frequency, Asynchronous Thread):** This is a heavier, slower module. It takes Image_N (both LR and HR) and queries the Google Maps database to find an *absolute GPS pose*. When a high-confidence match is found, its Absolute_GPS_Pose is sent to the Back-End as a global "anchor" constraint.
4. **Trajectory Optimization Back-End (Central Hub):** This component manages the complete flight trajectory as a pose graph.10 It continuously fuses two distinct, high-quality data streams:
* **On receiving Relative_Metric_Pose (T \< 5s):** It appends this pose to the graph, calculates the Pose_N_Est, and **sends this initial result to the user (AC-7, AC-8 met)**.
* **On receiving Absolute_GPS_Pose (T > 5s):** It adds this as a high-confidence "global anchor" constraint 12, triggers a full graph re-optimization to correct any minor biases, and **sends the Pose_N_Refined to the user (AC-8 refinement met)**.
###
### **VO "Trust Model" of GEORTOLS-SA**
In GEORTOLS-SA, the trust model:
* The **SA-VO Front-End** is now *highly trusted* for its local, frame-to-frame *metric* accuracy.
* The **CVGL Module** remains *highly trusted* for its *global* (GPS) accuracy.
Both components are operating in the same scale-aware, metric space. The Back-End's job is no longer to fix a broken, drifting VO. Instead, it performs a robust fusion of two independent, high-quality metric measurements.12
This model is self-correcting. If the user's predefined altitude $H$ is slightly incorrect (e.g., entered as 900m but is truly 880m), the SA-VO front-end will be *consistently* off by a small percentage. The periodic, high-confidence CVGL "anchors" will create a consistent, low-level "tension" in the pose graph. The graph optimizer (e.g., Ceres Solver) 3 will resolve this tension by slightly "pulling" the SA-VO poses to fit the global anchors, effectively *learning* and correcting for the altitude bias. This robust fusion is the key to meeting the 20-meter and 50-meter accuracy targets (AC-1, AC-2).
## **3.0 Core Component: The Scale-Aware Visual Odometry (SA-VO) Front-End**
This component is the new, critical engine of the system. Its sole task is to compute the *metric-scale* 6-DoF relative motion between consecutive frames, thereby eliminating scale drift at its source.
### **3.1 Rationale and Mechanism for Per-Frame Scale Recovery**
The SA-VO front-end implements a geometric algorithm to recover the absolute scale $s$ for *every* frame-to-frame transition. This algorithm directly leverages the query's "known altitude" ($H$) and "planar ground" constraints.5
The SA-VO algorithm for processing Image_N (relative to Image_N-1) is as follows:
1. **Feature Matching:** Extract and match robust features between Image_N and Image_N-1 using the selected feature matcher (see Section 3.2). This yields a set of corresponding 2D pixel coordinates.
2. **Essential Matrix:** Use RANSAC (Random Sample Consensus) and the camera intrinsic matrix $K$ to compute the Essential Matrix $E$ from the "inlier" correspondences.2
3. **Pose Decomposition:** Decompose $E$ to find the relative Rotation $R$ and the *unscaled* translation vector $t$, where the magnitude $||t||$ is fixed to 1.2
4. **Triangulation:** Triangulate the 3D-world points $X$ for all inlier features using the unscaled pose $$.15 These 3D points ($X_i$) are now in a local, *unscaled* coordinate system (i.e., we know the *shape* of the point cloud, but not its *size*).
5. **Ground Plane Fitting:** The query states "terrain height can be neglected," meaning we assume a planar ground. A *second* RANSAC pass is performed, this time fitting a 3D plane to the set of triangulated 3D points $X$. The inliers to this RANSAC are identified as the ground points $X_g$.5 This method is highly robust as it does not rely on a single point, but on the consensus of all visible ground features.16
6. **Unscaled Height ($h$):** From the fitted plane equation n^T X + d = 0, the parameter $d$ represents the perpendicular distance from the camera (at the coordinate system's origin) to the computed ground plane. This is our *unscaled* height $h$.
7. **Scale Computation:** We now have two values: the *real, metric* altitude $h$ (e.g., 900m) provided by the user, and our *computed, unscaled* altitude $h$. The absolute scale $s$ for this frame is the ratio of these two values: s = h / h.
8. **Metric Pose:** The final, metric-scale relative pose is $$, where the metric translation $T = s * t$. This high-confidence, scale-aware pose is sent to the Back-End.
### **3.2 Feature Matching Sub-System Analysis**
The success of the SA-VO algorithm depends *entirely* on the quality of the initial feature matches, especially in the low-texture agricultural terrain specified in the query. The system requires a matcher that is both robust (for sparse textures) and extremely fast (for AC-7).
The initial draft's choice of SuperGlue 17 is a strong, proven baseline. However, its successor, LightGlue 18, offers a critical, non-obvious advantage: **adaptivity**.
The UAV flight is specified as *mostly* straight, with high overlap. Sharp turns (AC-4) are "rather an exception." This means \~95% of our image pairs are "easy" to match, while 5% are "hard."
* SuperGlue uses a fixed-depth Graph Neural Network (GNN), spending the *same* (large) amount of compute on an "easy" pair as a "hard" pair.19 This is inefficient.
* LightGlue is *adaptive*.19 For an easy, high-overlap pair, it can exit early (e.g., at layer 3/9), returning a high-confidence match in a fraction of the time. For a "hard" low-overlap pair, it will use its full depth to get the best possible result.19
By using LightGlue, the system saves *enormous* amounts of computational budget on the 95% of "easy" frames, ensuring it *always* meets the \<5s budget (AC-7) and reserving that compute for the harder CVGL tasks. LightGlue is a "plug-and-play replacement" 19 that is faster, more accurate, and easier to train.19
### **Table 1: Analysis of State-of-the-Art Feature Matchers (For SA-VO Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** 17 | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems.22 | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue.19 - Training is complex.19 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.25 | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the \<5s time budget (AC-7). |
| **SuperPoint + LightGlue** 18 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.19 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy.19 - **Easier to Train:** Simpler architecture and loss.19 - Direct plug-and-play replacement for SuperGlue. | - Newer, less long-term-SLAM-proven than SuperGlue (though rapidly being adopted). | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.28 | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, preserving the budget for the 5% of hard (turn) frames, maximizing our ability to meet AC-7. |
### **3.3 Selected Approach (SA-VO): SuperPoint + LightGlue**
The SA-VO front-end will be built using:
* **Detector:** **SuperPoint** 24 to detect sparse, robust features on the Image_N_LR.
* **Matcher:** **LightGlue** 18 to match features from Image_N_LR to Image_N-1_LR.
This combination provides the SOTA robustness required for low-texture fields, while LightGlue's adaptive performance 19 is the key to meeting the \<5s (AC-7) real-time requirement.
## **4.0 Global Anchoring: The Cross-View Geolocalization (CVGL) Module**
With the SA-VO front-end handling metric scale, the CVGL module's task is refined. Its purpose is no longer to *correct scale*, but to provide *absolute global "anchor" poses*. This corrects for any accumulated bias (e.g., if the $h$ prior is off by 5m) and, critically, *relocalizes* the system after a persistent tracking loss (AC-4).
### **4.1 Hierarchical Retrieval-and-Match Pipeline**
This module runs asynchronously and is computationally heavy. A brute-force search against the entire Google Maps database is impossible. A two-stage hierarchical pipeline is required:
1. **Stage 1: Coarse Retrieval.** This is treated as an image retrieval problem.29
* A **Siamese CNN** 30 (or similar Dual-CNN architecture) is used to generate a compact "embedding vector" (a digital signature) for the Image_N_LR.
* An embedding database will be pre-computed for *all* Google Maps satellite tiles in the specified Eastern Ukraine operational area.
* The UAV image's embedding is then used to perform a very fast (e.g., FAISS library) similarity search against the satellite database, returning the *Top-K* (e.g., K=5) most likely-matching satellite tiles.
2. **Stage 2: Fine-Grained Pose.**
* *Only* for these Top-5 candidates, the system performs the heavy-duty **SuperPoint + LightGlue** matching.
* This match is *not* Image_N -> Image_N-1. It is Image_N -> Satellite_Tile_K.
* The match with the highest inlier count and lowest reprojection error (MRE \< 1.0, AC-10) is used to compute the precise 6-DoF pose of the UAV relative to that georeferenced satellite tile. This yields the final Absolute_GPS_Pose.
### **4.2 Critical Insight: Solving the Oblique-to-Nadir "Domain Gap"**
A critical, unaddressed failure mode exists. The query states the camera is **"not autostabilized"** [User Query]. On a fixed-wing UAV, this guarantees that during a bank or sharp turn (AC-4), the camera will *not* be nadir (top-down). It will be *oblique*, capturing the ground from an angle. The Google Maps reference, however, is *perfectly nadir*.32
This creates a severe "domain gap".33 A CVGL system trained *only* to match nadir-to-nadir images will *fail* when presented with an oblique UAV image.34 This means the CVGL module will fail *precisely* when it is needed most: during the sharp turns (AC-4) when SA-VO tracking is also lost.
The solution is to *close this domain gap* during training. Since the real-world UAV images will be oblique, the network must be taught to match oblique views to nadir ones.
Solution: Synthetic Data Generation for Robust Training
The Stage 1 Siamese CNN 30 must be trained on a custom, synthetically-generated dataset.37 The process is as follows:
1. Acquire nadir satellite imagery and a corresponding Digital Elevation Model (DEM) for the operational area.
2. Use this data to *synthetically render* the nadir satellite imagery from a wide variety of *oblique* viewpoints, simulating the UAV's roll and pitch.38
3. Create thousands of training pairs, each consisting of (Nadir_Satellite_Tile, Synthetically_Oblique_Tile_Angle_30_Deg).
4. Train the Siamese network 29 to learn that these two images—despite their *vastly* different appearances—are a *match*.
This process teaches the retrieval network to be *viewpoint-invariant*.35 It learns to ignore perspective distortion and match the true underlying ground features (road intersections, field boundaries). This is the *only* way to ensure the CVGL module can robustly relocalize the UAV during a sharp turn (AC-4).
## **5.0 Trajectory Fusion: The Robust Optimization Back-End**
This component is the system's central "brain." It runs continuously, fusing all incoming measurements (high-frequency/metric-scale SA-VO poses, low-frequency/globally-absolute CVGL poses) into a single, globally consistent trajectory. This component's design is dictated by the requirements for streaming (AC-8), refinement (AC-8), and outlier-rejection (AC-3).
### **5.1 Selected Strategy: Incremental Pose-Graph Optimization**
The user's requirements for "results...appear immediately" and "system could refine existing calculated results" [User Query] are a textbook description of a real-time SLAM back-end.11 A batch Structure from Motion (SfM) process, which requires all images upfront and can take hours, is unsuitable for the primary system.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (g2o 13, Ceres Solver 10, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (CVGL) data arrives (AC-8).11 - **Robust:** Can handle outliers via robust kernels.39 | - Initial estimate is less accurate than a full batch process. - Can drift *if* not anchored (though our SA-VO minimizes this). | - A graph optimization library (g2o, Ceres). - A robust cost function.41 | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming and asynchronous refinement. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process after the flight. |
The system's back-end will be built as an **Incremental Pose-Graph Optimizer** using **Ceres Solver**.10 Ceres is selected due to its large user community, robust documentation, excellent support for robust loss functions 10, and proven scalability for large-scale nonlinear least-squares problems.42
### **5.2 Mechanism for Automatic Outlier Rejection (AC-3, AC-5)**
The system must "correctly continue the work even in the presence of up to 350 meters of an outlier" (AC-3). A standard least-squares optimizer would be catastrophically corrupted by this event, as it would try to *average* this 350m error, pulling the *entire* 300km trajectory out of alignment.
A modern optimizer does not need to use brittle, hand-coded if-then logic to reject outliers. It can *mathematically* and *automatically* down-weight them using **Robust Loss Functions (Kernels)**.41
The mechanism is as follows:
1. The Ceres Back-End 10 maintains a graph of nodes (poses) and edges (constraints, or measurements).
2. A 350m outlier (AC-3) will create an edge with a *massive* error (residual).
3. A standard (quadratic) loss function $cost(error) = error^2$ would create a *catastrophic* cost, forcing the optimizer to ruin the entire graph to accommodate it.
4. Instead, the system will wrap its cost functions in a **Robust Loss Function**, such as **CauchyLoss** or **HuberLoss**.10
5. A robust loss function behaves quadratically for small errors (which it tries hard to fix) but becomes *sub-linear* for large errors. When it "sees" the 350m error, it mathematically *down-weights its influence*.43
6. The optimizer effectively *acknowledges* the 350m error but *refuses* to pull the entire graph to fix this one "insane" measurement. It automatically, and gracefully, treats the outlier as a "lost cause" and optimizes the 99.9% of "sane" measurements. This is the modern, robust solution to AC-3 and AC-5.
## **6.0 High-Resolution (6.2K) and Performance Optimization**
The system must simultaneously handle massive 6252x4168 (26-Megapixel) images and run on a modest RTX 2060 GPU [User Query] with a \<5s time limit (AC-7). These are opposing constraints.
### **6.1 The Multi-Scale Patch-Based Processing Pipeline**
Running *any* deep learning model (SuperPoint, LightGlue) on a full 6.2K image will be impossibly slow and will *immediately* cause a CUDA Out-of-Memory (OOM) error on a 6GB RTX 2060.45
The solution is not to process the full 6.2K image in real-time. Instead, a **multi-scale, patch-based pipeline** is required, where different components use the resolution best suited to their task.46
1. **For SA-VO (Real-time, \<5s):** The SA-VO front-end is concerned with *motion*, not fine-grained detail. The 6.2K Image_N_HR is *immediately* downscaled to a manageable 1536x1024 (Image_N_LR). The entire SA-VO (SuperPoint + LightGlue) pipeline runs *only* on this low-resolution, fast-to-process image. This is how the \<5s (AC-7) budget is met.
2. **For CVGL (High-Accuracy, Async):** The CVGL module, which runs asynchronously, is where the 6.2K detail is *selectively* used to meet the 20m (AC-2) accuracy target. It uses a "coarse-to-fine" 48 approach:
* **Step A (Coarse):** The Siamese CNN 30 runs on the *downscaled* 1536px Image_N_LR to get a coarse [Lat, Lon] guess.
* **Step B (Fine):** The system uses this coarse guess to fetch the corresponding *high-resolution* satellite tile.
* **Step C (Patching):** The system runs the SuperPoint detector on the *full 6.2K* Image_N_HR to find the Top 100 *most confident* feature keypoints. It then extracts 100 small (e.g., 256x256) *patches* from the full-resolution image, centered on these keypoints.49
* **Step D (Matching):** The system then matches *these small, full-resolution patches* against the high-res satellite tile.
This hybrid method provides the best of both worlds: the fine-grained matching accuracy 50 of the 6.2K image, but without the catastrophic OOM errors or performance penalties.45
### **6.2 Real-Time Deployment with TensorRT**
PyTorch is a research and training framework. Its default inference speed, even on an RTX 2060, is often insufficient to meet a \<5s production requirement.23
For the final production system, the key neural networks (SuperPoint, LightGlue, Siamese CNN) *must* be converted from their PyTorch-native format into a highly-optimized **NVIDIA TensorRT engine**.
* **Benefits:** TensorRT is an inference optimizer that applies graph optimizations, layer fusion, and precision reduction (e.g., to FP16).52 This can achieve a 2x-4x (or more) speedup over native PyTorch.28
* **Deployment:** The resulting TensorRT engine can be deployed via a C++ API 25, which is far more suitable for a robust, high-performance production system.
This conversion is a *mandatory* deployment step. It is what makes a 2-second inference (well within the 5-second AC-7 budget) *achievable* on the specified RTX 2060 hardware.
## **7.0 System Robustness: Failure Mode and Logic Escalation**
The system's logic is designed as a multi-stage escalation process to handle the specific failure modes in the acceptance criteria (AC-3, AC-4, AC-6), ensuring the >95% registration rate (AC-9).
### **Stage 1: Normal Operation (Tracking)**
* **Condition:** SA-VO(N-1 -> N) succeeds. The LightGlue match is high-confidence, and the computed scale $s$ is reasonable.
* **Logic:**
1. The Relative_Metric_Pose is sent to the Back-End.
2. The Pose_N_Est is calculated and sent to the user (\<5s).
3. The CVGL module is queued to run asynchronously to provide a Pose_N_Refined at a later time.
### **Stage 2: Transient SA-VO Failure (AC-3 Outlier Handling)**
* **Condition:** SA-VO(N-1 -> N) fails. This could be a 350m outlier (AC-3), a severely blurred image, or an image with no features (e.g., over a cloud). The LightGlue match fails, or the computed scale $s$ is nonsensical.
* **Logic (Frame Skipping):**
1. The system *buffers* Image_N and marks it as "tentatively lost."
2. When Image_N+1 arrives, the SA-VO front-end attempts to "bridge the gap" by matching SA-VO(N-1 -> N+1).
3. **If successful:** A Relative_Metric_Pose for N+1 is found. Image_N is officially marked as a rejected outlier (AC-5). The system "correctly continues the work" (AC-3 met).
4. **If fails:** The system repeats for SA-VO(N-1 -> N+2).
5. If this "bridging" fails for 3 consecutive frames, the system concludes it is not a transient outlier but a persistent tracking loss, and escalates to Stage 3.
### **Stage 3: Persistent Tracking Loss (AC-4 Sharp Turn Handling)**
* **Condition:** The "frame-skipping" in Stage 2 fails. This is the "sharp turn" scenario [AC-4] where there is \<5% overlap between Image_N-1 and Image_N+k.
* **Logic (Multi-Map "Chunking"):**
1. The Back-End declares a "Tracking Lost" state at Image_N and creates a *new, independent map chunk* ("Chunk 2").
2. The SA-VO Front-End is re-initialized at Image_N and begins populating this new chunk, tracking SA-VO(N -> N+1), SA-VO(N+1 -> N+2), etc.
3. Because the front-end is **Scale-Aware**, this new "Chunk 2" is *already in metric scale*. It is a "floating island" of *known size and shape*; it just is not anchored to the global GPS map.
* **Resolution (Asynchronous Relocalization):**
1. The **CVGL Module** is now tasked, high-priority, to find a *single* Absolute_GPS_Pose for *any* frame in this new "Chunk 2".
2. Once the CVGL module (which is robust to oblique views, per Section 4.2) finds one (e.g., for Image_N+20), the Back-End has all the information it needs.
3. **Merging:** The Back-End calculates the simple 6-DoF transformation (3D translation and rotation, scale=1) to align all of "Chunk 2" and merge it with "Chunk 1". This robustly handles the "correctly continue the work" criterion (AC-4).
### **Stage 4: Catastrophic Failure (AC-6 User Intervention)**
* **Condition:** The system has entered Stage 3 and is building "Chunk 2," but the **CVGL Module** has *also* failed for a prolonged period (e.g., 20% of the route, or 50+ consecutive frames). This is the "worst-case" scenario (e.g., heavy clouds *and* over a large, featureless lake). The system is "absolutely incapable" [User Query].
* **Logic:**
1. The system has a metric-scale "Chunk 2" but zero idea where it is in the world.
2. The Back-End triggers the AC-6 flag.
* **Resolution (User Input):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The UI displays the last known good image (from Chunk 1) and the new, "lost" image (e.g., Image_N+50).
3. The user clicks *one point* on the satellite map.
4. This user-provided [Lat, Lon] is *not* taken as ground truth. It is fed to the CVGL module as a *strong prior*, drastically narrowing its search area from "all of Ukraine" to "a 10km-radius circle."
5. This allows the CVGL module to re-acquire a lock, which triggers the Stage 3 merge, and the system continues.
## **8.0 Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated and how the system's compliance with all 10 acceptance criteria will be validated.
### **8.1 Generating Object-Level GPS (from Pixel Coordinate)**
This meets the requirement to find the "coordinates of the center of any object in these photos" [User Query]. The system provides this via a **Ray-Plane Intersection** method.
* **Inputs:**
1. The user clicks pixel coordinate $(u,v)$ on Image_N.
2. The system retrieves the refined, global 6-DoF pose $$ for Image_N from the Back-End.
3. The system uses the known camera intrinsic matrix $K$.
4. The system uses the known *global ground-plane equation* (e.g., $Z=150m$, based on the predefined altitude and start coordinate).
* **Method:**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} \= K^{-1} \cdot [u, v, 1]^T$.
2. **Transform Ray:** This ray direction is transformed into the *global* coordinate system using the pose's rotation matrix: $d_{global} \= R \cdot d_{cam}$.
3. **Define Ray:** A 3D ray is now defined, originating at the camera's global position $T$ (from the pose) and traveling in the direction $d_{global}$.
4. **Intersect:** The system solves the 3D line-plane intersection equation for this ray and the known global ground plane (e.g., find the intersection with $Z=150m$).
5. **Result:** The 3D intersection point $(X, Y, Z)$ is the *metric* world coordinate of the object on the ground.
6. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate. This process is immediate and can be performed for any pixel on any geolocated image.
### **8.2 Rigorous Validation Methodology**
A comprehensive test plan is required to validate all 10 acceptance criteria. The foundation of this is the creation of a **Ground-Truth Test Harness**.
* **Test Harness:**
1. **Ground-Truth Data:** Several test flights will be conducted in the operational area using a UAV equipped with a high-precision RTK/PPK GPS. This provides the "real GPS" (ground truth) for every image.
2. **Test Datasets:** Multiple test datasets will be curated from this ground-truth data:
* Test_Baseline_1000: A standard 1000-image flight.
* Test_Outlier_350m (AC-3): Test_Baseline_1000 with a single image from 350m away manually inserted at frame 30.
* Test_Sharp_Turn_5pct (AC-4): A sequence where frames 20-24 are manually deleted, simulating a \<5% overlap jump.
* Test_Catastrophic_Fail_20pct (AC-6): A sequence with 200 (20%) consecutive "bad" frames (e.g., pure sky, lens cap) inserted.
* Test_Full_3000: A full 3000-image sequence to test scalability and memory usage.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* Run Test_Baseline_1000. A test script will compare the system's *final refined GPS output* for each image against its *ground-truth GPS*.
* ASSERT (count(errors \< 50m) / 1000) \geq 0.80 (AC-1)
* ASSERT (count(errors \< 20m) / 1000) \geq 0.60 (AC-2)
* ASSERT (count(un-localized_images) / 1000) \< 0.10 (AC-5)
* ASSERT (count(localized_images) / 1000) > 0.95 (AC-9)
* **Test_MRE (AC-10):**
* ASSERT (BackEnd.final_MRE) \< 1.0 (AC-10)
* **Test_Performance (AC-7, AC-8):**
* Run Test_Full_3000 on the minimum-spec RTX 2060.
* Log timestamps for "Image In" -> "Initial Pose Out". ASSERT average_time \< 5.0s (AC-7).
* Log the output stream. ASSERT that >80% of images receive *two* poses: an "Initial" and a "Refined" (AC-8).
* **Test_Robustness (AC-3, AC-4, AC-6):**
* Run Test_Outlier_350m. ASSERT the system correctly continues and the final trajectory error for Image_31 is \< 50m (AC-3).
* Run Test_Sharp_Turn_5pct. ASSERT the system logs "Tracking Lost" and "Maps Merged," and the final trajectory is complete and accurate (AC-4).
* Run Test_Catastrophic_Fail_20pct. ASSERT the system correctly triggers the "ask for user input" event (AC-6).
_docs/01_solution/03_solution_draft.md
-259
View File
@@ -1,259 +0,0 @@
**GEORTEX-R: A Geospatial-Temporal Robust Extraction System for IMU-Denied UAV Geolocalization**
## **1.0 GEORTEX-R: System Architecture and Data Flow**
The GEORTEX-R system is an asynchronous, three-component software solution designed for deployment on an NVIDIA RTX 2060+ GPU. It is architected from the ground up to handle the specific, demonstrated challenges of IMU-denied localization in *non-planar terrain* (as seen in Images 1-9) and *temporally-divergent* (outdated) reference maps (AC-5).
The system's core design principle is the *decoupling of unscaled relative motion from global metric scale*. The front-end estimates high-frequency, robust, but *unscaled* motion. The back-end asynchronously provides sparse, high-confidence *metric* and *geospatial* anchors. The central hub fuses these two data streams into a single, globally-optimized, metric-scale trajectory.
### **1.1 Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate (Latitude, Longitude) for the first image.
3. **Camera Intrinsics ($K$):** A pre-calibrated camera intrinsic matrix.
4. **Altitude Prior ($H_{prior}$):** The *approximate* predefined metric altitude (e.g., 900 meters). This is used as a *prior* (a hint) for optimization, *not* a hard constraint.
5. **Geospatial API Access:** Credentials for an on-demand satellite and DEM provider (e.g., Copernicus, EOSDA).
### **1.2 Streaming Outputs**
1. **Initial Pose ($Pose\\_N\\_Est$):** An *unscaled* pose estimate. This is sent immediately to the UI for real-time visualization of the UAV's *path shape* (AC-7, AC-8).
2. **Refined Pose ($Pose\\_N\\_Refined$) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose (X, Y, Z, Qx, Qy, Qz, Qw) and its corresponding [Lat, Lon, Alt] coordinate. This is sent to the user whenever the Trajectory Optimization Hub re-converges, updating all past poses (AC-1, AC-2, AC-8).
### **1.3 Component Interaction and Data Flow**
The system is architected as three parallel-processing components:
1. **Image Ingestion & Pre-processing:** This module receives the new Image_N (up to 6.2K). It creates two copies:
* Image_N_LR (Low-Resolution, e.g., 1536x1024): Dispatched *immediately* to the V-SLAM Front-End for real-time processing.
* Image_N_HR (High-Resolution, 6.2K): Stored for asynchronous use by the Geospatial Anchoring Back-End (GAB).
2. **V-SLAM Front-End (High-Frequency Thread):** This component's sole task is high-speed, *unscaled* relative pose estimation. It tracks Image_N_LR against a *local map of keyframes*. It performs local bundle adjustment to minimize drift 12 and maintains a co-visibility graph of all keyframes. It sends Relative_Unscaled_Pose estimates to the Trajectory Optimization Hub (TOH).
3. **Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread):** This is the system's "anchor." When triggered by the TOH, it fetches *on-demand* geospatial data (satellite imagery and DEMs) from an external API.3 It then performs a robust *hybrid semantic-visual* search 5 to find an *absolute, metric, global pose* for a given keyframe, robust to outdated maps (AC-5) 5 and oblique views (AC-4).14 This Absolute_Metric_Anchor is sent to the TOH.
4. **Trajectory Optimization Hub (TOH) (Central Hub):** This component manages the complete flight trajectory as a **Sim(3) pose graph** (7-DoF). It continuously fuses two distinct data streams:
* **On receiving Relative_Unscaled_Pose (T \< 5s):** It appends this pose to the graph, calculates the Pose_N_Est, and sends this *unscaled* initial result to the user (AC-7, AC-8 met).
* **On receiving Absolute_Metric_Anchor (T > 5s):** This is the critical event. It adds this as a high-confidence *global metric constraint*. This anchor creates "tension" in the graph, which the optimizer (Ceres Solver 15) resolves by finding the *single global scale factor* that best fits all V-SLAM and CVGL measurements. It then triggers a full graph re-optimization, "stretching" the entire trajectory to the correct metric scale, and sends the new Pose_N_Refined stream to the user for all affected poses (AC-1, AC-2, AC-8 refinement met).
## **2.0 Core Component: The High-Frequency V-SLAM Front-End**
This component's sole task is to robustly and accurately compute the *unscaled* 6-DoF relative motion of the UAV and build a geometrically-consistent map of keyframes. It is explicitly designed to be more robust to drift than simple frame-to-frame odometry.
### **2.1 Rationale: Keyframe-Based Monocular SLAM**
The choice of a keyframe-based V-SLAM front-end over a frame-to-frame VO is deliberate and critical for system robustness.
* **Drift Mitigation:** Frame-to-frame VO is "prone to drift accumulation due to errors introduced by each frame-to-frame motion estimation".13 A single poor match permanently corrupts all future poses.
* **Robustness:** A keyframe-based system tracks new images against a *local map* of *multiple* previous keyframes, not just Image_N-1. This provides resilience to transient failures (e.g., motion blur, occlusion).
* **Optimization:** This architecture enables "local bundle adjustment" 12, a process where a sliding window of recent keyframes is continuously re-optimized, actively minimizing error and drift *before* it can accumulate.
* **Relocalization:** This architecture possesses *innate relocalization capabilities* (see Section 6.3), which is the correct, robust solution to the "sharp turn" (AC-4) requirement.
### **2.2 Feature Matching Sub-System**
The success of the V-SLAM front-end depends entirely on high-quality feature matches, especially in the sparse, low-texture agricultural terrain seen in the provided images (e.g., Image 6, Image 7). The system requires a matcher that is robust (for sparse textures 17) and extremely fast (for AC-7).
The selected approach is **SuperPoint + LightGlue**.
* **SuperPoint:** A SOTA (State-of-the-Art) feature detector proven to find robust, repeatable keypoints in challenging, low-texture conditions 17
* **LightGlue:** A highly optimized GNN-based matcher that is the successor to SuperGlue 19
The key advantage of selecting LightGlue 19 over SuperGlue 20 is its *adaptive nature*. The query states sharp turns (AC-4) are "rather an exception." This implies \~95% of image pairs are "easy" (high-overlap, straight flight) and 5% are "hard" (low-overlap, turns). SuperGlue uses a fixed-depth GNN, spending the *same* large amount of compute on an "easy" pair as a "hard" one. LightGlue is *adaptive*.19 For an "easy" pair, it can exit its GNN early, returning a high-confidence match in a fraction of the time. This saves *enormous* computational budget on the 95% of "easy" frames, ensuring the system *always* meets the \<5s budget (AC-7) and reserving that compute for the GAB.
#### **Table 1: Analysis of State-of-the-Art Feature Matchers (For V-SLAM Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** 20 | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems. | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue.19 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.21 | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the \<5s time budget (AC-7). |
| **SuperPoint + LightGlue** 17 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.19 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy. - SOTA "in practice" choice for large-scale matching.17 | - Newer, but rapidly being adopted and proven.21 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT.22 | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, maximizing our ability to meet AC-7. |
## **3.0 Core Component: The Geospatial Anchoring Back-End (GAB)**
This component is the system's "anchor to reality." It runs asynchronously to provide the *absolute, metric-scale* constraints needed to solve the trajectory. It is an *on-demand* system that solves three distinct "domain gaps": the hardware/scale gap, the temporal gap, and the viewpoint gap.
### **3.1 On-Demand Geospatial Data Retrieval**
A "pre-computed database" for all of Eastern Ukraine is operationally unfeasible on laptop-grade hardware.1 This design is replaced by an on-demand, API-driven workflow.
* **Mechanism:** When the TOH requests a global anchor, the GAB receives a *coarse* [Lat, Lon] estimate. The GAB then performs API calls to a geospatial data provider (e.g., EOSDA 3, Copernicus 8).
* **Dual-Retrieval:** The API query requests *two* distinct products for the specified Area of Interest (AOI):
1. **Visual Tile:** A high-resolution (e.g., 30-50cm) satellite ortho-image.26
2. **Terrain Tile:** The corresponding **Digital Elevation Model (DEM)**, such as the Copernicus GLO-30 (30m resolution) or SRTM (30m).7
This "Dual-Retrieval" mechanism is the central, enabling synergy of the new architecture. The **Visual Tile** is used by the CVGL (Section 3.2) to find the *geospatial pose*. The **DEM Tile** is used by the *output module* (Section 7.1) to perform high-accuracy **Ray-DEM Intersection**, solving the final output accuracy problem.
### **3.2 Hybrid Semantic-Visual Localization**
The "temporal gap" (evidenced by burn scars in Images 1-9) and "outdated maps" (AC-5) makes a purely visual CVGL system unreliable.5 The GAB solves this using a robust, two-stage *hybrid* matching pipeline.
1. **Stage 1: Coarse Visual Retrieval (Siamese CNN).** A lightweight Siamese CNN 14 is used to find the *approximate* location of the Image_N_LR *within* the large, newly-fetched satellite tile. This acts as a "candidate generator."
2. **Stage 2: Fine-Grained Semantic-Visual Fusion.** For the top candidates, the GAB performs a *dual-channel alignment*.
* **Visual Channel (Unreliable):** It runs SuperPoint+LightGlue on high-resolution *patches* (from Image_N_HR) against the satellite tile. This match may be *weak* due to temporal gaps.5
* **Semantic Channel (Reliable):** It extracts *temporally-invariant* semantic features (e.g., road-vectors, field-boundaries, tree-cluster-polygons, lake shorelines) from *both* the UAV image (using a segmentation model) and the satellite/OpenStreetMap data.5
* **Fusion:** A RANSAC-based optimizer finds the 6-DoF pose that *best aligns* this *hybrid* set of features.
This hybrid approach is robust to the exact failure mode seen in the images. When matching Image 3 (burn scars), the *visual* LightGlue match will be poor. However, the *semantic* features (the dirt road, the tree line) are *unchanged*. The optimizer will find a high-confidence pose by *trusting the semantic alignment* over the poor visual alignment, thereby succeeding despite the "outdated map" (AC-5).
### **3.3 Solution to Viewpoint Gap: Synthetic Oblique View Training**
This component is critical for handling "sharp turns" (AC-4). The camera *will* be oblique, not nadir, during turns.
* **Problem:** The GAB's Stage 1 Siamese CNN 14 will be matching an *oblique* UAV view to a *nadir* satellite tile. This "viewpoint gap" will cause a match failure.14
* **Mechanism (Synthetic Data Generation):** The network must be trained for *viewpoint invariance*.28
1. Using the on-demand DEMs (fetched in 3.1) and satellite tiles, the system can *synthetically render* the satellite imagery from *any* roll, pitch, and altitude.
2. The Siamese network is trained on (Nadir_Tile, Synthetic_Oblique_Tile) pairs.14
* **Result:** This process teaches the network to match the *underlying ground features*, not the *perspective distortion*. It ensures the GAB can relocalize the UAV *precisely* when it is needed most: during a sharp, banking turn (AC-4) when VO tracking has been lost.
## **4.0 Core Component: The Trajectory Optimization Hub (TOH)**
This component is the system's central "brain." It runs continuously, fusing all measurements (high-frequency/unscaled V-SLAM, low-frequency/metric-scale GAB anchors) into a single, globally consistent trajectory.
### **4.1 Incremental Sim(3) Pose-Graph Optimization**
The "planar ground" SA-VO (Finding 1) is removed. This component is its replacement. The system must *discover* the global scale, not *assume* it.
* **Selected Strategy:** An incremental pose-graph optimizer using **Ceres Solver**.15
* **The Sim(3) Insight:** The V-SLAM front-end produces *unscaled* 6-DoF ($SE(3)$) relative poses. The GAB produces *metric-scale* 6-DoF ($SE(3)$) *absolute* poses. These cannot be directly combined. The graph must be optimized in **Sim(3) (7-DoF)**, which adds a *single global scale factor $s$* as an optimizable variable.
* **Mechanism (Ceres Solver):**
1. **Nodes:** Each keyframe pose (7-DoF: $X, Y, Z, Qx, Qy, Qz, s$).
2. **Edge 1 (V-SLAM):** A relative pose constraint between Keyframe_i and Keyframe_j. The error is computed in Sim(3).
3. **Edge 2 (GAB):** An *absolute* pose constraint on Keyframe_k. This constraint *fixes* Keyframe_k's pose to the *metric* GPS coordinate and *fixes its scale $s$ to 1.0*.
* **Bootstrapping Scale:** The TOH graph "bootstraps" the scale.32 The GAB's $s=1.0$ anchor creates "tension" in the graph. The Ceres optimizer 15 resolves this tension by finding the *one* global scale $s$ for all V-SLAM nodes that minimizes the total error, effectively "stretching" the entire unscaled trajectory to fit the metric anchors. This is robust to *any* terrain.34
#### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (Ceres Solver 15, g2o 35, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (GAB) data arrives (AC-8).13 - **Robust:** Can handle outliers via robust kernels.15 | - Initial estimate is *unscaled* until a GAB anchor arrives. - Can drift *if* not anchored (though V-SLAM minimizes this). | - A graph optimization library (Ceres). - A robust cost function. | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming and asynchronous refinement. |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction and trajectory. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process. |
### **4.2 Automatic Outlier Rejection (AC-3, AC-5)**
The system must handle 350m outliers (AC-3) and \<10% bad GAB matches (AC-5).
* **Mechanism (Robust Loss Functions):** A standard least-squares optimizer (like Ceres 15) would be catastrophically corrupted by a 350m error. The solution is to wrap *all* constraints in a **Robust Loss Function (e.g., HuberLoss, CauchyLoss)**.15
* **Result:** A robust loss function mathematically *down-weights* the influence of constraints with large errors. When it "sees" the 350m error (AC-3), it effectively acknowledges the measurement but *refuses* to pull the entire 3000-image trajectory to fit this one "insane" data point. It automatically and gracefully *ignores* the outlier, optimizing the 99.9% of "sane" measurements. This is the modern, robust solution to AC-3 and AC-5.
## **5.0 High-Performance Compute & Deployment**
The system must run on an RTX 2060 (AC-7) and process 6.2K images. These are opposing constraints.
### **5.1 Multi-Scale, Patch-Based Processing Pipeline**
Running deep learning models (SuperPoint, LightGlue) on a full 6.2K (26-Megapixel) image will cause a CUDA Out-of-Memory (OOM) error and be impossibly slow.
* **Mechanism (Coarse-to-Fine):**
1. **For V-SLAM (Real-time, \<5s):** The V-SLAM front-end (Section 2.0) runs *only* on the Image_N_LR (e.g., 1536x1024) copy. This is fast enough to meet the AC-7 budget.
2. **For GAB (High-Accuracy, Async):** The GAB (Section 3.0) uses the full-resolution Image_N_HR *selectively* to meet the 20m accuracy (AC-2).
* It first runs its coarse Siamese CNN 27 on the Image_N_LR.
* It then runs the SuperPoint detector on the *full 6.2K* image to find the *most confident* feature keypoints.
* It then extracts small, 256x256 *patches* from the *full-resolution* image, centered on these keypoints.
* It matches *these small, full-resolution patches* against the high-res satellite tile.
* **Result:** This hybrid method provides the fine-grained matching accuracy of the 6.2K image (needed for AC-2) without the catastrophic OOM errors or performance penalties.
### **5.2 Mandatory Deployment: NVIDIA TensorRT Acceleration**
PyTorch is a research framework. For production, its inference speed is insufficient.
* **Requirement:** The key neural networks (SuperPoint, LightGlue, Siamese CNN) *must* be converted from PyTorch into a highly-optimized **NVIDIA TensorRT engine**.
* **Research Validation:** 23 demonstrates this process for LightGlue, achieving "2x-4x speed gains over compiled PyTorch." 22 and 21 provide open-source repositories for SuperPoint+LightGlue conversion to ONNX and TensorRT.
* **Result:** This is not an "optional" optimization. It is a *mandatory* deployment step. This conversion (which applies layer fusion, graph optimization, and FP16 precision) is what makes achieving the \<5s (AC-7) performance *possible* on the specified RTX 2060 hardware.36
## **6.0 System Robustness: Failure Mode Escalation Logic**
This logic defines the system's behavior during real-world failures, ensuring it meets criteria AC-3, AC-4, AC-6, and AC-9.
### **6.1 Stage 1: Normal Operation (Tracking)**
* **Condition:** V-SLAM front-end (Section 2.0) is healthy.
* **Logic:**
1. V-SLAM successfully tracks Image_N_LR against its local keyframe map.
2. A new Relative_Unscaled_Pose is sent to the TOH.
3. TOH sends Pose_N_Est (unscaled) to the user (\<5s).
4. If Image_N is selected as a new keyframe, the GAB (Section 3.0) is *queued* to find an Absolute_Metric_Anchor for it, which will trigger a Pose_N_Refined update later.
### **6.2 Stage 2: Transient VO Failure (Outlier Rejection)**
* **Condition:** Image_N is unusable (e.g., severe blur, sun-glare, 350m outlier per AC-3).
* **Logic (Frame Skipping):**
1. V-SLAM front-end fails to track Image_N_LR against the local map.
2. The system *discards* Image_N (marking it as a rejected outlier, AC-5).
3. When Image_N+1 arrives, the V-SLAM front-end attempts to track it against the *same* local keyframe map (from Image_N-1).
4. **If successful:** Tracking resumes. Image_N is officially an outlier. The system "correctly continues the work" (AC-3 met).
5. **If fails:** The system repeats for Image_N+2, N+3. If this fails for \~5 consecutive frames, it escalates to Stage 3.
### **6.3 Stage 3: Persistent VO Failure (Relocalization)**
* **Condition:** Tracking is lost for multiple frames. This is the "sharp turn" (AC-4) or "low overlap" (AC-4) scenario.
* **Logic (Keyframe-Based Relocalization):**
1. The V-SLAM front-end declares "Tracking Lost."
2. **Critically:** It does *not* create a "new map chunk."
3. Instead, it enters **Relocalization Mode**. For every new Image_N+k, it extracts features (SuperPoint) and queries the *entire* existing database of past keyframes for a match.
* **Resolution:** The UAV completes its sharp turn. Image_N+5 now has high overlap with Image_N-10 (from *before* the turn).
1. The relocalization query finds a strong match.
2. The V-SLAM front-end computes the 6-DoF pose of Image_N+5 relative to the *existing map*.
3. Tracking is *resumed* seamlessly. The system "correctly continues the work" (AC-4 met). This is vastly more robust than the previous "map-merging" logic.
### **6.4 Stage 4: Catastrophic Failure (User Intervention)**
* **Condition:** The system is in Stage 3 (Lost), but *also*, the **GAB (Section 3.0) has failed** to find *any* global anchors for a prolonged period (e.g., 20% of the route). This is the "absolutely incapable" scenario (AC-6), (e.g., heavy fog *and* over a featureless ocean).
* **Logic:**
1. The system has an *unscaled* trajectory, and *zero* idea where it is in the world.
2. The TOH triggers the AC-6 flag.
* **Resolution (User-Aided Prior):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The user clicks *one point* on a map.
3. This [Lat, Lon] is *not* taken as ground truth. It is fed to the **GAB (Section 3.1)** as a *strong prior* for its on-demand API query.
4. This narrows the GAB's search area from "all of Ukraine" to "a 5km radius." This *guarantees* the GAB's Dual-Retrieval (Section 3.1) will fetch the *correct* satellite and DEM tiles, allowing the Hybrid Matcher (Section 3.2) to find a high-confidence Absolute_Metric_Anchor, which in turn re-scales (Section 4.1) and relocalizes the entire trajectory.
## **7.0 Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated, specifically solving the "planar ground" output flaw, and how the system's compliance with all 10 ACs will be validated.
### **7.1 High-Accuracy Object Geolocalization via Ray-DEM Intersection**
The "Ray-Plane Intersection" method is inaccurate for non-planar terrain 37 and is replaced with a high-accuracy ray-tracing method. This is the correct method for geolocating an object on the *non-planar* terrain visible in Images 1-9.
* **Inputs:**
1. User clicks pixel coordinate $(u,v)$ on Image_N.
2. System retrieves the *final, refined, metric* 7-DoF pose $P = (R, T, s)$ for Image_N from the TOH.
3. The system uses the known camera intrinsic matrix $K$.
4. System retrieves the specific **30m DEM tile** 8 that was fetched by the GAB (Section 3.1) for this region of the map. This DEM is a 3D terrain mesh.
* **Algorithm (Ray-DEM Intersection):**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} = K^{-1} \\cdot [u, v, 1]^T$.
2. **Transform Ray:** This ray direction $d_{cam}$ and origin (0,0,0) are transformed into the *global, metric* coordinate system using the pose $P$. This yields a ray originating at $T$ and traveling in direction $R \\cdot d_{cam}$.
3. **Intersect:** The system performs a numerical *ray-mesh intersection* 39 to find the 3D point $(X, Y, Z)$ where this global ray *intersects the 3D terrain mesh* of the DEM.
4. **Result:** This 3D intersection point $(X, Y, Z)$ is the *metric* world coordinate of the object *on the actual terrain*.
5. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate.
This method correctly accounts for terrain. A pixel aimed at the top of a hill will intersect the DEM at a high Z-value. A pixel aimed at the ravine (Image 1) will intersect at a low Z-value. This is the *only* method that can reliably meet the 20m accuracy (AC-2) for object localization.
### **7.2 Rigorous Validation Methodology**
A comprehensive test plan is required. The foundation is a **Ground-Truth Test Harness** using the provided coordinates.csv.42
* **Test Harness:**
1. **Ground-Truth Data:** The file coordinates.csv 42 provides ground-truth [Lat, Lon] for 60 images (e.g., AD000001.jpg...AD000060.jpg).
2. **Test Datasets:**
* Test_Baseline_60 42: The 60 images and their coordinates.
* Test_Outlier_350m (AC-3): Test_Baseline_60 with a single, unrelated image inserted at frame 30.
* Test_Sharp_Turn_5pct (AC-4): A sequence where frames 20-24 are manually deleted, simulating a \<5% overlap jump.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* **Run:** Execute GEORTEX-R on Test_Baseline_60, providing AD000001.jpg's coordinate (48.275292, 37.385220) as the Start Coordinate 42
* **Script:** A validation script will compute the Haversine distance error between the *system's refined GPS output* for each image (2-60) and the *ground-truth GPS* from coordinates.csv.
* **ASSERT** (count(errors \< 50m) / 60) >= 0.80 **(AC-1 Met)**
* **ASSERT** (count(errors \< 20m) / 60) >= 0.60 **(AC-2 Met)**
* **ASSERT** (count(un-localized_images) / 60) \< 0.10 **(AC-5 Met)**
* **ASSERT** (count(localized_images) / 60) > 0.95 **(AC-9 Met)**
* **Test_MRE (AC-10):**
* **Run:** After Test_Baseline_60 completes.
* **ASSERT** TOH.final_Mean_Reprojection_Error \< 1.0 **(AC-10 Met)**
* **Test_Performance (AC-7, AC-8):**
* **Run:** Execute on a 1500-image sequence on the minimum-spec RTX 2060.
* **Log:** Log timestamps for "Image In" -> "Initial Pose Out".
* **ASSERT** average_time \< 5.0s **(AC-7 Met)**
* **Log:** Log the output stream.
* **ASSERT** >80% of images receive *two* poses: an "Initial" and a "Refined" **(AC-8 Met)**
* **Test_Robustness (AC-3, AC-4):**
* **Run:** Execute Test_Outlier_350m.
* **ASSERT** System logs "Stage 2: Discarding Outlier" and the final trajectory error for Image_31 is \< 50m **(AC-3 Met)**.
* **Run:** Execute Test_Sharp_Turn_5pct.
* **ASSERT** System logs "Stage 3: Tracking Lost" and "Relocalization Succeeded," and the final trajectory is complete and accurate **(AC-4 Met)**.
_docs/01_solution/04_solution_draft.md
-327
View File
@@ -1,327 +0,0 @@
## **The ATLAS-GEOFUSE System Architecture**
Multi-component architecture designed for high-performance, real-time geolocalization in IMU-denied, high-drift environments. Its architecture is explicitly designed around **pre-flight data caching** and **multi-map robustness**.
### **2.1 Core Design Principles**
1. **Pre-Flight Caching:** To meet the <5s (AC-7) real-time requirement, all network latency must be eliminated. The system mandates a "Pre-Flight" step (Section 3.0) where all geospatial data (satellite tiles, DEMs, vector data) for the Area of Interest (AOI) is downloaded from a viable open-source provider (e.g., Copernicus 6) and stored in a local database on the processing laptop. All real-time queries are made against this local cache.
2. **Decoupled Multi-Map SLAM:** The system separates *relative* motion from *absolute* scale. A Visual SLAM (V-SLAM) "Atlas" Front-End (Section 4.0) computes high-frequency, robust, but *unscaled* relative motion. A Local Geospatial Anchoring Back-End (GAB) (Section 5.0) provides sparse, high-confidence, *absolute metric* anchors by querying the local cache. A Trajectory Optimization Hub (TOH) (Section 6.0) fuses these two streams in a Sim(3) pose-graph to solve for the global 7-DoF trajectory (pose + scale).
3. **Multi-Map Robustness (Atlas):** To solve the "sharp turn" (AC-4) and "tracking loss" (AC-6) requirements, the V-SLAM front-end is based on an "Atlas" architecture.14 Tracking loss initiates a *new, independent map fragment*.13 The TOH is responsible for anchoring and merging *all* fragments geodetically 19 into a single, globally-consistent trajectory.
### **2.2 Component Interaction and Data Flow**
* **Component 1: Pre-Flight Caching Module (PCM) (Offline)**
* *Input:* User-defined Area of Interest (AOI) (e.g., a KML polygon).
* *Action:* Queries Copernicus 6 and OpenStreetMap APIs. Downloads and builds a local geospatial database (GeoPackage/SpatiaLite) containing satellite tiles, DEM tiles, and road/river vectors for the AOI.
* *Output:* A single, self-contained **Local Geo-Database file**.
* **Component 2: Image Ingestion & Pre-processing (Real-time)**
* *Input:* Image_N (up to 6.2K), Camera Intrinsics ($K$).
* *Action:* Creates two copies:
* **Image_N_LR** (Low-Resolution, e.g., 1536x1024): Dispatched *immediately* to the V-SLAM Front-End.
* **Image_N_HR** (High-Resolution, 6.2K): Stored for asynchronous use by the GAB.
* **Component 3: V-SLAM "Atlas" Front-End (High-Frequency Thread)**
* *Input:* Image_N_LR.
* *Action:* Tracks Image_N_LR against its *active map fragment*. Manages keyframes, local bundle adjustment 38, and the co-visibility graph. If tracking is lost (e.g., AC-4 sharp turn), it initializes a *new map fragment* 14 and continues tracking.
* *Output:* **Relative_Unscaled_Pose** and **Local_Point_Cloud** data, sent to the TOH.
* **Component 4: Local Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread)**
* *Input:* A keyframe (Image_N_HR) and its *unscaled* pose, triggered by the TOH.
* *Action:* Performs a visual-only, coarse-to-fine search 34 against the *Local Geo-Database*.
* *Output:* An **Absolute_Metric_Anchor** (a high-confidence [Lat, Lon, Alt] pose) for that keyframe, sent to the TOH.
* **Component 5: Trajectory Optimization Hub (TOH) (Central Hub Thread)**
* *Input:* (1) High-frequency Relative_Unscaled_Pose stream. (2) Low-frequency Absolute_Metric_Anchor stream.
* *Action:* Manages the complete flight trajectory as a **Sim(3) pose graph** 39 using Ceres Solver.19 Continuously fuses all data.
* *Output 1 (Real-time):* **Pose_N_Est** (unscaled) sent to UI (meets AC-7, AC-8).
* *Output 2 (Refined):* **Pose_N_Refined** (metric-scale, globally-optimized) sent to UI (meets AC-1, AC-2, AC-8).
### **2.3 System Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate (Latitude, Longitude).
3. **Camera Intrinsics ($K$):** Pre-calibrated camera intrinsic matrix.
4. **Local Geo-Database File:** The single file generated by the Pre-Flight Caching Module (Section 3.0).
### **2.4 Streaming Outputs (Meets AC-7, AC-8)**
1. **Initial Pose ($Pose_N^{Est}$):** An *unscaled* pose estimate. This is sent immediately (<5s, AC-7) to the UI for real-time visualization of the UAV's *path shape*.
2. **Refined Pose ($Pose_N^{Refined}$) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose (X, Y, Z, Qx, Qy, Qz, Qw) and its corresponding [Lat, Lon, Alt] coordinate. This is sent to the user whenever the TOH re-converges (e.g., after a new GAB anchor or map-merge), updating all past poses (AC-1, AC-2, AC-8 refinement met).
## **3.0 Pre-Flight Component: The Geospatial Caching Module (PCM)**
This component is a new, mandatory, pre-flight utility that solves the fatal flaws (Section 1.1, 1.2) of the GEORTEX-R design. It eliminates all real-time network latency (AC-7) and all ToS violations (AC-5), ensuring the project is both performant and legally viable.
### **3.1 Defining the Area of Interest (AOI)**
The system is designed for long-range flights. Given 3000 photos at 100m intervals, the maximum linear track is 300km. The user must provide a coarse "bounding box" or polygon (e.g., KML/GeoJSON format) of the intended flight area. The PCM will automatically add a generous buffer (e.g., 20km) to this AOI to account for navigational drift and ensure all necessary reference data is captured.
### **3.2 Legal & Viable Data Sources (Copernicus & OpenStreetMap)**
As established in 1.1, the system *must* use open-data providers. The PCM is architected to use the following:
1. **Visual/Terrain Data (Primary):** The **Copernicus Data Space Ecosystem** 6 is the primary source. The PCM will use the Copernicus Processing and Catalogue APIs 6 to query, process, and download two key products for the buffered AOI:
* **Sentinel-2 Satellite Imagery:** High-resolution (10m) visual tiles.
* **Copernicus GLO-30 DEM:** A 30m-resolution Digital Elevation Model.7 This DEM is *not* used for high-accuracy object localization (see 1.4), but as a coarse altitude *prior* for the TOH and for the critical dynamic-warping step (Section 5.3).
2. **Semantic Data (Secondary):** OpenStreetMap (OSM) data 40 for the AOI will be downloaded. This provides temporally-invariant vector data (roads, rivers, building footprints) which can be used as a secondary, optional verification layer for the GAB, especially in cases of extreme temporal divergence (e.g., new construction).42
### **3.3 Building the Local Geo-Database**
The PCM utility will process all downloaded data into a single, efficient, compressed file. A modern GeoPackage or SpatiaLite database is the ideal format. This database will contain the satellite tiles, DEM tiles, and vector features, all indexed by a common spatial grid (e.g., UTM).
This single file is then loaded by the main ATLAS-GEOFUSE application at runtime. The GAB's (Section 5.0) "API calls" are thus transformed from high-latency, unreliable HTTP requests 9 into high-speed, zero-latency local SQL queries, guaranteeing that data I/O is never the bottleneck for meeting the AC-7 performance requirement.
## **4.0 Core Component: The Multi-Map V-SLAM "Atlas" Front-End**
This component's sole task is to robustly and accurately compute the *unscaled* 6-DoF relative motion of the UAV and build a geometrically-consistent map of keyframes. It is explicitly designed to be more robust than simple frame-to-frame odometry and to handle catastrophic tracking loss (AC-4) gracefully.
### **4.1 Rationale: ORB-SLAM3 "Atlas" Architecture**
The system will implement a V-SLAM front-end based on the "Atlas" multi-map paradigm, as seen in SOTA systems like ORB-SLAM3.14 This is the industry-standard solution for robust, long-term navigation in environments where tracking loss is possible.13
The mechanism is as follows:
1. The system initializes and begins tracking on **Map_Fragment_0**, using the known start GPS as a metadata tag.
2. It tracks all new frames (Image_N_LR) against this active map.
3. **If tracking is lost** (e.g., a sharp turn (AC-4) or a persistent 350m outlier (AC-3)):
* The "Atlas" architecture does not fail. It declares Map_Fragment_0 "inactive," stores it, and *immediately initializes* **Map_Fragment_1** from the current frame.14
* Tracking *resumes instantly* on this new map fragment, ensuring the system "correctly continues the work" (AC-4).
This architecture converts the "sharp turn" failure case into a *standard operating procedure*. The system never "fails"; it simply fragments. The burden of stitching these fragments together is correctly moved from the V-SLAM front-end (which has no global context) to the TOH (Section 6.0), which *can* solve it using global-metric anchors.
### **4.2 Feature Matching Sub-System: SuperPoint + LightGlue**
The V-SLAM front-end's success depends entirely on high-quality feature matches, especially in the sparse, low-texture agricultural terrain seen in the user's images. The selected approach is **SuperPoint + LightGlue**.
* **SuperPoint:** A SOTA feature detector proven to find robust, repeatable keypoints in challenging, low-texture conditions.43
* **LightGlue:** A highly optimized GNN-based matcher that is the successor to SuperGlue.44
The choice of LightGlue over SuperGlue is a deliberate performance optimization. LightGlue is *adaptive*.46 The user query states sharp turns (AC-4) are "rather an exception." This implies \~95% of image pairs are "easy" (high-overlap, straight flight) and 5% are "hard" (low-overlap, turns). LightGlue's adaptive-depth GNN exits early on "easy" pairs, returning a high-confidence match in a fraction of the time. This saves *enormous* computational budget on the 95% of normal frames, ensuring the system *always* meets the <5s budget (AC-7) and reserving that compute for the GAB and TOH. This component will run on **Image_N_LR** (low-res) to guarantee performance, and will be accelerated via TensorRT (Section 7.0).
### **4.3 Keyframe Management and Local 3D Cloud**
The front-end will maintain a co-visibility graph of keyframes for its *active map fragment*. It will perform local Bundle Adjustment 38 continuously over a sliding window of recent keyframes to minimize drift *within* that fragment.
Crucially, it will triangulate features to create a **local, high-density 3D point cloud** for its map fragment.28 This point cloud is essential for two reasons:
1. It provides robust tracking (tracking against a 3D map, not just a 2D frame).
2. It serves as the **high-accuracy source** for the object localization output (Section 9.1), as established in 1.4, allowing the system to bypass the high-error external DEM.
#### **Table 1: Analysis of State-of-the-Art Feature Matchers (For V-SLAM Front-End)**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **SuperPoint + SuperGlue** | - SOTA robustness in low-texture, high-blur conditions. - GNN reasons about 3D scene context. - Proven in real-time SLAM systems. | - Computationally heavy (fixed-depth GNN). - Slower than LightGlue. | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT. | **Good.** A solid, baseline choice. Meets robustness needs but will heavily tax the <5s time budget (AC-7). |
| **SuperPoint + LightGlue** 44 | - **Adaptive Depth:** Faster on "easy" pairs, more accurate on "hard" pairs.46 - **Faster & Lighter:** Outperforms SuperGlue on speed and accuracy. - SOTA "in practice" choice for large-scale matching. | - Newer, but rapidly being adopted and proven.48 | - NVIDIA GPU (RTX 2060+). - PyTorch or TensorRT. | **Excellent (Selected).** The adaptive nature is *perfect* for this problem. It saves compute on the 95% of easy (straight) frames, maximizing our ability to meet AC-7. |
## **5.0 Core Component: The Local Geospatial Anchoring Back-End (GAB)**
This asynchronous component is the system's "anchor to reality." Its sole purpose is to find a high-confidence, *absolute-metric* pose for a given V-SLAM keyframe by matching it against the **local, pre-cached geo-database** (from Section 3.0). This component is a full replacement for the high-risk, high-latency GAB from the GEORTEX-R draft (see 1.2, 1.5).
### **5.1 Rationale: Local-First Query vs. On-Demand API**
As established in 1.2, all queries are made to the local SSD. This guarantees zero-latency I/O, which is a hard requirement for a real-time system, as external network latency is unacceptably high and variable.9 The GAB itself runs asynchronously and can take longer than 5s (e.g., 10-15s), but it must not be *blocked* by network I/O, which would stall the entire processing pipeline.
### **5.2 SOTA Visual-Only Coarse-to-Fine Localization**
This component implements a state-of-the-art, two-stage *visual-only* pipeline, which is lower-risk and more performant (see 1.5) than the GEORTEX-R's semantic-hybrid model. This approach is well-supported by SOTA research in aerial localization.34
1. **Stage 1 (Coarse): Global Descriptor Retrieval.**
* *Action:* When the TOH requests an anchor for Keyframe_k, the GAB first computes a *global descriptor* (a compact vector representation) for the *nadir-warped* (see 5.3) low-resolution Image_k_LR.
* *Technology:* A SOTA Visual Place Recognition (VPR) model like **SALAD** 49, **TransVLAD** 50, or **NetVLAD** 33 will be used. These are designed for this "image retrieval" task.45
* *Result:* This descriptor is used to perform a fast FAISS/vector search against the descriptors of the *local satellite tiles* (which were pre-computed and stored in the Geo-Database). This returns the Top-K (e.g., K=5) most likely satellite tiles in milliseconds.
2. **Stage 2 (Fine): Local Feature Matching.**
* *Action:* The system runs **SuperPoint+LightGlue** 43 to find pixel-level correspondences.
* *Performance:* This is *not* run on the *full* UAV image against the *full* satellite map. It is run *only* between high-resolution patches (from **Image_k_HR**) and the **Top-K satellite tiles** identified in Stage 1.
* *Result:* This produces a set of 2D-2D (image-to-map) feature matches. A PnP/RANSAC solver then computes a high-confidence 6-DoF pose. This pose is the **Absolute_Metric_Anchor** that is sent to the TOH.
### **5.3 Solving the Viewpoint Gap: Dynamic Feature Warping**
The GAB must solve the "viewpoint gap" 33: the UAV image is oblique (due to roll/pitch), while the satellite tiles are nadir (top-down).
The GEORTEX-R draft proposed a complex, high-risk deep learning solution. The ATLAS-GEOFUSE solution is far more elegant and requires zero R\&D:
1. The V-SLAM Front-End (Section 4.0) already *knows* the camera's *relative* 6-DoF pose, including its **roll and pitch** orientation relative to the *local map's ground plane*.
2. The *Local Geo-Database* (Section 3.0) contains a 30m-resolution DEM for the AOI.
3. When the GAB processes Keyframe_k, it *first* performs a **dynamic homography warp**. It projects the V-SLAM ground plane onto the coarse DEM, and then uses the known camera roll/pitch to calculate the perspective transform (homography) needed to *un-distort* the oblique UAV image into a synthetic *nadir-view*.
This *nadir-warped* UAV image is then used in the Coarse-to-Fine pipeline (5.2). It will now match the *nadir* satellite tiles with extremely high-fidelity. This method *eliminates* the viewpoint gap *without* training any new neural networks, leveraging the inherent synergy between the V-SLAM component and the GAB's pre-cached DEM.
## **6.0 Core Component: The Multi-Map Trajectory Optimization Hub (TOH)**
This component is the system's central "brain." It runs continuously, fusing all measurements (high-frequency/unscaled V-SLAM, low-frequency/metric-scale GAB anchors) from *all map fragments* into a single, globally consistent trajectory.
### **6.1 Incremental Sim(3) Pose-Graph Optimization**
The central challenge of monocular, IMU-denied SLAM is scale-drift. The V-SLAM front-end produces *unscaled* 6-DoF ($SE(3)$) relative poses.37 The GAB produces *metric-scale* 6-DoF ($SE(3)$) *absolute* poses. These cannot be directly combined.
The solution is that the graph *must* be optimized in **Sim(3) (7-DoF)**.39 This adds a *single global scale factor $s$* as an optimizable variable to each V-SLAM map fragment. The TOH will maintain a pose-graph using **Ceres Solver** 19, a SOTA optimization library.
The graph is constructed as follows:
1. **Nodes:** Each keyframe pose (7-DoF: $X, Y, Z, Qx, Qy, Qz, s$).
2. **Edge 1 (V-SLAM):** A relative pose constraint between Keyframe_i and Keyframe_j *within the same map fragment*. The error is computed in Sim(3).29
3. **Edge 2 (GAB):** An *absolute* pose constraint on Keyframe_k. This constraint *fixes* Keyframe_k's pose to the *metric* GPS coordinate from the GAB anchor and *fixes its scale $s$ to 1.0*.
The GAB's $s=1.0$ anchor creates "tension" in the graph. The Ceres optimizer 20 resolves this tension by finding the *one* global scale $s$ for all *other* V-SLAM nodes in that fragment that minimizes the total error. This effectively "stretches" or "shrinks" the entire unscaled V-SLAM fragment to fit the metric anchors, which is the core of monocular SLAM scale-drift correction.29
### **6.2 Geodetic Map-Merging via Absolute Anchors**
This is the robust solution to the "sharp turn" (AC-4) problem, replacing the flawed "relocalization" model from the original draft.
* **Scenario:** The UAV makes a sharp turn (AC-4). The V-SLAM front-end *loses tracking* on Map_Fragment_0 and *creates* Map_Fragment_1 (per Section 4.1). The TOH's pose graph now contains *two disconnected components*.
* **Mechanism (Geodetic Merging):**
1. The GAB (Section 5.0) is *queued* to find anchors for keyframes in *both* fragments.
2. The GAB returns Anchor_A for Keyframe_10 (in Map_Fragment_0) with GPS [Lat_A, Lon_A].
3. The GAB returns Anchor_B for Keyframe_50 (in Map_Fragment_1) with GPS ``.
4. The TOH adds *both* of these as absolute, metric constraints (Edge 2) to the global pose-graph.
* The graph optimizer 20 now has all the information it needs. It will solve for the 7-DoF pose of *both fragments*, placing them in their correct, globally-consistent metric positions. The two fragments are *merged geodetically* (i.e., by their global coordinates) even if they *never* visually overlap. This is a vastly more robust and modern solution than simple visual loop closure.19
### **6.3 Automatic Outlier Rejection (AC-3, AC-5)**
The system must be robust to 350m outliers (AC-3) and <10% bad GAB matches (AC-5). A standard least-squares optimizer (like Ceres 20) would be catastrophically corrupted by a 350m error.
This is a solved problem in modern graph optimization.19 The solution is to wrap *all* constraints (V-SLAM and GAB) in a **Robust Loss Function (e.g., HuberLoss, CauchyLoss)** within Ceres Solver.
A robust loss function mathematically *down-weights* the influence of constraints with large errors (high residuals). When the TOH "sees" the 350m error from a V-SLAM relative pose (AC-3) or a bad GAB anchor (AC-5), the robust loss function effectively acknowledges the measurement but *refuses* to pull the entire 3000-image trajectory to fit this one "insane" data point. It automatically and gracefully *ignores* the outlier, optimizing the 99.9% of "sane" measurements, thus meeting AC-3 and AC-5.
### **Table 2: Analysis of Trajectory Optimization Strategies**
| Approach (Tools/Library) | Advantages | Limitations | Requirements | Fitness for Problem Component |
| :---- | :---- | :---- | :---- | :---- |
| **Incremental SLAM (Pose-Graph Optimization)** (Ceres Solver 19, g2o, GTSAM) | - **Real-time / Online:** Provides immediate pose estimates (AC-7). - **Supports Refinement:** Explicitly designed to refine past poses when new "loop closure" (GAB) data arrives (AC-8). - **Robust:** Can handle outliers via robust kernels.19 | - Initial estimate is *unscaled* until a GAB anchor arrives. - Can drift *if* not anchored. | - A graph optimization library (Ceres). - A robust cost function (Huber). | **Excellent (Selected).** This is the *only* architecture that satisfies all user requirements for real-time streaming (AC-7) and asynchronous refinement (AC-8). |
| **Batch Structure from Motion (Global Bundle Adjustment)** (COLMAP, Agisoft Metashape) | - **Globally Optimal Accuracy:** Produces the most accurate possible 3D reconstruction. | - **Offline:** Cannot run in real-time or stream results. - High computational cost (minutes to hours). - Fails AC-7 and AC-8 completely. | - All images must be available before processing starts. - High RAM and CPU. | **Good (as an *Optional* Post-Processing Step).** Unsuitable as the primary online system, but could be offered as an optional, high-accuracy "Finalize Trajectory" batch process. |
## **7.0 High-Performance Compute & Deployment**
The system must run on an RTX 2060 (AC-7) while processing 6.2K images. These are opposing constraints that require a deliberate compute strategy to balance speed and accuracy.
### **7.1 Multi-Scale, Coarse-to-Fine Processing Pipeline**
The system must balance the conflicting demands of real-time speed (AC-7) and high accuracy (AC-2). This is achieved by running different components at different resolutions.
* **V-SLAM Front-End (Real-time, <5s):** This component (Section 4.0) runs *only* on the **Image_N_LR** (e.g., 1536x1024) copy. This is fast enough to meet the AC-7 budget.46
* **GAB (Asynchronous, High-Accuracy):** This component (Section 5.0) uses the full-resolution **Image_N_HR** *selectively* to meet the 20m accuracy (AC-2).
1. Stage 1 (Coarse) runs on the low-res, nadir-warped image.
2. Stage 2 (Fine) runs SuperPoint on the *full 6.2K* image to find the *most confident* keypoints. It then extracts small, 256x256 *patches* from the *full-resolution* image, centered on these keypoints.
3. It matches *these small, full-resolution patches* against the high-res satellite tile.
This hybrid, multi-scale method provides the fine-grained matching accuracy of the 6.2K image (needed for AC-2) without the catastrophic CUDA Out-of-Memory errors (an RTX 2060 has only 6GB VRAM 30) or performance penalties that full-resolution processing would entail.
### **7.2 Mandatory Deployment: NVIDIA TensorRT Acceleration**
The deep learning models (SuperPoint, LightGlue, NetVLAD) will be too slow in their native PyTorch framework to meet AC-7 on an RTX 2060.
This is not an "optional" optimization; it is a *mandatory* deployment step. The key neural networks *must* be converted from PyTorch into a highly-optimized **NVIDIA TensorRT engine**.
Research *specifically* on accelerating LightGlue with TensorRT shows **"2x-4x speed gains over compiled PyTorch"**.48 Other benchmarks confirm TensorRT provides 30-70% speedups for deep learning inference.52 This conversion (which applies layer fusion, graph optimization, and FP16/INT8 precision) is what makes achieving the <5s (AC-7) performance *possible* on the specified RTX 2060 hardware.
## **8.0 System Robustness: Failure Mode Escalation Logic**
This logic defines the system's behavior during real-world failures, ensuring it meets criteria AC-3, AC-4, AC-6, and AC-9, and is built upon the new "Atlas" multi-map architecture.
### **8.1 Stage 1: Normal Operation (Tracking)**
* **Condition:** V-SLAM front-end (Section 4.0) is healthy.
* **Logic:**
1. V-SLAM successfully tracks Image_N_LR against its *active map fragment*.
2. A new **Relative_Unscaled_Pose** is sent to the TOH (Section 6.0).
3. TOH sends **Pose_N_Est** (unscaled) to the user (AC-7, AC-8 met).
4. If Image_N is selected as a keyframe, the GAB (Section 5.0) is *queued* to find an anchor for it, which will trigger a **Pose_N_Refined** update later.
### **8.2 Stage 2: Transient VO Failure (Outlier Rejection)**
* **Condition:** Image_N is unusable (e.g., severe blur, sun-glare, or the 350m outlier from AC-3).
* **Logic (Frame Skipping):**
1. V-SLAM front-end fails to track Image_N_LR against the active map.
2. The system *discards* Image_N (marking it as a rejected outlier, AC-5).
3. When Image_N+1 arrives, the V-SLAM front-end attempts to track it against the *same* local keyframe map (from Image_N-1).
4. **If successful:** Tracking resumes. Image_N is officially an outlier. The system "correctly continues the work" (AC-3 met).
5. **If fails:** The system repeats for Image_N+2, N+3. If this fails for \~5 consecutive frames, it escalates to Stage 3.
### **8.3 Stage 3: Persistent VO Failure (New Map Initialization)**
* **Condition:** Tracking is lost for multiple frames. This is the **"sharp turn" (AC-4)** or "low overlap" (AC-4) scenario.
* **Logic (Atlas Multi-Map):**
1. The V-SLAM front-end (Section 4.0) declares "Tracking Lost."
2. It marks the current Map_Fragment_k as "inactive".13
3. It *immediately* initializes a **new** Map_Fragment_k+1 using the current frame (Image_N+5).
4. **Tracking resumes instantly** on this new, unscaled, un-anchored map fragment.
5. This "registering" of a new map ensures the system "correctly continues the work" (AC-4 met) and maintains the >95% registration rate (AC-9) by not counting this as a failure.
### **8.4 Stage 4: Map-Merging & Global Relocalization (GAB-Assisted)**
* **Condition:** The system is now tracking on Map_Fragment_k+1, while Map_Fragment_k is inactive. The TOH pose-graph (Section 6.0) is disconnected.
* **Logic (Geodetic Merging):**
1. The TOH queues the GAB (Section 5.0) to find anchors for *both* map fragments.
2. The GAB finds anchors for keyframes in *both* fragments.
3. The TOH (Section 6.2) receives these metric anchors, adds them to the graph, and the Ceres optimizer 20 *finds the global 7-DoF pose for both fragments*, merging them into a single, metrically-consistent trajectory.
### **8.5 Stage 5: Catastrophic Failure (User Intervention)**
* **Condition:** The system is in Stage 3 (Lost), *and* the GAB (Section 5.0) has *also* failed to find *any* global anchors for a new Map_Fragment_k+1 for a prolonged period (e.g., 20% of the route). This is the "absolutely incapable" scenario (AC-6), (e.g., flying over a large, featureless body of water or dense, uniform fog).
* **Logic:**
1. The system has an *unscaled, un-anchored* map fragment (Map_Fragment_k+1) and *zero* idea where it is in the world.
2. The TOH triggers the AC-6 flag.
* **Resolution (User-Aided Prior):**
1. The UI prompts the user: "Tracking lost. Please provide a coarse location for the *current* image."
2. The user clicks *one point* on a map.
3. This [Lat, Lon] is *not* taken as ground truth. It is fed to the **GAB (Section 5.0)** as a *strong spatial prior* for its *local database query* (Section 5.2).
4. This narrows the GAB's Stage 1 search area from "the entire AOI" to "a 5km radius around the user's click." This *guarantees* the GAB will find the correct satellite tile, find a high-confidence **Absolute_Metric_Anchor**, and allow the TOH (Stage 4) to re-scale 29 and geodetically-merge 20 this lost fragment, re-localizing the entire trajectory.
## **9.0 High-Accuracy Output Generation and Validation Strategy**
This section details how the final user-facing outputs are generated, specifically replacing the flawed "Ray-DEM" method (see 1.4) with a high-accuracy "Ray-Cloud" method to meet the 20m accuracy (AC-2).
### **9.1 High-Accuracy Object Geolocalization via Ray-Cloud Intersection**
As established in 1.4, using an external 30m DEM 21 for object localization introduces uncontrollable errors (up to 4m+22) that make meeting the 20m (AC-2) accuracy goal impossible. The system *must* use its *own*, internally-generated 3D map, which is locally far more accurate.25
* **Inputs:**
1. User clicks pixel coordinate $(u,v)$ on Image_N.
2. The system retrieves the **final, refined, metric 7-DoF Sim(3) pose** $P_{sim(3)} = (s, R, T)$ for the *map fragment* that Image_N belongs to. This transform $P_{sim(3)}$ maps the *local V-SLAM coordinate system* to the *global metric coordinate system*.
3. The system retrieves the *local, unscaled* **V-SLAM 3D point cloud** ($P_{local_cloud}$) generated by the Front-End (Section 4.3).
4. The known camera intrinsic matrix $K$.
* **Algorithm (Ray-Cloud Intersection):**
1. **Un-project Pixel:** The 2D pixel $(u,v)$ is un-projected into a 3D ray *direction* vector $d_{cam}$ in the camera's local coordinate system: $d_{cam} = K^{-1} \\cdot [u, v, 1]^T$.
2. **Transform Ray (Local):** This ray is transformed using the *local V-SLAM pose* of Image_N to get a ray in the *local map fragment's* coordinate system.
3. **Intersect (Local):** The system performs a numerical *ray-mesh intersection* (or nearest-neighbor search) to find the 3D point $P_{local}$ where this local ray *intersects the local V-SLAM point cloud* ($P_{local_cloud}$).25 This $P_{local}$ is *highly accurate* relative to the V-SLAM map.26
4. **Transform (Global):** This local 3D point $P_{local}$ is now transformed to the global, metric coordinate system using the 7-DoF Sim(3) transform from the TOH: $P_{metric} = s \\cdot (R \\cdot P_{local}) + T$.
5. **Result:** This 3D intersection point $P_{metric}$ is the *metric* world coordinate of the object.
6. **Convert:** This $(X, Y, Z)$ world coordinate is converted to a [Latitude, Longitude, Altitude] GPS coordinate.55
This method correctly isolates the error. The object's accuracy is now *only* dependent on the V-SLAM's geometric fidelity (AC-10 MRE < 1.0px) and the GAB's global anchoring (AC-1, AC-2). It *completely eliminates* the external 30m DEM error 22 from this critical, high-accuracy calculation.
### **9.2 Rigorous Validation Methodology**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** (e.g., using the provided coordinates.csv data).
* **Test Harness:**
1. **Ground-Truth Data:** coordinates.csv provides ground-truth [Lat, Lon] for a set of images.
2. **Test Datasets:**
* Test_Baseline: The ground-truth images and coordinates.
* Test_Outlier_350m (AC-3): Test_Baseline with a single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence where several frames are manually deleted to simulate <5% overlap.
* Test_Long_Route (AC-9): A 1500-image sequence.
* **Test Cases:**
* **Test_Accuracy (AC-1, AC-2, AC-5, AC-9):**
* **Run:** Execute ATLAS-GEOFUSE on Test_Baseline, providing the first image's coordinate as the Start Coordinate.
* **Script:** A validation script will compute the Haversine distance error between the *system's refined GPS output* ($Pose_N^{Refined}$) for each image and the *ground-truth GPS*.
* **ASSERT** (count(errors < 50m) / total_images) >= 0.80 **(AC-1 Met)**
* **ASSERT** (count(errors < 20m) / total_images) >= 0.60 **(AC-2 Met)**
* **ASSERT** (count(un-localized_images) / total_images) < 0.10 **(AC-5 Met)**
* **ASSERT** (count(localized_images) / total_images) > 0.95 **(AC-9 Met)**
* **Test_MRE (AC-10):**
* **Run:** After Test_Baseline completes.
* **ASSERT** TOH.final_Mean_Reprojection_Error < 1.0 **(AC-10 Met)**
* **Test_Performance (AC-7, AC-8):**
* **Run:** Execute on Test_Long_Route on the minimum-spec RTX 2060.
* **Log:** Log timestamps for "Image In" -> "Initial Pose Out" ($Pose_N^{Est}$).
* **ASSERT** average_time < 5.0s **(AC-7 Met)**
* **Log:** Log the output stream.
* **ASSERT** >80% of images receive *two* poses: an "Initial" and a "Refined" **(AC-8 Met)**
* **Test_Robustness (AC-3, AC-4, AC-6):**
* **Run:** Execute Test_Outlier_350m.
* **ASSERT** System logs "Stage 2: Discarding Outlier" or "Stage 3: New Map" *and* the final trajectory error for the *next* frame is < 50m **(AC-3 Met)**.
* **Run:** Execute Test_Sharp_Turn_5pct.
* **ASSERT** System logs "Stage 3: New Map Initialization" and "Stage 4: Geodetic Map-Merge," and the final trajectory is complete and accurate **(AC-4 Met)**.
* **Run:** Execute on a sequence with no GAB anchors possible for 20% of the route.
* **ASSERT** System logs "Stage 5: User Intervention Requested" **(AC-6 Met)**.
_docs/01_solution/05_solution_draft.md
-318
View File
@@ -1,318 +0,0 @@
# **ASTRAL System Architecture: A High-Fidelity Geopositioning Framework for IMU-Denied Aerial Operations**
## **2.0 The ASTRAL (Advanced Scale-Aware Trajectory-Refinement and Localization) System Architecture**
The ASTRAL architecture is a multi-map, decoupled, loosely-coupled system designed to solve the flaws identified in Section 1.0 and meet all 10 Acceptance Criteria.
### **2.1 Core Principles**
The ASTRAL architecture is built on three principles:
1. **Tiered Geospatial Database:** The system *cannot* rely on a single data source. It is architected around a *tiered* local database.
* **Tier-1 (Baseline):** Google Maps data. This is used to meet the 50m (AC-1) requirement and provide geolocalization.
* **Tier-2 (High-Accuracy):** A framework for ingesting *commercial, sub-meter* data (visual 4; and DEM 5). This tier is *required* to meet the 20m (AC-2) accuracy. The system will *run* on Tier-1 but *achieve* AC-2 when "fueled" with Tier-2 data.
2. **Viewpoint-Invariant Anchoring:** The system *rejects* geometric warping. The GAB (Section 5.0) is built on SOTA Visual Place Recognition (VPR) models that are *inherently* invariant to the oblique-to-nadir viewpoint change, decoupling it from the V-SLAM's unstable orientation.
3. **Continuously-Scaled Trajectory:** The system *rejects* the "single-scale-per-fragment" model. The TOH (Section 6.0) is a Sim(3) pose-graph optimizer 11 that models scale as a *per-keyframe optimizable parameter*.15 This allows the trajectory to "stretch" and "shrink" elastically to absorb continuous monocular scale drift.12
### **2.2 Component Interaction and Data Flow**
The system is multi-threaded and asynchronous, designed for real-time streaming (AC-7) and refinement (AC-8).
* **Component 1: Tiered GDB (Pre-Flight):**
* *Input:* User-defined Area of Interest (AOI).
* *Action:* Downloads and builds a local SpatiaLite/GeoPackage.
* *Output:* A single **Local-Geo-Database file** containing:
* Tier-1 (Google Maps) + GLO-30 DSM
* Tier-2 (Commercial) satellite tiles + WorldDEM DTM elevation tiles.
* A *pre-computed FAISS vector index* of global descriptors (e.g., SALAD 8) for *all* satellite tiles (see 3.4).
* **Component 2: Image Ingestion (Real-time):**
* *Input:* Image_N (up to 6.2K), Camera Intrinsics ($K$).
* *Action:* Creates Image_N_LR (Low-Res, e.g., 1536x1024) and Image_N_HR (High-Res, 6.2K).
* *Dispatch:* Image_N_LR -> V-SLAM. Image_N_HR -> GAB (for patches).
* **Component 3: "Atlas" V-SLAM Front-End (High-Frequency Thread):**
* *Input:* Image_N_LR.
* *Action:* Tracks Image_N_LR against the *active map fragment*. Manages keyframes and local BA. If tracking lost (AC-4, AC-6), it *initializes a new map fragment*.
* *Output:* Relative_Unscaled_Pose, Local_Point_Cloud, and Map_Fragment_ID -> TOH.
* **Component 4: VPR Geospatial Anchoring Back-End (GAB) (Low-Frequency, Asynchronous Thread):**
* *Input:* A keyframe (Image_N_LR, Image_N_HR) and its Map_Fragment_ID.
* *Action:* Performs SOTA two-stage VPR (Section 5.0) against the **Local-Geo-Database file**.
* *Output:* Absolute_Metric_Anchor ([Lat, Lon, Alt] pose) and its Map_Fragment_ID -> TOH.
* **Component 5: Scale-Aware Trajectory Optimization Hub (TOH) (Central Hub Thread):**
* *Input 1:* High-frequency Relative_Unscaled_Pose stream.
* *Input 2:* Low-frequency Absolute_Metric_Anchor stream.
* *Action:* Manages the *global Sim(3) pose-graph* 13 with *per-keyframe scale*.15
* *Output 1 (Real-time):* Pose_N_Est (unscaled) -> UI (Meets AC-7).
* *Output 2 (Refined):* Pose_N_Refined (metric-scale) -> UI (Meets AC-1, AC-2, AC-8).
### **2.3 System Inputs**
1. **Image Sequence:** Consecutively named images (FullHD to 6252x4168).
2. **Start Coordinate (Image 0):** A single, absolute GPS coordinate [Lat, Lon].
3. **Camera Intrinsics (K):** Pre-calibrated camera intrinsic matrix.
4. **Local-Geo-Database File:** The single file generated by Component 1.
### **2.4 Streaming Outputs (Meets AC-7, AC-8)**
1. **Initial Pose (Pose_N^{Est}):** An *unscaled* pose. This is the raw output from the V-SLAM Front-End, transformed by the *current best estimate* of the trajectory. It is sent immediately (<5s, AC-7) to the UI for real-time visualization of the UAV's *path shape*.
2. **Refined Pose (Pose_N^{Refined}) [Asynchronous]:** A globally-optimized, *metric-scale* 7-DoF pose. This is sent to the user *whenever the TOH re-converges* (e.g., after a new GAB anchor or a map-merge). This *re-writes* the history of poses (e.g., Pose_{N-100} to Pose_N), meeting the refinement (AC-8) and accuracy (AC-1, AC-2) requirements.
## **3.0 Component 1: The Tiered Pre-Flight Geospatial Database (GDB)**
This component is the implementation of the "Tiered Geospatial" principle. It is a mandatory pre-flight utility that solves both the *legal* problem (Flaw 1.4) and the *accuracy* problem (Flaw 1.1).
### **3.2 Tier-1 (Baseline): Google Maps and GLO-30 DEM**
This tier provides the baseline capability and satisfies AC-1.
* **Visual Data:** Google Maps (coarse Maxar)
* *Resolution:* 10m.
* *Geodetic Accuracy:* \~1 m to 20m
* *Purpose:* Meets AC-1 (80% < 50m error). Provides a robust baseline for coarse geolocalization.
* **Elevation Data:** Copernicus GLO-30 DEM
* *Resolution:* 30m.
* *Type:* DSM (Digital Surface Model).2 This is a *weakness*, as it includes buildings/trees.
* *Purpose:* Provides a coarse altitude prior for the TOH and the initial GAB search.
### **3.3 Tier-2 (High-Accuracy): Ingestion Framework for Commercial Data**
This is the *procurement and integration framework* required to meet AC-2.
* **Visual Data:** Commercial providers, e.g., Maxar (30-50cm) or Satellogic (70cm)
* *Resolution:* < 1m.
* *Geodetic Accuracy:* Typically < 5m.
* *Purpose:* Provides the high-resolution, high-accuracy reference needed for the GAB to achieve a sub-20m total error.
* **Elevation Data:** Commercial providers, e.g., WorldDEM Neo 5 or Elevation10.32
* *Resolution:* 5m-12m.
* *Vertical Accuracy:* < 4m.32
* *Type:* DTM (Digital Terrain Model).32
The use of a DTM (bare-earth) in Tier-2 is a critical advantage over the Tier-1 DSM (surface). The V-SLAM Front-End (Section 4.0) will triangulate a 3D point cloud of what it *sees*, which is the *ground* in fields or *tree-tops* in forests. The Tier-1 GLO-30 DSM 2 represents the *top* of the canopy/buildings. If the V-SLAM maps the *ground* (e.g., altitude 100m) and the GAB tries to anchor it to a DSM *prior* that shows a forest (e.g., altitude 120m), the 20m altitude discrepancy will introduce significant error into the TOH. The Tier-2 DTM (bare-earth) 5 provides a *vastly* superior altitude anchor, as it represents the same ground plane the V-SLAM is tracking, significantly improving the entire 7-DoF pose solution.
### **3.4 Local Database Generation: Pre-computing Global Descriptors**
This is the key performance optimization for the GAB. During the pre-flight caching step, the GDB utility does not just *store* tiles; it *processes* them.
For *every* satellite tile (e.g., 256x256m) in the AOI, the utility will load the tile into the VPR model (e.g., SALAD 8), compute its global descriptor (a compact feature vector), and store this vector in a high-speed vector index (e.g., FAISS).
This step moves 99% of the GAB's "Stage 1" (Coarse Retrieval) workload into an offline, pre-flight step. The *real-time* GAB query (Section 5.2) is now reduced to: (1) Compute *one* vector for the UAV image, and (2) Perform a very fast K-Nearest-Neighbor search on the pre-computed FAISS index. This is what makes a SOTA deep-learning GAB 6 fast enough to support the real-time refinement loop.
#### **Table 1: Geospatial Reference Data Analysis (Decision Matrix)**
| Data Product | Type | Resolution | Geodetic Accuracy (Horiz.) | Type | Cost | AC-2 (20m) Compliant? |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| Google Maps | Visual | 1m | 1m - 10m | N/A | Free | **Depending on the location** |
| Copernicus GLO-30 | Elevation | 30m | \~10-30m | **DSM** (Surface) | Free | **No (Fails Error Budget)** |
| **Tier-2: Maxar/Satellogic** | Visual | 0.3m - 0.7m | < 5 m (Est.) | N/A | Commercial | **Yes** |
| **Tier-2: WorldDEM Neo** | Elevation | 5m | < 4m | **DTM** (Bare-Earth) | Commercial | **Yes** |
## **4.0 Component 2: The "Atlas" Relative Motion Front-End**
This component's sole task is to robustly compute *unscaled* 6-DoF relative motion and handle tracking failures (AC-3, AC-4).
### **4.1 Feature Matching Sub-System: SuperPoint + LightGlue**
The system will use **SuperPoint** for feature detection and **LightGlue** for matching. This choice is driven by the project's specific constraints:
* **Rationale (Robustness):** The UAV flies over "eastern and southern parts of Ukraine," which includes large, low-texture agricultural areas. SuperPoint is a SOTA deep-learning detector renowned for its robustness and repeatability in these challenging, low-texture environments.
* **Rationale (Performance):** The RTX 2060 (AC-7) is a *hard* constraint with only 6GB VRAM.34 Performance is paramount. LightGlue is an SOTA matcher that provides a 4-10x speedup over its predecessor, SuperGlue. Its "adaptive" nature is a key optimization: it exits early on "easy" pairs (high-overlap, straight-flight) and spends more compute only on "hard" pairs (turns). This saves critical GPU budget on 95% of normal frames, ensuring the <5s (AC-7) budget is met.
This subsystem will run on the Image_N_LR (low-res) copy to guarantee it fits in VRAM and meets the real-time budget.
#### **Table 2: Analysis of State-of-the-Art Feature Matchers (V-SLAM Front-End)**
| Approach (Tools/Library) | Robustness (Low-Texture) | Speed (RTX 2060) | Fitness for Problem |
| :---- | :---- | :---- | :---- |
| ORB 33 (e.g., ORB-SLAM3) | Poor. Fails on low-texture. | Excellent (CPU/GPU) | **Good.** Fails robustness in target environment. |
| SuperPoint + SuperGlue | Excellent. | Good, but heavy. Fixed-depth GNN. 4-10x Slower than LightGlue.35 | **Good.** Robust, but risks AC-7 budget. |
| **SuperPoint + LightGlue** 35 | Excellent. | **Excellent.** Adaptive depth 35 saves budget. 4-10x faster. | **Excellent (Selected).** Balances robustness and performance. |
### **4.2 The "Atlas" Multi-Map Paradigm (Solution for AC-3, AC-4, AC-6)**
This architecture is the industry-standard solution for IMU-denied, long-term SLAM and is critical for robustness.
* **Mechanism (AC-4, Sharp Turn):**
1. The system is tracking on $Map_Fragment_0$.
2. The UAV makes a sharp turn (AC-4, <5% overlap). The V-SLAM *loses tracking*.
3. Instead of failing, the Atlas architecture *initializes a new map*: $Map_Fragment_1$.
4. Tracking *resumes instantly* on this new, unanchored map.
* **Mechanism (AC-3, 350m Outlier):**
1. The system is tracking. A 350m outlier $Image_N$ arrives.
2. The V-SLAM fails to match $Image_N$ (a "Transient VO Failure," see 7.3). It is *discarded*.
3. $Image_N+1$ arrives (back on track). V-SLAM re-acquires its location on $Map_Fragment_0$.
4. The system "correctly continues the work" (AC-3) by simply rejecting the outlier.
This design turns "catastrophic failure" (AC-3, AC-4) into a *standard operating procedure*. The "problem" of stitching the fragments ($Map_0$, $Map_1$) together is moved from the V-SLAM (which has no global context) to the TOH (which *can* solve it using GAB anchors, see 6.4).
### **4.3 Local Bundle Adjustment and High-Fidelity 3D Cloud**
The V-SLAM front-end will continuously run Local Bundle Adjustment (BA) over a sliding window of recent keyframes to minimize drift *within* that fragment. It will also triangulate a sparse, but high-fidelity, 3D point cloud for its *local map fragment*.
This 3D cloud serves a critical dual function:
1. It provides a robust 3D map for frame-to-map tracking, which is more stable than frame-to-frame odometry.
2. It serves as the **high-accuracy data source** for the object localization output (Section 7.2). This is the key to decoupling object-pointing accuracy from external DEM accuracy 19, a critical flaw in simpler designs.
## **5.0 Component 3: The Viewpoint-Invariant Geospatial Anchoring Back-End (GAB)**
This component *replaces* the draft's "Dynamic Warping" (Section 5.0) and implements the "Viewpoint-Invariant Anchoring" principle (Section 2.1).
### **5.1 Rationale: Viewpoint-Invariant VPR vs. Geometric Warping (Solves Flaw 1.2)**
As established in 1.2, geometrically warping the image using the V-SLAM's *drifty* roll/pitch estimate creates a *brittle*, high-risk failure spiral. The ASTRAL GAB *decouples* from the V-SLAM's orientation. It uses a SOTA VPR pipeline that *learns* to match oblique UAV images to nadir satellite images *directly*, at the feature level.6
### **5.2 Stage 1 (Coarse Retrieval): SOTA Global Descriptors**
When triggered by the TOH, the GAB takes Image_N_LR. It computes a *global descriptor* (a single feature vector) using a SOTA VPR model like **SALAD** 6 or **MixVPR**.7
This choice is driven by two factors:
1. **Viewpoint Invariance:** These models are SOTA for this exact task.
2. **Inference Speed:** They are extremely fast. SALAD reports < 3ms per image inference 8, and MixVPR is also noted for "fastest inference speed".37 This low overhead is essential for the AC-7 (<5s) budget.
This vector is used to query the *pre-computed FAISS vector index* (from 3.4), which returns the Top-K (e.g., K=5) most likely satellite tiles from the *entire AOI* in milliseconds.
#### **Table 3: Analysis of VPR Global Descriptors (GAB Back-End)**
| Model (Backbone) | Key Feature | Viewpoint Invariance | Inference Speed (ms) | Fitness for GAB |
| :---- | :---- | :---- | :---- | :---- |
| NetVLAD 7 (CNN) | Baseline | Poor. Not designed for oblique-to-nadir. | Moderate (\~20-50ms) | **Poor.** Fails robustness. |
| **SALAD** 8 (DINOv2) | Foundation Model.6 | **Excellent.** Designed for this. | **< 3ms**.8 Extremely fast. | **Excellent (Selected).** |
| **MixVPR** 36 (ResNet) | All-MLP aggregator.36 | **Very Good.**.7 | **Very Fast.**.37 | **Excellent (Selected).** |
### **5.3 Stage 2 (Fine): Local Feature Matching and Pose Refinement**
The system runs **SuperPoint+LightGlue** 35 to find pixel-level matches, but *only* between the UAV image and the **Top-K satellite tiles** identified in Stage 1.
A **Multi-Resolution Strategy** is employed to solve the VRAM bottleneck.
1. Stage 1 (Coarse) runs on the Image_N_LR.
2. Stage 2 (Fine) runs SuperPoint *selectively* on the Image_N_HR (6.2K) to get high-accuracy keypoints.
3. It then matches small, full-resolution *patches* from the full-res image, *not* the full image.
This hybrid approach is the *only* way to meet both AC-7 (speed) and AC-2 (accuracy). The 6.2K image *cannot* be processed in <5s on an RTX 2060 (6GB VRAM 34). But its high-resolution *pixels* are needed for the 20m *accuracy*. Using full-res *patches* provides the pixel-level accuracy without the VRAM/compute cost.
A PnP/RANSAC solver then computes a high-confidence 6-DoF pose. This pose, converted to [Lat, Lon, Alt], is the **$Absolute_Metric_Anchor$** sent to the TOH.
## **6.0 Component 4: The Scale-Aware Trajectory Optimization Hub (TOH)**
This component is the system's "brain" and implements the "Continuously-Scaled Trajectory" principle (Section 2.1). It *replaces* the draft's flawed "Single Scale" optimizer.
### **6.1 The $Sim(3)$ Pose-Graph as the Optimization Backbone**
The central challenge of IMU-denied monocular SLAM is *scale drift*.11 The V-SLAM (Component 3) produces 6-DoF poses, but they are *unscaled* ($SE(3)$). The GAB (Component 4) produces *metric* 6-DoF poses ($SE(3)$).
The solution is to optimize the *entire graph* in the 7-DoF "Similarity" group, **$Sim(3)$**.11 This adds a 7th degree of freedom (scale, $s$) to the poses. The optimization backbone will be **Ceres Solver** 14, a SOTA C++ library for large, complex non-linear least-squares problems.
### **6.2 Advanced Scale-Drift Correction: Modeling Scale as a Per-Keyframe Parameter (Solves Flaw 1.3)**
This is the *core* of the ASTRAL optimizer, solving Flaw 1.3. The draft's flawed model ($Pose_Graph(Fragment_i) = \\{Pose_1...Pose_n, s_i\\}$) is replaced by ASTRAL's correct model: $Pose_Graph = \\{ (Pose_1, s_1), (Pose_2, s_2),..., (Pose_N, s_N) \\}$.
The graph is constructed as follows:
* **Nodes:** Each keyframe pose is a 7-DoF $Sim(3)$ variable $\\{s_k, R_k, t_k\\}$.
* **Edge 1 (V-SLAM):** A *relative* $Sim(3)$ constraint between $Pose_k$ and $Pose_{k+1}$ from the V-SLAM Front-End.
* **Edge 2 (GAB):** An *absolute* $SE(3)$ constraint on $Pose_j$ from a GAB anchor. This constraint *fixes* the 6-DoF pose $(R_j, t_j)$ to the metric GAB value and *fixes its scale* $s_j = 1.0$.
This "per-keyframe scale" model 15 enables "elastic" trajectory refinement. When the graph is a long, unscaled "chain" of V-SLAM constraints, a GAB anchor (Edge 2) arrives at $Pose_{100}$, "nailing" it to the metric map and setting $s_{100} = 1.0$. As the V-SLAM continues, scale drifts. When a second anchor arrives at $Pose_{200}$ (setting $s_{200} = 1.0$), the Ceres optimizer 14 has a problem: the V-SLAM data *between* them has drifted.
The ASTRAL model *allows* the optimizer to solve for all intermediate scales (s_{101}, s_{102},..., s_{199}) as variables. The optimizer will find a *smooth, continuous* scale correction 15 that "elastically" stretches/shrinks the 100-frame sub-segment to *perfectly* fit both metric anchors. This *correctly* models the physics of scale drift 12 and is the *only* way to achieve the 20m accuracy (AC-2) and 1.0px MRE (AC-10).
### **6.3 Robust M-Estimation (Solution for AC-3, AC-5)**
A 350m outlier (AC-3) or a bad GAB match (AC-5) will add a constraint with a *massive* error. A standard least-squares optimizer 14 would be *catastrophically* corrupted, pulling the *entire* 3000-image trajectory to try and fit this one bad point.
This is a solved problem. All constraints (V-SLAM and GAB) *must* be wrapped in a **Robust Loss Function** (e.g., HuberLoss, CauchyLoss) within Ceres Solver. This function mathematically *down-weights* the influence of constraints with large errors (high residuals). It effectively tells the optimizer: "This measurement is insane. Ignore it." This provides automatic, graceful outlier rejection, meeting AC-3 and AC-5.
### **6.4 Geodetic Map-Merging (Solution for AC-4, AC-6)**
This mechanism is the robust solution to the "sharp turn" (AC-4) problem.
* **Scenario:** The UAV makes a sharp turn (AC-4). The V-SLAM (4.2) creates Map_Fragment_0 and Map_Fragment_1. The TOH's graph now has two *disconnected* components.
* **Mechanism (Geodetic Merging):**
1. The TOH queues the GAB (Section 5.0) to find anchors for *both* fragments.
2. GAB returns Anchor_A for Map_Fragment_0 and Anchor_B for Map_Fragment_1.
3. The TOH adds *both* of these as absolute, metric constraints (Edge 2) to the *single global pose-graph*.
4. The Ceres optimizer 14 now has all the information it needs. It solves for the 7-Dof pose of *both fragments*, placing them in their correct, globally-consistent metric positions.
The two fragments are *merged geodetically* (by their global coordinates 11) even if they *never* visually overlap. This is a vastly more robust solution to AC-4 and AC-6 than simple visual loop closure.
## **7.0 Performance, Deployment, and High-Accuracy Outputs**
### **7.1 Meeting the <5s Budget (AC-7): Mandatory Acceleration with NVIDIA TensorRT**
The system must run on an RTX 2060 (AC-7). This is a low-end, 6GB VRAM card 34, which is a *severe* constraint. Running three deep-learning models (SuperPoint, LightGlue, SALAD/MixVPR) plus a Ceres optimizer 38 will saturate this hardware.
* **Solution 1: Multi-Scale Pipeline.** As defined in 5.3, the system *never* processes a full 6.2K image on the GPU. It uses low-res for V-SLAM/GAB-Coarse and high-res *patches* for GAB-Fine.
* **Solution 2: Mandatory TensorRT Deployment.** Running these models in their native PyTorch framework will be too slow. All neural networks (SuperPoint, LightGlue, SALAD/MixVPR) *must* be converted from PyTorch into optimized **NVIDIA TensorRT engines**. Research *specifically* on accelerating LightGlue shows this provides **"2x-4x speed gains over compiled PyTorch"**.35 This 200-400% speedup is *not* an optimization; it is a *mandatory deployment step* to make the <5s (AC-7) budget *possible* on an RTX 2060.
### **7.2 High-Accuracy Object Geolocalization via Ray-Cloud Intersection (Solves AC-2/AC-10)**
The user must be able to find the GPS of an *object* in a photo. A simple approach of ray-casting from the camera and intersecting with the 30m GLO-30 DEM 2 is fatally flawed. The DEM error itself can be up to 30m 19, making AC-2 impossible.
The ASTRAL system uses a **Ray-Cloud Intersection** method that *decouples* object accuracy from external DEM accuracy.
* **Algorithm:**
1. The user clicks pixel (u,v) on Image_N.
2. The system retrieves the *final, refined, metric 7-DoF pose* P_{sim(3)} = (s, R, T) for Image_N from the TOH.
3. It also retrieves the V-SLAM's *local, high-fidelity 3D point cloud* (P_{local_cloud}) from Component 3 (Section 4.3).
4. **Step 1 (Local):** The pixel (u,v) is un-projected into a ray. This ray is intersected with the *local* P_{local_cloud}. This finds the 3D point $P_{local} *relative to the V-SLAM map*. The accuracy of this step is defined by AC-10 (MRE < 1.0px).
5. **Step 2 (Global):** This *highly-accurate* local point P_{local} is transformed into the global metric coordinate system using the *highly-accurate* refined pose from the TOH: P_{metric} = s * (R * P_{local}) + T.
6. **Step 3 (Convert):** P_{metric} (an X,Y,Z world coordinate) is converted to [Latitude, Longitude, Altitude].
This method correctly isolates error. The object's accuracy is now *only* dependent on the V-SLAM's internal geometry (AC-10) and the TOH's global pose accuracy (AC-1, AC-2). It *completely eliminates* the external 30m DEM error 2 from this critical, high-accuracy calculation.
### **7.3 Failure Mode Escalation Logic (Meets AC-3, AC-4, AC-6, AC-9)**
The system is built on a robust state machine to handle real-world failures.
* **Stage 1: Normal Operation (Tracking):** V-SLAM tracks, TOH optimizes.
* **Stage 2: Transient VO Failure (Outlier Rejection):**
* *Condition:* Image_N is a 350m outlier (AC-3) or severe blur.
* *Logic:* V-SLAM fails to track Image_N. System *discards* it (AC-5). Image_N+1 arrives, V-SLAM re-tracks.
* *Result:* **AC-3 Met.**
* **Stage 3: Persistent VO Failure (New Map Initialization):**
* *Condition:* "Sharp turn" (AC-4) or >5 frames of tracking loss.
* *Logic:* V-SLAM (Section 4.2) declares "Tracking Lost." Initializes *new* Map_Fragment_k+1. Tracking *resumes instantly*.
* *Result:* **AC-4 Met.** System "correctly continues the work." The >95% registration rate (AC-9) is met because this is *not* a failure, it's a *new registration*.
* **Stage 4: Map-Merging & Global Relocalization (GAB-Assisted):**
* *Condition:* System is on Map_Fragment_k+1, Map_Fragment_k is "lost."
* *Logic:* TOH (Section 6.4) receives GAB anchors for *both* fragments and *geodetically merges* them in the global optimizer.14
* *Result:* **AC-6 Met** (strategy to connect separate chunks).
* **Stage 5: Catastrophic Failure (User Intervention):**
* *Condition:* System is in Stage 3 (Lost) *and* the GAB has failed for 20% of the route. The "absolutely incapable" scenario (AC-6).
* *Logic:* TOH triggers the AC-6 flag. UI prompts user: "Please provide a coarse location for the *current* image."
* *Action:* This user-click is *not* taken as ground-truth. It is fed to the **GAB (Section 5.0)** as a *strong spatial prior*, narrowing its Stage 1 8 search from "the entire AOI" to "a 5km radius." This *guarantees* the GAB finds a match, which triggers Stage 4, re-localizing the system.
* *Result:* **AC-6 Met** (user input).
## **8.0 ASTRAL Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
_docs/01_solution/06_solution_draft.md
-282
View File
@@ -1,282 +0,0 @@
# **ASTRAL-Next: A Resilient, GNSS-Denied Geo-Localization Architecture for Wing-Type UAVs in Complex Semantic Environments**
## **1. Executive Summary and Operational Context**
The strategic necessity of operating Unmanned Aerial Vehicles (UAVs) in Global Navigation Satellite System (GNSS)-denied environments has precipitated a fundamental shift in autonomous navigation research. The specific operational profile under analysis—high-speed, fixed-wing UAVs operating without Inertial Measurement Units (IMU) over the visually homogenous and texture-repetitive terrain of Eastern and Southern Ukraine—presents a confluence of challenges that render traditional Simultaneous Localization and Mapping (SLAM) approaches insufficient. The target environment, characterized by vast agricultural expanses, seasonal variability, and potential conflict-induced terrain alteration, demands a navigation architecture that moves beyond simple visual odometry to a robust, multi-layered Absolute Visual Localization (AVL) system.
This report articulates the design and theoretical validation of **ASTRAL-Next**, a comprehensive architectural framework engineered to supersede the limitations of preliminary dead-reckoning solutions. By synthesizing state-of-the-art (SOTA) research emerging in 2024 and 2025, specifically leveraging **LiteSAM** for efficient cross-view matching 1, **AnyLoc** for universal place recognition 2, and **SuperPoint+LightGlue** for robust sequential tracking 1, the proposed system addresses the critical failure modes inherent in wing-type UAV flight dynamics. These dynamics include sharp banking maneuvers, significant pitch variations leading to ground sampling distance (GSD) disparities, and the potential for catastrophic track loss (the "kidnapped robot" problem).
The analysis indicates that relying solely on sequential image overlap is viable only for short-term trajectory smoothing. The core innovation of ASTRAL-Next lies in its "Hierarchical + Anchor" topology, which decouples the relative motion estimation from absolute global anchoring. This ensures that even during zero-overlap turns or 350-meter positional outliers caused by airframe tilt, the system can re-localize against a pre-cached satellite reference map within the required 5-second latency window.3 Furthermore, the system accounts for the semantic disconnect between live UAV imagery and potentially outdated satellite reference data (e.g., Google Maps) by prioritizing semantic geometry over pixel-level photometric consistency.
### **1.1 Operational Environment and Constraints Analysis**
The operational theater—specifically the left bank of the Dnipro River in Ukraine—imposes rigorous constraints on computer vision algorithms. The absence of IMU data removes the ability to directly sense acceleration and angular velocity, creating a scale ambiguity in monocular vision systems that must be resolved through external priors (altitude) and absolute reference data.
| Constraint Category | Specific Challenge | Implication for System Design |
| :---- | :---- | :---- |
| **Sensor Limitation** | **No IMU Data** | The system cannot distinguish between pure translation and camera rotation (pitch/roll) without visual references. Scale must be constrained via altitude priors and satellite matching.5 |
| **Flight Dynamics** | **Wing-Type UAV** | Unlike quadcopters, fixed-wing aircraft cannot hover. They bank to turn, causing horizon shifts and perspective distortions. "Sharp turns" result in 0% image overlap.6 |
| **Terrain Texture** | **Agricultural Fields** | Repetitive crop rows create aliasing for standard descriptors (SIFT/ORB). Feature matching requires context-aware deep learning methods (SuperPoint).7 |
| **Reference Data** | **Google Maps (2025)** | Public satellite data may be outdated or lower resolution than restricted military feeds. Matches must rely on invariant features (roads, tree lines) rather than ephemeral textures.9 |
| **Compute Hardware** | **NVIDIA RTX 2060/3070** | Algorithms must be optimized for TensorRT to meet the <5s per frame requirement. Heavy transformers (e.g., ViT-Huge) are prohibitive; efficient architectures (LiteSAM) are required.1 |
The confluence of these factors necessitates a move away from simple "dead reckoning" (accumulating relative movements) which drifts exponentially. Instead, ASTRAL-Next operates as a **Global-Local Hybrid System**, where a high-frequency visual odometry layer handles frame-to-frame continuity, while a parallel global localization layer periodically "resets" the drift by anchoring the UAV to the satellite map.
## **2. Architectural Critique of Legacy Approaches**
The initial draft solution ("ASTRAL") and similar legacy approaches typically rely on a unified SLAM pipeline, often attempting to use the same feature extractors for both sequential tracking and global localization. Recent literature highlights substantial deficiencies in this monolithic approach, particularly when applied to the specific constraints of this project.
### **2.1 The Failure of Classical Descriptors in Agricultural Settings**
Classical feature descriptors like SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) rely on detecting "corners" and "blobs" based on local pixel intensity gradients. In the agricultural landscapes of Eastern Ukraine, this approach faces severe aliasing. A field of sunflowers or wheat presents thousands of identical "blobs," causing the nearest-neighbor matching stage to generate a high ratio of outliers.8
Research demonstrates that deep-learning-based feature extractors, specifically SuperPoint, trained on large datasets of synthetic and real-world imagery, learn to identify interest points that are semantically significant (e.g., the intersection of a tractor path and a crop line) rather than just texturally distinct.1 Consequently, a redesign must replace SIFT/ORB with SuperPoint for the front-end tracking.
### **2.2 The Inadequacy of Dead Reckoning without IMU**
In a standard Visual-Inertial Odometry (VIO) system, the IMU provides a high-frequency prediction of the camera's pose, which the visual system then refines. Without an IMU, the system is purely Visual Odometry (VO). In VO, the scale of the world is unobservable from a single camera (monocular scale ambiguity). A 1-meter movement of a small object looks identical to a 10-meter movement of a large object.5
While the prompt specifies a "predefined altitude," relying on this as a static constant is dangerous due to terrain undulations and barometric drift. ASTRAL-Next must implement a Scale-Constrained Bundle Adjustment, treating the altitude not as a hard fact, but as a strong prior that prevents the scale drift common in monocular systems.5
### **2.3 Vulnerability to "Kidnapped Robot" Scenarios**
The requirement to recover from sharp turns where the "next photo doesn't overlap at all" describes the classic "Kidnapped Robot Problem" in robotics—where a robot is teleported to an unknown location and must relocalize.14
Sequential matching algorithms (optical flow, feature tracking) function on the assumption of overlap. When overlap is zero, these algorithms fail catastrophically. The legacy solution's reliance on continuous tracking makes it fragile to these flight dynamics. The redesigned architecture must incorporate a dedicated Global Place Recognition module that treats every frame as a potential independent query against the satellite database, independent of the previous frame's history.2
## **3. ASTRAL-Next: System Architecture and Methodology**
To meet the acceptance criteria—specifically the 80% success rate within 50m error and the <5 second processing time—ASTRAL-Next utilizes a tri-layer processing topology. These layers operate concurrently, feeding into a central state estimator.
### **3.1 The Tri-Layer Localization Strategy**
The architecture separates the concerns of continuity, recovery, and precision into three distinct algorithmic pathways.
| Layer | Functionality | Algorithm | Latency | Role in Acceptance Criteria |
| :---- | :---- | :---- | :---- | :---- |
| **L1: Sequential Tracking** | Frame-to-Frame Relative Pose | **SuperPoint + LightGlue** | \~50-100ms | Handles continuous flight, bridges small gaps (overlap < 5%), and maintains trajectory smoothness. Essential for the 100m spacing requirement. 1 |
| **L2: Global Re-Localization** | "Kidnapped Robot" Recovery | **AnyLoc (DINOv2 + VLAD)** | \~200ms | Detects location after sharp turns (0% overlap) or track loss. Matches current view to the satellite database tile. Addresses the sharp turn recovery criterion. 2 |
| **L3: Metric Refinement** | Precise GPS Anchoring | **LiteSAM / HLoc** | \~300-500ms | "Stitches" the UAV image to the satellite tile with pixel-level accuracy to reset drift. Ensures the "80% < 50m" and "60% < 20m" accuracy targets. 1 |
### **3.2 Data Flow and State Estimation**
The system utilizes a **Factor Graph Optimization** (using libraries like GTSAM) as the central "brain."
1. **Inputs:**
* **Relative Factors:** Provided by Layer 1 (Change in pose from $t-1$ to $t$).
* **Absolute Factors:** Provided by Layer 3 (Global GPS coordinate at $t$).
* **Priors:** Altitude constraint and Ground Plane assumption.
2. **Processing:** The factor graph optimizes the trajectory by minimizing the error between these conflicting constraints.
3. **Output:** A smoothed, globally consistent trajectory $(x, y, z, \\text{roll}, \\text{pitch}, \\text{yaw})$ for every image timestamp.
### **3.3 ZeroMQ Background Service Architecture**
As per the requirement, the system operates as a background service.
* **Communication Pattern:** The service utilizes a REP-REQ (Reply-Request) pattern for control commands (Start/Stop/Reset) and a PUB-SUB (Publish-Subscribe) pattern for the continuous stream of localization results.
* **Concurrency:** Layer 1 runs on a high-priority thread to ensure immediate feedback. Layers 2 and 3 run asynchronously; when a global match is found, the result is injected into the Factor Graph, which then "back-propagates" the correction to previous frames, refining the entire recent trajectory.
## **4. Layer 1: Robust Sequential Visual Odometry**
The first line of defense against localization loss is robust tracking between consecutive UAV images. Given the challenging agricultural environment, standard feature matching is prone to failure. ASTRAL-Next employs **SuperPoint** and **LightGlue**.
### **4.1 SuperPoint: Semantic Feature Detection**
SuperPoint is a fully convolutional neural network trained to detect interest points and compute their descriptors. Unlike SIFT, which uses handcrafted mathematics to find corners, SuperPoint is trained via self-supervision on millions of images.
* **Relevance to Ukraine:** In a wheat field, SIFT might latch onto hundreds of identical wheat stalks. SuperPoint, however, learns to prioritize more stable features, such as the boundary between the field and a dirt road, or a specific patch of discoloration in the crop canopy.1
* **Performance:** SuperPoint runs efficiently on the RTX 2060/3070, with inference times around 15ms per image when optimized with TensorRT.16
### **4.2 LightGlue: The Attention-Based Matcher**
**LightGlue** represents a paradigm shift from the traditional "Nearest Neighbor + RANSAC" matching pipeline. It is a deep neural network that takes two sets of SuperPoint features and jointly predicts the matches.
* **Mechanism:** LightGlue uses a transformer-based attention mechanism. It allows features in Image A to "look at" all features in Image B (and vice versa) to determine the best correspondence. Crucially, it has a "dustbin" mechanism to explicitly reject points that have no match (occlusion or field of view change).12
* **Addressing the <5% Overlap:** The user specifies handling overlaps of "less than 5%." Traditional RANSAC fails here because the inlier ratio is too low. LightGlue, however, can confidently identify the few remaining matches because its attention mechanism considers the global geometric context of the points. If only a single road intersection is visible in the corner of both images, LightGlue is significantly more likely to match it correctly than SIFT.8
* **Efficiency:** LightGlue is designed to be "light." It features an adaptive depth mechanism—if the images are easy to match, it exits early. If they are hard (low overlap), it uses more layers. This adaptability is perfect for the variable difficulty of the UAV flight path.19
## **5. Layer 2: Global Place Recognition (The "Kidnapped Robot" Solver)**
When the UAV executes a sharp turn, resulting in a completely new view (0% overlap), sequential tracking (Layer 1) is mathematically impossible. The system must recognize the new terrain solely based on its appearance. This is the domain of **AnyLoc**.
### **5.1 Universal Place Recognition with Foundation Models**
**AnyLoc** leverages **DINOv2**, a massive self-supervised vision transformer developed by Meta. DINOv2 is unique because it is not trained with labels; it is trained to understand the geometry and semantic layout of images.
* **Why DINOv2 for Satellite Matching:** Satellite images and UAV images have different "domains." The satellite image might be from summer (green), while the UAV flies in autumn (brown). DINOv2 features are remarkably invariant to these texture changes. It "sees" the shape of the road network or the layout of the field boundaries, rather than the color of the leaves.2
* **VLAD Aggregation:** AnyLoc extracts dense features from the image using DINOv2 and aggregates them using **VLAD** (Vector of Locally Aggregated Descriptors) into a single, compact vector (e.g., 4096 dimensions). This vector represents the "fingerprint" of the location.21
### **5.2 Implementation Strategy**
1. **Database Preparation:** Before the mission, the system downloads the satellite imagery for the operational bounding box (Eastern/Southern Ukraine). These images are tiled (e.g., 512x512 pixels with overlap) and processed through AnyLoc to generate a database of descriptors.
2. **Faiss Indexing:** These descriptors are indexed using **Faiss**, a library for efficient similarity search.
3. **In-Flight Retrieval:** When Layer 1 reports a loss of tracking (or periodically), the current UAV image is processed by AnyLoc. The resulting vector is queried against the Faiss index.
4. **Result:** The system retrieves the top-5 most similar satellite tiles. These tiles represent the coarse global location of the UAV (e.g., "You are in Grid Square B7").2
## **6. Layer 3: Fine-Grained Metric Localization (LiteSAM)**
Retrieving the correct satellite tile (Layer 2) gives a location error of roughly the tile size (e.g., 200 meters). To meet the "60% < 20m" and "80% < 50m" criteria, the system must precisely align the UAV image onto the satellite tile. ASTRAL-Next utilizes **LiteSAM**.
### **6.1 Justification for LiteSAM over TransFG**
While **TransFG** (Transformer for Fine-Grained recognition) is a powerful architecture for cross-view geo-localization, it is computationally heavy.23 **LiteSAM** (Lightweight Satellite-Aerial Matching) is specifically architected for resource-constrained platforms (like UAV onboard computers or efficient ground stations) while maintaining state-of-the-art accuracy.
* **Architecture:** LiteSAM utilizes a **Token Aggregation-Interaction Transformer (TAIFormer)**. It employs a convolutional token mixer (CTM) to model correlations between the UAV and satellite images.
* **Multi-Scale Processing:** LiteSAM processes features at multiple scales. This is critical because the UAV altitude varies (<1km), meaning the scale of objects in the UAV image will not perfectly match the fixed scale of the satellite image (Google Maps Zoom Level 19). LiteSAM's multi-scale approach inherently handles this discrepancy.1
* **Performance Data:** Empirical benchmarks on the **UAV-VisLoc** dataset show LiteSAM achieving an RMSE@30 (Root Mean Square Error within 30 meters) of 17.86 meters, directly supporting the project's accuracy requirements. Its inference time is approximately 61.98ms on standard GPUs, ensuring it fits within the overall 5-second budget.1
### **6.2 The Alignment Process**
1. **Input:** The UAV Image and the Top-1 Satellite Tile from Layer 2.
2. **Processing:** LiteSAM computes the dense correspondence field between the two images.
3. **Homography Estimation:** Using the correspondences, the system computes a homography matrix $H$ that maps pixels in the UAV image to pixels in the georeferenced satellite tile.
4. **Pose Extraction:** The camera's absolute GPS position is derived from this homography, utilizing the known GSD of the satellite tile.18
## **7. Satellite Data Management and Coordinate Systems**
The reliability of the entire system hinges on the quality and handling of the reference map data. The restriction to "Google Maps" necessitates a rigorous approach to coordinate transformation and data freshness management.
### **7.1 Google Maps Static API and Mercator Projection**
The Google Maps Static API delivers images without embedded georeferencing metadata (GeoTIFF tags). The system must mathematically derive the bounding box of each downloaded tile to assign coordinates to the pixels. Google Maps uses the **Web Mercator Projection (EPSG:3857)**.
The system must implement the following derivation to establish the **Ground Sampling Distance (GSD)**, or meters_per_pixel, which varies significantly with latitude:
$$ \\text{meters_per_pixel} = 156543.03392 \\times \\frac{\\cos(\\text{latitude} \\times \\frac{\\pi}{180})}{2^{\\text{zoom}}} $$
For the operational region (Ukraine, approx. Latitude 48N):
* At **Zoom Level 19**, the resolution is approximately 0.30 meters/pixel. This resolution is compatible with the input UAV imagery (Full HD at <1km altitude), providing sufficient detail for the LiteSAM matcher.24
**Bounding Box Calculation Algorithm:**
1. **Input:** Center Coordinate $(lat, lon)$, Zoom Level ($z$), Image Size $(w, h)$.
2. **Project to World Coordinates:** Convert $(lat, lon)$ to world pixel coordinates $(px, py)$ at the given zoom level.
3. **Corner Calculation:**
* px_{NW} = px - (w / 2)
* py_{NW} = py - (h / 2)
4. Inverse Projection: Convert $(px_{NW}, py_{NW})$ back to Latitude/Longitude to get the North-West corner. Repeat for South-East.
This calculation is critical. A precision error here translates directly to a systematic bias in the final GPS output.
### **7.2 Mitigating Data Obsolescence (The 2025 Problem)**
The provided research highlights that satellite imagery access over Ukraine is subject to restrictions and delays (e.g., Maxar restrictions in 2025).10 Google Maps data may be several years old.
* **Semantic Anchoring:** This reinforces the selection of **AnyLoc** (Layer 2) and **LiteSAM** (Layer 3). These algorithms are trained to ignore transient features (cars, temporary structures, vegetation color) and focus on persistent structural features (road geometry, building footprints).
* **Seasonality:** Research indicates that DINOv2 features (used in AnyLoc) exhibit strong robustness to seasonal changes (e.g., winter satellite map vs. summer UAV flight), maintaining high retrieval recall where pixel-based methods fail.17
## **8. Optimization and State Estimation (The "Brain")**
The individual outputs of the visual layers are noisy. Layer 1 drifts over time; Layer 3 may have occasional outliers. The **Factor Graph Optimization** fuses these inputs into a coherent trajectory.
### **8.1 Handling the 350-Meter Outlier (Tilt)**
The prompt specifies that "up to 350 meters of an outlier... could happen due to tilt." This large displacement masquerading as translation is a classic source of divergence in Kalman Filters.
* **Robust Cost Functions:** In the Factor Graph, the error terms for the visual factors are wrapped in a **Robust Kernel** (specifically the **Cauchy** or **Huber** kernel).
* *Mechanism:* Standard least-squares optimization penalizes errors quadratically ($e^2$). If a 350m error occurs, the penalty is massive, dragging the entire trajectory off-course. A robust kernel changes the penalty to be linear ($|e|$) or logarithmic after a certain threshold. This allows the optimizer to effectively "ignore" or down-weight the 350m jump if it contradicts the consensus of other measurements, treating it as a momentary outlier or solving for it as a rotation rather than a translation.19
### **8.2 The Altitude Soft Constraint**
To resolve the monocular scale ambiguity without IMU, the altitude ($h_{prior}$) is added as a **Unary Factor** to the graph.
* $E_{alt} = |
| z_{est} \- h_{prior} ||*{\\Sigma*{alt}}$
* $\\Sigma_{alt}$ (covariance) is set relatively high (soft constraint), allowing the visual odometry to adjust the altitude slightly to maintain consistency, but preventing the scale from collapsing to zero or exploding to infinity. This effectively creates an **Altimeter-Aided Monocular VIO** system, where the altimeter (virtual or barometric) replaces the accelerometer for scale determination.5
## **9. Implementation Specifications**
### **9.1 Hardware Acceleration (TensorRT)**
Meeting the <5 second per frame requirement on an RTX 2060 requires optimizing the deep learning models. Python/PyTorch inference is typically too slow due to overhead.
* **Model Export:** All core models (SuperPoint, LightGlue, LiteSAM) must be exported to **ONNX** (Open Neural Network Exchange) format.
* **TensorRT Compilation:** The ONNX models are then compiled into **TensorRT Engines**. This process performs graph fusion (combining multiple layers into one) and kernel auto-tuning (selecting the fastest GPU instructions for the specific RTX 2060/3070 architecture).26
* **Precision:** The models should be quantized to **FP16** (16-bit floating point). Research shows that FP16 inference on NVIDIA RTX cards offers a 2x-3x speedup with negligible loss in matching accuracy for these specific networks.16
### **9.2 Background Service Architecture (ZeroMQ)**
The system is encapsulated as a headless service.
**ZeroMQ Topology:**
* **Socket 1 (REP - Port 5555):** Command Interface. Accepts JSON messages:
* {"cmd": "START", "config": {"lat": 48.1, "lon": 37.5}}
* {"cmd": "USER_FIX", "lat": 48.22, "lon": 37.66} (Human-in-the-loop input).
* **Socket 2 (PUB - Port 5556):** Data Stream. Publishes JSON results for every frame:
* {"frame_id": 1024, "gps": [48.123, 37.123], "object_centers": [...], "status": "LOCKED", "confidence": 0.98}.
Asynchronous Pipeline:
The system utilizes a Python multiprocessing architecture. One process handles the camera/image ingest and ZeroMQ communication. A second process hosts the TensorRT engines and runs the Factor Graph. This ensures that the heavy computation of Bundle Adjustment does not block the receipt of new images or user commands.
## **10. Human-in-the-Loop Strategy**
The requirement stipulates that for the "20% of the route" where automation fails, the user must intervene. The system must proactively detect its own failure.
### **10.1 Failure Detection with PDM@K**
The system monitors the **PDM@K** (Positioning Distance Measurement) metric continuously.
* **Definition:** PDM@K measures the percentage of queries localized within $K$ meters.3
* **Real-Time Proxy:** In flight, we cannot know the true PDM (as we don't have ground truth). Instead, we use the **Marginal Covariance** from the Factor Graph. If the uncertainty ellipse for the current position grows larger than a radius of 50 meters, or if the **Image Registration Rate** (percentage of inliers in LightGlue/LiteSAM) drops below 10% for 3 consecutive frames, the system triggers a **Critical Failure Mode**.19
### **10.2 The User Interaction Workflow**
1. **Trigger:** Critical Failure Mode activated.
2. **Action:** The Service publishes a status {"status": "REQ_INPUT"} via ZeroMQ.
3. **Data Payload:** It sends the current UAV image and the top-3 retrieved satellite tiles (from Layer 2) to the client UI.
4. **User Input:** The user clicks a distinctive feature (e.g., a specific crossroad) in the UAV image and the corresponding point on the satellite map.
5. **Recovery:** This pair of points is treated as a **Hard Constraint** in the Factor Graph. The optimizer immediately snaps the trajectory to this user-defined anchor, resetting the covariance and effectively "healing" the localized track.19
## **11. Performance Evaluation and Benchmarks**
### **11.1 Accuracy Validation**
Based on the reported performance of the selected components in relevant datasets (UAV-VisLoc, AnyVisLoc):
* **LiteSAM** demonstrates an accuracy of 17.86m (RMSE) for cross-view matching. This aligns with the requirement that 60% of photos be within 20m error.18
* **AnyLoc** achieves high recall rates (Top-1 Recall > 85% on aerial benchmarks), supporting the recovery from sharp turns.2
* **Factor Graph Fusion:** By combining sequential and global measurements, the overall system error is expected to be lower than the individual component errors, satisfying the "80% within 50m" criterion.
### **11.2 Latency Analysis**
The breakdown of processing time per frame on an RTX 3070 is estimated as follows:
* **SuperPoint + LightGlue:** \~50ms.1
* **AnyLoc (Global Retrieval):** \~150ms (run only on keyframes or tracking loss).
* **LiteSAM (Metric Refinement):** \~60ms.1
* **Factor Graph Optimization:** \~100ms (using incremental updates/iSAM2).
* Total: \~360ms per frame (worst case with all layers active).
This is an order of magnitude faster than the 5-second limit, providing ample headroom for higher resolution processing or background tasks.
## **12.0 ASTRAL-Next Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Copernicus)** data 1 is sufficient. SOTA VPR 8 + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **8.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
_docs/01_solution/solution.md
-372
View File
@@ -1,372 +0,0 @@
# **ASTRAL-Next: A Resilient, GNSS-Denied Geo-Localization Architecture for Wing-Type UAVs in Complex Semantic Environments**
## **1. Executive Summary and Operational Context**
The strategic necessity of operating Unmanned Aerial Vehicles (UAVs) in Global Navigation Satellite System (GNSS)-denied environments has precipitated a fundamental shift in autonomous navigation research. The specific operational profile under analysis—high-speed, fixed-wing UAVs operating without Inertial Measurement Units (IMU) over the visually homogenous and texture-repetitive terrain of Eastern and Southern Ukraine—presents a confluence of challenges that render traditional Simultaneous Localization and Mapping (SLAM) approaches insufficient. The target environment, characterized by vast agricultural expanses, seasonal variability, and potential conflict-induced terrain alteration, demands a navigation architecture that moves beyond simple visual odometry to a robust, multi-layered Absolute Visual Localization (AVL) system.
This report articulates the design and theoretical validation of **ASTRAL-Next**, a comprehensive architectural framework engineered to supersede the limitations of preliminary dead-reckoning solutions. By synthesizing state-of-the-art (SOTA) research emerging in 2024 and 2025, specifically leveraging **LiteSAM** for efficient cross-view matching 1, **AnyLoc** for universal place recognition 2, and **SuperPoint+LightGlue** for robust sequential tracking 1, the proposed system addresses the critical failure modes inherent in wing-type UAV flight dynamics. These dynamics include sharp banking maneuvers, significant pitch variations leading to ground sampling distance (GSD) disparities, and the potential for catastrophic track loss (the "kidnapped robot" problem).
The analysis indicates that relying solely on sequential image overlap is viable only for short-term trajectory smoothing. The core innovation of ASTRAL-Next lies in its "Hierarchical + Anchor" topology, which decouples the relative motion estimation from absolute global anchoring. This ensures that even during zero-overlap turns or 350-meter positional outliers caused by airframe tilt, the system can re-localize against a pre-cached satellite reference map within the required 5-second latency window.3 Furthermore, the system accounts for the semantic disconnect between live UAV imagery and potentially outdated satellite reference data (e.g., Google Maps) by prioritizing semantic geometry over pixel-level photometric consistency.
### **1.1 Operational Environment and Constraints Analysis**
The operational theater—specifically the left bank of the Dnipro River in Ukraine—imposes rigorous constraints on computer vision algorithms. The absence of IMU data removes the ability to directly sense acceleration and angular velocity, creating a scale ambiguity in monocular vision systems that must be resolved through external priors (altitude) and absolute reference data.
| Constraint Category | Specific Challenge | Implication for System Design |
| :---- | :---- | :---- |
| **Sensor Limitation** | **No IMU Data** | The system cannot distinguish between pure translation and camera rotation (pitch/roll) without visual references. Scale must be constrained via altitude priors and satellite matching.5 |
| **Flight Dynamics** | **Wing-Type UAV** | Unlike quadcopters, fixed-wing aircraft cannot hover. They bank to turn, causing horizon shifts and perspective distortions. "Sharp turns" result in 0% image overlap.6 |
| **Terrain Texture** | **Agricultural Fields** | Repetitive crop rows create aliasing for standard descriptors (SIFT/ORB). Feature matching requires context-aware deep learning methods (SuperPoint).7 |
| **Reference Data** | **Google Maps (2025)** | Public satellite data may be outdated or lower resolution than restricted military feeds. Matches must rely on invariant features (roads, tree lines) rather than ephemeral textures.9 |
| **Compute Hardware** | **NVIDIA RTX 2060/3070** | Algorithms must be optimized for TensorRT to meet the <5s per frame requirement. Heavy transformers (e.g., ViT-Huge) are prohibitive; efficient architectures (LiteSAM) are required.1 |
The confluence of these factors necessitates a move away from simple "dead reckoning" (accumulating relative movements) which drifts exponentially. Instead, ASTRAL-Next operates as a **Global-Local Hybrid System**, where a high-frequency visual odometry layer handles frame-to-frame continuity, while a parallel global localization layer periodically "resets" the drift by anchoring the UAV to the satellite map.
## **2. Architectural Critique of Legacy Approaches**
The initial draft solution ("ASTRAL") and similar legacy approaches typically rely on a unified SLAM pipeline, often attempting to use the same feature extractors for both sequential tracking and global localization. Recent literature highlights substantial deficiencies in this monolithic approach, particularly when applied to the specific constraints of this project.
### **2.1 The Failure of Classical Descriptors in Agricultural Settings**
Classical feature descriptors like SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) rely on detecting "corners" and "blobs" based on local pixel intensity gradients. In the agricultural landscapes of Eastern Ukraine, this approach faces severe aliasing. A field of sunflowers or wheat presents thousands of identical "blobs," causing the nearest-neighbor matching stage to generate a high ratio of outliers.8
Research demonstrates that deep-learning-based feature extractors, specifically SuperPoint, trained on large datasets of synthetic and real-world imagery, learn to identify interest points that are semantically significant (e.g., the intersection of a tractor path and a crop line) rather than just texturally distinct.1 Consequently, a redesign must replace SIFT/ORB with SuperPoint for the front-end tracking.
### **2.2 The Inadequacy of Dead Reckoning without IMU**
In a standard Visual-Inertial Odometry (VIO) system, the IMU provides a high-frequency prediction of the camera's pose, which the visual system then refines. Without an IMU, the system is purely Visual Odometry (VO). In VO, the scale of the world is unobservable from a single camera (monocular scale ambiguity). A 1-meter movement of a small object looks identical to a 10-meter movement of a large object.5
While the prompt specifies a "predefined altitude," relying on this as a static constant is dangerous due to terrain undulations and barometric drift. ASTRAL-Next must implement a Scale-Constrained Bundle Adjustment, treating the altitude not as a hard fact, but as a strong prior that prevents the scale drift common in monocular systems.5
### **2.3 Vulnerability to "Kidnapped Robot" Scenarios**
The requirement to recover from sharp turns where the "next photo doesn't overlap at all" describes the classic "Kidnapped Robot Problem" in robotics—where a robot is teleported to an unknown location and must relocalize.14
Sequential matching algorithms (optical flow, feature tracking) function on the assumption of overlap. When overlap is zero, these algorithms fail catastrophically. The legacy solution's reliance on continuous tracking makes it fragile to these flight dynamics. The redesigned architecture must incorporate a dedicated Global Place Recognition module that treats every frame as a potential independent query against the satellite database, independent of the previous frame's history.2
## **3. ASTRAL-Next: System Architecture and Methodology**
To meet the acceptance criteria—specifically the 80% success rate within 50m error and the <5 second processing time—ASTRAL-Next utilizes a tri-layer processing topology. These layers operate concurrently, feeding into a central state estimator.
### **3.1 The Tri-Layer Localization Strategy**
The architecture separates the concerns of continuity, recovery, and precision into three distinct algorithmic pathways.
| Layer | Functionality | Algorithm | Latency | Role in Acceptance Criteria |
| :---- | :---- | :---- | :---- | :---- |
| **L1: Sequential Tracking** | Frame-to-Frame Relative Pose | **SuperPoint + LightGlue** | \~50-100ms | Handles continuous flight, bridges small gaps (overlap < 5%), and maintains trajectory smoothness. Essential for the 100m spacing requirement. 1 |
| **L2: Global Re-Localization** | "Kidnapped Robot" Recovery | **AnyLoc (DINOv2 + VLAD)** | \~200ms | Detects location after sharp turns (0% overlap) or track loss. Matches current view to the satellite database tile. Addresses the sharp turn recovery criterion. 2 |
| **L3: Metric Refinement** | Precise GPS Anchoring | **LiteSAM / HLoc** | \~300-500ms | "Stitches" the UAV image to the satellite tile with pixel-level accuracy to reset drift. Ensures the "80% < 50m" and "60% < 20m" accuracy targets. 1 |
### **3.2 Data Flow and State Estimation**
The system utilizes a **Factor Graph Optimization** (using libraries like GTSAM) as the central "brain."
1. **Inputs:**
* **Relative Factors:** Provided by Layer 1 (Change in pose from $t-1$ to $t$).
* **Absolute Factors:** Provided by Layer 3 (Global GPS coordinate at $t$).
* **Priors:** Altitude constraint and Ground Plane assumption.
2. **Processing:** The factor graph optimizes the trajectory by minimizing the error between these conflicting constraints.
3. **Output:** A smoothed, globally consistent trajectory $(x, y, z, \\text{roll}, \\text{pitch}, \\text{yaw})$ for every image timestamp.
### **3.3 Atlas Multi-Map Architecture**
ASTRAL-Next implements an **"Atlas" multi-map architecture** where route chunks/fragments are first-class entities, not just recovery mechanisms. This architecture is critical for handling sharp turns (AC-4) and disconnected route segments (AC-5).
**Core Principles:**
- **Chunks are the primary unit of operation**: When tracking is lost (sharp turn, 350m outlier), the system immediately creates a new chunk and continues processing.
- **Proactive chunk creation**: Chunks are created proactively on tracking loss, not reactively after matching failures.
- **Independent chunk processing**: Each chunk has its own subgraph in the factor graph, optimized independently with local consistency.
- **Chunk matching and merging**: Unanchored chunks are matched semantically (aggregate DINOv2 features) and with LiteSAM (with rotation sweeps), then merged into the global trajectory via Sim(3) similarity transformation.
**Chunk Lifecycle:**
1. **Chunk Creation**: On tracking loss, new chunk created immediately in factor graph.
2. **Chunk Building**: Frames processed within chunk using sequential VO, factors added to chunk's subgraph.
3. **Chunk Matching**: When chunk ready (5-20 frames), semantic matching attempted (aggregate DINOv2 descriptor more robust than single-image).
4. **Chunk LiteSAM Matching**: Candidate tiles matched with LiteSAM, rotation sweeps handle unknown orientation from sharp turns.
5. **Chunk Merging**: Successful matches anchor chunks, which are merged into global trajectory via Sim(3) transform (translation, rotation, scale).
**Benefits:**
- System never "fails" - it fragments and continues processing.
- Chunk semantic matching succeeds where single-image matching fails (featureless terrain).
- Multiple chunks can exist simultaneously and be matched/merged asynchronously.
- Reduces user input requests by 50-70% in challenging scenarios.
### **3.4 REST API Background Service Architecture**
As per the requirement, the system operates as a background service exposed via a REST API.
* **Communication Pattern:** The service utilizes **REST API endpoints** (FastAPI) for all control operations and **Server-Sent Events (SSE)** for real-time streaming of localization results. This architecture provides:
* **REST Endpoints:** `POST /flights` (create flight), `GET /flights/{id}` (status), `POST /flights/{id}/images/batch` (upload images), `POST /flights/{id}/user-fix` (human-in-the-loop input)
* **SSE Streaming:** `GET /flights/{id}/stream` provides continuous, real-time updates of frame processing results, refinements, and status changes
* **Standard HTTP/HTTPS:** Enables easy integration with web clients, mobile apps, and existing infrastructure without requiring specialized messaging libraries
* **Concurrency:** Layer 1 runs on a high-priority thread to ensure immediate feedback. Layers 2 and 3 run asynchronously; when a global match is found, the result is injected into the Factor Graph, which then "back-propagates" the correction to previous frames, refining the entire recent trajectory. Results are immediately pushed via SSE to connected clients.
* **Future Enhancement:** For multi-client online SaaS deployments, ZeroMQ (PUB-SUB pattern) can be added as an alternative transport layer to support high-throughput, multi-tenant scenarios with lower latency and better scalability than HTTP-based SSE.
## **4. Layer 1: Robust Sequential Visual Odometry**
The first line of defense against localization loss is robust tracking between consecutive UAV images. Given the challenging agricultural environment, standard feature matching is prone to failure. ASTRAL-Next employs **SuperPoint** and **LightGlue**.
### **4.1 SuperPoint: Semantic Feature Detection**
SuperPoint is a fully convolutional neural network trained to detect interest points and compute their descriptors. Unlike SIFT, which uses handcrafted mathematics to find corners, SuperPoint is trained via self-supervision on millions of images.
* **Relevance to Ukraine:** In a wheat field, SIFT might latch onto hundreds of identical wheat stalks. SuperPoint, however, learns to prioritize more stable features, such as the boundary between the field and a dirt road, or a specific patch of discoloration in the crop canopy.1
* **Performance:** SuperPoint runs efficiently on the RTX 2060/3070, with inference times around 15ms per image when optimized with TensorRT.16
### **4.2 LightGlue: The Attention-Based Matcher**
**LightGlue** represents a paradigm shift from the traditional "Nearest Neighbor + RANSAC" matching pipeline. It is a deep neural network that takes two sets of SuperPoint features and jointly predicts the matches.
* **Mechanism:** LightGlue uses a transformer-based attention mechanism. It allows features in Image A to "look at" all features in Image B (and vice versa) to determine the best correspondence. Crucially, it has a "dustbin" mechanism to explicitly reject points that have no match (occlusion or field of view change).12
* **Addressing the <5% Overlap:** The user specifies handling overlaps of "less than 5%." Traditional RANSAC fails here because the inlier ratio is too low. LightGlue, however, can confidently identify the few remaining matches because its attention mechanism considers the global geometric context of the points. If only a single road intersection is visible in the corner of both images, LightGlue is significantly more likely to match it correctly than SIFT.8
* **Efficiency:** LightGlue is designed to be "light." It features an adaptive depth mechanism—if the images are easy to match, it exits early. If they are hard (low overlap), it uses more layers. This adaptability is perfect for the variable difficulty of the UAV flight path.19
## **5. Layer 2: Global Place Recognition (The "Kidnapped Robot" Solver)**
When the UAV executes a sharp turn, resulting in a completely new view (0% overlap), sequential tracking (Layer 1) is mathematically impossible. The system must recognize the new terrain solely based on its appearance. This is the domain of **AnyLoc**.
### **5.1 Universal Place Recognition with Foundation Models**
**AnyLoc** leverages **DINOv2**, a massive self-supervised vision transformer developed by Meta. DINOv2 is unique because it is not trained with labels; it is trained to understand the geometry and semantic layout of images.
* **Why DINOv2 for Satellite Matching:** Satellite images and UAV images have different "domains." The satellite image might be from summer (green), while the UAV flies in autumn (brown). DINOv2 features are remarkably invariant to these texture changes. It "sees" the shape of the road network or the layout of the field boundaries, rather than the color of the leaves.2
* **VLAD Aggregation:** AnyLoc extracts dense features from the image using DINOv2 and aggregates them using **VLAD** (Vector of Locally Aggregated Descriptors) into a single, compact vector (e.g., 4096 dimensions). This vector represents the "fingerprint" of the location.21
### **5.2 Implementation Strategy**
1. **Database Preparation:** Before the mission, the system downloads the satellite imagery for the operational bounding box (Eastern/Southern Ukraine). These images are tiled (e.g., 512x512 pixels with overlap) and processed through AnyLoc to generate a database of descriptors.
2. **Faiss Indexing:** These descriptors are indexed using **Faiss**, a library for efficient similarity search.
3. **In-Flight Retrieval:** When Layer 1 reports a loss of tracking (or periodically), the current UAV image is processed by AnyLoc. The resulting vector is queried against the Faiss index.
4. **Result:** The system retrieves the top-5 most similar satellite tiles. These tiles represent the coarse global location of the UAV (e.g., "You are in Grid Square B7").2
### **5.3 Chunk-Based Processing**
When semantic matching fails on featureless terrain (plain agricultural fields), the system employs chunk-based processing as a more robust recovery strategy.
**Chunk Semantic Matching:**
- **Aggregate DINOv2 Features**: Instead of matching a single image, the system builds a route chunk (5-20 frames) using sequential VO and computes an aggregate DINOv2 descriptor from all chunk images.
- **Robustness**: Aggregate descriptors are more robust to featureless terrain where single-image matching fails. Multiple images provide more context and reduce false matches.
- **Implementation**: DINOv2 descriptors from all chunk images are aggregated (mean, VLAD, or max pooling) and queried against the Faiss index.
**Chunk LiteSAM Matching:**
- **Rotation Sweeps**: When matching chunks, the system rotates the entire chunk to all possible angles (0°, 30°, 60°, ..., 330°) because sharp turns change orientation and previous heading may not be relevant.
- **Aggregate Correspondences**: LiteSAM matches the entire chunk to satellite tiles, aggregating correspondences from multiple images for more robust matching.
- **Sim(3) Transform**: Successful matches provide Sim(3) transformation (translation, rotation, scale) for merging chunks into the global trajectory.
**Normal Operation:**
- Frames are processed within an active chunk context.
- Relative factors are added to the chunk's subgraph (not global graph).
- Chunks are optimized independently for local consistency.
- When chunks are anchored (GPS found), they are merged into the global trajectory.
**Chunk Merging:**
- Chunks are merged using Sim(3) similarity transformation, accounting for translation, rotation, and scale differences.
- This is critical for monocular VO where scale ambiguity exists.
- Merged chunks maintain global consistency while preserving internal consistency.
## **6. Layer 3: Fine-Grained Metric Localization (LiteSAM)**
Retrieving the correct satellite tile (Layer 2) gives a location error of roughly the tile size (e.g., 200 meters). To meet the "60% < 20m" and "80% < 50m" criteria, the system must precisely align the UAV image onto the satellite tile. ASTRAL-Next utilizes **LiteSAM**.
### **6.1 Justification for LiteSAM over TransFG**
While **TransFG** (Transformer for Fine-Grained recognition) is a powerful architecture for cross-view geo-localization, it is computationally heavy.23 **LiteSAM** (Lightweight Satellite-Aerial Matching) is specifically architected for resource-constrained platforms (like UAV onboard computers or efficient ground stations) while maintaining state-of-the-art accuracy.
* **Architecture:** LiteSAM utilizes a **Token Aggregation-Interaction Transformer (TAIFormer)**. It employs a convolutional token mixer (CTM) to model correlations between the UAV and satellite images.
* **Multi-Scale Processing:** LiteSAM processes features at multiple scales. This is critical because the UAV altitude varies (<1km), meaning the scale of objects in the UAV image will not perfectly match the fixed scale of the satellite image (Google Maps Zoom Level 19). LiteSAM's multi-scale approach inherently handles this discrepancy.1
* **Performance Data:** Empirical benchmarks on the **UAV-VisLoc** dataset show LiteSAM achieving an RMSE@30 (Root Mean Square Error within 30 meters) of 17.86 meters, directly supporting the project's accuracy requirements. Its inference time is approximately 61.98ms on standard GPUs, ensuring it fits within the overall 5-second budget.1
### **6.2 The Alignment Process**
1. **Input:** The UAV Image and the Top-1 Satellite Tile from Layer 2.
2. **Processing:** LiteSAM computes the dense correspondence field between the two images.
3. **Homography Estimation:** Using the correspondences, the system computes a homography matrix $H$ that maps pixels in the UAV image to pixels in the georeferenced satellite tile.
4. **Pose Extraction:** The camera's absolute GPS position is derived from this homography, utilizing the known GSD of the satellite tile.18
## **7. Satellite Data Management and Coordinate Systems**
The reliability of the entire system hinges on the quality and handling of the reference map data. The restriction to "Google Maps" necessitates a rigorous approach to coordinate transformation and data freshness management.
### **7.1 Google Maps Static API and Mercator Projection**
The Google Maps Static API delivers images without embedded georeferencing metadata (GeoTIFF tags). The system must mathematically derive the bounding box of each downloaded tile to assign coordinates to the pixels. Google Maps uses the **Web Mercator Projection (EPSG:3857)**.
The system must implement the following derivation to establish the **Ground Sampling Distance (GSD)**, or meters_per_pixel, which varies significantly with latitude:
$$ \\text{meters_per_pixel} = 156543.03392 \\times \\frac{\\cos(\\text{latitude} \\times \\frac{\\pi}{180})}{2^{\\text{zoom}}} $$
For the operational region (Ukraine, approx. Latitude 48N):
* At **Zoom Level 19**, the resolution is approximately 0.30 meters/pixel. This resolution is compatible with the input UAV imagery (Full HD at <1km altitude), providing sufficient detail for the LiteSAM matcher.24
**Bounding Box Calculation Algorithm:**
1. **Input:** Center Coordinate $(lat, lon)$, Zoom Level ($z$), Image Size $(w, h)$.
2. **Project to World Coordinates:** Convert $(lat, lon)$ to world pixel coordinates $(px, py)$ at the given zoom level.
3. **Corner Calculation:**
* px_{NW} = px - (w / 2)
* py_{NW} = py - (h / 2)
4. Inverse Projection: Convert $(px_{NW}, py_{NW})$ back to Latitude/Longitude to get the North-West corner. Repeat for South-East.
This calculation is critical. A precision error here translates directly to a systematic bias in the final GPS output.
### **7.2 Mitigating Data Obsolescence (The 2025 Problem)**
The provided research highlights that satellite imagery access over Ukraine is subject to restrictions and delays (e.g., Maxar restrictions in 2025).10 Google Maps data may be several years old.
* **Semantic Anchoring:** This reinforces the selection of **AnyLoc** (Layer 2) and **LiteSAM** (Layer 3). These algorithms are trained to ignore transient features (cars, temporary structures, vegetation color) and focus on persistent structural features (road geometry, building footprints).
* **Seasonality:** Research indicates that DINOv2 features (used in AnyLoc) exhibit strong robustness to seasonal changes (e.g., winter satellite map vs. summer UAV flight), maintaining high retrieval recall where pixel-based methods fail.17
## **8. Optimization and State Estimation (The "Brain")**
The individual outputs of the visual layers are noisy. Layer 1 drifts over time; Layer 3 may have occasional outliers. The **Factor Graph Optimization** fuses these inputs into a coherent trajectory.
### **8.1 Handling the 350-Meter Outlier (Tilt)**
The prompt specifies that "up to 350 meters of an outlier... could happen due to tilt." This large displacement masquerading as translation is a classic source of divergence in Kalman Filters.
* **Robust Cost Functions:** In the Factor Graph, the error terms for the visual factors are wrapped in a **Robust Kernel** (specifically the **Cauchy** or **Huber** kernel).
* *Mechanism:* Standard least-squares optimization penalizes errors quadratically ($e^2$). If a 350m error occurs, the penalty is massive, dragging the entire trajectory off-course. A robust kernel changes the penalty to be linear ($|e|$) or logarithmic after a certain threshold. This allows the optimizer to effectively "ignore" or down-weight the 350m jump if it contradicts the consensus of other measurements, treating it as a momentary outlier or solving for it as a rotation rather than a translation.19
### **8.2 The Altitude Soft Constraint**
To resolve the monocular scale ambiguity without IMU, the altitude ($h_{prior}$) is added as a **Unary Factor** to the graph.
* $E_{alt} = |
| z_{est} \- h_{prior} ||*{\\Sigma*{alt}}$
* $\\Sigma_{alt}$ (covariance) is set relatively high (soft constraint), allowing the visual odometry to adjust the altitude slightly to maintain consistency, but preventing the scale from collapsing to zero or exploding to infinity. This effectively creates an **Altimeter-Aided Monocular VIO** system, where the altimeter (virtual or barometric) replaces the accelerometer for scale determination.5
## **9. Implementation Specifications**
### **9.1 Hardware Acceleration (TensorRT)**
Meeting the <5 second per frame requirement on an RTX 2060 requires optimizing the deep learning models. Python/PyTorch inference is typically too slow due to overhead.
* **Model Export:** All core models (SuperPoint, LightGlue, LiteSAM) must be exported to **ONNX** (Open Neural Network Exchange) format.
* **TensorRT Compilation:** The ONNX models are then compiled into **TensorRT Engines**. This process performs graph fusion (combining multiple layers into one) and kernel auto-tuning (selecting the fastest GPU instructions for the specific RTX 2060/3070 architecture).26
* **Precision:** The models should be quantized to **FP16** (16-bit floating point). Research shows that FP16 inference on NVIDIA RTX cards offers a 2x-3x speedup with negligible loss in matching accuracy for these specific networks.16
### **9.2 Background Service Architecture (REST API + SSE)**
The system is encapsulated as a headless service exposed via REST API.
**REST API Architecture:**
* **FastAPI Framework:** Modern, high-performance Python web framework with automatic OpenAPI documentation
* **REST Endpoints:**
* `POST /flights` - Create flight with initial configuration (start GPS, camera params, altitude)
* `GET /flights/{flightId}` - Retrieve flight status and waypoints
* `POST /flights/{flightId}/images/batch` - Upload batch of 10-50 images for processing
* `POST /flights/{flightId}/user-fix` - Submit human-in-the-loop GPS anchor when system requests input
* `GET /flights/{flightId}/stream` - SSE stream for real-time frame results
* `DELETE /flights/{flightId}` - Cancel/delete flight
* **SSE Streaming:** Server-Sent Events provide real-time updates:
* Frame processing results: `{"event": "frame_processed", "data": {"frame_id": 1024, "gps": [48.123, 37.123], "confidence": 0.98}}`
* Refinement updates: `{"event": "frame_refined", "data": {"frame_id": 1000, "gps": [48.120, 37.120]}}`
* Status changes: `{"event": "status", "data": {"status": "REQ_INPUT", "message": "User input required"}}`
**Asynchronous Pipeline:**
The system utilizes a Python multiprocessing architecture. One process handles the REST API server and SSE streaming. A second process hosts the TensorRT engines and runs the Factor Graph. This ensures that the heavy computation of Bundle Adjustment does not block the receipt of new images or user commands. Results are immediately pushed to connected SSE clients.
**Future Multi-Client SaaS Enhancement:**
For production deployments requiring multiple concurrent clients and higher throughput, ZeroMQ can be added as an alternative transport layer:
* **ZeroMQ PUB-SUB:** For high-frequency result streaming to multiple subscribers
* **ZeroMQ REQ-REP:** For low-latency command/response patterns
* **Hybrid Approach:** REST API for control operations, ZeroMQ for data streaming in multi-tenant scenarios
## **10. Human-in-the-Loop Strategy**
The requirement stipulates that for the "20% of the route" where automation fails, the user must intervene. The system must proactively detect its own failure.
### **10.1 Failure Detection and Recovery Stages**
The system monitors the **PDM@K** (Positioning Distance Measurement) metric continuously.
* **Definition:** PDM@K measures the percentage of queries localized within $K$ meters.3
* **Real-Time Proxy:** In flight, we cannot know the true PDM (as we don't have ground truth). Instead, we use the **Marginal Covariance** from the Factor Graph. If the uncertainty ellipse for the current position grows larger than a radius of 50 meters, or if the **Image Registration Rate** (percentage of inliers in LightGlue/LiteSAM) drops below 10% for 3 consecutive frames, the system triggers a **Critical Failure Mode**.19
**Recovery Stages:**
1. **Stage 1: Progressive Tile Search (Single Image)**
- Attempts single-image semantic matching (DINOv2) and LiteSAM matching.
- Progressive tile grid expansion (1→4→9→16→25 tiles).
- Fast recovery for transient tracking loss.
2. **Stage 2: Chunk Building and Semantic Matching (Proactive)**
- **Immediately creates new chunk** when tracking lost (proactive, not reactive).
- Continues processing frames, building chunk with sequential VO.
- When chunk ready (5-20 frames), attempts chunk semantic matching.
- Aggregate DINOv2 descriptor more robust than single-image matching.
- Handles featureless terrain where single-image matching fails.
3. **Stage 3: Chunk LiteSAM Matching with Rotation Sweeps**
- After chunk semantic matching succeeds, attempts LiteSAM matching.
- Rotates entire chunk to all angles (0°, 30°, ..., 330°) for matching.
- Critical for sharp turns where orientation unknown.
- Aggregate correspondences from multiple images for robustness.
4. **Stage 4: User Input (Last Resort)**
- Only triggered if all chunk matching strategies fail.
- System requests user-provided GPS anchor.
- User anchor applied as hard constraint, processing resumes.
### **10.2 The User Interaction Workflow**
1. **Trigger:** Critical Failure Mode activated.
2. **Action:** The Service sends an SSE event `{"event": "user_input_needed", "data": {"status": "REQ_INPUT", "frame_id": 1024}}` to connected clients.
3. **Data Payload:** The client retrieves the current UAV image and top-3 retrieved satellite tiles via `GET /flights/{flightId}/frames/{frameId}/context` endpoint.
4. **User Input:** The user clicks a distinctive feature (e.g., a specific crossroad) in the UAV image and the corresponding point on the satellite map, then submits via `POST /flights/{flightId}/user-fix` with the GPS coordinate.
5. **Recovery:** This GPS coordinate is treated as a **Hard Constraint** in the Factor Graph. The optimizer immediately snaps the trajectory to this user-defined anchor, resetting the covariance and effectively "healing" the localized track. An SSE event confirms the recovery: `{"event": "user_fix_applied", "data": {"frame_id": 1024, "status": "PROCESSING"}}`.19
## **11. Performance Evaluation and Benchmarks**
### **11.1 Accuracy Validation**
Based on the reported performance of the selected components in relevant datasets (UAV-VisLoc, AnyVisLoc):
* **LiteSAM** demonstrates an accuracy of 17.86m (RMSE) for cross-view matching. This aligns with the requirement that 60% of photos be within 20m error.18
* **AnyLoc** achieves high recall rates (Top-1 Recall > 85% on aerial benchmarks), supporting the recovery from sharp turns.2
* **Factor Graph Fusion:** By combining sequential and global measurements, the overall system error is expected to be lower than the individual component errors, satisfying the "80% within 50m" criterion.
### **11.2 Latency Analysis**
The breakdown of processing time per frame on an RTX 3070 is estimated as follows:
* **SuperPoint + LightGlue:** \~50ms.1
* **AnyLoc (Global Retrieval):** \~150ms (run only on keyframes or tracking loss).
* **LiteSAM (Metric Refinement):** \~60ms.1
* **Factor Graph Optimization:** \~100ms (using incremental updates/iSAM2).
* Total: \~360ms per frame (worst case with all layers active).
This is an order of magnitude faster than the 5-second limit, providing ample headroom for higher resolution processing or background tasks.
## **12.0 ASTRAL-Next Validation Plan and Acceptance Criteria Matrix**
A comprehensive test plan is required to validate compliance with all 10 Acceptance Criteria. The foundation is a **Ground-Truth Test Harness** using project-provided ground-truth data.
### **Table 4: ASTRAL Component vs. Acceptance Criteria Compliance Matrix**
| ID | Requirement | ASTRAL Solution (Component) | Key Technology / Justification |
| :---- | :---- | :---- | :---- |
| **AC-1** | 80% of photos < 50m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Tier-1 (Google Maps)** data 1 is sufficient. SuperPoint + LightGlue + LiteSAM + Sim(3) graph 13 can achieve this. |
| **AC-2** | 60% of photos < 20m error | GDB (C-1) + GAB (C-5) + TOH (C-6) | **Requires Tier-2 (Commercial) Data**.4 Mitigates reference error.3 **Per-Keyframe Scale** 15 model in TOH minimizes drift error. |
| **AC-3** | Robust to 350m outlier | V-SLAM (C-3) + TOH (C-6) | **Stage 2 Failure Logic** (7.3) discards the frame. **Robust M-Estimation** (6.3) in Ceres 14 automatically rejects the constraint. |
| **AC-4** | Robust to sharp turns (<5% overlap) | V-SLAM (C-3) + TOH (C-6) | **"Atlas" Multi-Map** (4.2) initializes new map (Map_Fragment_k+1). **Geodetic Map-Merging** (6.4) in TOH re-connects fragments via GAB anchors. |
| **AC-5** | < 10% outlier anchors | TOH (C-6) | **Robust M-Estimation (Huber Loss)** (6.3) in Ceres 14 automatically down-weights and ignores high-residual (bad) GAB anchors. |
| **AC-6** | Connect route chunks; User input | V-SLAM (C-3) + TOH (C-6) + UI | **Geodetic Map-Merging** (6.4) connects chunks. **Stage 5 Failure Logic** (7.3) provides the user-input-as-prior mechanism. |
| **AC-7** | < 5 seconds processing/image | All Components | **Multi-Scale Pipeline** (5.3) (Low-Res V-SLAM, Hi-Res GAB patches). **Mandatory TensorRT Acceleration** (7.1) for 2-4x speedup.35 |
| **AC-8** | Real-time stream + async refinement | TOH (C-5) + Outputs (C-2.4) | Decoupled architecture provides Pose_N_Est (V-SLAM) in real-time and Pose_N_Refined (TOH) asynchronously as GAB anchors arrive. |
| **AC-9** | Image Registration Rate > 95% | V-SLAM (C-3) | **"Atlas" Multi-Map** (4.2). A "lost track" (AC-4) is *not* a registration failure; it's a *new map registration*. This ensures the rate > 95%. |
| **AC-10** | Mean Reprojection Error (MRE) < 1.0px | V-SLAM (C-3) + TOH (C-6) | Local BA (4.3) + Global BA (TOH14) + **Per-Keyframe Scale** (6.2) minimizes internal graph tension (Flaw 1.3), allowing the optimizer to converge to a low MRE. |
### **12.1 Rigorous Validation Methodology**
* **Test Harness:** A validation script will be created to compare the system's Pose_N^{Refined} output against a ground-truth coordinates.csv file, computing Haversine distance errors.
* **Test Datasets:**
* Test_Baseline: Standard flight.
* Test_Outlier_350m (AC-3): A single, unrelated image inserted.
* Test_Sharp_Turn_5pct (AC-4): A sequence with a 10-frame gap.
* Test_Long_Route (AC-9, AC-7): A 2000-image sequence.
* **Test Cases:**
* Test_Accuracy: Run Test_Baseline. ASSERT (count(errors < 50m) / total) >= 0.80 (AC-1). ASSERT (count(errors < 20m) / total) >= 0.60 (AC-2).
* Test_Robustness: Run Test_Outlier_350m and Test_Sharp_Turn_5pct. ASSERT system completes the run and Test_Accuracy assertions still pass on the valid frames.
* Test_Performance: Run Test_Long_Route on min-spec RTX 2060. ASSERT average_time(Pose_N^{Est} output) < 5.0s (AC-7).
* Test_MRE: ASSERT TOH.final_MRE < 1.0 (AC-10).
_docs/02_components/01_flight_api/01.01_feature_flight_management.md
-80
View File
@@ -1,80 +0,0 @@
# Feature: Flight Management
## Description
Core REST endpoints for flight lifecycle management including CRUD operations, status retrieval, and waypoint management. This feature provides the fundamental data operations for flight entities.
## Component APIs Implemented
- `create_flight(flight_data: FlightCreateRequest) -> FlightResponse`
- `get_flight(flight_id: str) -> FlightDetailResponse`
- `delete_flight(flight_id: str) -> DeleteResponse`
- `get_flight_status(flight_id: str) -> FlightStatusResponse`
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse`
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights` | Create new flight |
| GET | `/flights/{flightId}` | Get flight details |
| DELETE | `/flights/{flightId}` | Delete flight |
| GET | `/flights/{flightId}/status` | Get processing status |
| PUT | `/flights/{flightId}/waypoints/{waypointId}` | Update single waypoint |
| PUT | `/flights/{flightId}/waypoints/batch` | Batch update waypoints |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **Pydantic**: Request/response validation and serialization
- **Uvicorn**: ASGI server
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_gps_coordinates(lat, lon)` | Validate GPS coordinate ranges |
| `_validate_camera_params(params)` | Validate camera parameter values |
| `_validate_geofences(geofences)` | Validate geofence polygon data |
| `_build_flight_response(flight_data)` | Build response from F02 result |
| `_build_status_response(status_data)` | Build status response |
## Unit Tests
1. **create_flight validation**
- Valid request → calls F02.create_flight() and returns 201
- Missing required field (name) → returns 400
- Invalid GPS (lat > 90) → returns 400
- Invalid camera params → returns 400
2. **get_flight**
- Valid flight_id → returns 200 with flight data
- Non-existent flight_id → returns 404
3. **delete_flight**
- Valid flight_id → returns 200
- Non-existent flight_id → returns 404
- Processing flight → returns 409
4. **get_flight_status**
- Valid flight_id → returns 200 with status
- Non-existent flight_id → returns 404
5. **update_waypoint**
- Valid update → returns 200
- Invalid waypoint_id → returns 404
- Invalid coordinates → returns 400
6. **batch_update_waypoints**
- Valid batch → returns success with updated_count
- Empty batch → returns success, updated_count=0
- Partial failures → returns failed_ids
## Integration Tests
1. **Flight lifecycle**
- POST /flights → GET /flights/{id} → DELETE /flights/{id}
- Verify data consistency across operations
2. **Waypoint updates during processing**
- Create flight → Process frames → Verify waypoints updated
- Simulate refinement updates → Verify refined=true
3. **Concurrent operations**
- Create 10 flights concurrently → All succeed
- Batch update 500 waypoints → All succeed
_docs/02_components/01_flight_api/01.02_feature_image_upload.md
-70
View File
@@ -1,70 +0,0 @@
# Feature: Image Batch Upload
## Description
REST endpoint for uploading batches of UAV images for processing. Handles multipart form data with 10-50 images per request, validates sequence numbers, and queues images for async processing via F02.
## Component APIs Implemented
- `upload_image_batch(flight_id: str, batch: ImageBatch) -> BatchResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights/{flightId}/images/batch` | Upload batch of 10-50 images |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **python-multipart**: Multipart form data handling
- **Pydantic**: Validation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_batch_size(images)` | Validate batch contains 10-50 images |
| `_validate_sequence_numbers(metadata)` | Validate start/end sequence are valid |
| `_validate_sequence_continuity(flight_id, start_seq)` | Validate sequence continues from last batch |
| `_parse_multipart_images(request)` | Parse multipart form data into image list |
| `_validate_image_format(image)` | Validate image file is valid JPEG/PNG |
| `_build_batch_response(result)` | Build response with accepted sequences |
## Unit Tests
1. **Batch size validation**
- 20 images → accepted
- 9 images → returns 400 (too few)
- 51 images → returns 400 (too many)
2. **Sequence validation**
- Valid sequence (start=100, end=119) → accepted
- Invalid sequence (start > end) → returns 400
- Gap in sequence → returns 400
3. **Flight validation**
- Valid flight_id → accepted
- Non-existent flight_id → returns 404
4. **Image validation**
- Valid JPEG images → accepted
- Corrupted image → returns 400
- Non-image file → returns 400
5. **Size limits**
- 50 × 2MB images → accepted
- Batch > 500MB → returns 413
## Integration Tests
1. **Sequential batch uploads**
- Upload batch 1 (seq 0-49) → success
- Upload batch 2 (seq 50-99) → success
- Verify next_expected increments correctly
2. **Large image handling**
- Upload 50 × 8MB images → success within timeout
- Verify all images queued for processing
3. **Rate limiting**
- Rapid consecutive uploads → eventually returns 429
- Wait and retry → succeeds
4. **Concurrent uploads to same flight**
- Two clients upload different batches → both succeed
- Verify sequence integrity maintained
_docs/02_components/01_flight_api/01.03_feature_user_interaction.md
-68
View File
@@ -1,68 +0,0 @@
# Feature: User Interaction
## Description
REST endpoints for user-triggered operations: submitting GPS fixes for blocked flights and converting detected object pixel coordinates to GPS. These endpoints support the human-in-the-loop workflow when automated localization fails.
## Component APIs Implemented
- `submit_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResponse`
- `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights/{flightId}/user-fix` | Submit user-provided GPS anchor |
| POST | `/flights/{flightId}/frames/{frameId}/object-to-gps` | Convert pixel to GPS |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **Pydantic**: Request/response validation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_user_fix_request(fix_data)` | Validate pixel and GPS coordinates |
| `_validate_flight_blocked(flight_id)` | Verify flight is in blocked state |
| `_validate_frame_processed(flight_id, frame_id)` | Verify frame has pose in Factor Graph |
| `_validate_pixel_coordinates(pixel, resolution)` | Validate pixel within image bounds |
| `_build_user_fix_response(result)` | Build response with processing status |
| `_build_object_gps_response(result)` | Build GPS response with accuracy |
## Unit Tests
1. **submit_user_fix validation**
- Valid request for blocked flight → returns 200, processing_resumed=true
- Flight not blocked → returns 409
- Invalid GPS coordinates → returns 400
- Non-existent flight_id → returns 404
2. **submit_user_fix pixel validation**
- Pixel within image bounds → accepted
- Negative pixel coordinates → returns 400
- Pixel outside image bounds → returns 400
3. **convert_object_to_gps validation**
- Valid processed frame → returns GPS with accuracy
- Frame not yet processed → returns 409
- Non-existent frame_id → returns 404
- Invalid pixel coordinates → returns 400
4. **convert_object_to_gps accuracy**
- High confidence frame → low accuracy_meters
- Low confidence frame → high accuracy_meters
## Integration Tests
1. **User fix unblocks processing**
- Process until blocked → Submit user fix → Verify processing resumes
- Verify SSE `processing_resumed` event sent
2. **Object-to-GPS workflow**
- Process flight → Call object-to-gps for multiple pixels
- Verify GPS coordinates are spatially consistent
3. **User fix with invalid anchor**
- Submit fix with GPS far outside geofence
- Verify appropriate error handling
4. **Concurrent object-to-gps calls**
- Multiple clients request conversion simultaneously
- All receive correct responses
_docs/02_components/01_flight_api/01.04_feature_sse_streaming.md
-88
View File
@@ -1,88 +0,0 @@
# Feature: SSE Streaming
## Description
Server-Sent Events (SSE) endpoint for real-time streaming of flight processing results to clients. Manages SSE connections, handles keepalive, and supports client reconnection with event replay.
## Component APIs Implemented
- `create_sse_stream(flight_id: str) -> SSEStream`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/flights/{flightId}/stream` | Open SSE connection |
## SSE Events
| Event Type | Description |
|------------|-------------|
| `frame_processed` | New frame GPS calculated |
| `frame_refined` | Previous frame GPS refined |
| `search_expanded` | Search grid expanded |
| `user_input_needed` | User input required |
| `processing_blocked` | Processing halted |
| `flight_completed` | Flight processing finished |
## External Tools and Services
- **FastAPI**: Web framework with SSE support
- **sse-starlette**: SSE event streaming library
- **asyncio**: Async event handling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_flight_exists(flight_id)` | Verify flight exists before connecting |
| `_create_sse_response(flight_id)` | Create SSE StreamingResponse |
| `_event_generator(flight_id, client_id)` | Async generator yielding events |
| `_handle_client_disconnect(client_id)` | Cleanup on client disconnect |
| `_send_keepalive()` | Send ping every 30 seconds |
| `_replay_missed_events(last_event_id)` | Replay events since last_event_id |
| `_format_sse_event(event_type, data)` | Format event for SSE protocol |
## Unit Tests
1. **Connection establishment**
- Valid flight_id → SSE connection opens
- Non-existent flight_id → returns 404
- Invalid flight_id format → returns 400
2. **Event formatting**
- frame_processed event → correct JSON structure
- All event types → valid SSE format with id, event, data
3. **Keepalive**
- No events for 30s → ping sent
- Connection stays alive during idle periods
4. **Client disconnect handling**
- Client closes connection → resources cleaned up
- No memory leaks on disconnect
## Integration Tests
1. **Full event flow**
- Connect to stream → Upload images → Receive all frame_processed events
- Verify event count matches frames processed
2. **Event ordering**
- Process 100 frames → events received in order
- Event IDs are sequential
3. **Reconnection with replay**
- Connect → Process 50 frames → Disconnect → Reconnect with last_event_id
- Receive missed events since last_event_id
4. **user_input_needed flow**
- Process until blocked → Receive user_input_needed event
- Submit user fix → Receive processing_resumed confirmation
5. **Multiple concurrent clients**
- 10 clients connect to same flight
- All receive same events
- No cross-contamination between flights
6. **Long-running stream**
- Stream 2000 frames over extended period
- Connection remains stable
- All events delivered
7. **Flight completion**
- Process all frames → Receive flight_completed event
- Stream closes gracefully
_docs/02_components/01_flight_api/01._component_light_api.md
-704
View File
@@ -1,704 +0,0 @@
# Flight API
## Interface Definition
**Interface Name**: `IFlightAPI`
### Interface Methods
```python
class IFlightAPI(ABC):
@abstractmethod
def create_flight(self, flight_data: FlightCreateRequest) -> FlightResponse:
pass
@abstractmethod
def get_flight(self, flight_id: str) -> FlightDetailResponse:
pass
@abstractmethod
def delete_flight(self, flight_id: str) -> DeleteResponse:
pass
@abstractmethod
def update_waypoint(self, flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse:
pass
@abstractmethod
def batch_update_waypoints(self, flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse:
pass
@abstractmethod
def upload_image_batch(self, flight_id: str, batch: ImageBatch) -> BatchResponse:
pass
@abstractmethod
def submit_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResponse:
pass
@abstractmethod
def get_flight_status(self, flight_id: str) -> FlightStatusResponse:
pass
@abstractmethod
def create_sse_stream(self, flight_id: str) -> SSEStream:
pass
@abstractmethod
def convert_object_to_gps(self, flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse:
pass
```
## Component Description
### Responsibilities
- Expose REST API endpoints for complete flight lifecycle management
- Handle flight CRUD operations (create, read, update, delete)
- Manage waypoints and geofences within flights
- Handle satellite data prefetching on flight creation
- Accept batch image uploads (10-50 images per request)
- Accept user-provided GPS fixes for blocked flights
- Provide real-time status updates
- Stream results via Server-Sent Events (SSE)
### Scope
- FastAPI-based REST endpoints
- Request/response validation
- Coordinate with Flight Processor for all operations
- Multipart form data handling for image uploads
- SSE connection management
- Authentication and rate limiting
---
## Flight Management Endpoints
### `create_flight(flight_data: FlightCreateRequest) -> FlightResponse`
**REST Endpoint**: `POST /flights`
**Description**: Creates a new flight with initial waypoints, geofences, camera parameters, and triggers satellite data prefetching.
**Called By**:
- Client applications (Flight UI, Mission Planner UI)
**Input**:
```python
FlightCreateRequest:
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
**Output**:
```python
FlightResponse:
flight_id: str
status: str # "prefetching", "ready", "error"
message: Optional[str]
created_at: datetime
```
**Processing Flow**:
1. Validate request data
2. Call F02 Flight Processor → create_flight()
3. Flight Processor triggers satellite prefetch
4. Return flight_id immediately (prefetch is async)
**Error Conditions**:
- `400 Bad Request`: Invalid input data (missing required fields, invalid GPS coordinates)
- `409 Conflict`: Flight with same ID already exists
- `500 Internal Server Error`: Database or internal error
**Test Cases**:
1. **Valid flight creation**: Provide valid flight data → returns 201 with flight_id
2. **Missing required field**: Omit name → returns 400 with error message
3. **Invalid GPS coordinates**: Provide lat > 90 → returns 400
4. **Concurrent flight creation**: Multiple flights → all succeed
---
### `get_flight(flight_id: str) -> FlightDetailResponse`
**REST Endpoint**: `GET /flights/{flightId}`
**Description**: Retrieves complete flight information including all waypoints, geofences, and processing status.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightDetailResponse:
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: Flight ID does not exist
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Existing flight**: Valid flightId → returns 200 with complete flight data
2. **Non-existent flight**: Invalid flightId → returns 404
3. **Flight with many waypoints**: Flight with 2000+ waypoints → returns 200 with all data
---
### `delete_flight(flight_id: str) -> DeleteResponse`
**REST Endpoint**: `DELETE /flights/{flightId}`
**Description**: Deletes a flight and all associated waypoints, images, and processing data.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
DeleteResponse:
deleted: bool
flight_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight does not exist
- `409 Conflict`: Flight is currently being processed
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Delete existing flight**: Valid flightId → returns 200
2. **Delete non-existent flight**: Invalid flightId → returns 404
3. **Delete processing flight**: Active processing → returns 409
---
### `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/{waypointId}`
**Description**: Updates a specific waypoint within a flight. Used for per-frame GPS refinement.
**Called By**:
- Internal (F13 Result Manager for per-frame updates)
- Client applications (manual corrections)
**Input**:
```python
flight_id: str
waypoint_id: str
waypoint: Waypoint:
lat: float
lon: float
altitude: Optional[float]
confidence: float
timestamp: datetime
refined: bool
```
**Output**:
```python
UpdateResponse:
updated: bool
waypoint_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight or waypoint not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Update existing waypoint**: Valid data → returns 200
2. **Refinement update**: Refined coordinates → updates successfully
3. **Invalid coordinates**: lat > 90 → returns 400
4. **Non-existent waypoint**: Invalid waypoint_id → returns 404
---
### `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/batch`
**Description**: Updates multiple waypoints in a single request. Used for trajectory refinements.
**Called By**:
- Internal (F13 Result Manager for asynchronous refinement updates)
**Input**:
```python
flight_id: str
waypoints: List[Waypoint]
```
**Output**:
```python
BatchUpdateResponse:
success: bool
updated_count: int
failed_ids: List[str]
```
**Error Conditions**:
- `404 Not Found`: Flight not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Batch update 100 waypoints**: All succeed
2. **Partial failure**: 5 waypoints fail → returns failed_ids
3. **Empty batch**: Returns success=True, updated_count=0
4. **Large batch**: 500 waypoints → succeeds
---
## Image Processing Endpoints
### `upload_image_batch(flight_id: str, batch: ImageBatch) -> BatchResponse`
**REST Endpoint**: `POST /flights/{flightId}/images/batch`
**Description**: Uploads a batch of 10-50 UAV images for processing.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
ImageBatch: multipart/form-data
images: List[UploadFile]
metadata: BatchMetadata
start_sequence: int
end_sequence: int
```
**Output**:
```python
BatchResponse:
accepted: bool
sequences: List[int]
next_expected: int
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists
2. Validate batch size (10-50 images)
3. Validate sequence numbers (strict sequential)
4. Call F02 Flight Processor → queue_images(flight_id, batch)
5. F02 delegates to F05 Image Input Pipeline
6. Return immediately (processing is async)
**Error Conditions**:
- `400 Bad Request`: Invalid batch size, out-of-sequence images
- `404 Not Found`: flight_id doesn't exist
- `413 Payload Too Large`: Batch exceeds size limit
- `429 Too Many Requests`: Rate limit exceeded
**Test Cases**:
1. **Valid batch upload**: 20 images → returns 202 Accepted
2. **Out-of-sequence batch**: Sequence gap detected → returns 400
3. **Too many images**: 60 images → returns 400
4. **Large images**: 50 × 8MB images → successfully uploads
---
### `submit_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResponse`
**REST Endpoint**: `POST /flights/{flightId}/user-fix`
**Description**: Submits user-provided GPS anchor point to unblock failed localization.
**Called By**:
- Client applications (when user responds to `user_input_needed` event)
**Input**:
```python
UserFixRequest:
frame_id: int
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**:
```python
UserFixResponse:
accepted: bool
processing_resumed: bool
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists and is blocked
2. Call F02 Flight Processor → handle_user_fix(flight_id, fix_data)
3. F02 delegates to F11 Failure Recovery Coordinator
4. Coordinator applies anchor to Factor Graph
5. Resume processing pipeline
**Error Conditions**:
- `400 Bad Request`: Invalid fix data
- `404 Not Found`: flight_id or frame_id not found
- `409 Conflict`: Flight not in blocked state
**Test Cases**:
1. **Valid user fix**: Blocked flight → returns 200, processing resumes
2. **Fix for non-blocked flight**: Returns 409
3. **Invalid GPS coordinates**: Returns 400
---
### `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse`
**REST Endpoint**: `POST /flights/{flightId}/frames/{frameId}/object-to-gps`
**Description**: Converts object pixel coordinates to GPS. Used by external object detection systems (e.g., Azaion.Inference) to get GPS coordinates for detected objects.
**Called By**:
- External object detection systems (Azaion.Inference)
- Any system needing pixel-to-GPS conversion for a specific frame
**Input**:
```python
ObjectToGPSRequest:
pixel_x: float # X coordinate in image
pixel_y: float # Y coordinate in image
```
**Output**:
```python
ObjectGPSResponse:
gps: GPSPoint
accuracy_meters: float # Estimated accuracy
frame_id: int
pixel: Tuple[float, float]
```
**Processing Flow**:
1. Validate flight_id and frame_id exist
2. Validate frame has been processed (has pose in Factor Graph)
3. Call F02 Flight Processor → convert_object_to_gps(flight_id, frame_id, pixel)
4. F02 delegates to F13.image_object_to_gps(flight_id, frame_id, pixel)
5. Return GPS with accuracy estimate
**Error Conditions**:
- `400 Bad Request`: Invalid pixel coordinates
- `404 Not Found`: flight_id or frame_id not found
- `409 Conflict`: Frame not yet processed (no pose available)
**Test Cases**:
1. **Valid conversion**: Object at (1024, 768) → returns GPS
2. **Unprocessed frame**: Frame not in Factor Graph → returns 409
3. **Invalid pixel**: Negative coordinates → returns 400
---
### `get_flight_status(flight_id: str) -> FlightStatusResponse`
**REST Endpoint**: `GET /flights/{flightId}/status`
**Description**: Retrieves current processing status of a flight.
**Called By**:
- Client applications (polling for status)
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightStatusResponse:
status: str # "prefetching", "ready", "processing", "blocked", "completed", "failed"
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
**Test Cases**:
1. **Processing flight**: Returns current progress
2. **Blocked flight**: Returns blocked=true with search_grid_size
3. **Completed flight**: Returns status="completed" with final counts
---
### `create_sse_stream(flight_id: str) -> SSEStream`
**REST Endpoint**: `GET /flights/{flightId}/stream`
**Description**: Opens Server-Sent Events connection for real-time result streaming.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
SSE Stream with events:
- frame_processed
- frame_refined
- search_expanded
- user_input_needed
- processing_blocked
- flight_completed
```
**Processing Flow**:
1. Validate flight_id exists
2. Call F02 Flight Processor → create_client_stream(flight_id, client_id)
3. F02 delegates to F15 SSE Event Streamer → create_stream()
4. Return SSE stream to client
**Event Format**:
```json
{
"event": "frame_processed",
"data": {
"frame_id": 237,
"gps": {"lat": 48.123, "lon": 37.456},
"altitude": 800.0,
"confidence": 0.95,
"heading": 87.3,
"timestamp": "2025-11-24T10:30:00Z"
}
}
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
- Connection closed on client disconnect
**Test Cases**:
1. **Connect to stream**: Opens SSE connection successfully
2. **Receive frame events**: Process 100 frames → receive 100 events
3. **Receive user_input_needed**: Blocked frame → event sent
4. **Client reconnect**: Replay missed events from last_event_id
---
## Integration Tests
### Test 1: Complete Flight Lifecycle
1. POST /flights with valid data
2. GET /flights/{flightId} → verify data
3. GET /flights/{flightId}/stream (open SSE)
4. POST /flights/{flightId}/images/batch × 40
5. Receive frame_processed events via SSE
6. Receive flight_completed event
7. GET /flights/{flightId} → verify waypoints updated
8. DELETE /flights/{flightId}
### Test 2: User Fix Flow
1. Create flight and process images
2. Receive user_input_needed event
3. POST /flights/{flightId}/user-fix
4. Receive processing_resumed event
5. Continue receiving frame_processed events
### Test 3: Concurrent Flights
1. Create 10 flights concurrently
2. Upload batches to all flights in parallel
3. Stream results from all flights simultaneously
4. Verify no cross-contamination
### Test 4: Waypoint Updates
1. Create flight
2. Simulate per-frame updates via PUT /flights/{flightId}/waypoints/{waypointId} × 100
3. GET flight and verify all waypoints updated
4. Verify refined=true flag set
---
## Non-Functional Requirements
### Performance
- **create_flight**: < 500ms response (prefetch is async)
- **get_flight**: < 200ms for flights with < 2000 waypoints
- **update_waypoint**: < 100ms (critical for real-time updates)
- **upload_image_batch**: < 2 seconds for 50 × 2MB images
- **submit_user_fix**: < 200ms response
- **get_flight_status**: < 100ms
- **SSE latency**: < 500ms from event generation to client receipt
### Scalability
- Support 100 concurrent flight processing sessions
- Handle 1000+ concurrent SSE connections
- Handle flights with up to 3000 waypoints
- Support 10,000 requests per minute
### Reliability
- Request timeout: 30 seconds for batch uploads
- SSE keepalive: Ping every 30 seconds
- Automatic SSE reconnection with event replay
- Graceful handling of client disconnects
### Security
- API key authentication
- Rate limiting: 100 requests/minute per client
- Max upload size: 500MB per batch
- CORS configuration for web clients
- Input validation on all endpoints
- SQL injection prevention
---
## Dependencies
### Internal Components
- **F02 Flight Processor**: For ALL operations (flight CRUD, image batching, user fixes, SSE streams, object-to-GPS conversion). F01 is a thin REST layer that delegates all business logic to F02.
**Note**: F01 does NOT directly call F05, F11, F13, or F15. All operations are routed through F02 to maintain a single coordinator pattern.
### External Dependencies
- **FastAPI**: Web framework
- **Uvicorn**: ASGI server
- **Pydantic**: Validation
- **python-multipart**: Multipart form handling
---
## Data Models
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float # Latitude -90 to 90
lon: float # Longitude -180 to 180
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int # pixels
resolution_height: int # pixels
distortion_coefficients: Optional[List[float]] = None
```
### Polygon
```python
class Polygon(BaseModel):
north_west: GPSPoint
south_east: GPSPoint
```
### Geofences
```python
class Geofences(BaseModel):
polygons: List[Polygon]
```
### FlightCreateRequest
```python
class FlightCreateRequest(BaseModel):
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
### Waypoint
```python
class Waypoint(BaseModel):
id: str
lat: float
lon: float
altitude: Optional[float] = None
confidence: float
timestamp: datetime
refined: bool = False
```
### FlightDetailResponse
```python
class FlightDetailResponse(BaseModel):
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
### FlightStatusResponse
```python
class FlightStatusResponse(BaseModel):
status: str
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
### BatchMetadata
```python
class BatchMetadata(BaseModel):
start_sequence: int
end_sequence: int
batch_number: int
```
### BatchUpdateResponse
```python
class BatchUpdateResponse(BaseModel):
success: bool
updated_count: int
failed_ids: List[str]
errors: Optional[Dict[str, str]]
```
_docs/02_components/02_flight_processor/02.1.01_feature_flight_waypoint_management.md
-67
View File
@@ -1,67 +0,0 @@
# Feature: Flight & Waypoint Management
## Description
Core CRUD operations for flights and waypoints. Handles flight creation with satellite prefetching trigger, flight state management, waypoint updates, and data validation. This is the primary data management feature of the Flight Lifecycle Manager.
## Component APIs Implemented
### Flight Lifecycle
- `create_flight(flight_data: FlightData) -> str` - Creates flight, validates data, sets ENU origin, triggers satellite prefetching
- `get_flight(flight_id: str) -> Optional[Flight]` - Retrieves flight by ID
- `get_flight_state(flight_id: str) -> Optional[FlightState]` - Retrieves current flight state
- `delete_flight(flight_id: str) -> bool` - Deletes flight and associated resources
- `update_flight_status(flight_id: str, status: FlightStatusUpdate) -> bool` - Updates flight status
### Waypoint Management
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool` - Updates single waypoint
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResult` - Batch waypoint update
- `get_flight_metadata(flight_id: str) -> Optional[FlightMetadata]` - Retrieves flight metadata
### Validation
- `validate_waypoint(waypoint: Waypoint) -> ValidationResult` - Validates single waypoint
- `validate_geofence(geofence: Geofences) -> ValidationResult` - Validates geofence boundaries
- `validate_flight_continuity(waypoints: List[Waypoint]) -> ValidationResult` - Validates waypoint sequence continuity
## External Dependencies
- **F03 Flight Database** - Persistence layer for flights and waypoints
- **F04 Satellite Data Manager** - Triggered for prefetch_route_corridor on flight creation
- **F13 Coordinate Transformer** - Called for set_enu_origin on flight creation
## Internal Methods
### Flight Operations
- `_generate_flight_id()` - Generates unique flight identifier
- `_persist_flight(flight: Flight)` - Saves flight to F03
- `_load_flight(flight_id: str)` - Loads flight from F03
- `_delete_flight_resources(flight_id: str)` - Cleans up flight-related resources
### Validation Helpers
- `_validate_gps_bounds(lat: float, lon: float)` - Validates GPS coordinates within valid range
- `_validate_waypoint_sequence(waypoints: List[Waypoint])` - Checks waypoint ordering and gaps
- `_validate_geofence_polygon(geofence: Geofences)` - Validates geofence geometry
### Coordinate Setup
- `_setup_enu_origin(flight_id: str, start_gps: GPSPoint)` - Delegates to F13 for ENU origin setup
## Unit Tests
- Test flight creation with valid data returns flight_id
- Test flight creation with invalid geofence returns validation error
- Test flight creation triggers F04.prefetch_route_corridor
- Test flight creation calls F13.set_enu_origin
- Test get_flight returns None for non-existent flight
- Test delete_flight removes flight from database
- Test update_flight_status transitions state correctly
- Test validate_waypoint rejects out-of-bounds GPS
- Test validate_geofence rejects invalid polygon
- Test validate_flight_continuity detects excessive gaps
- Test batch_update_waypoints handles partial failures
- Test waypoint update validates before persisting
## Integration Tests
- Test flight creation end-to-end with F03 persistence
- Test flight creation triggers F04 satellite prefetching
- Test flight creation sets ENU origin via F13
- Test flight deletion cleans up all related data in F03
- Test concurrent flight creation handles ID generation correctly
- Test waypoint batch update maintains data consistency
_docs/02_components/02_flight_processor/02.1.02_feature_processing_delegation.md
-59
View File
@@ -1,59 +0,0 @@
# Feature: Processing Delegation
## Description
Delegation methods that route API calls from F01 Flight API to appropriate subsystems. Manages F02.2 Processing Engine instances, coordinates image queuing, handles user fixes, and provides access to SSE streaming and coordinate transformation services.
## Component APIs Implemented
- `queue_images(flight_id: str, batch: ImageBatch) -> BatchQueueResult` - Queues images via F05, ensures F02.2 engine is active
- `handle_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Forwards user fix to active F02.2 engine
- `create_client_stream(flight_id: str, client_id: str) -> StreamConnection` - Creates SSE stream via F15
- `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint` - Converts pixel to GPS via F13
## External Dependencies
- **F02.2 Flight Processing Engine** - Managed child component, created/retrieved per flight
- **F05 Image Input Pipeline** - Image batch queuing
- **F13 Coordinate Transformer** - Pixel-to-GPS conversion
- **F15 SSE Event Streamer** - Client stream creation
## Internal Methods
### Engine Management
- `_get_or_create_engine(flight_id: str) -> FlightProcessingEngine` - Retrieves existing or creates new F02.2 instance
- `_get_active_engine(flight_id: str) -> Optional[FlightProcessingEngine]` - Gets active engine or None
- `_engine_registry: Dict[str, FlightProcessingEngine]` - Registry of active engines per flight
### Image Queuing
- `_delegate_queue_batch(flight_id: str, batch: ImageBatch)` - Delegates to F05.queue_batch
- `_trigger_processing(engine: FlightProcessingEngine)` - Triggers async engine.start_processing
### User Fix Handling
- `_validate_fix_request(fix_data: UserFixRequest)` - Validates fix data before delegation
- `_apply_fix_to_engine(engine: FlightProcessingEngine, fix_data: UserFixRequest)` - Calls engine.apply_user_fix
### Stream Management
- `_delegate_stream_creation(flight_id: str, client_id: str)` - Delegates to F15
### Coordinate Conversion
- `_delegate_coordinate_transform(flight_id: str, frame_id: int, pixel: Tuple)` - Delegates to F13
## Unit Tests
- Test queue_images delegates to F05.queue_batch
- Test queue_images creates new engine if none exists
- Test queue_images retrieves existing engine for active flight
- Test queue_images triggers engine.start_processing
- Test handle_user_fix returns error for non-existent flight
- Test handle_user_fix delegates to correct engine
- Test create_client_stream delegates to F15
- Test convert_object_to_gps delegates to F13
- Test engine registry tracks active engines correctly
- Test multiple queue_images calls reuse same engine instance
## Integration Tests
- Test queue_images end-to-end with F05 and F02.2
- Test image batch flows through F05 to F02.2 processing
- Test user fix applied correctly through F02.2
- Test SSE stream creation and connection via F15
- Test coordinate conversion accuracy via F13
- Test engine cleanup on flight completion
- Test concurrent requests to same flight share engine
_docs/02_components/02_flight_processor/02.1.03_feature_system_initialization.md
-55
View File
@@ -1,55 +0,0 @@
# Feature: System Initialization
## Description
System startup routine that initializes all dependent components in correct order. Loads configuration, initializes ML models, prepares database connections, sets up satellite cache, and loads place recognition indexes. This is called once at service startup.
## Component APIs Implemented
- `initialize_system() -> bool` - Executes full system initialization sequence
## External Dependencies
- **F17 Configuration Manager** - Configuration loading
- **F16 Model Manager** - ML model initialization (SuperPoint, LightGlue, LiteSAM, DINOv2)
- **F03 Flight Database** - Database connection initialization
- **F04 Satellite Data Manager** - Satellite cache initialization
- **F08 Global Place Recognition** - Faiss index loading
## Internal Methods
### Initialization Sequence
- `_load_configuration()` - Loads config via F17, validates required settings
- `_initialize_models()` - Initializes ML models via F16 with TensorRT optimization
- `_initialize_database()` - Sets up F03 database connections and schema
- `_initialize_satellite_cache()` - Prepares F04 cache with operational region data
- `_load_place_recognition_indexes()` - Loads F08 Faiss indexes into memory
### Health Checks
- `_verify_gpu_availability()` - Checks CUDA/TensorRT availability
- `_verify_model_loading()` - Validates all models loaded correctly
- `_verify_database_connection()` - Tests database connectivity
- `_verify_index_integrity()` - Validates Faiss indexes are loadable
### Error Handling
- `_handle_initialization_failure(component: str, error: Exception)` - Logs and handles component init failures
- `_rollback_partial_initialization()` - Cleans up on partial initialization failure
## Unit Tests
- Test initialize_system calls F17 config loading
- Test initialize_system calls F16 model initialization
- Test initialize_system calls F03 database initialization
- Test initialize_system calls F04 cache initialization
- Test initialize_system calls F08 index loading
- Test initialize_system returns False on config load failure
- Test initialize_system returns False on model init failure
- Test initialize_system returns False on database init failure
- Test initialization order is correct (config first, then models, etc.)
- Test partial initialization failure triggers rollback
## Integration Tests
- Test full system initialization with all real components
- Test system startup on cold start (no cached data)
- Test system startup with existing database
- Test initialization with GPU unavailable falls back gracefully
- Test initialization timeout handling
- Test system ready state after successful initialization
- Test re-initialization after failure recovery
_docs/02_components/02_flight_processor/02.1._component_flight_lifecycle_manager.md
-146
View File
@@ -1,146 +0,0 @@
# Flight Lifecycle Manager
## Interface Definition
**Interface Name**: `IFlightLifecycleManager`
### Interface Methods
```python
class IFlightLifecycleManager(ABC):
# Flight Lifecycle
@abstractmethod
def create_flight(self, flight_data: FlightData) -> str:
pass
@abstractmethod
def get_flight(self, flight_id: str) -> Optional[Flight]:
pass
@abstractmethod
def get_flight_state(self, flight_id: str) -> Optional[FlightState]:
pass
@abstractmethod
def delete_flight(self, flight_id: str) -> bool:
pass
@abstractmethod
def update_flight_status(self, flight_id: str, status: FlightStatusUpdate) -> bool:
pass
# Waypoint Management
@abstractmethod
def update_waypoint(self, flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool:
pass
@abstractmethod
def batch_update_waypoints(self, flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResult:
pass
@abstractmethod
def get_flight_metadata(self, flight_id: str) -> Optional[FlightMetadata]:
pass
# Validation
@abstractmethod
def validate_waypoint(self, waypoint: Waypoint) -> ValidationResult:
pass
@abstractmethod
def validate_geofence(self, geofence: Geofences) -> ValidationResult:
pass
@abstractmethod
def validate_flight_continuity(self, waypoints: List[Waypoint]) -> ValidationResult:
pass
# API Delegation Methods (called by F01)
@abstractmethod
def queue_images(self, flight_id: str, batch: ImageBatch) -> BatchQueueResult:
"""Delegates to F05 Image Input Pipeline and triggers F02.2 Processing Engine."""
pass
@abstractmethod
def handle_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResult:
"""Delegates to F02.2 Processing Engine to apply fix."""
pass
@abstractmethod
def create_client_stream(self, flight_id: str, client_id: str) -> StreamConnection:
"""Delegates to F15 SSE Event Streamer."""
pass
@abstractmethod
def convert_object_to_gps(self, flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint:
"""Delegates to F13 Coordinate Transformer."""
pass
# System Initialization
@abstractmethod
def initialize_system(self) -> bool:
pass
```
## Component Description
### Responsibilities
- **Component ID**: F02.1
- **Flight Lifecycle**: Manage flight creation, persistence, and deletion.
- **System Entry Point**: Main interface for the REST API (F01).
- **Resource Management**: Initializes system components and manages `F02.2 Flight Processing Engine` instances.
- **Data Validation**: Validates inputs before processing.
- **Satellite Prefetching**: Triggers F04 prefetching on flight creation.
### Scope
- CRUD operations for Flights and Waypoints.
- Coordination of system startup.
- Hand-off of active processing tasks to F02.2.
- Does **NOT** contain the processing loop or recovery logic.
## API Methods
### `create_flight(flight_data: FlightData) -> str`
**Description**: Creates a new flight, validates data, sets ENU origin, and triggers satellite prefetching.
**Called By**: F01 Flight API
**Processing Flow**:
1. Generate flight_id.
2. Validate geofences and waypoints.
3. Call F13.set_enu_origin(flight_id, start_gps).
4. Trigger F04.prefetch_route_corridor().
5. Save to F03 Flight Database.
6. Return flight_id.
### `queue_images(flight_id: str, batch: ImageBatch) -> BatchQueueResult`
**Description**: Queues images and ensures a Processing Engine is active for this flight.
**Processing Flow**:
1. Delegate to F05.queue_batch().
2. Retrieve or create instance of `F02.2 Flight Processing Engine` for this flight.
3. Trigger `engine.start_processing()` (async).
4. Return result.
### `handle_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult`
**Description**: Forwards user fix to the active processing engine.
**Processing Flow**:
1. Get active engine for flight_id.
2. Call `engine.apply_user_fix(fix_data)`.
3. Return result.
### `initialize_system() -> bool`
**Description**: startup routine.
**Processing Flow**:
1. Load config (F17).
2. Initialize models (F16).
3. Initialize DB (F03).
4. Initialize Cache (F04).
5. Load Indexes (F08).
## Dependencies
- **F02.2 Flight Processing Engine**: Managed child component.
- **F03 Flight Database**: Persistence.
- **F04 Satellite Data Manager**: Prefetching.
- **F05 Image Input Pipeline**: Image queuing.
- **F13 Coordinate Transformer**: ENU origin setting.
- **F15 SSE Event Streamer**: Stream creation.
- **F17 Configuration Manager**: Config loading.
_docs/02_components/02_flight_processor/02.2.01_feature_frame_processing_loop.md
-43
View File
@@ -1,43 +0,0 @@
# Feature: Frame Processing Loop
## Description
Core frame-by-frame processing orchestration that runs the main visual odometry pipeline. Manages the continuous loop of fetching images, computing poses, updating the factor graph, and publishing results. Includes flight state machine management (Processing, Blocked, Recovering, Completed).
## Component APIs Implemented
- `start_processing(flight_id: str) -> None` - Starts the main processing loop in a background thread/task
- `stop_processing(flight_id: str) -> None` - Stops the processing loop gracefully
- `process_frame(flight_id: str, frame_id: int) -> FrameResult` - Processes a single frame through the pipeline
## External Dependencies
- **F05 Image Input Pipeline**: `get_next_image()` - Image source
- **F06 Image Rotation Manager**: `requires_rotation_sweep()` - Pre-processing checks
- **F07 Sequential Visual Odometry**: `compute_relative_pose()` - Motion estimation
- **F09 Metric Refinement**: `align_to_satellite()` - Drift correction
- **F10 Factor Graph Optimizer**: `add_relative_factor()`, `optimize_chunk()` - State estimation
- **F14 Result Manager**: `update_frame_result()` - Saving results
## Internal Methods
- `run_processing_loop(flight_id: str)` - Main loop: while images available, process each frame
- `_process_single_frame(flight_id: str, image: Image) -> FrameResult` - Single frame processing pipeline
- `_check_tracking_status(vo_result: VOResult) -> bool` - Determines if tracking is good or lost
- `_update_flight_status(flight_id: str, status: FlightStatus)` - Updates state machine
- `_get_flight_status(flight_id: str) -> FlightStatus` - Gets current flight status
- `_is_processing_active(flight_id: str) -> bool` - Checks if processing should continue
## Unit Tests
- Test `start_processing` initiates background loop
- Test `stop_processing` gracefully terminates active processing
- Test `process_frame` returns valid FrameResult with pose
- Test flight status transitions: Processing -> Completed on last frame
- Test flight status transitions: Processing -> Blocked on tracking loss
- Test processing stops when `stop_processing` called mid-flight
- Test processing handles empty image queue gracefully
- Test state machine rejects invalid transitions
## Integration Tests
- Test full pipeline: F05 -> F06 -> F07 -> F10 -> F14 flow
- Test processing 10 consecutive frames with good tracking
- Test real-time result publishing to F14/F15
- Test concurrent access to flight status from multiple threads
- Test processing resumes after Blocked -> Processing transition
_docs/02_components/02_flight_processor/02.2.02_feature_tracking_loss_recovery.md
-42
View File
@@ -1,42 +0,0 @@
# Feature: Tracking Loss Recovery
## Description
Handles recovery when visual odometry loses tracking. Implements progressive search strategy through F11 and manages user input workflow when automatic recovery fails. Coordinates the transition to BLOCKED status and back to PROCESSING upon successful recovery.
## Component APIs Implemented
- `handle_tracking_loss(flight_id: str, frame_id: int) -> RecoveryStatus` - Initiates and manages tracking loss recovery
- `apply_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Applies user-provided GPS anchor
## External Dependencies
- **F11 Failure Recovery Coordinator**:
- `start_search()` - Initiates progressive search
- `try_current_grid()` - Attempts matching on current tile grid
- `expand_search_radius()` - Expands search area
- `mark_found()` - Marks successful recovery
- `create_user_input_request()` - Creates request for user intervention
- `apply_user_anchor()` - Applies user-provided anchor
- **F15 SSE Event Streamer**: `send_user_input_request()` - Notifies client of required input
## Internal Methods
- `_run_progressive_search(flight_id: str, frame_id: int) -> Optional[RecoveryResult]` - Executes 1->25 tile progressive search
- `_request_user_input(flight_id: str, frame_id: int, request: UserInputRequest)` - Transitions to BLOCKED and notifies client
- `_validate_user_fix(fix_data: UserFixRequest) -> bool` - Validates user input data
- `_apply_fix_and_resume(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Applies fix and resumes processing
## Unit Tests
- Test `handle_tracking_loss` starts progressive search via F11
- Test progressive search expands from 1 to 25 tiles
- Test successful recovery returns RecoveryStatus.FOUND
- Test failed recovery transitions status to BLOCKED
- Test `apply_user_fix` validates input coordinates
- Test `apply_user_fix` returns success on valid anchor
- Test `apply_user_fix` transitions status to PROCESSING on success
- Test `apply_user_fix` rejects fix when not in BLOCKED status
## Integration Tests
- Test full recovery flow: tracking loss -> progressive search -> found
- Test full blocked flow: tracking loss -> search fails -> user input -> resume
- Test SSE notification sent when user input required
- Test recovery integrates with F11 search grid expansion
- Test user fix properly anchors subsequent frame processing
_docs/02_components/02_flight_processor/02.2.03_feature_chunk_lifecycle_orchestration.md
-37
View File
@@ -1,37 +0,0 @@
# Feature: Chunk Lifecycle Orchestration
## Description
Manages route chunk boundaries and creation during processing. Detects when new chunks should be created (tracking loss, sharp turns) and orchestrates chunk lifecycle through F12. Implements proactive chunk creation strategy where new chunks are created immediately on tracking loss rather than waiting for matching failures.
## Component APIs Implemented
- `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]` - Returns current active chunk for the flight
- `create_new_chunk(flight_id: str, frame_id: int) -> ChunkHandle` - Creates new chunk starting at specified frame
## External Dependencies
- **F12 Route Chunk Manager**:
- `get_active_chunk()` - Retrieves current chunk
- `create_chunk()` - Creates new chunk
- `add_frame_to_chunk()` - Adds processed frame to chunk
## Internal Methods
- `_detect_chunk_boundary(flight_id: str, frame_id: int, tracking_status: bool) -> bool` - Determines if chunk boundary detected
- `_should_create_chunk_on_tracking_loss(flight_id: str) -> bool` - Checks if proactive chunk creation needed
- `_create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle` - Creates chunk proactively on tracking loss
- `_add_frame_to_active_chunk(flight_id: str, frame_id: int, frame_result: FrameResult)` - Adds frame to current chunk
## Unit Tests
- Test `get_active_chunk` returns current chunk handle
- Test `get_active_chunk` returns None when no active chunk
- Test `create_new_chunk` delegates to F12 and returns handle
- Test chunk boundary detection on tracking loss
- Test proactive chunk creation triggers on tracking loss
- Test `_add_frame_to_active_chunk` delegates to F12
- Test multiple chunks can exist for same flight
## Integration Tests
- Test chunk creation flow: tracking loss -> new chunk -> frames added
- Test chunk boundary detection integrates with VO tracking status
- Test chunk handles are properly propagated to F10 factor graph
- Test chunk lifecycle across multiple tracking loss events
- Test chunk state consistency between F02.2 and F12
_docs/02_components/02_flight_processor/02.2._component_flight_processing_engine_spec.md
-108
View File
@@ -1,108 +0,0 @@
# Flight Processing Engine
## Interface Definition
**Interface Name**: `IFlightProcessingEngine`
### Interface Methods
```python
class IFlightProcessingEngine(ABC):
@abstractmethod
def start_processing(self, flight_id: str) -> None:
"""Starts the main processing loop in a background thread/task."""
pass
@abstractmethod
def stop_processing(self, flight_id: str) -> None:
pass
@abstractmethod
def process_frame(self, flight_id: str, frame_id: int) -> FrameResult:
pass
@abstractmethod
def apply_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResult:
pass
@abstractmethod
def handle_tracking_loss(self, flight_id: str, frame_id: int) -> RecoveryStatus:
pass
# Chunk Lifecycle (Delegates to F12 but orchestrated here)
@abstractmethod
def get_active_chunk(self, flight_id: str) -> Optional[ChunkHandle]:
pass
@abstractmethod
def create_new_chunk(self, flight_id: str, frame_id: int) -> ChunkHandle:
pass
```
## Component Description
### Responsibilities
- **Component ID**: F02.2
- **Processing Orchestration**: Runs the main loop: Image -> VO -> Graph -> Result.
- **State Machine**: Manages flight status (Processing, Blocked, Recovering, Completed).
- **Recovery Coordination**: Calls F11 methods and acts on return values.
- **Chunk Management**: Calls F12 to manage chunk lifecycle based on tracking status.
- **Background Tasks**: Manages background chunk matching tasks.
### Scope
- Per-flight processing logic.
- Interaction with Visual Pipeline (F06, F07, F09).
- Interaction with State Estimation (F10).
- Interaction with Recovery (F11).
- Result Publishing (F14, F15).
## Processing Flow
### `run_processing_loop(flight_id: str)`
1. **Loop**: While `F05.get_next_image()` returns image:
2. **Chunk Check**:
* `active_chunk = F12.get_active_chunk()`
* If `detect_chunk_boundary()`: `active_chunk = F12.create_chunk()`.
3. **Visual Odometry**:
* Call `F06.requires_rotation_sweep()`.
* Call `F07.compute_relative_pose()`.
4. **Tracking Check**:
* If tracking good:
* `F12.add_frame_to_chunk()`.
* `F09.align_to_satellite()` (optional drift correction).
* `F10.add_relative_factor()`.
* `F10.optimize_chunk()`.
* `F14.update_frame_result()`.
* If tracking lost:
* **Proactive Chunking**: `F11.create_chunk_on_tracking_loss()`.
* **Recovery**: Call `handle_tracking_loss()`.
### `handle_tracking_loss(flight_id, frame_id)`
1. Call `F11.start_search()`.
2. Loop progressive search (1..25 tiles):
* `result = F11.try_current_grid()`.
* If found: `F11.mark_found()`, break.
* If not found: `F11.expand_search_radius()`.
3. If still not found:
* `req = F11.create_user_input_request()`.
* `F15.send_user_input_request(req)`.
* Set status to **BLOCKED**.
* Wait for `apply_user_fix`.
### `apply_user_fix(fix_data)`
1. Call `F11.apply_user_anchor(fix_data)`.
2. If success:
* Set status to **PROCESSING**.
* Resume loop.
## Dependencies
- **F05 Image Input Pipeline**: Image source.
- **F06 Image Rotation Manager**: Pre-processing.
- **F07 Sequential Visual Odometry**: Motion estimation.
- **F09 Metric Refinement**: Satellite alignment.
- **F10 Factor Graph Optimizer**: State estimation.
- **F11 Failure Recovery Coordinator**: Recovery logic.
- **F12 Route Chunk Manager**: Chunk state.
- **F14 Result Manager**: Saving results.
- **F15 SSE Event Streamer**: Real-time updates.
_docs/02_components/03_flight_database/03.01_feature_flight_crud_operations.md
-114
View File
@@ -1,114 +0,0 @@
# Feature: Flight & Waypoint CRUD Operations
## Description
Core database infrastructure and CRUD operations for flights, waypoints, and geofences. Provides connection pooling, transaction support, and all primary data operations. This feature handles the main entities that define a flight and its route.
## Component APIs Implemented
- `execute_transaction(operations: List[Callable]) -> bool`
- `insert_flight(flight: Flight) -> str`
- `update_flight(flight: Flight) -> bool`
- `query_flights(filters: Dict[str, Any], limit: int, offset: int) -> List[Flight]`
- `get_flight_by_id(flight_id: str) -> Optional[Flight]`
- `delete_flight(flight_id: str) -> bool`
- `get_waypoints(flight_id: str, limit: Optional[int] = None) -> List[Waypoint]`
- `insert_waypoint(flight_id: str, waypoint: Waypoint) -> str`
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool`
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchResult`
## External Tools and Services
- **PostgreSQL**: Primary database
- **SQLAlchemy**: ORM and connection pooling
- **Alembic**: Schema migrations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_get_connection()` | Acquire connection from pool |
| `_release_connection(conn)` | Return connection to pool |
| `_execute_with_retry(query, params, retries=3)` | Execute query with automatic retry on transient errors |
| `_build_flight_from_row(row)` | Map database row to Flight object |
| `_build_waypoint_from_row(row)` | Map database row to Waypoint object |
| `_serialize_camera_params(params)` | Serialize CameraParameters to JSONB |
| `_deserialize_camera_params(jsonb)` | Deserialize JSONB to CameraParameters |
| `_build_filter_query(filters)` | Build WHERE clause from filter dict |
## Unit Tests
1. **Connection management**
- Pool returns connections correctly
- Connection recycling after timeout
- Health check on stale connections
2. **execute_transaction**
- All operations succeed → commit
- One operation fails → rollback all
- Connection error → retry and succeed
- Nested transactions handled correctly
3. **insert_flight**
- Valid flight with 100 waypoints → all persisted
- Duplicate flight_id → raises IntegrityError
- Partial failure mid-insert → complete rollback
- Empty waypoints list → flight created with no waypoints
4. **update_flight**
- Existing flight → returns True, fields updated
- Non-existent flight → returns False
- Only specified fields updated, others unchanged
5. **query_flights**
- Filter by name → returns matching flights
- Filter by status → returns matching flights
- Pagination (offset=100, limit=50) → returns correct slice
- No matches → returns empty list
6. **get_flight_by_id**
- Existing flight → returns complete Flight with waypoints
- Non-existent flight → returns None
- Large flight (3000 waypoints) → returns within 150ms
7. **delete_flight**
- Existing flight → returns True, cascade to all tables
- Non-existent flight → returns False
- Verify cascade deletes waypoints, geofences, etc.
8. **get_waypoints**
- All waypoints (limit=None) → returns complete list
- Limited (limit=100) → returns first 100
- Non-existent flight → returns empty list
9. **insert_waypoint**
- Valid insertion → returns waypoint_id
- Non-existent flight → raises ForeignKeyError
10. **update_waypoint**
- Existing waypoint → returns True
- Non-existent waypoint → returns False
- Concurrent updates → no data corruption
11. **batch_update_waypoints**
- Batch of 100 → all succeed
- Partial failure → returns failed_ids
- Empty batch → returns success with updated_count=0
## Integration Tests
1. **Complete flight lifecycle**
- insert_flight() → update_waypoint() × 100 → get_flight_by_id() → delete_flight()
- Verify all data persisted and cascade delete works
2. **High-frequency waypoint updates**
- Insert flight with 2000 waypoints
- Concurrent update_waypoint() calls (100/sec)
- Verify throughput > 200 ops/sec
- Verify no data corruption
3. **Transaction atomicity**
- Begin transaction with 5 operations
- Fail on operation 3
- Verify operations 1-2 rolled back
4. **Connection pool behavior**
- Exhaust pool → new requests wait
- Pool recovery after connection failures
- Connection reuse efficiency
_docs/02_components/03_flight_database/03.02_feature_processing_state_persistence.md
-103
View File
@@ -1,103 +0,0 @@
# Feature: Processing State Persistence
## Description
Persistence layer for flight processing state, frame results, and heading history. Supports crash recovery by persisting processing progress and enables temporal smoothing through heading history tracking. Critical for maintaining processing continuity and data integrity during frame refinement operations.
## Component APIs Implemented
- `save_flight_state(flight_state: FlightState) -> bool`
- `load_flight_state(flight_id: str) -> Optional[FlightState]`
- `query_processing_history(filters: Dict[str, Any]) -> List[FlightState]`
- `save_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
- `get_frame_results(flight_id: str) -> List[FrameResult]`
- `save_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
- `get_heading_history(flight_id: str, last_n: Optional[int] = None) -> List[HeadingRecord]`
- `get_latest_heading(flight_id: str) -> Optional[float]`
## External Tools and Services
- **PostgreSQL**: Primary database
- **SQLAlchemy**: ORM and connection pooling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_flight_state_from_row(row)` | Map database row to FlightState object |
| `_build_frame_result_from_row(row)` | Map database row to FrameResult object |
| `_build_heading_record_from_row(row)` | Map database row to HeadingRecord object |
| `_upsert_flight_state(state)` | Insert or update flight state (single source of truth) |
| `_upsert_frame_result(flight_id, result)` | Insert or update frame result (handles refinement updates) |
## Unit Tests
1. **save_flight_state**
- New state → created successfully
- Update existing state → overwrites previous
- Verify all fields persisted correctly
- Verify heading NOT stored in flight_state (use heading_history)
2. **load_flight_state**
- Existing state → returns FlightState object
- Non-existent flight → returns None
- Verify all fields deserialized correctly
3. **query_processing_history**
- Filter by date range → returns flights in range
- Filter by status → returns matching flights
- Combined filters → returns intersection
- No matches → returns empty list
4. **save_frame_result**
- New frame → persisted successfully
- Update on refinement → overwrites with refined=True
- Verify GPS, altitude, heading, confidence persisted
- Verify timestamp and updated_at set correctly
5. **get_frame_results**
- Flight with 500 frames → returns all results ordered by frame_id
- No results → returns empty list
- Performance: 2000 frames returned within 100ms
6. **save_heading**
- New heading → persisted correctly
- Overwrite same frame_id → updates value
- Verify timestamp persisted
- Heading range validation (0-360)
7. **get_heading_history**
- All headings (last_n=None) → returns complete history
- Last 10 headings → returns 10 most recent by timestamp
- Non-existent flight → returns empty list
- Ordered by frame_id descending
8. **get_latest_heading**
- Has history → returns latest heading value
- No history → returns None
- After multiple saves → returns most recent
## Integration Tests
1. **Processing state recovery**
- Save state with frames_processed=250
- Simulate crash (close connection)
- Reconnect and load_flight_state()
- Verify state intact, processing can resume
2. **Frame result refinement flow**
- save_frame_result() with refined=False
- Later, save_frame_result() with refined=True
- get_frame_results() → shows refined=True
- Verify GPS coordinates updated
3. **Heading history for smoothing**
- save_heading() × 100 frames
- get_heading_history(last_n=10)
- Verify correct 10 headings returned for smoothing calculation
- Verify ordering correct for temporal analysis
4. **High-frequency persistence**
- Concurrent save_frame_result() and save_heading() calls
- Measure throughput > 200 ops/sec
- Verify no data loss or corruption
5. **Transactional consistency**
- Transaction: update frame_result + update waypoint
- Verify atomic update or complete rollback
_docs/02_components/03_flight_database/03.03_feature_auxiliary_data_persistence.md
-99
View File
@@ -1,99 +0,0 @@
# Feature: Auxiliary Data Persistence
## Description
Persistence layer for supporting data including image metadata and chunk state. Image metadata enables image pipeline to track stored files, while chunk state persistence is critical for crash recovery of route chunk processing. These operations support other components' specific data requirements.
## Component APIs Implemented
- `save_image_metadata(flight_id: str, frame_id: int, file_path: str, metadata: Dict) -> bool`
- `get_image_path(flight_id: str, frame_id: int) -> Optional[str]`
- `get_image_metadata(flight_id: str, frame_id: int) -> Optional[Dict]`
- `save_chunk_state(flight_id: str, chunk: ChunkHandle) -> bool`
- `load_chunk_states(flight_id: str) -> List[ChunkHandle]`
- `delete_chunk_state(flight_id: str, chunk_id: str) -> bool`
## External Tools and Services
- **PostgreSQL**: Primary database with JSONB support
- **SQLAlchemy**: ORM and connection pooling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_chunk_handle_from_row(row)` | Map database row to ChunkHandle object |
| `_serialize_chunk_frames(frames)` | Serialize frame list to JSONB array |
| `_deserialize_chunk_frames(jsonb)` | Deserialize JSONB array to frame list |
| `_serialize_metadata(metadata)` | Serialize metadata dict to JSONB |
| `_deserialize_metadata(jsonb)` | Deserialize JSONB to metadata dict |
| `_upsert_chunk_state(flight_id, chunk)` | Insert or update chunk state |
## Unit Tests
1. **save_image_metadata**
- New image → metadata persisted with file_path
- Overwrite same frame_id → updates metadata
- Verify JSONB serialization of metadata dict
- Verify uploaded_at timestamp set
2. **get_image_path**
- Existing image → returns file path string
- Non-existent frame → returns None
- Non-existent flight → returns None
3. **get_image_metadata**
- Existing image → returns metadata dict
- Verify deserialization of original_name, width, height, file_size
- Non-existent → returns None
4. **save_chunk_state**
- New chunk → persisted successfully
- Update existing chunk → state updated
- Verify all fields: chunk_id, start_frame_id, end_frame_id, frames
- Verify anchor fields: has_anchor, anchor_frame_id, anchor_gps
- Verify matching_status persisted
- Verify frames JSONB array serialization
5. **load_chunk_states**
- Flight with 3 chunks → returns all chunk handles
- No chunks → returns empty list
- Verify correct deserialization of ChunkHandle fields
- Verify frames list deserialized from JSONB
6. **delete_chunk_state**
- Existing chunk → deleted, returns True
- Non-existent chunk → returns False
- Verify other chunks unaffected
## Integration Tests
1. **Image metadata lifecycle**
- save_image_metadata() for 100 frames
- get_image_path() for each → all paths returned
- Delete flight → cascade deletes image metadata
- Verify no orphan records
2. **Chunk state crash recovery**
- Create flight, save 5 chunk states
- Simulate crash (close connection)
- Reconnect, load_chunk_states()
- Verify all 5 chunks restored with correct state
- Verify frames lists intact
3. **Chunk lifecycle operations**
- save_chunk_state() → active chunk
- Update chunk: add frames, set has_anchor=True
- save_chunk_state() → verify update
- delete_chunk_state() after merge → verify removed
4. **Concurrent chunk operations**
- Multiple chunks saved concurrently
- Verify no data corruption
- Verify unique chunk_ids enforced
5. **JSONB query performance**
- Save chunks with large frames arrays (500+ frames)
- load_chunk_states() performance within 50ms
- Verify JSONB indexing effectiveness
6. **Foreign key constraints**
- save_chunk_state with invalid anchor_frame_id → proper error handling
- Verify FK constraint fk_anchor_frame enforced
- Delete referenced image → anchor_frame_id set to NULL
_docs/02_components/03_flight_database/03._component_flight_database.md
-1059
View File
File diff suppressed because it is too large Load Diff
_docs/02_components/04_satellite_data_manager/04.01_feature_tile_cache_management.md
-41
View File
@@ -1,41 +0,0 @@
# Feature: Tile Cache Management
## Description
Manages persistent disk-based caching of satellite tiles with flight-specific organization. Provides storage, retrieval, and cleanup of cached tiles to minimize redundant API calls and enable offline access to prefetched data.
## Component APIs Implemented
- `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
- `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
- `clear_flight_cache(flight_id: str) -> bool`
## External Tools and Services
- **diskcache**: Persistent cache library for disk storage management
- **opencv-python**: Image serialization (PNG encoding/decoding)
- **numpy**: Image array handling
## Internal Methods
- `_generate_cache_path(flight_id: str, tile_coords: TileCoords) -> Path`: Generates cache file path following pattern `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
- `_ensure_cache_directory(flight_id: str, zoom: int) -> bool`: Creates cache directory structure if not exists
- `_serialize_tile(tile_data: np.ndarray) -> bytes`: Encodes tile array to PNG bytes
- `_deserialize_tile(data: bytes) -> Optional[np.ndarray]`: Decodes PNG bytes to tile array
- `_update_cache_index(flight_id: str, tile_coords: TileCoords, action: str) -> None`: Updates cache index for tracking
- `_check_global_cache(tile_coords: TileCoords) -> Optional[np.ndarray]`: Fallback lookup in shared cache
## Unit Tests
1. **cache_tile_success**: Cache new tile → file created at correct path
2. **cache_tile_overwrite**: Cache existing tile → file updated
3. **cache_tile_disk_error**: Simulate disk full → returns False
4. **get_cached_tile_hit**: Tile exists → returns np.ndarray
5. **get_cached_tile_miss**: Tile not exists → returns None
6. **get_cached_tile_corrupted**: Invalid file → returns None, logs warning
7. **get_cached_tile_global_fallback**: Not in flight cache, found in global → returns tile
8. **clear_flight_cache_success**: Flight with tiles → all files removed
9. **clear_flight_cache_nonexistent**: No such flight → returns True (no-op)
10. **cache_path_generation**: Various tile coords → correct paths generated
## Integration Tests
1. **cache_round_trip**: cache_tile() then get_cached_tile() → returns identical data
2. **multi_flight_isolation**: Cache tiles for flight A and B → each retrieves only own tiles
3. **clear_does_not_affect_others**: Clear flight A → flight B cache intact
4. **large_cache_handling**: Cache 1000 tiles → all retrievable
_docs/02_components/04_satellite_data_manager/04.02_feature_tile_coordinate_operations.md
-44
View File
@@ -1,44 +0,0 @@
# Feature: Tile Coordinate Operations
## Description
Handles all tile coordinate calculations including GPS-to-tile conversion, tile grid computation, and grid expansion for progressive search. Delegates core Web Mercator projection math to H06 Web Mercator Utils to maintain single source of truth.
## Component APIs Implemented
- `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
- `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
- `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
- `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
## External Tools and Services
None (pure computation, delegates to H06)
## Internal Dependencies
- **H06 Web Mercator Utils**: Core projection calculations
- `H06.latlon_to_tile()` for coordinate conversion
- `H06.compute_tile_bounds()` for bounding box calculation
## Internal Methods
- `_compute_grid_offset(grid_size: int) -> int`: Calculates offset from center for symmetric grid (e.g., 3×3 → offset 1)
- `_grid_size_to_dimensions(grid_size: int) -> Tuple[int, int]`: Maps grid_size (1,4,9,16,25) to (rows, cols)
- `_generate_grid_tiles(center: TileCoords, rows: int, cols: int) -> List[TileCoords]`: Generates all tile coords in grid
## Unit Tests
1. **compute_tile_coords_ukraine**: Ukraine GPS coords at zoom 19 → valid tile coords
2. **compute_tile_coords_origin**: lat=0, lon=0 → correct center tile
3. **compute_tile_coords_edge_cases**: lat=90, lon=180, lon=-180 → handled correctly
4. **compute_tile_bounds_zoom19**: Zoom 19 tile → GSD ≈ 0.3 m/pixel
5. **compute_tile_bounds_corners**: Returns valid GPS for all 4 corners
6. **get_tile_grid_1**: grid_size=1 → returns [center]
7. **get_tile_grid_4**: grid_size=4 → returns 4 tiles (2×2)
8. **get_tile_grid_9**: grid_size=9 → returns 9 tiles (3×3) centered
9. **get_tile_grid_25**: grid_size=25 → returns 25 tiles (5×5)
10. **expand_search_grid_1_to_4**: Returns 3 new tiles only
11. **expand_search_grid_4_to_9**: Returns 5 new tiles only
12. **expand_search_grid_9_to_16**: Returns 7 new tiles only
13. **expand_search_grid_no_duplicates**: Expanded tiles not in original set
## Integration Tests
1. **h06_delegation_verify**: compute_tile_coords() result matches direct H06.latlon_to_tile()
2. **grid_bounds_coverage**: get_tile_grid(9) → all 9 tile bounds form contiguous area
3. **expand_completes_grid**: get_tile_grid(4) + expand_search_grid(4,9) == get_tile_grid(9)
_docs/02_components/04_satellite_data_manager/04.03_feature_tile_fetching.md
-51
View File
@@ -1,51 +0,0 @@
# Feature: Tile Fetching
## Description
Handles HTTP-based satellite tile retrieval from external provider API with multiple fetching patterns: single tile, grid, progressive expansion, and route corridor prefetching. Integrates with cache for performance optimization and supports parallel fetching for throughput.
## Component APIs Implemented
- `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
- `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
- `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
- `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
## External Tools and Services
- **Satellite Provider API**: HTTP tile source (`GET /api/satellite/tiles/latlon`)
- **httpx** or **requests**: HTTP client with async support
- **numpy**: Image array handling
## Internal Dependencies
- **01_feature_tile_cache_management**: cache_tile, get_cached_tile
- **02_feature_tile_coordinate_operations**: compute_tile_coords, get_tile_grid
## Internal Methods
- `_fetch_from_api(tile_coords: TileCoords) -> Optional[np.ndarray]`: HTTP GET to satellite provider, handles response parsing
- `_fetch_with_retry(tile_coords: TileCoords, max_retries: int = 3) -> Optional[np.ndarray]`: Wraps _fetch_from_api with retry logic
- `_fetch_tiles_parallel(tiles: List[TileCoords], max_concurrent: int = 20) -> Dict[str, np.ndarray]`: Parallel fetching with connection pooling
- `_compute_corridor_tiles(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> List[TileCoords]`: Calculates tiles covering route corridor polygon
- `_generate_tile_id(tile_coords: TileCoords) -> str`: Creates unique tile identifier string
## Unit Tests
1. **fetch_tile_cache_hit**: Tile in cache → returns immediately, no HTTP call
2. **fetch_tile_cache_miss**: Not cached → HTTP fetch, cache, return
3. **fetch_tile_api_error**: HTTP 500 → returns None
4. **fetch_tile_invalid_coords**: Invalid GPS → returns None
5. **fetch_tile_retry_success**: First attempt fails, second succeeds → returns tile
6. **fetch_tile_retry_exhausted**: All 3 attempts fail → returns None
7. **fetch_tile_grid_2x2**: grid_size=4 → returns dict with 4 tiles
8. **fetch_tile_grid_3x3**: grid_size=9 → returns dict with 9 tiles
9. **fetch_tile_grid_partial_failure**: 2 of 9 tiles fail → returns 7 tiles
10. **fetch_tile_grid_all_cached**: All tiles cached → no HTTP calls
11. **prefetch_route_corridor_success**: 10 waypoints → prefetches tiles, returns True
12. **prefetch_route_corridor_partial_failure**: Some tiles fail → continues, returns True
13. **prefetch_route_corridor_complete_failure**: All tiles fail → returns False
14. **progressive_fetch_yields_sequence**: [1,4,9] → yields 3 dicts in order
15. **progressive_fetch_early_termination**: Break after 4 → doesn't fetch 9,16,25
## Integration Tests
1. **fetch_and_cache_verify**: fetch_tile() → get_cached_tile() returns same data
2. **progressive_search_simulation**: progressive_fetch with simulated match on grid 9
3. **grid_expansion_no_refetch**: fetch_tile_grid(4) then expand → no duplicate fetches
4. **corridor_prefetch_coverage**: prefetch_route_corridor → all corridor tiles cached
5. **concurrent_fetch_stress**: Fetch 100 tiles in parallel → all complete within timeout
_docs/02_components/04_satellite_data_manager/04._component_satellite_data_manager.md
-562
View File
@@ -1,562 +0,0 @@
# Satellite Data Manager
## Interface Definition
**Interface Name**: `ISatelliteDataManager`
### Interface Methods
```python
class ISatelliteDataManager(ABC):
@abstractmethod
def fetch_tile(self, lat: float, lon: float, zoom: int) -> Optional[np.ndarray]:
pass
@abstractmethod
def fetch_tile_grid(self, center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]:
pass
@abstractmethod
def prefetch_route_corridor(self, waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool:
pass
@abstractmethod
def progressive_fetch(self, center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]:
pass
@abstractmethod
def cache_tile(self, flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool:
pass
@abstractmethod
def get_cached_tile(self, flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]:
pass
@abstractmethod
def get_tile_grid(self, center: TileCoords, grid_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_coords(self, lat: float, lon: float, zoom: int) -> TileCoords:
pass
@abstractmethod
def expand_search_grid(self, center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_bounds(self, tile_coords: TileCoords) -> TileBounds:
pass
@abstractmethod
def clear_flight_cache(self, flight_id: str) -> bool:
pass
```
## Component Description
### Responsibilities
- Fetch satellite tiles from external provider API
- Manage local tile cache per flight
- Calculate tile coordinates and grid layouts
- Support progressive tile grid expansion (1→4→9→16→25)
- Handle Web Mercator projection calculations
- Coordinate corridor prefetching for flight routes
### Scope
- **HTTP client** for Satellite Provider API
- **Local caching** with disk storage
- **Grid calculations** for search patterns
- **Tile coordinate transformations** (GPS↔Tile coordinates)
- **Progressive retrieval** for "kidnapped robot" recovery
## API Methods
### `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
**Description**: Fetches a single satellite tile by GPS coordinates.
**Called By**:
- F09 Metric Refinement (single tile for drift correction)
- Internal (during prefetching)
**Input**:
```python
lat: float # Latitude
lon: float # Longitude
zoom: int # Zoom level (19 for 0.3m/pixel at Ukraine latitude)
```
**Output**:
```python
np.ndarray: Tile image (H×W×3 RGB) or None if failed
```
**HTTP Request**:
```
GET /api/satellite/tiles/latlon?lat={lat}&lon={lon}&zoom={zoom}
```
**Processing Flow**:
1. Convert GPS to tile coordinates
2. Check cache
3. If not cached, fetch from satellite provider
4. Cache tile
5. Return tile image
**Error Conditions**:
- Returns `None`: Tile unavailable, HTTP error
- Logs errors for monitoring
**Test Cases**:
1. **Cache hit**: Tile in cache → returns immediately
2. **Cache miss**: Fetches from API → caches → returns
3. **API error**: Returns None
4. **Invalid coordinates**: Returns None
---
### `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
**Description**: Fetches NxN grid of tiles centered on GPS coordinates.
**Called By**:
- F09 Metric Refinement (for progressive search)
- F11 Failure Recovery Coordinator
**Input**:
```python
center_lat: float
center_lon: float
grid_size: int # 1, 4 (2×2), 9 (3×3), 16 (4×4), or 25 (5×5)
zoom: int
```
**Output**:
```python
Dict[str, np.ndarray] # tile_id -> tile_image
```
**Processing Flow**:
1. Compute tile grid centered on coordinates
2. For each tile in grid:
- Check cache
- If not cached, fetch from API
3. Return dict of tiles
**HTTP Request** (if using batch endpoint):
```
GET /api/satellite/tiles/batch?tiles=[...]
```
**Error Conditions**:
- Returns partial dict if some tiles fail
- Empty dict if all tiles fail
**Test Cases**:
1. **2×2 grid**: Returns 4 tiles
2. **3×3 grid**: Returns 9 tiles
3. **5×5 grid**: Returns 25 tiles
4. **Partial failure**: Some tiles unavailable → returns available tiles
5. **All cached**: Fast retrieval without HTTP requests
---
### `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
**Description**: Prefetches satellite tiles along route corridor for a flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight creation)
**Input**:
```python
waypoints: List[GPSPoint] # Rough route waypoints
corridor_width_m: float # Corridor width in meters (e.g., 1000m)
zoom: int
```
**Output**:
```python
bool: True if prefetch completed, False on error
```
**Processing Flow**:
1. For each waypoint pair:
- Calculate corridor polygon
- Determine tiles covering corridor
2. Fetch tiles (async, parallel)
3. Cache all tiles with flight_id reference
**Algorithm**:
- Use H06 Web Mercator Utils for tile calculations
- Parallel fetching (10-20 concurrent requests)
- Progress tracking for monitoring
**Error Conditions**:
- Returns `False`: Major error preventing prefetch
- Logs warnings for individual tile failures
**Test Cases**:
1. **Simple route**: 10 waypoints → prefetches 50-100 tiles
2. **Long route**: 50 waypoints → prefetches 200-500 tiles
3. **Partial failure**: Some tiles fail → continues, returns True
4. **Complete failure**: All tiles fail → returns False
---
### `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
**Description**: Progressively fetches expanding tile grids for "kidnapped robot" recovery.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search)
**Input**:
```python
center_lat: float
center_lon: float
grid_sizes: List[int] # e.g., [1, 4, 9, 16, 25]
zoom: int
```
**Output**:
```python
Iterator yielding Dict[str, np.ndarray] for each grid size
```
**Processing Flow**:
1. For each grid_size in sequence:
- Fetch tile grid
- Yield tiles
- If match found by caller, iterator can be stopped
**Usage Pattern**:
```python
for tiles in progressive_fetch(lat, lon, [1, 4, 9, 16, 25], 19):
if litesam_match_found(tiles):
break # Stop expanding search
```
**Test Cases**:
1. **Progressive search**: Yields 1, then 4, then 9 tiles
2. **Early termination**: Match on 4 tiles → doesn't fetch 9, 16, 25
3. **Full search**: No match → fetches all grid sizes
---
### `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
**Description**: Caches a satellite tile to disk with flight_id association.
**Called By**:
- Internal (after fetching tiles)
**Input**:
```python
flight_id: str # Flight this tile belongs to
tile_coords: TileCoords:
x: int
y: int
zoom: int
tile_data: np.ndarray
```
**Output**:
```python
bool: True if cached successfully
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Create flight cache directory if not exists
3. Serialize tile_data (PNG format)
4. Write to disk cache directory
5. Update cache index with flight_id association
**Error Conditions**:
- Returns `False`: Disk write error, space full
**Test Cases**:
1. **Cache new tile**: Writes successfully
2. **Overwrite existing**: Updates tile
3. **Disk full**: Returns False
---
### `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
**Description**: Retrieves a cached tile from disk, checking flight-specific cache first.
**Called By**:
- Internal (before fetching from API)
- F09 Metric Refinement (direct cache lookup)
**Input**:
```python
flight_id: str # Flight to check cache for
tile_coords: TileCoords
```
**Output**:
```python
Optional[np.ndarray]: Tile image or None if not cached
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Check flight-specific cache first
3. If not found, check global cache (shared tiles)
4. If file exists, load and deserialize
5. Return tile_data or None
**Error Conditions**:
- Returns `None`: Not cached, corrupted file
**Test Cases**:
1. **Cache hit**: Returns tile quickly
2. **Cache miss**: Returns None
3. **Corrupted cache**: Returns None, logs warning
---
### `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
**Description**: Calculates tile coordinates for NxN grid centered on a tile.
**Called By**:
- Internal (for grid fetching)
- F11 Failure Recovery Coordinator
**Input**:
```python
center: TileCoords
grid_size: int # 1, 4, 9, 16, 25
```
**Output**:
```python
List[TileCoords] # List of tile coordinates in grid
```
**Algorithm**:
- For grid_size=9 (3×3): tiles from center-1 to center+1 in both x and y
- For grid_size=16 (4×4): asymmetric grid with center slightly off-center
**Test Cases**:
1. **1-tile grid**: Returns [center]
2. **4-tile grid (2×2)**: Returns 4 tiles
3. **9-tile grid (3×3)**: Returns 9 tiles centered
4. **25-tile grid (5×5)**: Returns 25 tiles
---
### `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
**Description**: Converts GPS coordinates to tile coordinates.
**Called By**:
- All methods that need tile coordinates from GPS
**Input**:
```python
lat: float
lon: float
zoom: int
```
**Output**:
```python
TileCoords:
x: int
y: int
zoom: int
```
**Algorithm** (Web Mercator):
```python
n = 2 ** zoom
x = int((lon + 180) / 360 * n)
lat_rad = lat * π / 180
y = int((1 - log(tan(lat_rad) + sec(lat_rad)) / π) / 2 * n)
```
**Test Cases**:
1. **Ukraine coordinates**: Produces valid tile coords
2. **Edge cases**: lat=0, lon=0, lat=90, lon=180
---
### `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
**Description**: Returns only NEW tiles when expanding from current grid to larger grid.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search optimization)
**Input**:
```python
center: TileCoords
current_size: int # e.g., 4
new_size: int # e.g., 9
```
**Output**:
```python
List[TileCoords] # Only tiles not in current_size grid
```
**Purpose**: Avoid re-fetching tiles already tried in smaller grid.
**Test Cases**:
1. **4→9 expansion**: Returns 5 new tiles (9-4)
2. **9→16 expansion**: Returns 7 new tiles
3. **1→4 expansion**: Returns 3 new tiles
---
### `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
**Description**: Computes GPS bounding box of a tile.
**Called By**:
- F09 Metric Refinement (for homography calculations)
- H06 Web Mercator Utils (shared calculation)
**Input**:
```python
tile_coords: TileCoords
```
**Output**:
```python
TileBounds:
nw: GPSPoint # North-West corner
ne: GPSPoint # North-East corner
sw: GPSPoint # South-West corner
se: GPSPoint # South-East corner
center: GPSPoint
gsd: float # Ground Sampling Distance (meters/pixel)
```
**Algorithm**:
- Inverse Web Mercator projection
- GSD calculation: `156543.03392 * cos(lat * π/180) / 2^zoom`
**Test Cases**:
1. **Zoom 19 at Ukraine**: GSD ≈ 0.3 m/pixel
2. **Tile bounds**: Valid GPS coordinates
---
### `clear_flight_cache(flight_id: str) -> bool`
**Description**: Clears cached tiles for a completed flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (cleanup after flight completion)
**Input**:
```python
flight_id: str
```
**Output**:
```python
bool: True if cleared successfully
```
**Processing Flow**:
1. Find all tiles associated with flight_id
2. Delete tile files
3. Update cache index
**Test Cases**:
1. **Clear flight cache**: Removes all associated tiles
2. **Non-existent flight**: Returns True (no-op)
## Integration Tests
### Test 1: Prefetch and Retrieval
1. prefetch_route_corridor() with 20 waypoints
2. Verify tiles cached
3. get_cached_tile() for each tile → all hit cache
4. clear_flight_cache() → cache cleared
### Test 2: Progressive Search Simulation
1. progressive_fetch() with [1, 4, 9, 16, 25]
2. Simulate match on 9 tiles
3. Verify only 1, 4, 9 fetched (not 16, 25)
### Test 3: Grid Expansion
1. fetch_tile_grid(4) → 4 tiles
2. expand_search_grid(4, 9) → 5 new tiles
3. Verify no duplicate fetches
## Non-Functional Requirements
### Performance
- **fetch_tile**: < 200ms (cached: < 10ms)
- **fetch_tile_grid(9)**: < 1 second
- **prefetch_route_corridor**: < 30 seconds for 500 tiles
- **Cache lookup**: < 5ms
### Scalability
- Cache 10,000+ tiles per flight
- Support 100 concurrent tile fetches
- Handle 10GB+ cache size
### Reliability
- Retry failed HTTP requests (3 attempts)
- Graceful degradation on partial failures
- Cache corruption recovery
## Dependencies
### Internal Components
- **H06 Web Mercator Utils**: Tile coordinate calculations
**Note on Tile Coordinate Calculations**: F04 delegates ALL tile coordinate calculations to H06 Web Mercator Utils:
- `compute_tile_coords()` → internally calls `H06.latlon_to_tile()`
- `compute_tile_bounds()` → internally calls `H06.compute_tile_bounds()`
- `get_tile_grid()` → uses H06 for coordinate math
This ensures single source of truth for Web Mercator projection logic and avoids duplication with H06.
### External Dependencies
- **Satellite Provider API**: HTTP tile source
- **requests** or **httpx**: HTTP client
- **numpy**: Image handling
- **opencv-python**: Image I/O
- **diskcache**: Persistent cache
## Data Models
### TileCoords
```python
class TileCoords(BaseModel):
x: int
y: int
zoom: int
```
### TileBounds
```python
class TileBounds(BaseModel):
nw: GPSPoint
ne: GPSPoint
sw: GPSPoint
se: GPSPoint
center: GPSPoint
gsd: float # meters/pixel
```
### CacheConfig
```python
class CacheConfig(BaseModel):
cache_dir: str = "./satellite_cache"
max_size_gb: int = 50
eviction_policy: str = "lru"
ttl_days: int = 30
```
_docs/02_components/05_image_input_pipeline/05.01_feature_batch_queue_management.md
-61
View File
@@ -1,61 +0,0 @@
# Feature: Batch Queue Management
## Description
Handles ingestion, validation, and FIFO queuing of image batches. This feature manages the entry point for all images into the system, ensuring sequence integrity and proper queuing for downstream processing.
## Component APIs Implemented
- `queue_batch(flight_id: str, batch: ImageBatch) -> bool`
- `validate_batch(batch: ImageBatch) -> ValidationResult`
- `process_next_batch(flight_id: str) -> Optional[ProcessedBatch]`
## External Tools and Services
- **H08 Batch Validator**: Delegated validation for naming convention, sequence continuity, format, dimensions
- **Pillow**: Image decoding and metadata extraction
- **opencv-python**: Image I/O operations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_add_to_queue(flight_id, batch)` | Adds validated batch to flight's FIFO queue |
| `_dequeue_batch(flight_id)` | Removes and returns next batch from queue |
| `_check_sequence_continuity(flight_id, batch)` | Validates batch continues from last processed sequence |
| `_decode_images(batch)` | Decompresses/decodes raw image bytes to ImageData |
| `_extract_metadata(image_bytes)` | Extracts EXIF, dimensions from raw image |
| `_get_queue_capacity(flight_id)` | Returns remaining queue capacity for backpressure |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_queue_batch_valid` | Valid batch queued successfully, returns True |
| `test_queue_batch_sequence_gap` | Batch with sequence gap from last processed → ValidationError |
| `test_queue_batch_invalid_naming` | Non-consecutive filenames → ValidationError |
| `test_queue_batch_queue_full` | Queue at capacity → QueueFullError |
| `test_validate_batch_size_min` | 9 images → invalid (min 10) |
| `test_validate_batch_size_max` | 51 images → invalid (max 50) |
| `test_validate_batch_naming_convention` | ADxxxxxx.jpg format validated |
| `test_validate_batch_invalid_format` | IMG_0001.jpg → invalid |
| `test_validate_batch_non_consecutive` | AD000101, AD000103 → invalid |
| `test_validate_batch_file_format` | JPEG/PNG accepted, others rejected |
| `test_validate_batch_dimensions` | Within 640x480 to 6252x4168 |
| `test_validate_batch_file_size` | < 10MB per image |
| `test_process_next_batch_dequeue` | Returns ProcessedBatch with decoded images |
| `test_process_next_batch_empty_queue` | Empty queue → returns None |
| `test_process_next_batch_corrupted_image` | Corrupted image skipped, others processed |
| `test_process_next_batch_metadata_extraction` | EXIF and dimensions extracted correctly |
| `test_fifo_order` | Multiple batches processed in queue order |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_batch_flow_queue_to_process` | queue_batch → process_next_batch → verify ImageData list |
| `test_multiple_batches_fifo` | Queue 5 batches, process in order, verify sequence maintained |
| `test_batch_validation_with_h08` | Integration with H08 Batch Validator |
| `test_concurrent_queue_access` | Multiple flights queuing simultaneously |
| `test_backpressure_handling` | Queue fills up, backpressure signal returned |
_docs/02_components/05_image_input_pipeline/05.02_feature_image_storage_retrieval.md
-70
View File
@@ -1,70 +0,0 @@
# Feature: Image Storage and Retrieval
## Description
Handles persistent storage of processed images and provides retrieval mechanisms for sequential processing, random access, and metadata queries. Manages disk storage structure, maintains sequence tracking per flight, and provides processing status information.
## Component APIs Implemented
- `store_images(flight_id: str, images: List[ImageData]) -> bool`
- `get_next_image(flight_id: str) -> Optional[ImageData]`
- `get_image_by_sequence(flight_id: str, sequence: int) -> Optional[ImageData]`
- `get_image_metadata(flight_id: str, sequence: int) -> Optional[ImageMetadata]`
- `get_processing_status(flight_id: str) -> ProcessingStatus`
## External Tools and Services
- **F03 Flight Database**: Metadata persistence, flight state queries
- **opencv-python**: Image I/O (cv2.imread, cv2.imwrite)
- **numpy**: Image array handling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_create_flight_directory(flight_id)` | Creates storage directory structure for flight |
| `_write_image(flight_id, filename, image_data)` | Writes single image to disk |
| `_update_metadata_index(flight_id, metadata_list)` | Updates metadata.json with new image metadata |
| `_load_image_from_disk(flight_id, filename)` | Reads image file and returns np.ndarray |
| `_construct_filename(sequence)` | Converts sequence number to ADxxxxxx.jpg format |
| `_get_sequence_tracker(flight_id)` | Gets/initializes current sequence position for flight |
| `_increment_sequence(flight_id)` | Advances sequence counter after get_next_image |
| `_load_metadata_from_index(flight_id, sequence)` | Reads metadata from index without loading image |
| `_calculate_processing_rate(flight_id)` | Computes images/second processing rate |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_store_images_success` | All images written to correct paths |
| `test_store_images_creates_directory` | Flight directory created if not exists |
| `test_store_images_updates_metadata` | metadata.json updated with image info |
| `test_store_images_disk_full` | Storage error returns False |
| `test_get_next_image_sequential` | Returns images in sequence order |
| `test_get_next_image_increments_counter` | Sequence counter advances after each call |
| `test_get_next_image_end_of_sequence` | Returns None when no more images |
| `test_get_next_image_missing_file` | Handles missing image gracefully |
| `test_get_image_by_sequence_valid` | Returns correct image for sequence number |
| `test_get_image_by_sequence_invalid` | Invalid sequence returns None |
| `test_get_image_by_sequence_constructs_filename` | Sequence 101 → AD000101.jpg |
| `test_get_image_metadata_fast` | Returns metadata without loading full image |
| `test_get_image_metadata_missing` | Missing image returns None |
| `test_get_image_metadata_contains_fields` | Returns sequence, filename, dimensions, file_size, timestamp, exif |
| `test_get_processing_status_counts` | Accurate total_images, processed_images counts |
| `test_get_processing_status_current_sequence` | Reflects current processing position |
| `test_get_processing_status_queued_batches` | Includes queue depth |
| `test_get_processing_status_rate` | Processing rate calculation |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_store_then_retrieve_sequential` | store_images → get_next_image × N → all images retrieved |
| `test_store_then_retrieve_by_sequence` | store_images → get_image_by_sequence → correct image |
| `test_metadata_persistence_f03` | store_images → metadata persisted to F03 Flight Database |
| `test_crash_recovery_resume` | Restart processing from last stored sequence |
| `test_concurrent_retrieval` | Multiple consumers retrieving images simultaneously |
| `test_storage_large_batch` | Store and retrieve 3000 images for single flight |
| `test_multiple_flights_isolation` | Multiple flights don't interfere with each other's storage |
| `test_status_updates_realtime` | Status reflects current state during active processing |
_docs/02_components/05_image_input_pipeline/05._component_image_input_pipeline.md
-455
View File
@@ -1,455 +0,0 @@
# Image Input Pipeline
## Interface Definition
**Interface Name**: `IImageInputPipeline`
### Interface Methods
```python
class IImageInputPipeline(ABC):
@abstractmethod
def queue_batch(self, flight_id: str, batch: ImageBatch) -> bool:
pass
@abstractmethod
def process_next_batch(self, flight_id: str) -> Optional[ProcessedBatch]:
pass
@abstractmethod
def validate_batch(self, batch: ImageBatch) -> ValidationResult:
pass
@abstractmethod
def store_images(self, flight_id: str, images: List[ImageData]) -> bool:
pass
@abstractmethod
def get_next_image(self, flight_id: str) -> Optional[ImageData]:
pass
@abstractmethod
def get_image_by_sequence(self, flight_id: str, sequence: int) -> Optional[ImageData]:
pass
@abstractmethod
def get_image_metadata(self, flight_id: str, sequence: int) -> Optional[ImageMetadata]:
pass
@abstractmethod
def get_processing_status(self, flight_id: str) -> ProcessingStatus:
pass
```
## Component Description
### Responsibilities
- Unified image ingestion, validation, storage, and retrieval
- FIFO batch queuing for processing
- Validate consecutive naming (AD000001, AD000002, etc.)
- Validate sequence integrity (strict sequential ordering)
- Image persistence with indexed retrieval
- Metadata extraction (EXIF, dimensions)
### Scope
- Batch queue management
- Image validation
- Disk storage management
- Sequential processing coordination
- Metadata management
## API Methods
### `queue_batch(flight_id: str, batch: ImageBatch) -> bool`
**Description**: Queues a batch of images for processing (FIFO).
**Called By**:
- F02.1 Flight Lifecycle Manager (via F01 Flight API image upload route)
**Input**:
```python
flight_id: str
batch: ImageBatch:
images: List[bytes] # Raw image data
filenames: List[str] # e.g., ["AD000101.jpg", "AD000102.jpg", ...]
start_sequence: int # 101
end_sequence: int # 150
```
**Output**:
```python
bool: True if queued successfully
```
**Processing Flow**:
1. Validate batch using H08 Batch Validator
2. Check sequence continuity (no gaps)
3. Add to FIFO queue for flight_id
4. Return immediately (async processing)
**Error Conditions**:
- `ValidationError`: Sequence gap, invalid naming
- `QueueFullError`: Queue capacity exceeded
**Test Cases**:
1. **Valid batch**: Queued successfully
2. **Sequence gap**: Batch 101-150, expecting 51-100 → error
3. **Invalid naming**: Non-consecutive names → error
4. **Queue full**: Returns error with backpressure signal
---
### `process_next_batch(flight_id: str) -> Optional[ProcessedBatch]`
**Description**: Dequeues and processes the next batch from FIFO queue.
**Called By**:
- Internal processing loop (background worker)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ProcessedBatch:
images: List[ImageData]
batch_id: str
start_sequence: int
end_sequence: int
```
**Processing Flow**:
1. Dequeue next batch
2. Decompress/decode images
3. Extract metadata (EXIF, dimensions)
4. Store images to disk
5. Return ProcessedBatch for pipeline
**Error Conditions**:
- Returns `None`: Queue empty
- `ImageCorruptionError`: Invalid image data
**Test Cases**:
1. **Process batch**: Dequeues, returns ImageData list
2. **Empty queue**: Returns None
3. **Corrupted image**: Logs error, skips image
---
### `validate_batch(batch: ImageBatch) -> ValidationResult`
**Description**: Validates batch integrity and sequence continuity.
**Called By**:
- Internal (before queuing)
- H08 Batch Validator (delegated validation)
**Input**:
```python
batch: ImageBatch
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
1. **Batch size**: 10 <= len(images) <= 50
2. **Naming convention**: ADxxxxxx.jpg (6 digits)
3. **Sequence continuity**: Consecutive numbers
4. **File format**: JPEG or PNG
5. **Image dimensions**: 640x480 to 6252x4168
6. **File size**: < 10MB per image
**Test Cases**:
1. **Valid batch**: Returns valid=True
2. **Too few images**: 5 images → invalid
3. **Too many images**: 60 images → invalid
4. **Non-consecutive**: AD000101, AD000103 → invalid
5. **Invalid naming**: IMG_0001.jpg → invalid
---
### `store_images(flight_id: str, images: List[ImageData]) -> bool`
**Description**: Persists images to disk with indexed storage.
**Called By**:
- Internal (after processing batch)
**Input**:
```python
flight_id: str
images: List[ImageData]
```
**Output**:
```python
bool: True if stored successfully
```
**Storage Structure**:
```
/image_storage/
{flight_id}/
AD000001.jpg
AD000002.jpg
metadata.json
```
**Processing Flow**:
1. Create flight directory if not exists
2. Write each image to disk
3. Update metadata index
4. Persist to F03 Database Layer (metadata only)
**Error Conditions**:
- `StorageError`: Disk full, permission error
**Test Cases**:
1. **Store batch**: All images written successfully
2. **Disk full**: Returns False
3. **Verify storage**: Images retrievable after storage
---
### `get_next_image(flight_id: str) -> Optional[ImageData]`
**Description**: Gets the next image in sequence for processing.
**Called By**:
- F02.2 Flight Processing Engine (main processing loop)
- F06 Image Rotation Manager (via F02.2)
- F07 Sequential VO (via F02.2)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ImageData:
flight_id: str
sequence: int
filename: str
image: np.ndarray # Loaded image
metadata: ImageMetadata
```
**Processing Flow**:
1. Track current sequence number for flight
2. Load next image from disk
3. Increment sequence counter
4. Return ImageData
**Error Conditions**:
- Returns `None`: No more images
- `ImageNotFoundError`: Expected image missing
**Test Cases**:
1. **Get sequential images**: Returns images in order
2. **End of sequence**: Returns None
3. **Missing image**: Handles gracefully
---
### `get_image_by_sequence(flight_id: str, sequence: int) -> Optional[ImageData]`
**Description**: Retrieves a specific image by sequence number.
**Called By**:
- F11 Failure Recovery Coordinator (for user fix)
- F13 Result Manager (for refinement)
**Input**:
```python
flight_id: str
sequence: int
```
**Output**:
```python
Optional[ImageData]
```
**Processing Flow**:
1. Construct filename from sequence (ADxxxxxx.jpg)
2. Load from disk
3. Load metadata
4. Return ImageData
**Error Conditions**:
- Returns `None`: Image not found
**Test Cases**:
1. **Get specific image**: Returns correct image
2. **Invalid sequence**: Returns None
---
### `get_image_metadata(flight_id: str, sequence: int) -> Optional[ImageMetadata]`
**Description**: Retrieves metadata without loading full image (lightweight).
**Called By**:
- F02.1 Flight Lifecycle Manager (status checks)
- F13 Result Manager (metadata-only queries)
**Input**:
```python
flight_id: str
sequence: int
```
**Output**:
```python
ImageMetadata:
sequence: int
filename: str
dimensions: Tuple[int, int] # (width, height)
file_size: int # bytes
timestamp: datetime
exif_data: Optional[Dict]
```
**Test Cases**:
1. **Get metadata**: Returns quickly without loading image
2. **Missing image**: Returns None
---
### `get_processing_status(flight_id: str) -> ProcessingStatus`
**Description**: Gets current processing status for a flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (status queries via F01 Flight API)
- F02.2 Flight Processing Engine (processing loop status)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ProcessingStatus:
flight_id: str
total_images: int
processed_images: int
current_sequence: int
queued_batches: int
processing_rate: float # images/second
```
**Processing Flow**:
1. Get flight state via F03 Flight Database.get_flight(flight_id).status
2. Combine with internal queue status
3. Return ProcessingStatus
**Test Cases**:
1. **Get status**: Returns accurate counts
2. **During processing**: Updates in real-time
## Integration Tests
### Test 1: Batch Processing Flow
1. queue_batch() with 50 images
2. process_next_batch() → returns batch
3. store_images() → persists to disk
4. get_next_image() × 50 → retrieves all sequentially
5. Verify metadata
### Test 2: Multiple Batches
1. queue_batch() × 5 (250 images total)
2. process_next_batch() × 5
3. Verify FIFO order maintained
4. Verify sequence continuity
### Test 3: Error Handling
1. Queue batch with sequence gap
2. Verify validation error
3. Queue valid batch → succeeds
4. Simulate disk full → storage fails gracefully
## Non-Functional Requirements
### Performance
- **queue_batch**: < 100ms
- **process_next_batch**: < 2 seconds for 50 images
- **get_next_image**: < 50ms
- **get_image_by_sequence**: < 50ms
- **Processing throughput**: 10-20 images/second
### Scalability
- Support 3000 images per flight
- Handle 10 concurrent flights
- Manage 100GB+ image storage
### Reliability
- Crash recovery (resume processing from last sequence)
- Atomic batch operations
- Data integrity validation
## Dependencies
### Internal Components
- **F03 Flight Database**: For metadata persistence and flight state information
- **H08 Batch Validator**: For batch validation (naming convention, sequence continuity, format, dimensions)
### External Dependencies
- **opencv-python**: Image I/O
- **Pillow**: Image processing
- **numpy**: Image arrays
## Data Models
### ImageBatch
```python
class ImageBatch(BaseModel):
images: List[bytes]
filenames: List[str]
start_sequence: int
end_sequence: int
batch_number: int
```
### ImageData
```python
class ImageData(BaseModel):
flight_id: str
sequence: int
filename: str
image: np.ndarray
metadata: ImageMetadata
```
### ImageMetadata
```python
class ImageMetadata(BaseModel):
sequence: int
filename: str
dimensions: Tuple[int, int]
file_size: int
timestamp: datetime
exif_data: Optional[Dict]
```
### ProcessingStatus
```python
class ProcessingStatus(BaseModel):
flight_id: str
total_images: int
processed_images: int
current_sequence: int
queued_batches: int
processing_rate: float
```
_docs/02_components/06_image_rotation_manager/06.01_feature_image_rotation_core.md
-39
View File
@@ -1,39 +0,0 @@
# Feature: Image Rotation Core
## Description
Pure image rotation operations without state. Provides utility functions to rotate single images and batches of images by specified angles around their center. This is the foundation for rotation sweeps and pre-rotation before matching.
## Component APIs Implemented
- `rotate_image_360(image: np.ndarray, angle: float) -> np.ndarray`
- `rotate_chunk_360(chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]`
## External Tools and Services
- **opencv-python**: `cv2.warpAffine` for rotation transformation
- **numpy**: Matrix operations for rotation matrix construction
## Internal Methods
- `_build_rotation_matrix(center: Tuple[float, float], angle: float) -> np.ndarray`: Constructs 2x3 affine rotation matrix
- `_get_image_center(image: np.ndarray) -> Tuple[float, float]`: Calculates image center coordinates
## Unit Tests
### rotate_image_360
1. **Rotate 0°**: Input image equals output image (identity)
2. **Rotate 90°**: Image rotated 90° clockwise, dimensions preserved
3. **Rotate 180°**: Image inverted correctly
4. **Rotate 270°**: Image rotated 270° clockwise
5. **Rotate 45°**: Diagonal rotation with black fill at corners
6. **Rotate 360°**: Equivalent to 0° rotation
7. **Negative angle**: -90° equivalent to 270°
8. **Large angle normalization**: 450° equivalent to 90°
### rotate_chunk_360
1. **Empty chunk**: Returns empty list
2. **Single image chunk**: Equivalent to rotate_image_360
3. **Multiple images**: All images rotated by same angle
4. **Image independence**: Original chunk images unchanged
5. **Consistent dimensions**: All output images have same dimensions as input
## Integration Tests
None - this feature is stateless and has no external dependencies beyond opencv/numpy.
_docs/02_components/06_image_rotation_manager/06.02_feature_heading_management.md
-70
View File
@@ -1,70 +0,0 @@
# Feature: Heading Management
## Description
Manages UAV heading state per flight. Tracks current heading, maintains heading history, detects sharp turns, and determines when rotation sweeps are required. This is the stateful core of the rotation manager.
## Component APIs Implemented
- `get_current_heading(flight_id: str) -> Optional[float]`
- `update_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
- `detect_sharp_turn(flight_id: str, new_heading: float) -> bool`
- `requires_rotation_sweep(flight_id: str) -> bool`
## External Tools and Services
None - pure Python state management.
## Internal Methods
- `_normalize_angle(angle: float) -> float`: Normalizes angle to 0-360 range
- `_calculate_angle_delta(angle1: float, angle2: float) -> float`: Calculates smallest delta between two angles (handles wraparound)
- `_get_flight_state(flight_id: str) -> HeadingHistory`: Gets or creates heading state for flight
- `_add_to_history(flight_id: str, heading: float)`: Adds heading to circular history buffer
- `_set_sweep_required(flight_id: str, required: bool)`: Sets sweep required flag (used after tracking loss)
## Unit Tests
### get_current_heading
1. **New flight**: Returns None (no heading set)
2. **After update**: Returns last updated heading
3. **Multiple flights**: Each flight has independent heading
### update_heading
1. **First heading**: Sets initial heading, returns True
2. **Update heading**: Overwrites previous heading
3. **Angle normalization**: 370° stored as 10°
4. **Negative normalization**: -30° stored as 330°
5. **History tracking**: Heading added to history list
6. **History limit**: Only last 10 headings kept
### detect_sharp_turn
1. **No current heading**: Returns False (can't detect turn)
2. **Small turn (15°)**: 60° → 75° returns False
3. **Sharp turn (60°)**: 60° → 120° returns True
4. **Exactly 45°**: Returns False (threshold is >45)
5. **Exactly 46°**: Returns True
6. **Wraparound small**: 350° → 20° returns False (30° delta)
7. **Wraparound sharp**: 350° → 60° returns True (70° delta)
8. **180° turn**: 0° → 180° returns True
### requires_rotation_sweep
1. **First frame (no heading)**: Returns True
2. **Heading known, no flags**: Returns False
3. **Tracking loss flag set**: Returns True
4. **Sharp turn detected recently**: Returns True
5. **After successful match**: Returns False
## Integration Tests
### Test 1: Heading Lifecycle
1. Create new flight
2. get_current_heading → None
3. requires_rotation_sweep → True
4. update_heading(heading=45°)
5. get_current_heading → 45°
6. requires_rotation_sweep → False
### Test 2: Sharp Turn Flow
1. update_heading(heading=90°)
2. detect_sharp_turn(new_heading=100°) → False
3. detect_sharp_turn(new_heading=180°) → True
4. Set sweep required flag
5. requires_rotation_sweep → True
_docs/02_components/06_image_rotation_manager/06.03_feature_rotation_sweep_orchestration.md
-75
View File
@@ -1,75 +0,0 @@
# Feature: Rotation Sweep Orchestration
## Description
Coordinates rotation sweeps by rotating images at 30° steps and delegating matching to an injected matcher (F09 Metric Refinement). Calculates precise angles from homography matrices after successful matches. This feature ties together image rotation and heading management with external matching.
## Component APIs Implemented
- `try_rotation_steps(flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]`
- `try_chunk_rotation_steps(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]`
- `calculate_precise_angle(homography: np.ndarray, initial_angle: float) -> float`
## External Tools and Services
- **H07 Image Rotation Utils**: Angle extraction from homography
- **IImageMatcher (injected)**: F09 Metric Refinement for align_to_satellite and align_chunk_to_satellite
## Internal Methods
- `_get_rotation_steps() -> List[float]`: Returns [0, 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330]
- `_extract_rotation_from_homography(homography: np.ndarray) -> float`: Extracts rotation component from 3x3 homography
- `_combine_angles(initial_angle: float, delta_angle: float) -> float`: Combines step angle with homography delta, normalizes result
- `_select_best_result(results: List[Tuple[float, AlignmentResult]]) -> Tuple[float, AlignmentResult]`: Selects highest confidence match if multiple found
## Unit Tests
### calculate_precise_angle
1. **Identity homography**: Returns initial_angle unchanged
2. **Small rotation delta**: initial=60°, homography shows +2.5° → returns 62.5°
3. **Negative delta**: initial=30°, homography shows -3° → returns 27°
4. **Large delta normalization**: initial=350°, delta=+20° → returns 10°
5. **Invalid homography (singular)**: Returns initial_angle as fallback
6. **Near-zero homography**: Returns initial_angle as fallback
### try_rotation_steps (with mock matcher)
1. **Match at 0°**: First rotation matches, returns RotationResult with initial_angle=0
2. **Match at 60°**: Third rotation matches, returns RotationResult with initial_angle=60
3. **Match at 330°**: Last rotation matches, returns RotationResult with initial_angle=330
4. **No match**: All 12 rotations fail, returns None
5. **Multiple matches**: Returns highest confidence result
6. **Heading updated**: After match, flight heading is updated
7. **Confidence threshold**: Match below threshold rejected
### try_chunk_rotation_steps (with mock matcher)
1. **Match at 0°**: First rotation matches chunk
2. **Match at 120°**: Returns RotationResult with initial_angle=120
3. **No match**: All 12 rotations fail, returns None
4. **Chunk consistency**: All images rotated by same angle before matching
5. **Does not update heading**: Chunk matching doesn't affect flight state
## Integration Tests
### Test 1: First Frame Rotation Sweep
1. Create flight with no heading
2. Call try_rotation_steps with image, satellite tile, mock matcher
3. Mock matcher returns match at 60° rotation
4. Verify RotationResult.initial_angle = 60
5. Verify RotationResult.precise_angle refined from homography
6. Verify flight heading updated to precise_angle
### Test 2: Full Sweep No Match
1. Call try_rotation_steps with mock matcher that never matches
2. Verify all 12 rotations attempted (0°, 30°, ..., 330°)
3. Verify returns None
4. Verify flight heading unchanged
### Test 3: Chunk Rotation Sweep
1. Create chunk with 10 images
2. Call try_chunk_rotation_steps with mock matcher
3. Mock matcher returns match at 90° rotation
4. Verify all 10 images were rotated before matching call
5. Verify RotationResult returned with correct angles
### Test 4: Precise Angle Calculation
1. Perform rotation sweep, match at 60° step
2. Homography indicates +2.3° additional rotation
3. Verify precise_angle = 62.3°
4. Verify heading updated to 62.3° (not 60°)
_docs/02_components/06_image_rotation_manager/06._component_image_rotation_manager.md
-554
View File
@@ -1,554 +0,0 @@
# Image Rotation Manager
## Interface Definition
**Interface Name**: `IImageRotationManager`
### Interface Methods
```python
class IImageRotationManager(ABC):
@abstractmethod
def rotate_image_360(self, image: np.ndarray, angle: float) -> np.ndarray:
pass
@abstractmethod
def try_rotation_steps(self, flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]:
"""
Performs rotation sweep.
'matcher' is an injected dependency (usually F09) to avoid direct coupling.
"""
pass
@abstractmethod
def calculate_precise_angle(self, homography: np.ndarray, initial_angle: float) -> float:
pass
@abstractmethod
def get_current_heading(self, flight_id: str) -> Optional[float]:
pass
@abstractmethod
def update_heading(self, flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool:
pass
@abstractmethod
def detect_sharp_turn(self, flight_id: str, new_heading: float) -> bool:
pass
@abstractmethod
def requires_rotation_sweep(self, flight_id: str) -> bool:
pass
@abstractmethod
def rotate_chunk_360(self, chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]:
pass
@abstractmethod
def try_chunk_rotation_steps(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]:
pass
```
## Component Description
### Responsibilities
- Image rotation utility.
- Heading tracking.
- Coordination of rotation sweeps.
### Decoupling Fix
- **Problem**: F06 previously depended directly on `F09 Metric Refinement`.
- **Fix**: Methods `try_rotation_steps` and `try_chunk_rotation_steps` now accept a `matcher` argument conforming to `IImageMatcher`.
- **IImageMatcher Interface**:
```python
class IImageMatcher(ABC):
def align_to_satellite(self, uav_image, satellite_tile, tile_bounds) -> AlignmentResult: pass
def align_chunk_to_satellite(self, chunk_images, satellite_tile, tile_bounds) -> ChunkAlignmentResult: pass
```
- **Runtime**: F02.2 injects F09 instance when calling F06 methods.
### Scope
- Image rotation operations (pure rotation, no matching)
- UAV heading tracking and history
- Sharp turn detection
- Rotation sweep coordination (rotates images, delegates matching to F09 Metric Refinement)
- Precise angle calculation from homography (extracted from F09 results)
- **Chunk-level rotation (all images rotated by same angle)**
## API Methods
### `rotate_image_360(image: np.ndarray, angle: float) -> np.ndarray`
**Description**: Rotates an image by specified angle around center.
**Called By**:
- Internal (during rotation sweep)
- H07 Image Rotation Utils (may delegate to)
**Input**:
```python
image: np.ndarray # Input image (H×W×3)
angle: float # Rotation angle in degrees (0-360)
```
**Output**:
```python
np.ndarray # Rotated image (same dimensions)
```
**Processing Details**:
- Rotation around image center
- Preserves image dimensions
- Fills borders with black or extrapolation
**Error Conditions**:
- None (always returns rotated image)
**Test Cases**:
1. **Rotate 90°**: Image rotated correctly
2. **Rotate 0°**: Image unchanged
3. **Rotate 180°**: Image inverted
4. **Rotate 45°**: Diagonal rotation
---
### `try_rotation_steps(flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]`
**Description**: Performs 30° rotation sweep, rotating image at each step and delegating matching to F09 Metric Refinement.
**Called By**:
- Internal (when requires_rotation_sweep() returns True)
- Main processing loop (first frame or sharp turn)
**Input**:
```python
flight_id: str
frame_id: int # Frame identifier for heading persistence
image: np.ndarray # UAV image
satellite_tile: np.ndarray # Satellite reference tile
tile_bounds: TileBounds # GPS bounds and GSD of satellite tile (passed to F09)
timestamp: datetime # Timestamp for heading persistence
matcher: IImageMatcher # Injected matcher (F09)
```
**About tile_bounds**: `TileBounds` contains the GPS bounding box of the satellite tile:
- `nw`, `ne`, `sw`, `se`: GPS coordinates of tile corners
- `center`: GPS coordinate of tile center
- `gsd`: Ground Sampling Distance (meters/pixel)
The caller (F02 Flight Processor) obtains tile_bounds by calling `F04.compute_tile_bounds(tile_coords)` before calling this method. F06 passes tile_bounds to F09.align_to_satellite() which uses it to convert pixel coordinates to GPS.
**Output**:
```python
RotationResult:
matched: bool
initial_angle: float # Best matching step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
```
**Algorithm**:
```
For angle in [0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°]:
rotated_image = rotate_image_360(image, angle)
result = matcher.align_to_satellite(rotated_image, satellite_tile, tile_bounds)
if result.matched and result.confidence > threshold:
precise_angle = calculate_precise_angle(result.homography, angle)
update_heading(flight_id, frame_id, precise_angle, timestamp)
return RotationResult(matched=True, initial_angle=angle, precise_angle=precise_angle, ...)
return None # No match found
```
**Processing Flow**:
1. For each 30° step:
- Rotate image via rotate_image_360()
- Call matcher.align_to_satellite(rotated_image, satellite_tile, tile_bounds)
- Check if match found
2. If match found:
- Calculate precise angle from homography via calculate_precise_angle()
- Update UAV heading via update_heading()
- Return RotationResult
3. If no match:
- Return None (triggers progressive search expansion)
**Error Conditions**:
- Returns `None`: No match found in any rotation
- This is expected behavior (leads to progressive search)
**Test Cases**:
1. **Match at 60°**: Finds match, returns result
2. **Match at 0°**: No rotation needed, finds match
3. **No match**: All 12 rotations tried, returns None
4. **Multiple matches**: Returns best confidence
---
### `calculate_precise_angle(homography: np.ndarray, initial_angle: float) -> float`
**Description**: Calculates precise rotation angle from homography matrix point shifts.
**Called By**:
- Internal (after LiteSAM match in rotation sweep)
**Input**:
```python
homography: np.ndarray # 3×3 homography matrix from LiteSAM
initial_angle: float # 30° step angle that matched
```
**Output**:
```python
float: Precise rotation angle (e.g., 62.3° refined from 60° step)
```
**Algorithm**:
1. Extract rotation component from homography
2. Calculate angle from rotation matrix
3. Refine initial_angle with delta from homography
**Uses**: H07 Image Rotation Utils for angle calculation
**Error Conditions**:
- Falls back to initial_angle if calculation fails
**Test Cases**:
1. **Refine 60°**: Returns 62.5° (small delta)
2. **Refine 0°**: Returns 3.2° (small rotation)
3. **Invalid homography**: Returns initial_angle
---
### `get_current_heading(flight_id: str) -> Optional[float]`
**Description**: Gets current UAV heading angle for a flight.
**Called By**:
- F06 Internal (to check if pre-rotation needed)
- Main processing loop (before LiteSAM)
- F11 Failure Recovery Coordinator (logging)
**Input**:
```python
flight_id: str
```
**Output**:
```python
Optional[float]: Heading angle in degrees (0-360), or None if not initialized
```
**Error Conditions**:
- Returns `None`: First frame, heading not yet determined
**Test Cases**:
1. **After first frame**: Returns heading angle
2. **Before first frame**: Returns None
3. **During flight**: Returns current heading
---
### `update_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
**Description**: Updates UAV heading angle after successful match.
**Called By**:
- Internal (after rotation sweep match)
- Internal (after normal LiteSAM match with small rotation delta)
**Input**:
```python
flight_id: str
frame_id: int # Frame identifier for database persistence
heading: float # New heading angle (0-360)
timestamp: datetime # Timestamp for database persistence
```
**Output**:
```python
bool: True if updated successfully
```
**Processing Flow**:
1. Normalize angle to 0-360 range
2. Add to heading history (last 10 headings)
3. Update current_heading for flight
4. Return True (caller F02 is responsible for persistence via F03)
**Note**: Heading persistence is the caller's responsibility (F02 Flight Processor calls F03.save_heading() after receiving the updated heading).
**Test Cases**:
1. **Update heading**: Sets new heading
2. **Angle normalization**: 370° → 10°
3. **History tracking**: Maintains last 10 headings
---
### `detect_sharp_turn(flight_id: str, new_heading: float) -> bool`
**Description**: Detects if UAV made a sharp turn (>45° heading change).
**Called By**:
- Internal (before deciding if rotation sweep needed)
- Main processing loop
**Input**:
```python
flight_id: str
new_heading: float # Proposed new heading
```
**Output**:
```python
bool: True if sharp turn detected (>45° change)
```
**Algorithm**:
```python
current = get_current_heading(flight_id)
if current is None:
return False
delta = abs(new_heading - current)
if delta > 180: # Handle wraparound
delta = 360 - delta
return delta > 45
```
**Test Cases**:
1. **Small turn**: 60° → 75° → False (15° delta)
2. **Sharp turn**: 60° → 120° → True (60° delta)
3. **Wraparound**: 350° → 20° → False (30° delta)
4. **180° turn**: 0° → 180° → True
---
### `requires_rotation_sweep(flight_id: str) -> bool`
**Description**: Determines if rotation sweep is needed for current frame.
**Called By**:
- Main processing loop (before each frame)
- F11 Failure Recovery Coordinator (after tracking loss)
**Input**:
```python
flight_id: str
```
**Output**:
```python
bool: True if rotation sweep required
```
**Conditions for sweep**:
1. **First frame**: heading not initialized
2. **Sharp turn detected**: >45° heading change from VO
3. **Tracking loss**: LiteSAM failed to match in previous frame
4. **User flag**: Manual trigger (rare)
**Test Cases**:
1. **First frame**: Returns True
2. **Second frame, no turn**: Returns False
3. **Sharp turn detected**: Returns True
4. **Tracking loss**: Returns True
---
### `rotate_chunk_360(chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]`
**Description**: Rotates all images in a chunk by the same angle.
**Called By**:
- Internal (during try_chunk_rotation_steps)
- F11 Failure Recovery Coordinator (chunk rotation sweeps)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
angle: float # Rotation angle in degrees (0-360)
```
**Output**:
```python
List[np.ndarray] # Rotated images (same dimensions)
```
**Processing Flow**:
1. For each image in chunk:
- rotate_image_360(image, angle) → rotated_image
2. Return list of rotated images
**Performance**:
- Rotation time: ~20ms × N images
- For 10 images: ~200ms total
**Test Cases**:
1. **Rotate chunk**: All images rotated correctly
2. **Angle consistency**: All images rotated by same angle
3. **Image preservation**: Original images unchanged
---
### `try_chunk_rotation_steps(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]`
**Description**: Performs 30° rotation sweep on entire chunk, rotating all images at each step and delegating matching to F09 Metric Refinement.
**Called By**:
- F11 Failure Recovery Coordinator (chunk matching with rotation)
**Input**:
```python
chunk_images: List[np.ndarray] # Chunk images
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of satellite tile (for F09)
matcher: IImageMatcher # Injected matcher
```
**Output**:
```python
RotationResult:
matched: bool
initial_angle: float # Best matching step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
```
**Algorithm**:
```
For angle in [0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°]:
rotated_chunk = rotate_chunk_360(chunk_images, angle)
result = matcher.align_chunk_to_satellite(rotated_chunk, satellite_tile, tile_bounds)
if result.matched and result.confidence > threshold:
precise_angle = calculate_precise_angle(result.homography, angle)
return RotationResult(matched=True, initial_angle=angle, precise_angle=precise_angle, ...)
return None # No match found
```
**Processing Flow**:
1. For each 30° step:
- Rotate all chunk images via rotate_chunk_360()
- Call matcher.align_chunk_to_satellite(rotated_chunk, satellite_tile, tile_bounds)
- Check if match found
2. If match found:
- Calculate precise angle from homography via calculate_precise_angle()
- Return RotationResult
3. If no match:
- Return None
**Performance**:
- 12 rotations × chunk matching via F09 (~60ms) = ~720ms
- Acceptable for chunk matching (async operation)
**Test Cases**:
1. **Match at 60°**: Finds match, returns result
2. **Match at 0°**: No rotation needed, finds match
3. **No match**: All 12 rotations tried, returns None
4. **Multiple matches**: Returns best confidence
## Integration Tests
### Test 1: First Frame Rotation Sweep
1. First frame arrives (no heading set)
2. requires_rotation_sweep() → True
3. try_rotation_steps(flight_id, frame_id=1, image, satellite_tile, tile_bounds, timestamp=now()) → rotates 12 times
4. F09 Metric Refinement called for each rotation
5. Match found at 60° step
6. calculate_precise_angle() → 62.3°
7. update_heading(flight_id, frame_id=1, heading=62.3°, timestamp=now())
8. Subsequent frames use 62.3° heading
### Test 2: Normal Frame Processing
1. Heading known (90°)
2. requires_rotation_sweep() → False
3. Pre-rotate image to 90°
4. LiteSAM match succeeds with small delta (+2.5°)
5. update_heading(flight_id, frame_id=237, heading=92.5°, timestamp=now())
### Test 3: Sharp Turn Detection
1. UAV heading 45°
2. Next frame shows 120° heading (from VO estimate)
3. detect_sharp_turn() → True (75° delta)
4. requires_rotation_sweep() → True
5. Perform rotation sweep → find match at 120° step
### Test 4: Tracking Loss Recovery
1. F09 Metric Refinement fails to match (no overlap after turn)
2. requires_rotation_sweep() → True
3. try_rotation_steps(flight_id, frame_id, image, satellite_tile, tile_bounds, timestamp) with all 12 rotations
4. F09 called for each rotation step
5. Match found → heading updated
### Test 5: Chunk Rotation Sweeps
1. Build chunk with 10 images (unknown orientation)
2. try_chunk_rotation_steps(chunk_images, satellite_tile, tile_bounds) with all 12 rotations
3. F09 Metric Refinement called for each rotation
4. Match found at 120° step
5. Precise angle calculated (122.5°)
6. Verify all images rotated consistently
## Non-Functional Requirements
### Performance
- **rotate_image_360**: < 20ms per rotation
- **try_rotation_steps**: < 1.2 seconds (12 rotations × 100ms LiteSAM)
- **calculate_precise_angle**: < 10ms
- **get_current_heading**: < 1ms
- **update_heading**: < 5ms
### Accuracy
- **Angle precision**: ±0.5° for precise angle calculation
- **Sharp turn detection**: 100% accuracy for >45° turns
### Reliability
- Rotation sweep always completes all 12 steps
- Graceful handling of no-match scenarios
- Heading history preserved across failures
## Dependencies
### Internal Components
- **H07 Image Rotation Utils**: For image rotation and angle calculations
- **Injected Matcher (F09)**.
**Note**:
- `TileBounds` data model is imported from F09 Metric Refinement.
- F06 does NOT call F04 directly. The caller (F02 or F11) provides satellite tiles and tile_bounds.
- F06 does NOT persist heading to database. The caller (F02) is responsible for calling F03.save_heading().
- Chunk rotation orchestration (calling try_chunk_rotation_steps in recovery flow) is done by F11 Failure Recovery Coordinator.
### External Dependencies
- **opencv-python**: Image rotation (`cv2.warpAffine`)
- **numpy**: Matrix operations
## Data Models
### RotationResult
```python
class RotationResult(BaseModel):
matched: bool
initial_angle: float # 30° step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
inlier_count: int
```
### HeadingHistory
```python
class HeadingHistory(BaseModel):
flight_id: str
current_heading: float
heading_history: List[float] # Last 10 headings
last_update: datetime
sharp_turns: int # Count of sharp turns detected
```
### RotationConfig
```python
class RotationConfig(BaseModel):
step_angle: float = 30.0 # Degrees
sharp_turn_threshold: float = 45.0 # Degrees
confidence_threshold: float = 0.7 # For accepting match
history_size: int = 10 # Number of headings to track
```
_docs/02_components/07_sequential_visual_odometry/07.01_feature_combined_neural_inference.md
-129
View File
@@ -1,129 +0,0 @@
# Feature: Combined Neural Inference
## Description
Single-pass SuperPoint+LightGlue TensorRT inference for feature extraction and matching. Takes two images as input and outputs matched keypoints directly, eliminating intermediate feature transfer overhead.
## Component APIs Implemented
- `extract_and_match(image1: np.ndarray, image2: np.ndarray) -> Matches`
## External Tools and Services
- **Combined SuperPoint+LightGlue TensorRT Engine**: Single model combining extraction and matching
- **F16 Model Manager**: Provides pre-loaded TensorRT engine instance
- **Reference**: [D_VINS](https://github.com/kajo-kurisu/D_VINS/) for TensorRT optimization patterns
## Internal Methods
### `_preprocess_images(image1: np.ndarray, image2: np.ndarray) -> Tuple[np.ndarray, np.ndarray]`
Converts images to grayscale if needed, normalizes pixel values, resizes to model input dimensions.
### `_run_combined_inference(img1_tensor: np.ndarray, img2_tensor: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]`
Executes combined TensorRT engine. Returns matched keypoints from both images and match confidence scores.
**Internal Pipeline** (within single inference):
1. SuperPoint extracts keypoints + descriptors from both images
2. LightGlue performs attention-based matching with adaptive depth
3. Dustbin mechanism filters unmatched features
4. Returns only matched keypoint pairs
### `_filter_matches_by_confidence(keypoints1: np.ndarray, keypoints2: np.ndarray, scores: np.ndarray, threshold: float) -> Matches`
Filters low-confidence matches, constructs final Matches object.
## Architecture Notes
### Combined Model Benefits
- Single GPU memory transfer (both images together)
- No intermediate descriptor serialization
- Optimized attention layers for batch processing
- Adaptive depth exits early for easy (high-overlap) pairs
### TensorRT Engine Configuration
```
Input shapes:
image1: (1, 1, H, W) - grayscale
image2: (1, 1, H, W) - grayscale
Output shapes:
keypoints1: (1, M, 2) - matched keypoints from image1
keypoints2: (1, M, 2) - matched keypoints from image2
scores: (1, M) - match confidence scores
```
### Model Export (reference from D_VINS)
```bash
trtexec --onnx='superpoint_lightglue_combined.onnx' \
--fp16 \
--minShapes=image1:1x1x480x752,image2:1x1x480x752 \
--optShapes=image1:1x1x480x752,image2:1x1x480x752 \
--maxShapes=image1:1x1x480x752,image2:1x1x480x752 \
--saveEngine=sp_lg_combined.engine \
--warmUp=500 --duration=10
```
## Unit Tests
### Test: Grayscale Conversion
- Input: Two RGB images (H×W×3)
- Verify: _preprocess_images returns grayscale tensors
### Test: Grayscale Passthrough
- Input: Two grayscale images (H×W)
- Verify: _preprocess_images returns unchanged
### Test: High Overlap Matching
- Input: Two images with >50% overlap
- Verify: Returns 500+ matches
- Verify: Inference time ~35-50ms (adaptive depth fast path)
### Test: Low Overlap Matching
- Input: Two images with 5-10% overlap
- Verify: Returns 20-50 matches
- Verify: Inference time ~80ms (full depth)
### Test: No Overlap Handling
- Input: Two non-overlapping images
- Verify: Returns <10 matches
- Verify: No exception raised
### Test: Confidence Score Range
- Input: Any valid image pair
- Verify: All scores in [0, 1] range
### Test: Empty/Invalid Image Handling
- Input: Black/invalid image pair
- Verify: Returns empty Matches (never raises exception)
### Test: High Resolution Images
- Input: Two 6252×4168 images
- Verify: Preprocessing resizes appropriately
- Verify: Completes within performance budget
### Test: Output Shape Consistency
- Input: Any valid image pair
- Verify: keypoints1.shape[0] == keypoints2.shape[0] == scores.shape[0]
## Integration Tests
### Test: Model Manager Integration
- Verify: Successfully retrieves combined SP+LG engine from F16
- Verify: Engine loaded with correct TensorRT backend
### Test: Performance Budget
- Input: Two FullHD images
- Verify: Combined inference completes in <80ms on RTX 2060
### Test: Adaptive Depth Behavior
- Input: High overlap pair, then low overlap pair
- Verify: High overlap completes faster than low overlap
### Test: Agricultural Texture Handling
- Input: Two wheat field images with repetitive patterns
- Verify: Produces valid matches despite repetitive textures
### Test: Memory Efficiency
- Verify: Single GPU memory allocation vs two separate models
- Verify: No intermediate descriptor buffer allocation
### Test: Batch Consistency
- Input: Same image pair multiple times
- Verify: Consistent match results (deterministic)
_docs/02_components/07_sequential_visual_odometry/07.02_feature_geometric_pose_estimation.md
-133
View File
@@ -1,133 +0,0 @@
# Feature: Geometric Pose Estimation
## Description
Estimates camera motion from matched keypoints using Essential Matrix decomposition. Orchestrates the full visual odometry pipeline and provides tracking quality assessment. Pure geometric computation (non-ML).
## Component APIs Implemented
- `compute_relative_pose(prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]`
- `estimate_motion(matches: Matches, camera_params: CameraParameters) -> Optional[Motion]`
## External Tools and Services
- **opencv-python**: Essential Matrix estimation via RANSAC, matrix decomposition
- **numpy**: Matrix operations, coordinate normalization
- **F17 Configuration Manager**: Camera parameters (focal length, principal point)
- **H01 Camera Model**: Coordinate normalization utilities
## Internal Methods
### `_normalize_keypoints(keypoints: np.ndarray, camera_params: CameraParameters) -> np.ndarray`
Normalizes pixel coordinates to camera-centered coordinates using intrinsic matrix K.
```
normalized = K^(-1) @ [x, y, 1]^T
```
### `_estimate_essential_matrix(points1: np.ndarray, points2: np.ndarray) -> Tuple[Optional[np.ndarray], np.ndarray]`
RANSAC-based Essential Matrix estimation. Returns E matrix and inlier mask.
- Uses cv2.findEssentialMat with RANSAC
- Requires minimum 8 point correspondences
- Returns None if insufficient inliers
### `_decompose_essential_matrix(E: np.ndarray, points1: np.ndarray, points2: np.ndarray, camera_params: CameraParameters) -> Tuple[np.ndarray, np.ndarray]`
Decomposes Essential Matrix into rotation R and translation t.
- Uses cv2.recoverPose
- Selects correct solution via cheirality check
- Translation is unit vector (scale ambiguous)
### `_compute_tracking_quality(inlier_count: int, total_matches: int) -> Tuple[float, bool]`
Computes confidence score and tracking_good flag:
- **Good**: inlier_count > 50, inlier_ratio > 0.5 → confidence > 0.8
- **Degraded**: inlier_count 20-50 → confidence 0.4-0.8
- **Lost**: inlier_count < 20 → tracking_good = False
### `_build_relative_pose(motion: Motion, matches: Matches) -> RelativePose`
Constructs RelativePose dataclass from motion estimate and match statistics.
## Unit Tests
### Test: Keypoint Normalization
- Input: Pixel coordinates and camera params (fx=1000, cx=640, cy=360)
- Verify: Output centered at principal point, scaled by focal length
### Test: Essential Matrix Estimation - Good Data
- Input: 100+ inlier correspondences from known motion
- Verify: Returns valid Essential Matrix
- Verify: det(E) ≈ 0
- Verify: Singular values satisfy 2σ₁ ≈ σ₂, σ₃ ≈ 0
### Test: Essential Matrix Estimation - Insufficient Points
- Input: <8 point correspondences
- Verify: Returns None
### Test: Essential Matrix Decomposition
- Input: Valid Essential Matrix from known motion
- Verify: Returns valid rotation (det(R) = 1, R^T R = I)
- Verify: Translation is unit vector (||t|| = 1)
### Test: Tracking Quality - Good
- Input: inlier_count=100, total_matches=150
- Verify: tracking_good=True
- Verify: confidence > 0.8
### Test: Tracking Quality - Degraded
- Input: inlier_count=30, total_matches=50
- Verify: tracking_good=True
- Verify: 0.4 < confidence < 0.8
### Test: Tracking Quality - Lost
- Input: inlier_count=10, total_matches=20
- Verify: tracking_good=False
### Test: Scale Ambiguity Marker
- Input: Any valid motion estimate
- Verify: translation vector has unit norm (||t|| = 1)
- Verify: scale_ambiguous flag is True
### Test: Pure Rotation Handling
- Input: Matches from pure rotational motion (no translation)
- Verify: Returns valid pose
- Verify: translation ≈ [0, 0, 0] or arbitrary unit vector
### Test: Forward Motion
- Input: Matches from forward camera motion
- Verify: translation z-component is positive
## Integration Tests
### Test: Full Pipeline - Normal Flight
- Input: Consecutive frames with 50% overlap
- Verify: Returns valid RelativePose
- Verify: inlier_count > 100
- Verify: Total time < 150ms
### Test: Full Pipeline - Low Overlap
- Input: Frames with 5% overlap
- Verify: Returns valid RelativePose
- Verify: inlier_count > 20
### Test: Full Pipeline - Tracking Loss
- Input: Non-overlapping frames (sharp turn)
- Verify: Returns None
- Verify: No exception raised
### Test: Configuration Manager Integration
- Verify: Successfully retrieves camera_params from F17
- Verify: Parameters match expected resolution and focal length
### Test: Camera Model Integration
- Verify: H01 normalization produces correct coordinates
- Verify: Consistent with opencv undistortion
### Test: Pipeline Orchestration
- Verify: extract_and_match called once (combined inference)
- Verify: estimate_motion called with correct params
- Verify: Returns RelativePose with all fields populated
### Test: Agricultural Environment
- Input: Wheat field images with repetitive texture
- Verify: Pipeline succeeds with reasonable inlier count
### Test: Known Motion Validation
- Input: Synthetic image pair with known ground truth motion
- Verify: Estimated rotation within ±2° of ground truth
- Verify: Estimated translation direction within ±5° of ground truth
_docs/02_components/07_sequential_visual_odometry/07._component_sequential_visual_odometry.md
-301
View File
@@ -1,301 +0,0 @@
# Sequential Visual Odometry
## Interface Definition
**Interface Name**: `ISequentialVisualOdometry`
### Interface Methods
```python
class ISequentialVisualOdometry(ABC):
@abstractmethod
def compute_relative_pose(self, prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]:
pass
@abstractmethod
def extract_and_match(self, image1: np.ndarray, image2: np.ndarray) -> Matches:
pass
@abstractmethod
def estimate_motion(self, matches: Matches, camera_params: CameraParameters) -> Optional[Motion]:
pass
```
**Note**: F07 is chunk-agnostic. It only computes relative poses between images. The caller (F02.2 Flight Processing Engine) determines which chunk the frames belong to and routes factors to the appropriate subgraph via F12 → F10.
## Component Description
### Responsibilities
- Combined SuperPoint+LightGlue neural network inference for extraction and matching
- Handle <5% overlap scenarios via LightGlue attention mechanism
- Estimate relative pose (translation + rotation) between frames
- Return relative pose factors for Factor Graph Optimizer
- Detect tracking loss (low inlier count)
### Scope
- Frame-to-frame visual odometry
- Feature-based motion estimation using combined neural network
- Handles low overlap and challenging agricultural environments
- Provides relative measurements for trajectory optimization
- **Chunk-agnostic**: F07 doesn't know about chunks. Caller (F02.2) routes results to appropriate chunk subgraph.
### Architecture Notes
- Uses combined SuperPoint+LightGlue TensorRT model (single inference pass)
- Reference: [D_VINS](https://github.com/kajo-kurisu/D_VINS/) for TensorRT optimization patterns
- Model outputs matched keypoints directly from two input images
## API Methods
### `compute_relative_pose(prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]`
**Description**: Computes relative camera pose between consecutive frames.
**Called By**:
- Main processing loop (per-frame)
**Input**:
```python
prev_image: np.ndarray # Previous frame (t-1)
curr_image: np.ndarray # Current frame (t)
```
**Output**:
```python
RelativePose:
translation: np.ndarray # (x, y, z) in meters
rotation: np.ndarray # 3×3 rotation matrix or quaternion
confidence: float # 0.0 to 1.0
inlier_count: int
total_matches: int
tracking_good: bool
```
**Processing Flow**:
1. extract_and_match(prev_image, curr_image) → matches
2. estimate_motion(matches, camera_params) → motion
3. Return RelativePose
**Tracking Quality Indicators**:
- **Good tracking**: inlier_count > 50, inlier_ratio > 0.5
- **Degraded tracking**: inlier_count 20-50
- **Tracking loss**: inlier_count < 20
**Error Conditions**:
- Returns `None`: Tracking lost (insufficient matches)
**Test Cases**:
1. **Good overlap (>50%)**: Returns reliable pose
2. **Low overlap (5-10%)**: Still succeeds with LightGlue
3. **<5% overlap**: May return None (tracking loss)
4. **Agricultural texture**: Handles repetitive patterns
---
### `extract_and_match(image1: np.ndarray, image2: np.ndarray) -> Matches`
**Description**: Single-pass neural network inference combining SuperPoint feature extraction and LightGlue matching.
**Called By**:
- Internal (during compute_relative_pose)
**Input**:
```python
image1: np.ndarray # First image (H×W×3 or H×W)
image2: np.ndarray # Second image (H×W×3 or H×W)
```
**Output**:
```python
Matches:
matches: np.ndarray # (M, 2) - indices [idx1, idx2]
scores: np.ndarray # (M,) - match confidence scores
keypoints1: np.ndarray # (M, 2) - matched keypoints from image 1
keypoints2: np.ndarray # (M, 2) - matched keypoints from image 2
```
**Processing Details**:
- Uses F16 Model Manager to get combined SuperPoint+LightGlue TensorRT engine
- Single inference pass processes both images
- Converts to grayscale internally if needed
- SuperPoint extracts keypoints + 256-dim descriptors
- LightGlue performs attention-based matching with adaptive depth
- "Dustbin" mechanism handles unmatched features
**Performance** (TensorRT on RTX 2060):
- Combined inference: ~50-80ms (vs ~65-115ms separate)
- Faster for high-overlap pairs (adaptive depth exits early)
**Test Cases**:
1. **High overlap**: ~35ms, 500+ matches
2. **Low overlap (<5%)**: ~80ms, 20-50 matches
3. **No overlap**: Few or no matches (< 10)
---
### `estimate_motion(matches: Matches, camera_params: CameraParameters) -> Optional[Motion]`
**Description**: Estimates camera motion from matched keypoints using Essential Matrix.
**Called By**:
- Internal (during compute_relative_pose)
**Input**:
```python
matches: Matches
camera_params: CameraParameters:
focal_length: float
principal_point: Tuple[float, float]
resolution: Tuple[int, int]
```
**Output**:
```python
Motion:
translation: np.ndarray # (x, y, z) - unit vector (scale ambiguous)
rotation: np.ndarray # 3×3 rotation matrix
inliers: np.ndarray # Boolean mask of inlier matches
inlier_count: int
```
**Algorithm**:
1. Normalize keypoint coordinates using camera intrinsics
2. Estimate Essential Matrix using RANSAC
3. Decompose Essential Matrix → [R, t]
4. Return motion with inlier mask
**Scale Ambiguity**:
- Monocular VO has inherent scale ambiguity
- Translation is unit vector (direction only, magnitude = 1)
- **F07 does NOT resolve scale** - it only outputs unit translation vectors
- Scale resolution is handled by F10 Factor Graph Optimizer, which uses:
- Altitude priors (soft constraints)
- GSD-based expected displacement calculations (via H02)
- Absolute GPS anchors from F09 Metric Refinement
**Critical Handoff to F10**:
The caller (F02.2) must pass the unit translation to F10 for scale resolution:
```python
vo_result = F07.compute_relative_pose(prev_image, curr_image)
# vo_result.translation is a UNIT VECTOR (||t|| = 1)
# F02.2 passes to F10 which scales using:
# 1. altitude = F17.get_operational_altitude(flight_id)
# 2. gsd = H02.compute_gsd(altitude, camera_params)
# 3. expected_displacement = frame_spacing * gsd
# 4. scaled_translation = vo_result.translation * expected_displacement
F10.add_relative_factor(flight_id, frame_i, frame_j, vo_result, covariance)
```
**Error Conditions**:
- Returns `None`: Insufficient inliers (< 8 points for Essential Matrix)
**Test Cases**:
1. **Good matches**: Returns motion with high inlier count
2. **Low inliers**: May return None
3. **Degenerate motion**: Handles pure rotation
## Integration Tests
### Test 1: Normal Flight Sequence
1. Load consecutive frames with 50% overlap
2. compute_relative_pose() → returns valid pose
3. Verify translation direction reasonable
4. Verify inlier_count > 100
### Test 2: Low Overlap Scenario
1. Load frames with 5% overlap
2. compute_relative_pose() → still succeeds
3. Verify inlier_count > 20
4. Verify combined model finds matches despite low overlap
### Test 3: Tracking Loss
1. Load frames with 0% overlap (sharp turn)
2. compute_relative_pose() → returns None
3. Verify tracking_good = False
4. Trigger global place recognition
### Test 4: Agricultural Texture
1. Load images of wheat fields (repetitive texture)
2. compute_relative_pose() → SuperPoint handles better than SIFT
3. Verify match quality
### Test 5: VO with Chunk Routing
1. Create chunk_1 and chunk_2 via F12
2. compute_relative_pose() for frames in chunk_1, F02.2 routes to chunk_1
3. compute_relative_pose() for frames in chunk_2, F02.2 routes to chunk_2
4. Verify F02.2 calls F12.add_frame_to_chunk() with correct chunk_id
5. Verify chunks optimized independently via F10
## Non-Functional Requirements
### Performance
- **compute_relative_pose**: < 150ms total
- Combined SP+LG inference: ~50-80ms
- Motion estimation: ~10ms
- **Frame rate**: 5-10 FPS processing (meets <5s requirement)
### Accuracy
- **Relative rotation**: ±2° error
- **Relative translation direction**: ±5° error
- **Inlier ratio**: >50% for good tracking
### Reliability
- Handle 100m spacing between frames
- Survive temporary tracking degradation
- Recover from brief occlusions
## Dependencies
### Internal Components
- **F16 Model Manager**: For combined SuperPoint+LightGlue TensorRT model
- **F17 Configuration Manager**: For camera parameters
- **H01 Camera Model**: For coordinate normalization
- **H05 Performance Monitor**: For timing measurements
**Note**: F07 is chunk-agnostic and does NOT depend on F10 Factor Graph Optimizer. F07 only computes relative poses between images and returns them to the caller (F02.2). The caller (F02.2) determines which chunk the frames belong to and routes factors to the appropriate subgraph via F12 → F10.
### External Dependencies
- **Combined SuperPoint+LightGlue**: Single TensorRT engine for extraction + matching
- **opencv-python**: Essential Matrix estimation
- **numpy**: Matrix operations
## Data Models
### Matches
```python
class Matches(BaseModel):
matches: np.ndarray # (M, 2) - pairs of indices
scores: np.ndarray # (M,) - match confidence
keypoints1: np.ndarray # (M, 2)
keypoints2: np.ndarray # (M, 2)
```
### RelativePose
```python
class RelativePose(BaseModel):
translation: np.ndarray # (3,) - unit vector
rotation: np.ndarray # (3, 3) or (4,) quaternion
confidence: float
inlier_count: int
total_matches: int
tracking_good: bool
scale_ambiguous: bool = True
chunk_id: Optional[str] = None
```
### Motion
```python
class Motion(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3)
inliers: np.ndarray # Boolean mask
inlier_count: int
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float
principal_point: Tuple[float, float]
resolution: Tuple[int, int]
```
_docs/02_components/08_global_place_recognition/08.01_feature_index_management.md
-64
View File
@@ -1,64 +0,0 @@
# Feature: Index Management
## Description
Load and manage pre-built satellite descriptor database (Faiss index). The semantic index is built by the satellite data provider offline using DINOv2 + VLAD - F08 only loads and validates the index. This is the foundation for all place recognition queries.
## Component APIs Implemented
- `load_index(flight_id: str, index_path: str) -> bool`
## External Tools and Services
- **H04 Faiss Index Manager**: For `load_index()`, `validate_index()` operations
- **Faiss**: Facebook similarity search library
## Internal Methods
### `_validate_index_integrity(index) -> bool`
Validates loaded Faiss index: checks descriptor dimensions (4096 or 8192), verifies tile count matches metadata.
### `_load_tile_metadata(metadata_path: str) -> Dict[int, TileMetadata]`
Loads tile_id → gps_center, bounds mapping from JSON file provided by satellite provider.
### `_verify_metadata_alignment(index, metadata: Dict) -> bool`
Ensures metadata entries match index size (same number of tiles).
## Unit Tests
### Test: Load Valid Index
- Input: Valid index file path from satellite provider
- Verify: Returns True, index operational
### Test: Index Not Found
- Input: Non-existent path
- Verify: Raises `IndexNotFoundError`
### Test: Corrupted Index
- Input: Corrupted/truncated index file
- Verify: Raises `IndexCorruptedError`
### Test: Dimension Validation
- Input: Index with wrong descriptor dimensions
- Verify: Raises `IndexCorruptedError` with descriptive message
### Test: Metadata Mismatch
- Input: Index with 1000 entries, metadata with 500 entries
- Verify: Raises `MetadataMismatchError`
### Test: Empty Metadata File
- Input: Valid index, empty metadata JSON
- Verify: Raises `MetadataMismatchError`
### Test: Load Performance
- Input: Index with 10,000 tiles
- Verify: Load completes in <10 seconds
## Integration Tests
### Test: Faiss Manager Integration
- Verify: Successfully delegates index loading to H04 Faiss Index Manager
- Verify: Index accessible for subsequent queries
### Test: Query After Load
- Setup: Load valid index
- Action: Query with random descriptor
- Verify: Returns valid matches (not empty, valid indices)
_docs/02_components/08_global_place_recognition/08.02_feature_descriptor_computation.md
-95
View File
@@ -1,95 +0,0 @@
# Feature: Descriptor Computation
## Description
Compute global location descriptors using DINOv2 + VLAD aggregation. Supports both single-image descriptors and aggregate chunk descriptors for robust matching. DINOv2 features are semantic and invariant to season/texture changes, critical for UAV-to-satellite domain gap.
## Component APIs Implemented
- `compute_location_descriptor(image: np.ndarray) -> np.ndarray`
- `compute_chunk_descriptor(chunk_images: List[np.ndarray]) -> np.ndarray`
## External Tools and Services
- **F16 Model Manager**: Provides DINOv2 inference engine via `get_inference_engine("DINOv2")`
- **DINOv2**: Meta's foundation vision model for semantic feature extraction
- **numpy**: Array operations and L2 normalization
## Internal Methods
### `_preprocess_image(image: np.ndarray) -> np.ndarray`
Resizes and normalizes image for DINOv2 input (typically 224x224 or 518x518).
### `_extract_dense_features(preprocessed: np.ndarray) -> np.ndarray`
Runs DINOv2 inference, extracts dense feature map from multiple spatial locations.
### `_vlad_aggregate(dense_features: np.ndarray, codebook: np.ndarray) -> np.ndarray`
Applies VLAD (Vector of Locally Aggregated Descriptors) aggregation using pre-trained cluster centers.
### `_l2_normalize(descriptor: np.ndarray) -> np.ndarray`
L2-normalizes descriptor vector for cosine similarity search.
### `_aggregate_chunk_descriptors(descriptors: List[np.ndarray], strategy: str) -> np.ndarray`
Aggregates multiple descriptors into one using strategy: "mean" (default), "vlad", or "max".
## Unit Tests
### compute_location_descriptor
#### Test: Output Dimensions
- Input: UAV image (any resolution)
- Verify: Returns 4096-dim or 8192-dim vector
#### Test: Normalization
- Input: Any valid image
- Verify: Output L2 norm equals 1.0
#### Test: Deterministic Output
- Input: Same image twice
- Verify: Identical descriptors (no randomness)
#### Test: Season Invariance
- Input: Two images of same location, different seasons
- Verify: Cosine similarity > 0.7
#### Test: Location Discrimination
- Input: Two images of different locations
- Verify: Cosine similarity < 0.5
#### Test: Domain Invariance
- Input: UAV image and satellite image of same location
- Verify: Cosine similarity > 0.6 (cross-domain)
### compute_chunk_descriptor
#### Test: Empty Chunk
- Input: Empty list
- Verify: Raises ValueError
#### Test: Single Image Chunk
- Input: List with one image
- Verify: Equivalent to compute_location_descriptor
#### Test: Multiple Images Mean Aggregation
- Input: 5 images from chunk
- Verify: Descriptor is mean of individual descriptors
#### Test: Aggregated Normalization
- Input: Any chunk
- Verify: Output L2 norm equals 1.0
#### Test: Chunk More Robust Than Single
- Input: Featureless terrain chunk (10 images)
- Verify: Chunk descriptor has lower variance than individual descriptors
## Integration Tests
### Test: Model Manager Integration
- Verify: Successfully retrieves DINOv2 from F16 Model Manager
- Verify: Model loaded with correct TensorRT/ONNX backend
### Test: Performance Budget
- Input: FullHD image
- Verify: Single descriptor computed in ~150ms
### Test: Chunk Performance
- Input: 10 images
- Verify: Chunk descriptor in <2s (parallelizable)
_docs/02_components/08_global_place_recognition/08.03_feature_candidate_retrieval.md
-130
View File
@@ -1,130 +0,0 @@
# Feature: Candidate Retrieval
## Description
Query satellite database and retrieve ranked tile candidates. Orchestrates the full place recognition pipeline: descriptor computation → database query → candidate ranking. Supports both single-image and chunk-based retrieval for "kidnapped robot" recovery.
## Component APIs Implemented
- `query_database(descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]`
- `rank_candidates(candidates: List[TileCandidate]) -> List[TileCandidate]`
- `retrieve_candidate_tiles(image: np.ndarray, top_k: int) -> List[TileCandidate]`
- `retrieve_candidate_tiles_for_chunk(chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]`
## External Tools and Services
- **H04 Faiss Index Manager**: For `search()` operation
- **F04 Satellite Data Manager**: For tile metadata retrieval (gps_center, bounds) after Faiss returns indices
- **F12 Route Chunk Manager**: Provides chunk images for chunk-based retrieval
## Internal Methods
### `_distance_to_similarity(distance: float) -> float`
Converts L2 distance to normalized similarity score [0, 1].
### `_retrieve_tile_metadata(indices: List[int]) -> List[TileMetadata]`
Fetches GPS center and bounds for tile indices from F04 Satellite Data Manager.
### `_apply_spatial_reranking(candidates: List[TileCandidate], dead_reckoning_estimate: Optional[GPSPoint]) -> List[TileCandidate]`
Re-ranks candidates based on proximity to dead-reckoning estimate if available.
### `_apply_trajectory_reranking(candidates: List[TileCandidate], previous_trajectory: Optional[List[GPSPoint]]) -> List[TileCandidate]`
Favors tiles that continue the previous trajectory direction.
### `_filter_by_geofence(candidates: List[TileCandidate], geofence: Optional[BoundingBox]) -> List[TileCandidate]`
Removes candidates outside operational geofence.
## Unit Tests
### query_database
#### Test: Returns Top-K Matches
- Input: Valid descriptor, top_k=5
- Verify: Returns exactly 5 DatabaseMatch objects
#### Test: Ordered by Distance
- Input: Any descriptor
- Verify: Matches sorted by ascending distance
#### Test: Similarity Score Range
- Input: Any query
- Verify: All similarity_score values in [0, 1]
#### Test: Empty Database
- Input: Query when no index loaded
- Verify: Returns empty list (not exception)
#### Test: Query Performance
- Input: Large database (10,000 tiles)
- Verify: Query completes in <50ms
### rank_candidates
#### Test: Preserves Order Without Heuristics
- Input: Candidates without dead-reckoning estimate
- Verify: Order unchanged (similarity-based)
#### Test: Spatial Reranking Applied
- Input: Candidates + dead-reckoning estimate
- Verify: Closer tile promoted in ranking
#### Test: Tie Breaking
- Input: Two candidates with similar similarity scores
- Verify: Spatial proximity breaks tie
#### Test: Geofence Filtering
- Input: Candidates with some outside geofence
- Verify: Out-of-bounds candidates removed
### retrieve_candidate_tiles
#### Test: End-to-End Single Image
- Input: UAV image, top_k=5
- Verify: Returns 5 TileCandidate with valid gps_center
#### Test: Correct Tile in Top-5
- Input: UAV image with known location
- Verify: Correct tile appears in top-5 (Recall@5 test)
#### Test: Performance Budget
- Input: FullHD UAV image
- Verify: Total time <200ms (descriptor ~150ms + query ~50ms)
### retrieve_candidate_tiles_for_chunk
#### Test: End-to-End Chunk
- Input: 10 chunk images, top_k=5
- Verify: Returns 5 TileCandidate
#### Test: Chunk More Accurate Than Single
- Input: Featureless terrain images
- Verify: Chunk retrieval finds correct tile where single-image fails
#### Test: Recall@5 > 90%
- Input: Various chunk scenarios
- Verify: Correct tile in top-5 at least 90% of test cases
## Integration Tests
### Test: Faiss Manager Integration
- Verify: query_database correctly delegates to H04 Faiss Index Manager
### Test: Satellite Data Manager Integration
- Verify: Tile metadata correctly retrieved from F04 after Faiss query
### Test: Full Pipeline Single Image
- Setup: Load index, prepare UAV image
- Action: retrieve_candidate_tiles()
- Verify: Returns valid candidates with GPS coordinates
### Test: Full Pipeline Chunk
- Setup: Load index, prepare chunk images
- Action: retrieve_candidate_tiles_for_chunk()
- Verify: Returns valid candidates, more robust than single-image
### Test: Season Invariance
- Setup: Satellite tiles from summer, UAV image from autumn
- Action: retrieve_candidate_tiles()
- Verify: Correct match despite appearance change
### Test: Recall@5 Benchmark
- Input: Test dataset of 100 UAV images with ground truth
- Verify: Recall@5 > 85% for single-image, > 90% for chunk
_docs/02_components/08_global_place_recognition/08._component_global_place_recognition.md
-424
View File
@@ -1,424 +0,0 @@
# Global Place Recognition
## Interface Definition
**Interface Name**: `IGlobalPlaceRecognition`
### Interface Methods
```python
class IGlobalPlaceRecognition(ABC):
@abstractmethod
def retrieve_candidate_tiles(self, image: np.ndarray, top_k: int) -> List[TileCandidate]:
pass
@abstractmethod
def compute_location_descriptor(self, image: np.ndarray) -> np.ndarray:
pass
@abstractmethod
def query_database(self, descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]:
pass
@abstractmethod
def rank_candidates(self, candidates: List[TileCandidate]) -> List[TileCandidate]:
pass
@abstractmethod
def load_index(self, flight_id: str, index_path: str) -> bool:
pass
@abstractmethod
def retrieve_candidate_tiles_for_chunk(self, chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]:
pass
@abstractmethod
def compute_chunk_descriptor(self, chunk_images: List[np.ndarray]) -> np.ndarray:
pass
```
## Component Description
### Responsibilities
- AnyLoc (DINOv2 + VLAD) for coarse localization after tracking loss
- "Kidnapped robot" recovery after sharp turns
- Compute image descriptors robust to season/appearance changes
- Query Faiss index of satellite tile descriptors
- Return top-k candidate tile regions for progressive refinement
- **Load pre-built satellite descriptor index** (index is built by satellite provider, NOT by F08)
- **Chunk semantic matching (aggregate DINOv2 features)**
- **Chunk descriptor computation for robust matching**
### Scope
- Global localization (not frame-to-frame)
- Appearance-based place recognition
- Handles domain gap (UAV vs satellite imagery)
- Semantic feature extraction (DINOv2)
- Efficient similarity search (Faiss)
- **Chunk-level matching (more robust than single-image)**
## API Methods
### `retrieve_candidate_tiles(image: np.ndarray, top_k: int) -> List[TileCandidate]`
**Description**: Retrieves top-k candidate satellite tiles for a UAV image.
**Called By**:
- F11 Failure Recovery Coordinator (after tracking loss)
**Input**:
```python
image: np.ndarray # UAV image
top_k: int # Number of candidates (typically 5)
```
**Output**:
```python
List[TileCandidate]:
tile_id: str
gps_center: GPSPoint
similarity_score: float
rank: int
```
**Processing Flow**:
1. compute_location_descriptor(image) → descriptor
2. query_database(descriptor, top_k) → database_matches
3. Retrieve tile metadata for matches
4. rank_candidates() → sorted by similarity
5. Return top-k candidates
**Error Conditions**:
- Returns empty list: Database not initialized, query failed
**Test Cases**:
1. **UAV image over Ukraine**: Returns relevant tiles
2. **Different season**: DINOv2 handles appearance change
3. **Top-1 accuracy**: Correct tile in top-5 > 85%
---
### `compute_location_descriptor(image: np.ndarray) -> np.ndarray`
**Description**: Computes global descriptor using DINOv2 + VLAD aggregation.
**Called By**:
- Internal (during retrieve_candidate_tiles)
- System initialization (for satellite database)
**Input**:
```python
image: np.ndarray # UAV or satellite image
```
**Output**:
```python
np.ndarray: Descriptor vector (4096-dim or 8192-dim)
```
**Algorithm (AnyLoc)**:
1. Extract DINOv2 features (dense feature map)
2. Apply VLAD (Vector of Locally Aggregated Descriptors) aggregation
3. L2-normalize descriptor
4. Return compact global descriptor
**Processing Details**:
- Uses F16 Model Manager to get DINOv2 model
- Dense features: extracts from multiple spatial locations
- VLAD codebook: pre-trained cluster centers
- Semantic features: invariant to texture/color changes
**Performance**:
- Inference time: ~150ms for DINOv2 + VLAD
**Test Cases**:
1. **Same location, different season**: Similar descriptors
2. **Different locations**: Dissimilar descriptors
3. **UAV vs satellite**: Domain-invariant features
---
### `query_database(descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]`
**Description**: Queries Faiss index for most similar satellite tiles.
**Called By**:
- Internal (during retrieve_candidate_tiles)
**Input**:
```python
descriptor: np.ndarray # Query descriptor
top_k: int
```
**Output**:
```python
List[DatabaseMatch]:
index: int # Tile index in database
distance: float # L2 distance
similarity_score: float # Normalized score
```
**Processing Details**:
- Uses H04 Faiss Index Manager
- Index type: IVF (Inverted File) or HNSW for fast search
- Distance metric: L2 (Euclidean)
- Query time: ~10-50ms for 10,000+ tiles
**Error Conditions**:
- Returns empty list: Query failed
**Test Cases**:
1. **Query satellite database**: Returns top-5 matches
2. **Large database (10,000 tiles)**: Fast retrieval (<50ms)
---
### `rank_candidates(candidates: List[TileCandidate]) -> List[TileCandidate]`
**Description**: Re-ranks candidates based on additional heuristics.
**Called By**:
- Internal (during retrieve_candidate_tiles)
**Input**:
```python
candidates: List[TileCandidate] # Initial ranking by similarity
```
**Output**:
```python
List[TileCandidate] # Re-ranked list
```
**Re-ranking Factors**:
1. **Similarity score**: Primary factor
2. **Spatial proximity**: Prefer tiles near dead-reckoning estimate
3. **Previous trajectory**: Favor continuation of route
4. **Geofence constraints**: Within operational area
**Test Cases**:
1. **Spatial re-ranking**: Closer tile promoted
2. **Similar scores**: Spatial proximity breaks tie
---
### `load_index(flight_id: str, index_path: str) -> bool`
**Description**: Loads pre-built satellite descriptor database from file. **Note**: The semantic index (DINOv2 descriptors + Faiss index) MUST be provided by the satellite data provider. F08 does NOT build the index - it only loads it.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight initialization, index_path from F04 Satellite Data Manager)
**Input**:
```python
index_path: str # Path to pre-built Faiss index file from satellite provider
```
**Output**:
```python
bool: True if database loaded successfully
```
**Processing Flow**:
1. Load pre-built Faiss index from index_path
2. Load tile metadata (tile_id → gps_center, bounds mapping)
3. Validate index integrity (check descriptor dimensions, tile count)
4. Return True if loaded successfully
**Satellite Provider Responsibility**:
- Satellite provider builds the semantic index offline using DINOv2 + VLAD
- Provider delivers index file along with satellite tiles
- **Index format**: Faiss IVF1000 (Inverted File with 1000 clusters) + tile metadata JSON
- Provider is responsible for index updates when satellite data changes
- Index is rebuilt by provider whenever new satellite tiles are fetched on demand
- Supported providers: Maxar, Google Maps, Copernicus, etc.
**Error Conditions**:
- Raises `IndexNotFoundError`: Index file not found
- Raises `IndexCorruptedError`: Index file corrupted or invalid format
- Raises `MetadataMismatchError`: Metadata doesn't match index
**Performance**:
- **Load time**: <10 seconds for 10,000+ tiles
**Test Cases**:
1. **Load valid index**: Completes successfully, index operational
2. **Index not found**: Raises IndexNotFoundError
3. **Corrupted index**: Raises IndexCorruptedError
4. **Index query after load**: Works correctly
---
### `retrieve_candidate_tiles_for_chunk(chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]`
**Description**: Retrieves top-k candidate satellite tiles for a chunk using aggregate descriptor.
**Called By**:
- F11 Failure Recovery Coordinator (chunk semantic matching)
- F12 Route Chunk Manager (chunk matching coordination)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
top_k: int # Number of candidates (typically 5)
```
**Output**:
```python
List[TileCandidate]:
tile_id: str
gps_center: GPSPoint
similarity_score: float
rank: int
```
**Processing Flow**:
1. compute_chunk_descriptor(chunk_images) → aggregate descriptor
2. query_database(descriptor, top_k) → database_matches
3. Retrieve tile metadata for matches
4. rank_candidates() → sorted by similarity
5. Return top-k candidates
**Advantages over Single-Image Matching**:
- Aggregate descriptor more robust to featureless terrain
- Multiple images provide more context
- Better handles plain fields where single-image matching fails
**Test Cases**:
1. **Chunk matching**: Returns relevant tiles
2. **Featureless terrain**: Succeeds where single-image fails
3. **Top-1 accuracy**: Correct tile in top-5 > 90% (better than single-image)
---
### `compute_chunk_descriptor(chunk_images: List[np.ndarray]) -> np.ndarray`
**Description**: Computes aggregate DINOv2 descriptor from multiple chunk images.
**Called By**:
- Internal (during retrieve_candidate_tiles_for_chunk)
- F12 Route Chunk Manager (chunk descriptor computation - delegates to F08)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
```
**Output**:
```python
np.ndarray: Aggregated descriptor vector (4096-dim or 8192-dim)
```
**Algorithm**:
1. For each image in chunk:
- compute_location_descriptor(image) → descriptor (DINOv2 + VLAD)
2. Aggregate descriptors:
- **Mean aggregation**: Average all descriptors
- **VLAD aggregation**: Use VLAD codebook for aggregation
- **Max aggregation**: Element-wise maximum
3. L2-normalize aggregated descriptor
4. Return composite descriptor
**Aggregation Strategy**:
- **Mean**: Simple average (default)
- **VLAD**: More sophisticated, preserves spatial information
- **Max**: Emphasizes strongest features
**Performance**:
- Descriptor computation: ~150ms × N images (can be parallelized)
- Aggregation: ~10ms
**Test Cases**:
1. **Compute descriptor**: Returns aggregated descriptor
2. **Multiple images**: Descriptor aggregates correctly
3. **Descriptor quality**: More robust than single-image descriptor
## Integration Tests
### Test 1: Place Recognition Flow
1. Load UAV image from sharp turn
2. retrieve_candidate_tiles(top_k=5)
3. Verify correct tile in top-5
4. Pass candidates to F11 Failure Recovery
### Test 2: Season Invariance
1. Satellite tiles from summer
2. UAV images from autumn
3. retrieve_candidate_tiles() → correct match despite appearance change
### Test 3: Index Loading
1. Prepare pre-built index file from satellite provider
2. load_index(index_path)
3. Verify Faiss index loaded correctly
4. Query with test image → returns matches
### Test 4: Chunk Semantic Matching
1. Build chunk with 10 images (plain field scenario)
2. compute_chunk_descriptor() → aggregate descriptor
3. retrieve_candidate_tiles_for_chunk() → returns candidates
4. Verify correct tile in top-5 (where single-image matching failed)
5. Verify chunk matching more robust than single-image
## Non-Functional Requirements
### Performance
- **retrieve_candidate_tiles**: < 200ms total
- Descriptor computation: ~150ms
- Database query: ~50ms
- **compute_location_descriptor**: ~150ms
- **query_database**: ~10-50ms
### Accuracy
- **Recall@5**: > 85% (correct tile in top-5)
- **Recall@1**: > 60% (correct tile is top-1)
### Scalability
- Support 10,000+ satellite tiles in database
- Fast query even with large database
## Dependencies
### Internal Components
- **F16 Model Manager**: For DINOv2 inference engine via `get_inference_engine("DINOv2")`.
- **H04 Faiss Index Manager**: For similarity search via `load_index()`, `search()`. Critical for `query_database()`.
- **F04 Satellite Data Manager**: For tile metadata retrieval after Faiss search returns tile indices.
- **F12 Route Chunk Manager**: For chunk image retrieval during chunk descriptor computation.
### External Dependencies
- **DINOv2**: Foundation vision model
- **Faiss**: Similarity search library
- **numpy**: Array operations
## Data Models
### TileCandidate
```python
class TileCandidate(BaseModel):
tile_id: str
gps_center: GPSPoint
bounds: TileBounds
similarity_score: float
rank: int
spatial_score: Optional[float]
```
### DatabaseMatch
```python
class DatabaseMatch(BaseModel):
index: int
tile_id: str
distance: float
similarity_score: float
```
### SatelliteTile
```python
class SatelliteTile(BaseModel):
tile_id: str
image: np.ndarray
gps_center: GPSPoint
bounds: TileBounds
descriptor: Optional[np.ndarray]
```
_docs/02_components/09_metric_refinement/09.01_feature_single_image_alignment.md
-55
View File
@@ -1,55 +0,0 @@
# Feature: Single Image Alignment
## Description
Core UAV-to-satellite cross-view matching for individual frames using LiteSAM. Computes precise GPS coordinates by aligning a pre-rotated UAV image to a georeferenced satellite tile through homography estimation.
## Component APIs Implemented
- `align_to_satellite(uav_image, satellite_tile, tile_bounds) -> AlignmentResult`
- `compute_homography(uav_image, satellite_tile) -> Optional[np.ndarray]`
- `extract_gps_from_alignment(homography, tile_bounds, image_center) -> GPSPoint`
- `compute_match_confidence(alignment) -> float`
## External Tools and Services
- **LiteSAM**: Cross-view matching model (TAIFormer encoder, CTM correlation)
- **opencv-python**: RANSAC homography estimation, image operations
- **numpy**: Matrix operations, coordinate transformations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_extract_features(image)` | Extract multi-scale features using LiteSAM TAIFormer encoder |
| `_compute_correspondences(uav_features, sat_features)` | Compute dense correspondence field via CTM |
| `_estimate_homography_ransac(correspondences)` | Estimate 3×3 homography using RANSAC |
| `_refine_homography(homography, correspondences)` | Non-linear refinement of homography |
| `_validate_match(homography, inliers)` | Check inlier count/ratio thresholds |
| `_pixel_to_gps(pixel, tile_bounds)` | Convert satellite pixel coordinates to GPS |
| `_compute_inlier_ratio(inliers, total)` | Calculate inlier ratio for confidence |
| `_compute_spatial_distribution(inliers)` | Assess inlier spatial distribution quality |
| `_compute_reprojection_error(homography, correspondences)` | Calculate mean reprojection error |
## Unit Tests
1. **Feature extraction**: LiteSAM encoder produces valid feature tensors
2. **Correspondence computation**: CTM produces dense correspondence field
3. **Homography estimation**: RANSAC returns valid 3×3 matrix for good correspondences
4. **Homography estimation failure**: Returns None for insufficient correspondences (<15 inliers)
5. **GPS extraction accuracy**: Pixel-to-GPS conversion within expected tolerance
6. **Confidence high**: Returns >0.8 for inlier_ratio >0.6, inlier_count >50, MRE <0.5px
7. **Confidence medium**: Returns 0.5-0.8 for moderate match quality
8. **Confidence low**: Returns <0.5 for poor matches
9. **Reprojection error calculation**: Correctly computes mean pixel error
10. **Spatial distribution scoring**: Penalizes clustered inliers
## Integration Tests
1. **Single tile drift correction**: Load UAV image + satellite tile → align_to_satellite() returns GPS within 20m of ground truth
2. **Progressive search (4 tiles)**: align_to_satellite() on 2×2 grid, first 3 fail, 4th succeeds
3. **Rotation sensitivity**: Unrotated image (>45°) fails; pre-rotated image succeeds
4. **Multi-scale robustness**: Different GSD (UAV 0.1m/px, satellite 0.3m/px) → match succeeds
5. **Altitude variation**: UAV at various altitudes (<1km) → consistent GPS accuracy
6. **Performance benchmark**: align_to_satellite() completes in ~60ms (TensorRT)
_docs/02_components/09_metric_refinement/09.02_feature_chunk_alignment.md
-51
View File
@@ -1,51 +0,0 @@
# Feature: Chunk Alignment
## Description
Batch UAV-to-satellite matching that aggregates correspondences from multiple images in a chunk for more robust geo-localization. Handles scenarios where single-image matching fails (featureless terrain, partial occlusions). Returns Sim(3) transform for the entire chunk.
## Component APIs Implemented
- `align_chunk_to_satellite(chunk_images, satellite_tile, tile_bounds) -> ChunkAlignmentResult`
- `match_chunk_homography(chunk_images, satellite_tile) -> Optional[np.ndarray]`
## External Tools and Services
- **LiteSAM**: Cross-view matching model (TAIFormer encoder, CTM correlation)
- **opencv-python**: RANSAC homography estimation
- **numpy**: Matrix operations, feature aggregation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_extract_chunk_features(chunk_images)` | Extract features from all chunk images |
| `_aggregate_features(features_list)` | Combine features via mean/max pooling |
| `_aggregate_correspondences(correspondences_list)` | Merge correspondences from multiple images |
| `_estimate_chunk_homography(aggregated_correspondences)` | Estimate homography from aggregate data |
| `_compute_sim3_transform(homography, tile_bounds)` | Extract translation, rotation, scale |
| `_get_chunk_center_gps(homography, tile_bounds, chunk_images)` | GPS of middle frame center |
| `_validate_chunk_match(inliers, confidence)` | Check chunk-specific thresholds (>30 inliers) |
## Unit Tests
1. **Feature aggregation**: Mean pooling produces valid combined features
2. **Correspondence aggregation**: Merges correspondences from N images correctly
3. **Chunk homography estimation**: Returns valid 3×3 matrix for aggregate correspondences
4. **Chunk homography failure**: Returns None for insufficient aggregate correspondences
5. **Sim(3) extraction**: Correctly decomposes homography into translation, rotation, scale
6. **Chunk center GPS**: Returns GPS of middle frame's center pixel
7. **Chunk confidence high**: Returns >0.7 for >50 inliers
8. **Chunk confidence medium**: Returns 0.5-0.7 for 30-50 inliers
9. **Chunk validation**: Rejects matches with <30 inliers
## Integration Tests
1. **Chunk LiteSAM matching**: 10 images from plain field → align_chunk_to_satellite() returns GPS within 20m
2. **Chunk vs single-image robustness**: Featureless terrain where single-image fails, chunk succeeds
3. **Chunk rotation sweeps**: Unknown orientation → try rotations (0°, 30°, ..., 330°) → match at correct angle
4. **Sim(3) transform correctness**: Verify transform aligns chunk trajectory to satellite coordinates
5. **Multi-scale chunk matching**: GSD mismatch handled correctly
6. **Performance benchmark**: 10-image chunk alignment completes within acceptable time
7. **Partial occlusion handling**: Some images occluded → chunk still matches successfully
_docs/02_components/09_metric_refinement/09._component_metric_refinement.md
-462
View File
@@ -1,462 +0,0 @@
# Metric Refinement
## Interface Definition
**Interface Name**: `IMetricRefinement`
### Interface Methods
```python
class IMetricRefinement(ABC):
@abstractmethod
def align_to_satellite(self, uav_image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[AlignmentResult]:
pass
@abstractmethod
def compute_homography(self, uav_image: np.ndarray, satellite_tile: np.ndarray) -> Optional[np.ndarray]:
pass
@abstractmethod
def extract_gps_from_alignment(self, homography: np.ndarray, tile_bounds: TileBounds, image_center: Tuple[int, int]) -> GPSPoint:
pass
@abstractmethod
def compute_match_confidence(self, alignment: AlignmentResult) -> float:
pass
@abstractmethod
def align_chunk_to_satellite(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[ChunkAlignmentResult]:
pass
@abstractmethod
def match_chunk_homography(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray) -> Optional[np.ndarray]:
pass
```
## Component Description
### Responsibilities
- LiteSAM for precise UAV-to-satellite cross-view matching
- **Requires pre-rotated images** from Image Rotation Manager
- Compute homography mapping UAV image to satellite tile
- Extract absolute GPS coordinates from alignment
- Process against single tile (drift correction) or tile grid (progressive search)
- Achieve <20m accuracy requirement
- **Chunk-to-satellite matching (more robust than single-image)**
- **Chunk homography computation**
### Scope
- Cross-view geo-localization (UAV↔satellite)
- Handles altitude variations (<1km)
- Multi-scale processing for different GSDs
- Domain gap (UAV downward vs satellite nadir view)
- **Critical**: Fails if rotation >45° (handled by F06)
- **Chunk-level matching (aggregate correspondences from multiple images)**
## API Methods
### `align_to_satellite(uav_image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[AlignmentResult]`
**Description**: Aligns UAV image to satellite tile, returning GPS location.
**Called By**:
- F06 Image Rotation Manager (during rotation sweep)
- F11 Failure Recovery Coordinator (progressive search)
- F02.2 Flight Processing Engine (drift correction with single tile)
**Input**:
```python
uav_image: np.ndarray # Pre-rotated UAV image
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of the satellite tile
```
**Output**:
```python
AlignmentResult:
matched: bool
homography: np.ndarray # 3×3 transformation matrix
gps_center: GPSPoint # UAV image center GPS
confidence: float
inlier_count: int
total_correspondences: int
```
**Processing Flow**:
1. Extract features from both images using LiteSAM encoder
2. Compute dense correspondence field
3. Estimate homography from correspondences
4. Validate match quality (inlier count, reprojection error)
5. If valid match:
- Extract GPS from homography using tile_bounds
- Return AlignmentResult
6. If no match:
- Return None
**Match Criteria**:
- **Good match**: inlier_count > 30, confidence > 0.7
- **Weak match**: inlier_count 15-30, confidence 0.5-0.7
- **No match**: inlier_count < 15
**Error Conditions**:
- Returns `None`: No match found, rotation >45° (should be pre-rotated)
**Test Cases**:
1. **Good alignment**: Returns GPS within 20m of ground truth
2. **Altitude variation**: Handles GSD mismatch
3. **Rotation >45°**: Fails (by design, requires pre-rotation)
4. **Multi-scale**: Processes at multiple scales
---
### `compute_homography(uav_image: np.ndarray, satellite_tile: np.ndarray) -> Optional[np.ndarray]`
**Description**: Computes homography transformation from UAV to satellite.
**Called By**:
- Internal (during align_to_satellite)
**Input**:
```python
uav_image: np.ndarray
satellite_tile: np.ndarray
```
**Output**:
```python
Optional[np.ndarray]: 3×3 homography matrix or None
```
**Algorithm (LiteSAM)**:
1. Extract multi-scale features using TAIFormer
2. Compute correlation via Convolutional Token Mixer (CTM)
3. Generate dense correspondences
4. Estimate homography using RANSAC
5. Refine with non-linear optimization
**Homography Properties**:
- Maps pixels from UAV image to satellite image
- Accounts for: scale, rotation, perspective
- 8 DoF (degrees of freedom)
**Error Conditions**:
- Returns `None`: Insufficient correspondences
**Test Cases**:
1. **Valid correspondence**: Returns 3×3 matrix
2. **Insufficient features**: Returns None
---
### `extract_gps_from_alignment(homography: np.ndarray, tile_bounds: TileBounds, image_center: Tuple[int, int]) -> GPSPoint`
**Description**: Extracts GPS coordinates from homography and tile georeferencing.
**Called By**:
- Internal (during align_to_satellite)
- F06 Image Rotation Manager (for precise angle calculation)
**Input**:
```python
homography: np.ndarray # 3×3 matrix
tile_bounds: TileBounds # GPS bounds of satellite tile
image_center: Tuple[int, int] # Center pixel of UAV image
```
**Output**:
```python
GPSPoint:
lat: float
lon: float
```
**Algorithm**:
1. Apply homography to UAV image center point
2. Get pixel coordinates in satellite tile
3. Convert satellite pixel to GPS using tile_bounds and GSD
4. Return GPS coordinates
**Uses**: tile_bounds parameter, H02 GSD Calculator
**Test Cases**:
1. **Center alignment**: UAV center → correct GPS
2. **Corner alignment**: UAV corner → correct GPS
3. **Multiple points**: All points consistent
---
### `compute_match_confidence(alignment: AlignmentResult) -> float`
**Description**: Computes match confidence score from alignment quality.
**Called By**:
- Internal (during align_to_satellite)
- F11 Failure Recovery Coordinator (to decide if match acceptable)
**Input**:
```python
alignment: AlignmentResult
```
**Output**:
```python
float: Confidence score (0.0 to 1.0)
```
**Confidence Factors**:
1. **Inlier ratio**: inliers / total_correspondences
2. **Inlier count**: Absolute number of inliers
3. **Reprojection error**: Mean error of inliers (in pixels)
4. **Spatial distribution**: Inliers well-distributed vs clustered
**Thresholds**:
- **High confidence (>0.8)**: inlier_ratio > 0.6, inlier_count > 50, MRE < 0.5px
- **Medium confidence (0.5-0.8)**: inlier_ratio > 0.4, inlier_count > 30
- **Low confidence (<0.5)**: Reject match
**Test Cases**:
1. **Good match**: confidence > 0.8
2. **Weak match**: confidence 0.5-0.7
3. **Poor match**: confidence < 0.5
---
### `align_chunk_to_satellite(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[ChunkAlignmentResult]`
**Description**: Aligns entire chunk to satellite tile, returning GPS location.
**Called By**:
- F06 Image Rotation Manager (during chunk rotation sweep)
- F11 Failure Recovery Coordinator (chunk LiteSAM matching)
**Input**:
```python
chunk_images: List[np.ndarray] # Pre-rotated chunk images (5-20 images)
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of the satellite tile
```
**Output**:
```python
ChunkAlignmentResult:
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint # GPS of chunk center (middle frame)
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform
```
**Processing Flow**:
1. For each image in chunk:
- Extract features using LiteSAM encoder
- Compute correspondences with satellite tile
2. Aggregate correspondences from all images
3. Estimate homography from aggregate correspondences
4. Validate match quality (inlier count, reprojection error)
5. If valid match:
- Extract GPS from chunk center using tile_bounds
- Compute Sim(3) transform (translation, rotation, scale)
- Return ChunkAlignmentResult
6. If no match:
- Return None
**Match Criteria**:
- **Good match**: inlier_count > 50, confidence > 0.7
- **Weak match**: inlier_count 30-50, confidence 0.5-0.7
- **No match**: inlier_count < 30
**Advantages over Single-Image Matching**:
- More correspondences (aggregate from multiple images)
- More robust to featureless terrain
- Better handles partial occlusions
- Higher confidence scores
**Test Cases**:
1. **Chunk alignment**: Returns GPS within 20m of ground truth
2. **Featureless terrain**: Succeeds where single-image fails
3. **Rotation >45°**: Fails (requires pre-rotation via F06)
4. **Multi-scale**: Handles GSD mismatch
---
### `match_chunk_homography(chunk_images: List[np.ndarray], satellite_tile: np.ndarray) -> Optional[np.ndarray]`
**Description**: Computes homography transformation from chunk to satellite.
**Called By**:
- Internal (during align_chunk_to_satellite)
**Input**:
```python
chunk_images: List[np.ndarray]
satellite_tile: np.ndarray
```
**Output**:
```python
Optional[np.ndarray]: 3×3 homography matrix or None
```
**Algorithm (LiteSAM)**:
1. Extract multi-scale features from all chunk images using TAIFormer
2. Aggregate features (mean or max pooling)
3. Compute correlation via Convolutional Token Mixer (CTM)
4. Generate dense correspondences
5. Estimate homography using RANSAC
6. Refine with non-linear optimization
**Homography Properties**:
- Maps pixels from chunk center to satellite image
- Accounts for: scale, rotation, perspective
- 8 DoF (degrees of freedom)
**Test Cases**:
1. **Valid correspondence**: Returns 3×3 matrix
2. **Insufficient features**: Returns None
3. **Aggregate correspondences**: More robust than single-image
## Integration Tests
### Test 1: Single Tile Drift Correction
1. Load UAV image and expected satellite tile
2. Pre-rotate UAV image to known heading
3. align_to_satellite() → returns GPS
4. Verify GPS within 20m of ground truth
### Test 2: Progressive Search (4 tiles)
1. Load UAV image from sharp turn
2. Get 2×2 tile grid from F04
3. align_to_satellite() for each tile (with tile_bounds)
4. First 3 tiles: No match
5. 4th tile: Match found → GPS extracted
### Test 3: Rotation Sensitivity
1. Rotate UAV image by 60° (not pre-rotated)
2. align_to_satellite() → returns None (fails as expected)
3. Pre-rotate to 60°
4. align_to_satellite() → succeeds
### Test 4: Multi-Scale Robustness
1. UAV at 500m altitude (GSD=0.1m/pixel)
2. Satellite at zoom 19 (GSD=0.3m/pixel)
3. LiteSAM handles scale difference → match succeeds
### Test 5: Chunk LiteSAM Matching
1. Build chunk with 10 images (plain field scenario)
2. Pre-rotate chunk to known heading
3. align_chunk_to_satellite() → returns GPS
4. Verify GPS within 20m of ground truth
5. Verify chunk matching more robust than single-image
### Test 6: Chunk Rotation Sweeps
1. Build chunk with unknown orientation
2. Try chunk rotation steps (0°, 30°, ..., 330°)
3. align_chunk_to_satellite() for each rotation
4. Match found at 120° → GPS extracted
5. Verify Sim(3) transform computed correctly
## Non-Functional Requirements
### Performance
- **align_to_satellite**: ~60ms per tile (TensorRT optimized)
- **Progressive search 25 tiles**: ~1.5 seconds total (25 × 60ms)
- Meets <5s per frame requirement
### Accuracy
- **GPS accuracy**: 60% of frames < 20m error, 80% < 50m error
- **Mean Reprojection Error (MRE)**: < 1.0 pixels
- **Alignment success rate**: > 95% when rotation correct
### Reliability
- Graceful failure when no match
- Robust to altitude variations (<1km)
- Handles seasonal appearance changes (to extent possible)
## Dependencies
### Internal Components
- **F12 Route Chunk Manager**: For chunk image retrieval and chunk operations
- **F16 Model Manager**: For LiteSAM model
- **H01 Camera Model**: For projection operations
- **H02 GSD Calculator**: For coordinate transformations
- **H05 Performance Monitor**: For timing
**Critical Dependency on F06 Image Rotation Manager**:
- F09 requires pre-rotated images (rotation <45° from north)
- Caller (F06 or F11) must pre-rotate images using F06.rotate_image_360() before calling F09.align_to_satellite()
- If rotation >45°, F09 will fail to match (by design)
- F06 handles the rotation sweep (trying 0°, 30°, 60°, etc.) and calls F09 for each rotation
**Note**: tile_bounds is passed as parameter from caller (F02.2 Flight Processing Engine gets it from F04 Satellite Data Manager)
### External Dependencies
- **LiteSAM**: Cross-view matching model
- **opencv-python**: Homography estimation
- **numpy**: Matrix operations
## Data Models
### AlignmentResult
```python
class AlignmentResult(BaseModel):
matched: bool
homography: np.ndarray # (3, 3)
gps_center: GPSPoint
confidence: float
inlier_count: int
total_correspondences: int
reprojection_error: float # Mean error in pixels
```
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float
lon: float
```
### TileBounds
```python
class TileBounds(BaseModel):
nw: GPSPoint
ne: GPSPoint
sw: GPSPoint
se: GPSPoint
center: GPSPoint
gsd: float # Ground Sampling Distance (m/pixel)
```
### LiteSAMConfig
```python
class LiteSAMConfig(BaseModel):
model_path: str
confidence_threshold: float = 0.7
min_inliers: int = 15
max_reprojection_error: float = 2.0 # pixels
multi_scale_levels: int = 3
chunk_min_inliers: int = 30 # Higher threshold for chunk matching
```
### ChunkAlignmentResult
```python
class ChunkAlignmentResult(BaseModel):
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform # Translation, rotation, scale
reprojection_error: float # Mean error in pixels
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/10_factor_graph_optimizer/10.01_feature_core_factor_management.md
-125
View File
@@ -1,125 +0,0 @@
# Feature: Core Factor Management
## Name
Core Factor Management
## Description
Provides the fundamental building blocks for factor graph construction by adding relative pose measurements (from VO), absolute GPS measurements (from LiteSAM or user anchors), and altitude priors. Handles scale resolution for monocular VO via GSD computation and applies robust kernels (Huber/Cauchy) for outlier handling.
## Component APIs Implemented
- `add_relative_factor(flight_id, frame_i, frame_j, relative_pose, covariance) -> bool`
- `add_absolute_factor(flight_id, frame_id, gps, covariance, is_user_anchor) -> bool`
- `add_altitude_prior(flight_id, frame_id, altitude, covariance) -> bool`
## External Tools and Services
- **GTSAM**: BetweenFactor, PriorFactor, UnaryFactor creation
- **H02 GSD Calculator**: Scale resolution for monocular VO (GSD computation)
- **H03 Robust Kernels**: Huber/Cauchy loss functions for outlier handling
- **F17 Configuration Manager**: Camera parameters, altitude, frame spacing
## Internal Methods
### `_scale_relative_translation(flight_id, translation, frame_i, frame_j) -> np.ndarray`
Scales unit translation vector from VO using GSD and expected displacement.
- Retrieves altitude and camera params from F17
- Calls H02.compute_gsd() to get ground sampling distance
- Computes expected_displacement = frame_spacing × GSD
- Returns scaled_translation = unit_translation × expected_displacement
### `_create_between_factor(frame_i, frame_j, pose, covariance) -> gtsam.BetweenFactor`
Creates GTSAM BetweenFactor for relative pose measurement.
- Creates Pose3 from translation and rotation
- Wraps with robust kernel (Huber) for outlier handling
- Returns configured factor
### `_create_prior_factor(frame_id, position, covariance, is_hard_constraint) -> gtsam.PriorFactor`
Creates GTSAM PriorFactor for absolute position constraint.
- Converts GPS (lat/lon) to ENU coordinates
- Sets covariance based on source (5m for user anchor, 20-50m for LiteSAM)
- Returns configured factor
### `_create_altitude_factor(frame_id, altitude, covariance) -> gtsam.UnaryFactor`
Creates GTSAM UnaryFactor for Z-coordinate constraint.
- Creates soft constraint on altitude
- Prevents scale drift in monocular VO
### `_apply_robust_kernel(factor, kernel_type, threshold) -> gtsam.Factor`
Wraps factor with robust kernel for outlier handling.
- Supports Huber (default) and Cauchy kernels
- Critical for 350m outlier handling
### `_gps_to_enu(gps, reference_origin) -> np.ndarray`
Converts GPS coordinates to local ENU (East-North-Up) frame.
- Uses reference origin (first frame or flight start GPS)
- Returns (x, y, z) in meters
### `_ensure_graph_initialized(flight_id) -> bool`
Ensures factor graph exists for flight, creates if needed.
- Creates new GTSAM iSAM2 instance if not exists
- Initializes reference origin for ENU conversion
## Unit Tests
### Test: add_relative_factor_scales_translation
- Arrange: Mock H02, F17 with known GSD values
- Act: Call add_relative_factor with unit translation
- Assert: Factor added with correctly scaled translation
### Test: add_relative_factor_applies_huber_kernel
- Arrange: Create factor graph
- Act: Add relative factor
- Assert: Factor has Huber kernel applied
### Test: add_absolute_factor_converts_gps_to_enu
- Arrange: Set reference origin
- Act: Add absolute factor with known GPS
- Assert: Factor position in correct ENU coordinates
### Test: add_absolute_factor_sets_covariance_by_source
- Arrange: Create factor graph
- Act: Add user anchor (is_user_anchor=True)
- Assert: Covariance is 5m (high confidence)
### Test: add_absolute_factor_litesam_covariance
- Arrange: Create factor graph
- Act: Add LiteSAM match (is_user_anchor=False)
- Assert: Covariance is 20-50m
### Test: add_altitude_prior_creates_soft_constraint
- Arrange: Create factor graph
- Act: Add altitude prior
- Assert: UnaryFactor created with correct covariance
### Test: add_relative_factor_returns_false_on_invalid_flight
- Arrange: No graph for flight_id
- Act: Call add_relative_factor
- Assert: Returns False (or creates graph per implementation)
### Test: scale_resolution_uses_f17_config
- Arrange: Mock F17 with specific altitude, camera params
- Act: Add relative factor
- Assert: H02 called with correct parameters from F17
## Integration Tests
### Test: incremental_trajectory_building
1. Initialize graph with first frame
2. Add 100 relative factors from VO
3. Add 100 altitude priors
4. Verify all factors added successfully
5. Verify graph structure correct
### Test: outlier_handling_350m
1. Add normal relative factors (10 frames)
2. Add 350m outlier factor (simulating tilt error)
3. Add more normal factors
4. Verify outlier factor weight reduced by Huber kernel
5. Verify graph remains stable
### Test: mixed_factor_types
1. Add relative factors
2. Add absolute GPS factors
3. Add altitude priors
4. Verify all factor types coexist correctly
5. Verify graph ready for optimization
_docs/02_components/10_factor_graph_optimizer/10.02_feature_trajectory_optimization.md
-125
View File
@@ -1,125 +0,0 @@
# Feature: Trajectory Optimization & Retrieval
## Name
Trajectory Optimization & Retrieval
## Description
Provides core optimization functionality using GTSAM's iSAM2 for incremental updates and Levenberg-Marquardt for batch optimization. Retrieves optimized trajectory poses and marginal covariances for uncertainty quantification. Supports both real-time incremental optimization (after each frame) and batch refinement (when new absolute factors added).
## Component APIs Implemented
- `optimize(flight_id, iterations) -> OptimizationResult`
- `get_trajectory(flight_id) -> Dict[int, Pose]`
- `get_marginal_covariance(flight_id, frame_id) -> np.ndarray`
## External Tools and Services
- **GTSAM**: iSAM2 incremental optimizer, Levenberg-Marquardt batch solver
- **numpy**: Matrix operations for covariance extraction
## Internal Methods
### `_run_isam2_update(flight_id, iterations) -> OptimizationResult`
Runs incremental iSAM2 optimization on flight's factor graph.
- Updates only affected nodes (fast, <100ms)
- Used for real-time frame-by-frame optimization
- Returns optimization statistics
### `_run_batch_optimization(flight_id, iterations) -> OptimizationResult`
Runs full Levenberg-Marquardt optimization on entire graph.
- Re-optimizes all poses (slower, ~500ms for 100 frames)
- Used when new absolute factors added for back-propagation
- Returns optimization statistics
### `_check_convergence(prev_error, curr_error, threshold) -> bool`
Checks if optimization has converged.
- Compares error reduction against threshold
- Returns True if error reduction < threshold
### `_extract_poses_from_graph(flight_id) -> Dict[int, Pose]`
Extracts all optimized poses from GTSAM Values.
- Converts GTSAM Pose3 to internal Pose model
- Includes position, orientation, timestamp
### `_compute_marginal_covariance(flight_id, frame_id) -> np.ndarray`
Computes marginal covariance for specific frame.
- Uses GTSAM Marginals class
- Returns 6x6 covariance matrix [x, y, z, roll, pitch, yaw]
### `_get_optimized_frames(flight_id, prev_values, curr_values) -> List[int]`
Determines which frames had pose updates.
- Compares previous and current values
- Returns list of frame IDs with significant changes
### `_compute_mean_reprojection_error(flight_id) -> float`
Computes mean reprojection error across all factors.
- Used for quality assessment
- Should be < 1.0 pixels per AC
## Unit Tests
### Test: optimize_incremental_updates_affected_nodes
- Arrange: Create graph with 10 frames, add new factor to frame 8
- Act: Call optimize with few iterations
- Assert: Only frames 8-10 significantly updated
### Test: optimize_batch_updates_all_frames
- Arrange: Create graph with drift, add absolute GPS at frame 50
- Act: Call optimize with batch mode
- Assert: All frames updated (back-propagation)
### Test: optimize_returns_convergence_status
- Arrange: Create well-constrained graph
- Act: Call optimize
- Assert: OptimizationResult.converged == True
### Test: get_trajectory_returns_all_poses
- Arrange: Create graph with 100 frames, optimize
- Act: Call get_trajectory
- Assert: Returns dict with 100 poses
### Test: get_trajectory_poses_in_enu
- Arrange: Create graph with known reference
- Act: Get trajectory
- Assert: Positions in ENU coordinates
### Test: get_marginal_covariance_returns_6x6
- Arrange: Create and optimize graph
- Act: Call get_marginal_covariance for frame
- Assert: Returns (6, 6) numpy array
### Test: get_marginal_covariance_reduces_after_absolute_factor
- Arrange: Create graph, get covariance for frame 50
- Act: Add absolute GPS factor at frame 50, re-optimize
- Assert: New covariance smaller than before
### Test: optimize_respects_iteration_limit
- Arrange: Create complex graph
- Act: Call optimize with iterations=5
- Assert: iterations_used <= 5
## Integration Tests
### Test: drift_correction_with_absolute_gps
1. Build trajectory with VO only (100 frames) - will drift
2. Add absolute GPS factor at frame 50
3. Optimize (batch mode)
4. Verify trajectory corrects
5. Verify frames 1-49 also corrected (back-propagation)
### Test: user_anchor_immediate_refinement
1. Build trajectory to frame 237
2. Add user anchor (high confidence, is_user_anchor=True)
3. Optimize
4. Verify trajectory snaps to anchor
5. Verify low covariance at anchor frame
### Test: performance_incremental_under_100ms
1. Create graph with 50 frames
2. Add new relative factor
3. Time incremental optimization
4. Assert: < 100ms
### Test: performance_batch_under_500ms
1. Create graph with 100 frames
2. Time batch optimization
3. Assert: < 500ms
_docs/02_components/10_factor_graph_optimizer/10.03_feature_chunk_subgraph_operations.md
-139
View File
@@ -1,139 +0,0 @@
# Feature: Chunk Subgraph Operations
## Name
Chunk Subgraph Operations
## Description
Manages chunk-level factor graph subgraphs for handling tracking loss recovery. Creates independent subgraphs for route chunks (map fragments), adds factors to chunk-specific subgraphs, anchors chunks with GPS measurements, and optimizes chunks independently. F10 provides low-level factor graph operations only; F12 owns chunk metadata (status, is_active, etc.).
## Component APIs Implemented
- `create_chunk_subgraph(flight_id, chunk_id, start_frame_id) -> bool`
- `add_relative_factor_to_chunk(flight_id, chunk_id, frame_i, frame_j, relative_pose, covariance) -> bool`
- `add_chunk_anchor(flight_id, chunk_id, frame_id, gps, covariance) -> bool`
- `get_chunk_trajectory(flight_id, chunk_id) -> Dict[int, Pose]`
- `optimize_chunk(flight_id, chunk_id, iterations) -> OptimizationResult`
## External Tools and Services
- **GTSAM**: Subgraph management, BetweenFactor, PriorFactor
- **H03 Robust Kernels**: Huber kernel for chunk factors
## Internal Methods
### `_create_subgraph(flight_id, chunk_id) -> gtsam.NonlinearFactorGraph`
Creates new GTSAM subgraph for chunk.
- Initializes empty factor graph
- Stores in chunk subgraph dictionary
### `_initialize_chunk_origin(flight_id, chunk_id, start_frame_id) -> bool`
Initializes first frame pose in chunk's local coordinate system.
- Sets origin pose at identity (or relative to previous chunk)
- Adds prior factor for origin frame
### `_add_factor_to_subgraph(flight_id, chunk_id, factor) -> bool`
Adds factor to chunk's specific subgraph.
- Verifies chunk exists
- Appends factor to chunk's graph
- Marks chunk as needing optimization
### `_get_chunk_subgraph(flight_id, chunk_id) -> Optional[gtsam.NonlinearFactorGraph]`
Retrieves chunk's subgraph if exists.
- Returns None if chunk not found
### `_extract_chunk_poses(flight_id, chunk_id) -> Dict[int, Pose]`
Extracts all optimized poses from chunk's subgraph.
- Poses in chunk's local coordinate system
- Converts GTSAM Pose3 to internal Pose model
### `_optimize_subgraph(flight_id, chunk_id, iterations) -> OptimizationResult`
Runs Levenberg-Marquardt on chunk's isolated subgraph.
- Optimizes only chunk's factors
- Does not affect other chunks
- Returns optimization result
### `_anchor_chunk_to_global(flight_id, chunk_id, frame_id, enu_position) -> bool`
Adds GPS anchor to chunk's subgraph as PriorFactor.
- Converts GPS to ENU (may need separate reference or global reference)
- Adds prior factor for anchor frame
- Returns success status
### `_track_chunk_frames(flight_id, chunk_id, frame_id) -> bool`
Tracks which frames belong to which chunk.
- Updates frame-to-chunk mapping
- Used for factor routing
## Unit Tests
### Test: create_chunk_subgraph_returns_true
- Arrange: Valid flight_id
- Act: Call create_chunk_subgraph
- Assert: Returns True, subgraph created
### Test: create_chunk_subgraph_initializes_origin
- Arrange: Create chunk
- Act: Check subgraph
- Assert: Has prior factor for start_frame_id
### Test: add_relative_factor_to_chunk_success
- Arrange: Create chunk
- Act: Add relative factor to chunk
- Assert: Returns True, factor in chunk's subgraph
### Test: add_relative_factor_to_chunk_nonexistent_chunk
- Arrange: No chunk created
- Act: Add relative factor to nonexistent chunk
- Assert: Returns False
### Test: add_chunk_anchor_success
- Arrange: Create chunk with frames
- Act: Add GPS anchor
- Assert: Returns True, prior factor added
### Test: get_chunk_trajectory_returns_poses
- Arrange: Create chunk, add factors, optimize
- Act: Call get_chunk_trajectory
- Assert: Returns dict with chunk poses
### Test: get_chunk_trajectory_empty_chunk
- Arrange: Create chunk without factors
- Act: Call get_chunk_trajectory
- Assert: Returns dict with only origin pose
### Test: optimize_chunk_success
- Arrange: Create chunk with factors
- Act: Call optimize_chunk
- Assert: Returns OptimizationResult with converged=True
### Test: optimize_chunk_isolation
- Arrange: Create chunk_1 and chunk_2 with factors
- Act: Optimize chunk_1
- Assert: chunk_2 poses unchanged
### Test: multiple_chunks_simultaneous
- Arrange: Create 3 chunks
- Act: Add factors to each
- Assert: All chunks maintain independent state
## Integration Tests
### Test: multi_chunk_creation_and_isolation
1. Create chunk_1 with frames 1-10
2. Create chunk_2 with frames 20-30 (disconnected)
3. Add relative factors to each chunk
4. Verify chunks optimized independently
5. Verify factors isolated to respective chunks
### Test: chunk_anchoring
1. Create chunk with frames 1-10
2. Add relative factors
3. Add chunk_anchor at frame 5
4. Optimize chunk
5. Verify trajectory consistent with anchor
6. Verify has_anchor reflected (via F12 query)
### Test: chunk_trajectory_local_coordinates
1. Create chunk
2. Add relative factors
3. Get chunk trajectory
4. Verify poses in chunk's local coordinate system
5. Verify origin at identity or start frame
_docs/02_components/10_factor_graph_optimizer/10.04_feature_chunk_merging_global_optimization.md
-142
View File
@@ -1,142 +0,0 @@
# Feature: Chunk Merging & Global Optimization
## Name
Chunk Merging & Global Optimization
## Description
Provides Sim(3) similarity transformation for merging chunk subgraphs and global optimization across all chunks. Handles scale, rotation, and translation differences between chunks (critical for monocular VO with scale ambiguity). Called by F12 Route Chunk Manager after chunk matching succeeds.
## Component APIs Implemented
- `merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform: Sim3Transform) -> bool`
- `optimize_global(flight_id, iterations) -> OptimizationResult`
## External Tools and Services
- **GTSAM**: Graph merging, global optimization
- **numpy/scipy**: Sim(3) transformation computation
## Internal Methods
### `_apply_sim3_transform(poses: Dict[int, Pose], transform: Sim3Transform) -> Dict[int, Pose]`
Applies Sim(3) similarity transformation to all poses.
- Translation: p' = s * R * p + t
- Rotation: R' = R_transform * R_original
- Scale: All positions scaled by transform.scale
- Critical for merging chunks with different scales
### `_merge_subgraphs(flight_id, source_chunk_id, dest_chunk_id) -> bool`
Merges source chunk's subgraph into destination chunk's subgraph.
- Copies all factors from source to destination
- Updates node keys to avoid conflicts
- Preserves factor graph structure
### `_update_frame_to_chunk_mapping(flight_id, source_chunk_id, dest_chunk_id) -> bool`
Updates frame-to-chunk mapping after merge.
- All frames from source_chunk now belong to dest_chunk
- Used for routing future factors
### `_transform_chunk_factors(flight_id, chunk_id, transform: Sim3Transform) -> bool`
Transforms all factors in chunk's subgraph by Sim(3).
- Updates BetweenFactor translations with scale
- Updates PriorFactor positions
- Preserves covariances (may need adjustment for scale)
### `_validate_merge_preconditions(flight_id, new_chunk_id, main_chunk_id) -> bool`
Validates that merge can proceed.
- Verifies both chunks exist
- Verifies new_chunk has anchor (required for merge)
- Returns False if preconditions not met
### `_collect_all_chunk_subgraphs(flight_id) -> List[gtsam.NonlinearFactorGraph]`
Collects all chunk subgraphs for global optimization.
- Returns list of all subgraphs
- Filters out merged/inactive chunks (via F12)
### `_run_global_optimization(flight_id, iterations) -> OptimizationResult`
Runs optimization across all anchored chunks.
- Transforms chunks to global coordinate system
- Runs Levenberg-Marquardt on combined graph
- Updates all chunk trajectories
### `_compute_sim3_from_correspondences(source_poses, dest_poses) -> Sim3Transform`
Computes Sim(3) transformation from pose correspondences.
- Uses Horn's method or similar for rigid + scale estimation
- Returns translation, rotation, scale
### `_update_all_chunk_trajectories(flight_id) -> bool`
Updates all chunk trajectories after global optimization.
- Extracts poses from global graph
- Distributes to chunk-specific storage
## Unit Tests
### Test: merge_chunk_subgraphs_success
- Arrange: Create chunk_1 (anchored), chunk_2 (with anchor)
- Act: Call merge_chunk_subgraphs with Sim3 transform
- Assert: Returns True, chunks merged
### Test: merge_chunk_subgraphs_unanchored_fails
- Arrange: Create chunk_1, chunk_2 (no anchor)
- Act: Call merge_chunk_subgraphs
- Assert: Returns False
### Test: merge_applies_sim3_transform
- Arrange: Create chunks with known poses, define transform
- Act: Merge chunks
- Assert: new_chunk poses transformed correctly
### Test: merge_updates_frame_mapping
- Arrange: Create chunks, merge
- Act: Query frame-to-chunk mapping
- Assert: new_chunk frames now belong to main_chunk
### Test: optimize_global_returns_result
- Arrange: Create multiple anchored chunks
- Act: Call optimize_global
- Assert: Returns OptimizationResult with all frames
### Test: optimize_global_achieves_consistency
- Arrange: Create chunks with overlapping anchors
- Act: Optimize global
- Assert: Anchor frames align within tolerance
### Test: sim3_transform_handles_scale
- Arrange: Poses with scale=2.0 difference
- Act: Apply Sim(3) transform
- Assert: Scaled positions match expected
### Test: sim3_transform_handles_rotation
- Arrange: Poses with 90° rotation difference
- Act: Apply Sim(3) transform
- Assert: Rotated poses match expected
## Integration Tests
### Test: chunk_anchoring_and_merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Add relative factors to each
3. Add chunk_anchor to chunk_2 (frame 25)
4. Optimize chunk_2 → local consistency improved
5. Merge chunk_2 into chunk_1 with Sim(3) transform
6. Optimize global → both chunks globally consistent
7. Verify final trajectory coherent across chunks
### Test: simultaneous_multi_chunk_processing
1. Create 3 chunks simultaneously (disconnected segments)
2. Process frames in each chunk independently
3. Anchor each chunk asynchronously
4. Merge chunks as anchors become available
5. Verify final global trajectory consistent
### Test: global_optimization_performance
1. Create 5 chunks with 20 frames each
2. Anchor all chunks
3. Time optimize_global
4. Assert: < 500ms for 100 total frames
### Test: scale_resolution_across_chunks
1. Create chunk_1 with scale drift (monocular VO)
2. Create chunk_2 with different scale drift
3. Anchor both chunks
4. Merge with Sim(3) accounting for scale
5. Verify merged trajectory has consistent scale
_docs/02_components/10_factor_graph_optimizer/10.05_feature_multi_flight_graph_lifecycle.md
-152
View File
@@ -1,152 +0,0 @@
# Feature: Multi-Flight Graph Lifecycle
## Name
Multi-Flight Graph Lifecycle
## Description
Manages flight-scoped factor graph state for concurrent multi-flight processing. Each flight maintains an independent factor graph (Dict[str, FactorGraph] keyed by flight_id), enabling parallel processing without cross-contamination. Provides cleanup when flights are deleted.
## Component APIs Implemented
- `delete_flight_graph(flight_id) -> bool`
## External Tools and Services
- **GTSAM**: Factor graph instance management
## Internal Methods
### `_get_or_create_flight_graph(flight_id) -> FlightGraphState`
Gets existing flight graph state or creates new one.
- Manages Dict[str, FlightGraphState] internally
- Initializes iSAM2 instance, reference origin, chunk subgraphs
- Thread-safe access
### `_initialize_flight_graph(flight_id) -> FlightGraphState`
Initializes new factor graph state for flight.
- Creates new iSAM2 instance
- Initializes empty chunk subgraph dictionary
- Sets reference origin to None (set on first absolute factor)
### `_cleanup_flight_resources(flight_id) -> bool`
Cleans up all resources associated with flight.
- Removes factor graph from memory
- Cleans up chunk subgraphs
- Clears frame-to-chunk mappings
- Releases GTSAM objects
### `_validate_flight_exists(flight_id) -> bool`
Validates that flight graph exists.
- Returns True if flight_id in graph dictionary
- Used as guard for all operations
### `_get_all_flight_ids() -> List[str]`
Returns list of all active flight IDs.
- Used for administrative/monitoring purposes
### `_get_flight_statistics(flight_id) -> FlightGraphStats`
Returns statistics about flight's factor graph.
- Number of frames, factors, chunks
- Memory usage estimate
- Last optimization time
## Data Structures
### `FlightGraphState`
Internal state container for a flight's factor graph.
```python
class FlightGraphState:
flight_id: str
isam2: gtsam.ISAM2
values: gtsam.Values
reference_origin: Optional[GPSPoint] # ENU reference
chunk_subgraphs: Dict[str, gtsam.NonlinearFactorGraph]
chunk_values: Dict[str, gtsam.Values]
frame_to_chunk: Dict[int, str]
created_at: datetime
last_optimized: Optional[datetime]
```
### `FlightGraphStats`
Statistics for monitoring.
```python
class FlightGraphStats:
flight_id: str
num_frames: int
num_factors: int
num_chunks: int
num_active_chunks: int
estimated_memory_mb: float
last_optimization_time_ms: float
```
## Unit Tests
### Test: delete_flight_graph_success
- Arrange: Create flight graph with factors
- Act: Call delete_flight_graph
- Assert: Returns True, graph removed
### Test: delete_flight_graph_nonexistent
- Arrange: No graph for flight_id
- Act: Call delete_flight_graph
- Assert: Returns False (graceful handling)
### Test: delete_flight_graph_cleans_chunks
- Arrange: Create flight with multiple chunks
- Act: Delete flight graph
- Assert: All chunk subgraphs cleaned up
### Test: flight_isolation_no_cross_contamination
- Arrange: Create flight_1, flight_2
- Act: Add factors to flight_1
- Assert: flight_2 graph unchanged
### Test: concurrent_flight_processing
- Arrange: Create flight_1, flight_2
- Act: Add factors to both concurrently
- Assert: Both maintain independent state
### Test: get_or_create_creates_new
- Arrange: No existing graph
- Act: Call internal _get_or_create_flight_graph
- Assert: New graph created with default state
### Test: get_or_create_returns_existing
- Arrange: Graph exists for flight_id
- Act: Call internal _get_or_create_flight_graph
- Assert: Same graph instance returned
### Test: flight_statistics_accurate
- Arrange: Create flight with known factors
- Act: Get flight statistics
- Assert: Counts match expected
## Integration Tests
### Test: multi_flight_concurrent_processing
1. Create flight_1, flight_2, flight_3 concurrently
2. Add factors to each flight in parallel
3. Optimize each flight independently
4. Verify each flight has correct trajectory
5. Delete flight_2
6. Verify flight_1, flight_3 unaffected
### Test: flight_cleanup_memory_release
1. Create flight with 1000 frames
2. Optimize trajectory
3. Note memory usage
4. Delete flight graph
5. Verify memory released (within tolerance)
### Test: reference_origin_per_flight
1. Create flight_1 with start GPS at point A
2. Create flight_2 with start GPS at point B
3. Add absolute factors to each
4. Verify ENU coordinates relative to respective origins
### Test: chunk_cleanup_on_flight_delete
1. Create flight with 5 chunks
2. Add factors and anchors to chunks
3. Delete flight
4. Verify all chunk subgraphs cleaned up
5. Verify no memory leaks
_docs/02_components/10_factor_graph_optimizer/10._component_factor_graph_optimizer.md
-830
View File
@@ -1,830 +0,0 @@
# Factor Graph Optimizer
## Interface Definition
**Interface Name**: `IFactorGraphOptimizer`
### Interface Methods
```python
class IFactorGraphOptimizer(ABC):
"""
GTSAM-based factor graph optimizer for trajectory estimation.
## Multi-Flight Support
All state-modifying methods require `flight_id` parameter.
Each flight maintains an independent factor graph state.
F10 internally manages Dict[str, FactorGraph] keyed by flight_id.
This enables:
- Concurrent processing of multiple flights
- Flight-scoped optimization without cross-contamination
- Independent cleanup via delete_flight_graph()
"""
@abstractmethod
def add_relative_factor(self, flight_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def add_absolute_factor(self, flight_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool:
pass
@abstractmethod
def add_altitude_prior(self, flight_id: str, frame_id: int, altitude: float, covariance: float) -> bool:
pass
@abstractmethod
def optimize(self, flight_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def get_trajectory(self, flight_id: str) -> Dict[int, Pose]:
pass
@abstractmethod
def get_marginal_covariance(self, flight_id: str, frame_id: int) -> np.ndarray:
pass
# Chunk operations - F10 only manages factor graph subgraphs
# F12 owns chunk metadata (status, is_active, etc.)
@abstractmethod
def create_chunk_subgraph(self, flight_id: str, chunk_id: str, start_frame_id: int) -> bool:
pass
@abstractmethod
def add_relative_factor_to_chunk(self, flight_id: str, chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def add_chunk_anchor(self, flight_id: str, chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def merge_chunk_subgraphs(self, flight_id: str, new_chunk_id: str, main_chunk_id: str, transform: Sim3Transform) -> bool:
"""Merges new_chunk INTO main_chunk. Extends main_chunk with new_chunk's subgraph."""
pass
@abstractmethod
def get_chunk_trajectory(self, flight_id: str, chunk_id: str) -> Dict[int, Pose]:
pass
@abstractmethod
def optimize_chunk(self, flight_id: str, chunk_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def optimize_global(self, flight_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def delete_flight_graph(self, flight_id: str) -> bool:
"""Cleanup factor graph when flight is deleted."""
pass
```
## Component Description
### Responsibilities
- GTSAM-based fusion of relative and absolute measurements
- Incremental optimization (iSAM2) for real-time performance
- Robust kernels (Huber/Cauchy) for 350m outlier handling
- Scale resolution through altitude priors and absolute GPS
- Trajectory smoothing and global consistency
- Back-propagation of refinements to previous frames
- **Low-level factor graph chunk operations** (subgraph creation, factor addition, optimization)
- **Sim(3) transformation for chunk merging**
### Chunk Responsibility Clarification
**F10 provides low-level factor graph operations only**:
- `create_chunk_subgraph()`: Creates subgraph in factor graph (returns bool, not ChunkHandle)
- `add_relative_factor_to_chunk()`: Adds factors to chunk's subgraph
- `add_chunk_anchor()`: Adds GPS anchor to chunk's subgraph
- `merge_chunk_subgraphs()`: Applies Sim(3) transform and merges subgraphs
- `optimize_chunk()`, `optimize_global()`: Runs optimization
**F10 does NOT own chunk metadata** - only factor graph data structures.
**F12 is the source of truth for ALL chunk state** (see F12 spec):
- ChunkHandle with all metadata (is_active, has_anchor, matching_status)
- Chunk lifecycle management
- Chunk readiness determination
- High-level chunk queries
- F12 calls F10 for factor graph operations
**F11 coordinates recovery** (see F11 spec):
- Triggers chunk creation via F12
- Coordinates matching workflows
- Emits chunk-related events
### Scope
- Non-linear least squares optimization
- Factor graph representation of SLAM problem
- Handles monocular scale ambiguity
- Real-time incremental updates
- Asynchronous batch refinement
- **Multi-chunk factor graph with independent subgraphs**
- **Chunk-level optimization and global merging**
- **Sim(3) similarity transformation for chunk alignment**
### Design Pattern: Composition Over Complex Interface
F10 uses **composition** to keep the interface manageable. Rather than exposing 20+ methods in a monolithic interface, complex operations are composed from simpler primitives:
**Primitive Operations**:
- `add_relative_factor()`, `add_absolute_factor()`, `add_altitude_prior()` - Factor management
- `optimize()`, `get_trajectory()`, `get_marginal_covariance()` - Core optimization
**Chunk Operations** (composed from primitives):
- `create_new_chunk()`, `add_relative_factor_to_chunk()`, `add_chunk_anchor()` - Chunk factor management
- `merge_chunks()`, `optimize_chunk()`, `optimize_global()` - Chunk optimization
**Callers compose these primitives** for complex workflows (e.g., F11 composes anchor + merge + optimize_global).
## API Methods
### `add_relative_factor(frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool`
**Description**: Adds relative pose measurement between consecutive frames.
**Called By**:
- F07 Sequential VO (frame-to-frame odometry)
**Input**:
```python
frame_i: int # Previous frame ID
frame_j: int # Current frame ID (typically frame_i + 1)
relative_pose: RelativePose:
translation: np.ndarray # (3,) - unit vector (scale ambiguous from VO)
rotation: np.ndarray # (3, 3) or quaternion
covariance: np.ndarray # (6, 6) - uncertainty
```
**Scale Resolution**:
F07 returns unit translation vectors due to monocular scale ambiguity. F10 resolves scale by:
1. Using altitude prior to constrain Z-axis
2. **Computing expected displacement**: Call H02 GSD Calculator.compute_gsd() to get GSD
- GSD = (sensor_width × altitude) / (focal_length × resolution_width)
- expected_displacement ≈ frame_spacing × GSD (typically ~100m)
3. Scaling: scaled_translation = unit_translation × expected_displacement
4. Global refinement using absolute GPS factors from F09 LiteSAM
**Explicit Flow**:
```python
# In add_relative_factor():
# altitude comes from F17 Configuration Manager (predefined operational altitude)
# focal_length, sensor_width from F17 Configuration Manager
config = F17.get_flight_config(flight_id)
altitude = config.altitude # Predefined altitude, NOT from EXIF
gsd = H02.compute_gsd(altitude, config.camera_params.focal_length,
config.camera_params.sensor_width,
config.camera_params.resolution_width)
expected_displacement = frame_spacing * gsd # ~100m typical at 300m altitude
scaled_translation = relative_pose.translation * expected_displacement
# Add scaled_translation to factor graph
```
**Note**: Altitude comes from F17 Configuration Manager (predefined operational altitude), NOT from EXIF metadata. The problem statement specifies images don't have GPS metadata.
**Output**:
```python
bool: True if factor added successfully
```
**Processing Flow**:
1. Create BetweenFactor in GTSAM
2. Apply robust kernel (Huber) to handle outliers
3. Add to factor graph
4. Mark graph as needing optimization
**Robust Kernel**:
- **Huber loss**: Downweights large errors (>threshold)
- **Critical** for 350m outlier handling from tilt
**Test Cases**:
1. **Normal motion**: Factor added, contributes to optimization
2. **Large displacement** (350m outlier): Huber kernel reduces weight
3. **Consecutive factors**: Chain of relative factors builds trajectory
---
### `add_absolute_factor(frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool`
**Description**: Adds absolute GPS measurement for drift correction or user anchor.
**Called By**:
- F09 Metric Refinement (after LiteSAM alignment)
- F11 Failure Recovery Coordinator (user-provided anchors)
**Input**:
```python
frame_id: int
gps: GPSPoint:
lat: float
lon: float
covariance: np.ndarray # (2, 2) or (3, 3) - GPS uncertainty
is_user_anchor: bool # True for user-provided fixes (high confidence)
```
**Output**:
```python
bool: True if factor added
```
**Processing Flow**:
1. Convert GPS to local ENU coordinates (East-North-Up)
2. Create PriorFactor or UnaryFactor
3. Set covariance (low for user anchors, higher for LiteSAM)
4. Add to factor graph
5. Trigger optimization (immediate for user anchors)
**Covariance Settings**:
- **User anchor**: σ = 5m (high confidence)
- **LiteSAM match**: σ = 20-50m (depends on confidence)
**Test Cases**:
1. **LiteSAM GPS**: Adds absolute factor, corrects drift
2. **User anchor**: High confidence, immediately refines trajectory
3. **Multiple absolute factors**: Graph optimizes to balance all
---
### `add_altitude_prior(frame_id: int, altitude: float, covariance: float) -> bool`
**Description**: Adds altitude constraint to resolve monocular scale ambiguity.
**Called By**:
- Main processing loop (for each frame)
**Input**:
```python
frame_id: int
altitude: float # Predefined altitude in meters
covariance: float # Altitude uncertainty (e.g., 50m)
```
**Output**:
```python
bool: True if prior added
```
**Processing Flow**:
1. Create UnaryFactor for Z-coordinate
2. Set as soft constraint (not hard constraint)
3. Add to factor graph
**Purpose**:
- Resolves scale ambiguity in monocular VO
- Prevents scale drift (trajectory collapsing or exploding)
- Soft constraint allows adjustment based on absolute GPS
**Test Cases**:
1. **Without altitude prior**: Scale drifts over time
2. **With altitude prior**: Scale stabilizes
3. **Conflicting measurements**: Optimizer balances VO and altitude
---
### `optimize(iterations: int) -> OptimizationResult`
**Description**: Runs optimization to refine trajectory.
**Called By**:
- Main processing loop (incremental after each frame)
- Asynchronous refinement thread (batch optimization)
**Input**:
```python
iterations: int # Max iterations (typically 5-10 for incremental, 50-100 for batch)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int] # Frames with updated poses
```
**Processing Details**:
- **Incremental** (iSAM2): Updates only affected nodes
- **Batch**: Re-optimizes entire trajectory when new absolute factors added
- **Robust M-estimation**: Automatically downweights outliers
**Optimization Algorithm** (Levenberg-Marquardt):
1. Linearize factor graph around current estimate
2. Solve linear system
3. Update pose estimates
4. Check convergence (error reduction < threshold)
**Test Cases**:
1. **Incremental optimization**: Fast (<100ms), local update
2. **Batch optimization**: Slower (~500ms), refines entire trajectory
3. **Convergence**: Error reduces, converges within iterations
---
### `get_trajectory() -> Dict[int, Pose]`
**Description**: Retrieves complete optimized trajectory.
**Called By**:
- F13 Result Manager (for publishing results)
- F12 Coordinate Transformer (for GPS conversion)
**Input**: None
**Output**:
```python
Dict[int, Pose]:
frame_id -> Pose:
position: np.ndarray # (x, y, z) in ENU
orientation: np.ndarray # Quaternion or rotation matrix
timestamp: datetime
```
**Processing Flow**:
1. Extract all pose estimates from graph
2. Convert to appropriate coordinate system
3. Return dictionary
**Test Cases**:
1. **After optimization**: Returns all frame poses
2. **Refined trajectory**: Poses updated after batch optimization
---
### `get_marginal_covariance(frame_id: int) -> np.ndarray`
**Description**: Gets uncertainty (covariance) of a pose estimate.
**Called By**:
- F11 Failure Recovery Coordinator (to detect high uncertainty)
**Input**:
```python
frame_id: int
```
**Output**:
```python
np.ndarray: (6, 6) covariance matrix [x, y, z, roll, pitch, yaw]
```
**Purpose**:
- Uncertainty quantification
- Trigger user input when uncertainty too high (> 50m radius)
**Test Cases**:
1. **Well-constrained pose**: Small covariance
2. **Unconstrained pose**: Large covariance
3. **After absolute factor**: Covariance reduces
---
### `create_new_chunk(chunk_id: str, start_frame_id: int) -> ChunkHandle`
**Description**: Creates a new map fragment/chunk with its own subgraph.
**Called By**:
- F02 Flight Processor (when tracking lost)
- F12 Route Chunk Manager (chunk lifecycle)
**Input**:
```python
chunk_id: str # Unique chunk identifier
start_frame_id: int # First frame in chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
**Processing Flow**:
1. Create new subgraph for chunk
2. Initialize first frame pose in chunk's local coordinate system
3. Mark chunk as active
4. Return ChunkHandle
**Test Cases**:
1. **Create chunk**: Returns ChunkHandle with is_active=True
2. **Multiple chunks**: Can create multiple chunks simultaneously
3. **Chunk isolation**: Factors added to chunk don't affect other chunks
---
### `get_chunk_for_frame(frame_id: int) -> Optional[ChunkHandle]`
**Description**: Gets the chunk containing the specified frame (low-level factor graph query).
**Called By**:
- F12 Route Chunk Manager (for internal queries)
- F07 Sequential VO (to determine chunk context for factor graph operations)
**Note**: This is a low-level method for factor graph operations. For high-level chunk queries, use F12.get_active_chunk(flight_id).
**Input**:
```python
frame_id: int
```
**Output**:
```python
Optional[ChunkHandle] # Active chunk or None if frame not in any chunk
```
**Test Cases**:
1. **Frame in active chunk**: Returns ChunkHandle
2. **Frame not in chunk**: Returns None
3. **Multiple chunks**: Returns correct chunk for frame
---
### `add_relative_factor_to_chunk(chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool`
**Description**: Adds relative pose measurement to a specific chunk's subgraph.
**Called By**:
- F07 Sequential VO (chunk-scoped operations)
**Input**:
```python
chunk_id: str
frame_i: int # Previous frame ID
frame_j: int # Current frame ID
relative_pose: RelativePose
covariance: np.ndarray # (6, 6)
```
**Output**:
```python
bool: True if factor added successfully
```
**Processing Flow**:
1. Verify chunk exists and is active
2. Create BetweenFactor in chunk's subgraph
3. Apply robust kernel (Huber)
4. Add to chunk's factor graph
5. Mark chunk as needing optimization
**Test Cases**:
1. **Add to active chunk**: Factor added successfully
2. **Add to inactive chunk**: Returns False
3. **Multiple chunks**: Factors isolated to respective chunks
---
### `add_chunk_anchor(chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool`
**Description**: Adds absolute GPS anchor to a chunk, enabling global localization.
**Called By**:
- F09 Metric Refinement (after chunk LiteSAM matching)
- F11 Failure Recovery Coordinator (chunk matching)
**Input**:
```python
chunk_id: str
frame_id: int # Frame within chunk to anchor
gps: GPSPoint
covariance: np.ndarray # (2, 2) or (3, 3)
```
**Output**:
```python
bool: True if anchor added
```
**Processing Flow**:
1. Convert GPS to ENU coordinates
2. Create PriorFactor in chunk's subgraph
3. Mark chunk as anchored
4. Trigger chunk optimization
5. Enable chunk merging
**Test Cases**:
1. **Anchor unanchored chunk**: Anchor added, has_anchor=True
2. **Anchor already anchored chunk**: Updates anchor
3. **Chunk optimization**: Chunk optimized after anchor
---
### `merge_chunk_subgraphs(flight_id: str, new_chunk_id: str, main_chunk_id: str, transform: Sim3Transform) -> bool`
**Description**: Merges new_chunk INTO main_chunk using Sim(3) similarity transformation. Extends main_chunk with new_chunk's frames.
**Called By**:
- F12 Route Chunk Manager (via merge_chunks method)
**Input**:
```python
flight_id: str # Flight identifier
new_chunk_id: str # New chunk being merged (source, typically newer/recently anchored)
main_chunk_id: str # Main chunk being extended (destination, typically older/established)
transform: Sim3Transform:
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or quaternion
scale: float
```
**Output**:
```python
bool: True if merge successful
```
**Processing Flow**:
1. Verify both chunks exist and new_chunk is anchored
2. Apply Sim(3) transform to all poses in new_chunk
3. Merge new_chunk's subgraph into main_chunk's subgraph
4. Update frame-to-chunk mapping (new_chunk frames now belong to main_chunk)
5. Mark new_chunk subgraph as merged (F12 handles state updates)
6. Optimize merged graph globally
**Sim(3) Transformation**:
- Accounts for translation, rotation, and scale differences
- Critical for merging chunks with different scales (monocular VO)
- Preserves internal consistency of both chunks
**Test Cases**:
1. **Merge anchored chunks**: new_chunk merged into main_chunk successfully
2. **Merge unanchored chunk**: Returns False
3. **Global consistency**: Merged trajectory is globally consistent
---
### `get_chunk_trajectory(chunk_id: str) -> Dict[int, Pose]`
**Description**: Retrieves optimized trajectory for a specific chunk.
**Called By**:
- F12 Route Chunk Manager (chunk state queries)
- F14 Result Manager (result publishing)
**Input**:
```python
chunk_id: str
```
**Output**:
```python
Dict[int, Pose] # Frame ID -> Pose in chunk's local coordinate system
```
**Test Cases**:
1. **Get chunk trajectory**: Returns all poses in chunk
2. **Empty chunk**: Returns empty dict
3. **After optimization**: Returns optimized poses
---
### `get_all_chunks() -> List[ChunkHandle]`
**Description**: Retrieves all chunks in the factor graph.
**Called By**:
- F11 Failure Recovery Coordinator (chunk matching coordination)
- F12 Route Chunk Manager (chunk state queries)
**Input**: None
**Output**:
```python
List[ChunkHandle] # All chunks (active and inactive)
```
**Test Cases**:
1. **Multiple chunks**: Returns all chunks
2. **No chunks**: Returns empty list
3. **Mixed states**: Returns chunks in various states (active, anchored, merged)
---
### `optimize_chunk(chunk_id: str, iterations: int) -> OptimizationResult`
**Description**: Optimizes a specific chunk's subgraph independently.
**Called By**:
- F02 Flight Processor (after chunk anchor added)
- F12 Route Chunk Manager (periodic chunk optimization)
**Input**:
```python
chunk_id: str
iterations: int # Max iterations (typically 5-10 for incremental)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int]
mean_reprojection_error: float
```
**Processing Flow**:
1. Extract chunk's subgraph
2. Run Levenberg-Marquardt optimization
3. Update poses in chunk's local coordinate system
4. Return optimization result
**Test Cases**:
1. **Optimize active chunk**: Chunk optimized successfully
2. **Optimize anchored chunk**: Optimization improves consistency
3. **Chunk isolation**: Other chunks unaffected
---
### `optimize_global(iterations: int) -> OptimizationResult`
**Description**: Optimizes all chunks and performs global merging.
**Called By**:
- Background optimization task (periodic)
- F11 Failure Recovery Coordinator (after chunk matching)
**Input**:
```python
iterations: int # Max iterations (typically 50-100 for global)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int] # All frames across all chunks
mean_reprojection_error: float
```
**Processing Flow**:
1. Collect all chunks
2. For anchored chunks, transform to global coordinate system
3. Optimize merged global graph
4. Update all chunk trajectories
5. Return global optimization result
**Test Cases**:
1. **Global optimization**: All chunks optimized together
2. **Multiple anchored chunks**: Global consistency achieved
3. **Performance**: Completes within acceptable time (<500ms)
## Integration Tests
### Test 1: Incremental Trajectory Building
1. Initialize graph with first frame
2. Add relative factors from VO × 100
3. Add altitude priors × 100
4. Optimize incrementally after each frame
5. Verify smooth trajectory
### Test 2: Drift Correction with Absolute GPS
1. Build trajectory with VO only (will drift)
2. Add absolute GPS factor at frame 50
3. Optimize → trajectory corrects
4. Verify frames 1-49 also corrected (back-propagation)
### Test 3: Outlier Handling
1. Add normal relative factors
2. Add 350m outlier factor (tilt error)
3. Optimize with robust kernel
4. Verify outlier downweighted, trajectory smooth
### Test 4: User Anchor Integration
1. Processing blocked at frame 237
2. User provides anchor (high confidence)
3. add_absolute_factor(is_user_anchor=True)
4. Optimize → trajectory snaps to anchor
### Test 5: Multi-Chunk Creation and Isolation
1. Create chunk_1 with frames 1-10
2. Create chunk_2 with frames 20-30 (disconnected)
3. Add relative factors to each chunk
4. Verify chunks optimized independently
5. Verify factors isolated to respective chunks
### Test 6: Chunk Anchoring and Merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Add chunk_anchor to chunk_2 (frame 25)
3. Optimize chunk_2 → local consistency improved
4. Merge chunk_2 into chunk_1 with Sim(3) transform
5. Optimize global → both chunks globally consistent
6. Verify final trajectory coherent across chunks
### Test 7: Simultaneous Multi-Chunk Processing
1. Create 3 chunks simultaneously (disconnected segments)
2. Process frames in each chunk independently
3. Anchor each chunk asynchronously
4. Merge chunks as anchors become available
5. Verify final global trajectory consistent
## Non-Functional Requirements
### Performance
- **Incremental optimize**: < 100ms per frame (iSAM2)
- **Batch optimize**: < 500ms for 100 frames
- **get_trajectory**: < 10ms
- Real-time capable: 10 FPS processing
### Accuracy
- **Mean Reprojection Error (MRE)**: < 1.0 pixels
- **GPS accuracy**: Meet 80% < 50m, 60% < 20m criteria
- **Trajectory smoothness**: No sudden jumps (except user anchors)
### Reliability
- Numerical stability for 2000+ frame trajectories
- Graceful handling of degenerate configurations
- Robust to missing/corrupted measurements
## Dependencies
### Internal Components
- **H03 Robust Kernels**: For Huber/Cauchy loss functions
- **H02 GSD Calculator**: For GSD computation and scale resolution
### External Dependencies
- **GTSAM**: Graph optimization library
- **numpy**: Matrix operations
- **scipy**: Sparse matrix operations (optional)
## Data Models
### Pose
```python
class Pose(BaseModel):
frame_id: int
position: np.ndarray # (3,) - [x, y, z] in ENU
orientation: np.ndarray # (4,) quaternion or (3,3) rotation matrix
timestamp: datetime
covariance: Optional[np.ndarray] # (6, 6)
```
### RelativePose
```python
class RelativePose(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or (4,)
covariance: np.ndarray # (6, 6)
```
### OptimizationResult
```python
class OptimizationResult(BaseModel):
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int]
mean_reprojection_error: float
```
### FactorGraphConfig
```python
class FactorGraphConfig(BaseModel):
robust_kernel_type: str = "Huber" # or "Cauchy"
huber_threshold: float = 1.0 # pixels
cauchy_k: float = 0.1
isam2_relinearize_threshold: float = 0.1
isam2_relinearize_skip: int = 1
max_chunks: int = 100 # Maximum number of simultaneous chunks
chunk_merge_threshold: float = 0.1 # Error threshold for chunk merging
```
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/11_failure_recovery_coordinator/11.01_feature_confidence_assessment.md
-45
View File
@@ -1,45 +0,0 @@
# Feature: Confidence Assessment
## Description
Assesses tracking confidence from VO and LiteSAM results and determines when tracking is lost. This is the foundation for all recovery decisions - every frame's results pass through confidence assessment to detect degradation.
## Component APIs Implemented
- `check_confidence(vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment`
- `detect_tracking_loss(confidence: ConfidenceAssessment) -> bool`
## External Tools and Services
None - pure computation based on input metrics.
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_compute_vo_confidence(vo_result)` | Calculates confidence from VO inlier count and ratio |
| `_compute_litesam_confidence(litesam_result)` | Calculates confidence from LiteSAM match score |
| `_compute_overall_confidence(vo_conf, litesam_conf)` | Weighted combination of confidence scores |
| `_determine_tracking_status(overall_conf, inlier_count)` | Maps confidence to "good"/"degraded"/"lost" |
## Thresholds
- **Good**: VO inliers > 50, LiteSAM confidence > 0.7
- **Degraded**: VO inliers 20-50
- **Lost**: VO inliers < 20
## Unit Tests
1. **Good tracking metrics** → Returns "good" status with high confidence
2. **Low VO inliers (25)** → Returns "degraded" status
3. **Very low VO inliers (10)** → Returns "lost" status
4. **High VO, low LiteSAM** → Returns appropriate degraded assessment
5. **LiteSAM result None** → Confidence based on VO only
6. **Edge case at threshold boundary (20 inliers)** → Correct status determination
## Integration Tests
1. **Normal flight sequence** → Confidence remains "good" throughout
2. **Low overlap frame** → Confidence drops to "degraded", recovers
3. **Sharp turn sequence** → Confidence drops to "lost", triggers recovery flow
_docs/02_components/11_failure_recovery_coordinator/11.02_feature_progressive_search.md
-59
View File
@@ -1,59 +0,0 @@
# Feature: Progressive Search
## Description
Coordinates progressive tile search when tracking is lost. Expands search grid from 1→4→9→16→25 tiles, trying LiteSAM matching at each level. Manages search session state and tracks progress.
## Component APIs Implemented
- `start_search(flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession`
- `expand_search_radius(session: SearchSession) -> List[TileCoords]`
- `try_current_grid(session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]`
- `mark_found(session: SearchSession, result: AlignmentResult) -> bool`
- `get_search_status(session: SearchSession) -> SearchStatus`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F04 Satellite Data Manager | `expand_search_grid()`, `compute_tile_bounds()` |
| F09 Metric Refinement | `align_to_satellite()` for tile matching |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_create_search_session(flight_id, frame_id, gps)` | Initializes SearchSession with grid_size=1 |
| `_get_next_grid_size(current_size)` | Returns next size in sequence: 1→4→9→16→25 |
| `_compute_tile_sequence(session)` | Returns tiles for current grid around center GPS |
| `_try_single_tile_match(uav_image, tile, tile_bounds)` | Wraps F09 alignment call |
| `_is_match_acceptable(result, threshold)` | Validates alignment confidence meets threshold |
## Search Grid Progression
| Level | Grid Size | Tiles | New Tiles Added |
|-------|-----------|-------|-----------------|
| 1 | 1×1 | 1 | 1 |
| 2 | 2×2 | 4 | 3 |
| 3 | 3×3 | 9 | 5 |
| 4 | 4×4 | 16 | 7 |
| 5 | 5×5 | 25 | 9 |
## Unit Tests
1. **start_search** → Creates session with grid_size=1, correct center GPS
2. **expand_search_radius from 1** → Returns 3 new tiles, grid_size becomes 4
3. **expand_search_radius from 4** → Returns 5 new tiles, grid_size becomes 9
4. **expand_search_radius at max (25)** → Returns empty list, exhausted=true
5. **try_current_grid match found** → Returns AlignmentResult, session.found=true
6. **try_current_grid no match** → Returns None
7. **mark_found** → Sets session.found=true
8. **get_search_status** → Returns correct current_grid_size, found, exhausted flags
## Integration Tests
1. **Match on first grid** → start_search → try_current_grid → match found immediately
2. **Match on 3rd expansion** → Search through 1→4→9 tiles, match found at grid_size=9
3. **Full exhaustion** → All 25 tiles searched, no match, exhausted=true
4. **Concurrent searches** → Multiple flight search sessions operate independently
_docs/02_components/11_failure_recovery_coordinator/11.03_feature_user_input_handling.md
-47
View File
@@ -1,47 +0,0 @@
# Feature: User Input Handling
## Description
Handles human-in-the-loop recovery when all automated strategies are exhausted. Creates user input requests with candidate tiles and applies user-provided GPS anchors to the factor graph.
## Component APIs Implemented
- `create_user_input_request(flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest`
- `apply_user_anchor(flight_id: str, frame_id: int, anchor: UserAnchor) -> bool`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F10 Factor Graph Optimizer | `add_absolute_factor()`, `optimize()` for anchor application |
| F08 Global Place Recognition | Candidate tiles (passed in, not called directly) |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_generate_request_id()` | Creates unique request ID |
| `_get_uav_image_for_frame(flight_id, frame_id)` | Retrieves UAV image for user display |
| `_validate_anchor(anchor)` | Validates anchor data (GPS bounds, pixel within image) |
| `_convert_anchor_to_factor(anchor, frame_id)` | Transforms UserAnchor to factor graph format |
## Request/Response Flow
1. **Request Creation**: F11 creates UserInputRequest → F02.2 sends via F15 → Client receives
2. **Anchor Application**: Client sends anchor → F01 API → F02.2 calls F11.apply_user_anchor()
## Unit Tests
1. **create_user_input_request** → Returns UserInputRequest with correct flight_id, frame_id, candidates
2. **create_user_input_request generates unique ID** → Multiple calls produce unique request_ids
3. **apply_user_anchor valid** → Returns true, calls F10.add_absolute_factor with is_user_anchor=true
4. **apply_user_anchor invalid GPS** → Returns false, no factor added
5. **apply_user_anchor invalid pixel coords** → Returns false, rejected
6. **apply_user_anchor triggers optimization** → F10.optimize() called after factor addition
## Integration Tests
1. **Full user input flow** → Search exhausted → create_user_input_request → user provides anchor → apply_user_anchor → processing resumes
2. **User anchor improves trajectory** → Before/after anchor application, trajectory accuracy improves
3. **Multiple user anchors** → Can apply anchors to different frames in sequence
_docs/02_components/11_failure_recovery_coordinator/11.04_feature_chunk_recovery_coordination.md
-72
View File
@@ -1,72 +0,0 @@
# Feature: Chunk Recovery Coordination
## Description
Coordinates chunk-based recovery when tracking is lost. Creates chunks proactively, orchestrates semantic matching via F08, LiteSAM matching with rotation sweeps via F06/F09, and merges recovered chunks into the main trajectory. Includes background processing of unanchored chunks.
## Component APIs Implemented
- `create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle`
- `try_chunk_semantic_matching(chunk_id: str) -> Optional[List[TileCandidate]]`
- `try_chunk_litesam_matching(chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]`
- `merge_chunk_to_trajectory(flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool`
- `process_unanchored_chunks(flight_id: str) -> None`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F12 Route Chunk Manager | `create_chunk()`, `get_chunk_images()`, `get_chunk_frames()`, `mark_chunk_anchored()`, `merge_chunks()`, `get_chunks_for_matching()`, `is_chunk_ready_for_matching()`, `mark_chunk_matching()` |
| F10 Factor Graph Optimizer | Chunk creation, anchor application (via F12) |
| F04 Satellite Data Manager | Tile retrieval for matching |
| F08 Global Place Recognition | `retrieve_candidate_tiles_for_chunk()` |
| F06 Image Rotation Manager | `try_chunk_rotation_steps()` for rotation sweeps |
| F09 Metric Refinement | `align_chunk_to_satellite()` (via F06) |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_determine_merge_target(chunk_id)` | Finds temporal predecessor chunk for merging |
| `_compute_sim3_transform(alignment_result)` | Builds Sim3Transform from alignment |
| `_select_anchor_frame(chunk_id)` | Selects best frame in chunk for anchoring (middle or highest confidence) |
| `_handle_chunk_matching_failure(chunk_id)` | Decides retry vs user input request on failure |
## Rotation Sweep Strategy
- 12 rotation angles: 0°, 30°, 60°, ..., 330°
- Critical for chunks from sharp turns (unknown orientation)
- Returns best matching rotation angle in ChunkAlignmentResult
## Chunk Lifecycle States
| Status | Description |
|--------|-------------|
| `unanchored` | Chunk created, no GPS anchor |
| `matching` | Matching in progress |
| `anchored` | GPS anchor established |
| `merged` | Merged into main trajectory |
## Unit Tests
1. **create_chunk_on_tracking_loss** → Returns ChunkHandle with correct flight_id, start_frame_id, is_active=true
2. **try_chunk_semantic_matching success** → Returns List[TileCandidate] from F08
3. **try_chunk_semantic_matching no match** → Returns None
4. **try_chunk_litesam_matching match at 0°** → Returns ChunkAlignmentResult with rotation_angle=0
5. **try_chunk_litesam_matching match at 120°** → Returns ChunkAlignmentResult with rotation_angle=120
6. **try_chunk_litesam_matching no match** → Returns None after trying all rotations and tiles
7. **merge_chunk_to_trajectory** → Returns true, F12.merge_chunks called with correct parameters
8. **merge_chunk_to_trajectory determines correct merge target** → Merges to temporal predecessor
9. **_determine_merge_target no predecessor** → Returns "main" for first chunk
## Integration Tests
1. **Proactive chunk creation** → Tracking lost → chunk created immediately → processing continues in new chunk
2. **Chunk semantic matching** → Chunk with 10 frames → semantic matching finds candidates where single-frame fails
3. **Chunk LiteSAM with rotation** → Unknown orientation chunk → rotation sweep finds match at correct angle
4. **Full chunk recovery flow** → create_chunk → semantic matching → LiteSAM matching → merge
5. **Multiple chunk merging** → chunk_1 (frames 1-10), chunk_2 (frames 20-30) → both recovered and merged
6. **Background processing** → 3 unanchored chunks → process_unanchored_chunks matches and merges asynchronously
7. **Non-blocking processing** → Frame processing continues while chunk matching runs in background
8. **Chunk matching failure** → All candidates fail → user input requested for chunk
_docs/02_components/11_failure_recovery_coordinator/11._component_failure_recovery_coordinator.md
-784
View File
@@ -1,784 +0,0 @@
# Failure Recovery Coordinator
## Interface Definition
**Interface Name**: `IFailureRecoveryCoordinator`
### Interface Methods
```python
class IFailureRecoveryCoordinator(ABC):
# Status Checks
@abstractmethod
def check_confidence(self, vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment:
pass
@abstractmethod
def detect_tracking_loss(self, confidence: ConfidenceAssessment) -> bool:
pass
# Search & Recovery (Synchronous returns, NO EVENTS)
@abstractmethod
def start_search(self, flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession:
pass
@abstractmethod
def expand_search_radius(self, session: SearchSession) -> List[TileCoords]:
pass
@abstractmethod
def try_current_grid(self, session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]:
pass
@abstractmethod
def mark_found(self, session: SearchSession, result: AlignmentResult) -> bool:
pass
@abstractmethod
def get_search_status(self, session: SearchSession) -> SearchStatus:
pass
@abstractmethod
def create_user_input_request(self, flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest:
"""Returns request object. Caller (F02.2) is responsible for sending to F15."""
pass
@abstractmethod
def apply_user_anchor(self, flight_id: str, frame_id: int, anchor: UserAnchor) -> bool:
pass
# Chunk Recovery
@abstractmethod
def create_chunk_on_tracking_loss(self, flight_id: str, frame_id: int) -> ChunkHandle:
pass
@abstractmethod
def try_chunk_semantic_matching(self, chunk_id: str) -> Optional[List[TileCandidate]]:
pass
@abstractmethod
def try_chunk_litesam_matching(self, chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]:
pass
@abstractmethod
def merge_chunk_to_trajectory(self, flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool:
pass
@abstractmethod
def process_unanchored_chunks(self, flight_id: str) -> None:
"""Background task logic. Should be invoked/managed by F02.2."""
pass
```
## Component Description
### Responsibilities
- **Pure Logic Component**: Coordinates recovery strategies but delegates execution and communication to F02.2.
- **No Events**: Returns status objects or booleans. F02.2 decides state transitions based on these returns.
- **Chunk Orchestration**: Coordinates F12 and F10 operations during recovery but does not own the thread/task.
- Monitor confidence metrics (inlier count, MRE, covariance)
- Detect tracking loss and trigger recovery
- Coordinate progressive tile search (1→4→9→16→25)
- Handle human-in-the-loop when all strategies exhausted
- Apply user-provided anchors to Factor Graph
- **Proactive chunk creation on tracking loss** (via F12)
- **Chunk LiteSAM matching with rotation sweeps**
- **Chunk merging orchestration**
- **Background chunk matching processing**
### Interaction Pattern (Direct Calls)
**Caller**: F02.2 Flight Processing Engine
1. **Tracking Loss**:
* F02.2 calls `start_search()`.
* F02.2 calls `try_current_grid()` iteratively.
* If found, F02.2 calls `mark_found()`.
* If exhausted, F02.2 calls `create_user_input_request()` -> gets `Request` object -> F02.2 calls F15.
2. **Chunk Matching**:
* F02.2 calls `create_chunk_on_tracking_loss()`.
* F02.2 (background task) calls `try_chunk_semantic_matching()`.
* F02.2 calls `try_chunk_litesam_matching()`.
* F02.2 calls `merge_chunk_to_trajectory()`.
### External SSE Events (to Clients)
F11 does NOT directly send events to clients. F02.2 orchestrates client communication based on F11's return values:
- When F11.create_user_input_request() returns `UserInputRequest` → F02.2 calls F15.send_user_input_request()
- When F11.merge_chunk_to_trajectory() returns True → F02.2 calls F14.update_results_after_chunk_merge()
- When recovery fails (all F11 methods return None/False) → F02.2 calls F15.send_processing_blocked()
This separation ensures:
1. F11 is a pure business logic component (no I/O dependencies)
2. F02.2 controls all coordination and state management
3. F14/F15 handle all client-facing communication
### Scope
- Confidence monitoring
- Progressive search coordination
- User input request/response handling
- Recovery strategy orchestration
- Integration point for F04, F06, F08, F09, F10
- **Chunk lifecycle and matching coordination**
- **Multi-chunk simultaneous processing**
### Architecture Note: Single Responsibility Consideration
F11 has extensive responsibilities including progressive search, chunk creation coordination, chunk matching coordination (semantic + LiteSAM), chunk merging coordination, and user input handling. While this represents significant scope, these remain together in a single component because:
1. **Tight Coupling**: All recovery strategies share state (SearchSession, chunk status) and need coordinated fallback logic (progressive search → chunk matching → user input).
2. **Sequential Dependency**: Chunk semantic matching feeds into chunk LiteSAM matching, which feeds into chunk merging. Splitting these would create circular dependencies or complex event choreography.
3. **Pure Logic Pattern**: F11 remains a pure logic component (no I/O, no threads). All execution is delegated to F02.2, keeping F11 focused on decision-making rather than coordination.
4. **Single Point of Recovery Policy**: Having all recovery strategies in one component makes the fallback logic explicit and maintainable. The alternative—multiple coordinators—would require a meta-coordinator.
**Future Consideration**: If F11 grows beyond ~1000 lines or if specific recovery strategies need independent evolution, consider splitting into:
- `F11a_search_coordinator` - Progressive search, user input handling
- `F11b_chunk_recovery_coordinator` - Chunk semantic matching, LiteSAM matching, merging
For now, the component remains unified with clear internal organization (search methods, chunk methods).
## API Methods
### `check_confidence(vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment`
**Description**: Assesses tracking confidence from VO and LiteSAM results.
**Called By**: Main processing loop (per frame)
**Input**:
```python
vo_result: RelativePose
litesam_result: Optional[AlignmentResult]
```
**Output**:
```python
ConfidenceAssessment:
overall_confidence: float # 0-1
vo_confidence: float
litesam_confidence: float
inlier_count: int
tracking_status: str # "good", "degraded", "lost"
```
**Confidence Metrics**:
- VO inlier count and ratio
- LiteSAM match confidence
- Factor graph marginal covariance
- Reprojection error
**Thresholds**:
- **Good**: VO inliers > 50, LiteSAM confidence > 0.7
- **Degraded**: VO inliers 20-50
- **Lost**: VO inliers < 20
**Test Cases**:
1. Good tracking → "good" status
2. Low overlap → "degraded"
3. Sharp turn → "lost"
---
### `detect_tracking_loss(confidence: ConfidenceAssessment) -> bool`
**Description**: Determines if tracking is lost.
**Called By**: Main processing loop
**Input**: `ConfidenceAssessment`
**Output**: `bool` - True if tracking lost
**Test Cases**:
1. Confidence good → False
2. Confidence lost → True
---
### `start_search(flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession`
**Description**: Initiates progressive search session.
**Called By**: Main processing loop (when tracking lost)
**Input**:
```python
flight_id: str
frame_id: int
estimated_gps: GPSPoint # Dead-reckoning estimate
```
**Output**:
```python
SearchSession:
session_id: str
flight_id: str
frame_id: int
center_gps: GPSPoint
current_grid_size: int # Starts at 1
max_grid_size: int # 25
found: bool
```
**Processing Flow**:
1. Create search session
2. Set center from estimated_gps
3. Set current_grid_size = 1
4. Return session
**Test Cases**:
1. Start search → session created with grid_size=1
---
### `expand_search_radius(session: SearchSession) -> List[TileCoords]`
**Description**: Expands search grid to next size (1→4→9→16→25).
**Called By**: Internal (after try_current_grid fails)
**Input**: `SearchSession`
**Output**: `List[TileCoords]` - Tiles for next grid size
**Processing Flow**:
1. Increment current_grid_size (1→4→9→16→25)
2. Call F04.expand_search_grid() to get new tiles only
3. Return new tile coordinates
**Test Cases**:
1. Expand 1→4 → returns 3 new tiles
2. Expand 4→9 → returns 5 new tiles
3. At grid_size=25 → no more expansion
---
### `try_current_grid(session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]`
**Description**: Tries LiteSAM matching on current tile grid.
**Called By**: Internal (progressive search loop)
**Input**:
```python
session: SearchSession
tiles: Dict[str, np.ndarray] # From F04
```
**Output**: `Optional[AlignmentResult]` - Match result or None
**Processing Flow**:
1. Get UAV image for frame_id
2. For each tile in grid:
- Get tile_bounds via F04.compute_tile_bounds(tile_coords)
- Call F09.align_to_satellite(uav_image, tile, tile_bounds)
- If match found with confidence > threshold:
- mark_found(session, result)
- Return result
3. Return None if no match
**Test Cases**:
1. Match on 3rd tile → returns result
2. No match in grid → returns None
---
### `mark_found(session: SearchSession, result: AlignmentResult) -> bool`
**Description**: Marks search session as successful.
**Called By**: Internal
**Input**:
```python
session: SearchSession
result: AlignmentResult
```
**Output**: `bool` - True
**Processing Flow**:
1. Set session.found = True
2. Log success (grid_size where found)
3. Resume processing
---
### `get_search_status(session: SearchSession) -> SearchStatus`
**Description**: Gets current search status.
**Called By**: F01 REST API (for status endpoint)
**Output**:
```python
SearchStatus:
current_grid_size: int
found: bool
exhausted: bool # Reached grid_size=25 without match
```
---
### `create_user_input_request(flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest`
**Description**: Creates user input request when all search strategies exhausted.
**Called By**: Internal (when grid_size=25 and no match)
**Input**:
```python
flight_id: str
frame_id: int
candidate_tiles: List[TileCandidate] # Top-5 from G08
```
**Output**:
```python
UserInputRequest:
request_id: str
flight_id: str
frame_id: int
uav_image: np.ndarray
candidate_tiles: List[TileCandidate]
message: str
```
**Processing Flow**:
1. Get UAV image for frame_id
2. Get top-5 candidates from F08
3. Create request
4. Return Request object (Caller F02.2 handles sending)
**Test Cases**:
1. All search failed → creates request
2. Request returned to caller
---
### `apply_user_anchor(flight_id: str, frame_id: int, anchor: UserAnchor) -> bool`
**Description**: Applies user-provided GPS anchor.
**Called By**: F01 REST API (user-fix endpoint)
**Input**:
```python
flight_id: str
frame_id: int
anchor: UserAnchor:
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**: `bool` - True if applied
**Processing Flow**:
1. Validate anchor data
2. Call F10.add_absolute_factor(frame_id, gps, is_user_anchor=True)
3. F10.optimize() → refines trajectory
4. Return True (Caller F02.2 updates status)
**Test Cases**:
1. Valid anchor → applied, processing resumes
2. Invalid anchor → rejected
---
### `create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle`
**Description**: Creates a new chunk proactively when tracking is lost.
**Called By**:
- F02.2 Flight Processing Engine (when tracking lost detected)
**Input**:
```python
flight_id: str
frame_id: int # First frame in new chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str
```
**Processing Flow**:
1. Call F12 Route Chunk Manager.create_chunk()
2. F12 creates chunk in F10 Factor Graph Optimizer
3. Mark chunk as active
4. Return ChunkHandle
**Proactive Behavior**:
- Chunk created immediately (not waiting for matching to fail)
- Processing continues in new chunk
- Matching attempted asynchronously
**Test Cases**:
1. **Create chunk**: Chunk created successfully
2. **Proactive creation**: Chunk created before matching attempts
3. **Continue processing**: Processing continues in new chunk
---
### `try_chunk_semantic_matching(chunk_id: str) -> Optional[List[TileCandidate]]`
**Description**: Attempts semantic matching for a chunk using aggregate DINOv2 descriptor.
**Called By**:
- Internal (when chunk ready for matching)
- process_unanchored_chunks() (background task)
**Input**:
```python
chunk_id: str
```
**Output**:
```python
Optional[List[TileCandidate]]: Top-k candidate tiles or None if matching failed
```
**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. Call F08 Global Place Recognition.retrieve_candidate_tiles_for_chunk()
3. F08 computes chunk descriptor and queries Faiss
4. Return candidate tiles if found, None otherwise
**Test Cases**:
1. **Chunk matching**: Returns candidate tiles
2. **Featureless terrain**: Succeeds where single-image fails
3. **No match**: Returns None
---
### `try_chunk_litesam_matching(chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]`
**Description**: Attempts LiteSAM matching for chunk with rotation sweeps.
**Called By**:
- Internal (after chunk semantic matching succeeds)
- process_unanchored_chunks() (background task)
**Input**:
```python
chunk_id: str
candidate_tiles: List[TileCandidate] # From chunk semantic matching
```
**Output**:
```python
Optional[ChunkAlignmentResult]: Match result or None
```
**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. For each candidate tile:
- Get tile from F04 Satellite Data Manager
- Call F06.try_chunk_rotation_steps() (12 rotations: 0°, 30°, ..., 330°)
- F06 calls F09.align_chunk_to_satellite() for each rotation
- If match found with confidence > threshold:
- Return ChunkAlignmentResult
3. Return None if no match found
**Rotation Sweeps**:
- Critical for chunks from sharp turns (unknown orientation)
- Tries all 12 rotation angles
- Returns best matching rotation
**Test Cases**:
1. **Match on first tile**: Returns alignment result
2. **Match on 3rd tile**: Returns alignment result
3. **Match at 120° rotation**: Returns result with correct rotation angle
4. **No match**: Returns None
---
### `merge_chunk_to_trajectory(flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool`
**Description**: Merges chunk into main trajectory after successful matching.
**Called By**:
- Internal (after chunk LiteSAM matching succeeds)
- process_unanchored_chunks() (which has flight_id from chunk handle)
**Input**:
```python
flight_id: str # Flight identifier (available from chunk handle)
chunk_id: str
alignment_result: ChunkAlignmentResult:
chunk_center_gps: GPSPoint
transform: Sim3Transform
confidence: float
```
**Output**:
```python
bool: True if merge successful
```
**Processing Flow**:
1. Get chunk frames via F12.get_chunk_frames(chunk_id) → merged_frames
2. Get chunk anchor frame (middle frame or best frame)
3. Call F12.mark_chunk_anchored() with GPS (F12 coordinates with F10)
4. **Determine merge target**:
- Main chunk is typically the temporal predecessor (previous chunk by frame_id order)
- If no predecessor: merge to main trajectory (main_chunk_id="main")
- F11 determines main_chunk based on chunk frame_id ordering
5. Call F12.merge_chunks(main_chunk_id, chunk_id, transform)
- Note: `merge_chunks(main_chunk, new_chunk)` merges new_chunk INTO main_chunk
- chunk_id (source) is merged into target_chunk_id
6. F12 handles chunk state updates (deactivation, status updates)
7. F10 optimizes merged graph globally (via F12.merge_chunks())
8. Return True (Caller F02.2 coordinates result updates)
**Sim(3) Transform**:
- Translation: GPS offset
- Rotation: From alignment result
- Scale: Resolved from altitude and GSD
**Test Cases**:
1. **Merge successful**: Chunks merged successfully
2. **Global consistency**: Merged trajectory globally consistent
3. **Multiple chunks**: Can merge multiple chunks sequentially
---
### `process_unanchored_chunks(flight_id: str) -> None`
**Description**: Background task that periodically attempts matching for unanchored chunks.
**Called By**:
- Background thread (periodic, e.g., every 5 seconds)
- F02.2 Flight Processing Engine (after frame processing)
**Input**:
```python
flight_id: str
```
**Output**: None (runs asynchronously)
**Processing Flow**:
```
while flight_active:
unanchored_chunks = F12.get_chunks_for_matching(flight_id)
for chunk in unanchored_chunks:
if F12.is_chunk_ready_for_matching(chunk.chunk_id):
F12.mark_chunk_matching(chunk.chunk_id)
candidates = try_chunk_semantic_matching(chunk.chunk_id)
if candidates:
alignment = try_chunk_litesam_matching(chunk.chunk_id, candidates)
if alignment:
merge_chunk_to_trajectory(chunk.flight_id, chunk.chunk_id, alignment)
sleep(5 seconds)
```
**Background Processing**:
- **Trigger**: Started by F02.2 Flight Processing Engine after flight creation
- Runs asynchronously, doesn't block frame processing
- Periodically checks for ready chunks (every 5 seconds)
- Attempts matching and merging
- Reduces user input requests
- **Lifecycle**: Starts when flight becomes active, stops when flight completed
**Chunk Failure Flow**:
When chunk matching fails after trying all candidates:
1. Emit `ChunkMatchingFailed` event with chunk_id and flight_id (to F02.2 internal queue)
2. F02.2 receives event and decides next action:
- **Option A**: Wait for more frames and retry later (chunk may gain more distinctive features)
- **Option B**: Create user input request for the chunk
3. If user input requested:
- F02.2 calls F15.send_user_input_request() with chunk context
- Client receives via SSE
- User provides GPS anchor for one frame in chunk
- F02.2 receives user fix via handle_user_fix()
- F11.apply_user_anchor() anchors the chunk
- Processing resumes
**Test Cases**:
1. **Background matching**: Unanchored chunks matched asynchronously
2. **Chunk merging**: Chunks merged when matches found
3. **Non-blocking**: Frame processing continues during matching
4. **Chunk matching fails**: Failure handled, user input requested if needed
## Integration Tests
### Test 1: Progressive Search Flow
1. Tracking lost detected
2. start_search() → grid_size=1
3. try_current_grid(1 tile) → no match
4. expand_search_radius() → grid_size=4
5. try_current_grid(4 tiles) → match found
6. mark_found() → success
### Test 2: Full Search Exhaustion
1. start_search()
2. try grids: 1→4→9→16→25, all fail
3. create_user_input_request()
4. User provides anchor
5. apply_user_anchor() → processing resumes
### Test 3: Confidence Monitoring
1. Normal frames → confidence good
2. Low overlap frame → confidence degraded
3. Sharp turn → tracking lost, trigger search
### Test 4: Proactive Chunk Creation
1. Tracking lost detected
2. create_chunk_on_tracking_loss() → chunk created immediately
3. Processing continues in new chunk
4. Verify chunk matching attempted asynchronously
### Test 5: Chunk Semantic Matching
1. Build chunk with 10 images (plain field)
2. try_chunk_semantic_matching() → returns candidate tiles
3. Verify chunk matching succeeds where single-image fails
### Test 6: Chunk LiteSAM Matching with Rotation
1. Build chunk with unknown orientation
2. try_chunk_litesam_matching() with candidate tiles
3. Rotation sweep finds match at 120°
4. Returns ChunkAlignmentResult with correct GPS
### Test 7: Chunk Merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Anchor chunk_2 via chunk matching
3. merge_chunk_to_trajectory() → chunks merged
4. Verify global trajectory consistent
### Test 8: Background Chunk Processing
1. Create 3 unanchored chunks
2. process_unanchored_chunks() runs in background
3. Chunks matched and merged asynchronously
4. Frame processing continues uninterrupted
## Non-Functional Requirements
### Performance
- **check_confidence**: < 10ms
- **Progressive search (25 tiles)**: < 1.5s total
- **User input latency**: < 500ms from creation to SSE event
### Reliability
- Always exhausts all search strategies before requesting user input
- Guarantees processing block when awaiting user input
- Graceful recovery from all failure modes
## Dependencies
### Internal Components
- **F12 Route Chunk Manager**: Chunk operations.
- **F10 Factor Graph Optimizer**: Anchor application.
- **F04 Satellite Data Manager**: Search grids.
- **F08 Global Place Recognition**: Candidate retrieval.
- **F09 Metric Refinement**: Alignment.
- **F06 Image Rotation Manager**: Rotation sweeps.
### External Dependencies
- None
## Data Models
### ConfidenceAssessment
```python
class ConfidenceAssessment(BaseModel):
overall_confidence: float
vo_confidence: float
litesam_confidence: float
inlier_count: int
tracking_status: str
```
### SearchSession
```python
class SearchSession(BaseModel):
session_id: str
flight_id: str
frame_id: int
center_gps: GPSPoint
current_grid_size: int
max_grid_size: int
found: bool
exhausted: bool
```
### UserInputRequest
```python
class UserInputRequest(BaseModel):
request_id: str
flight_id: str
frame_id: int
uav_image: np.ndarray
candidate_tiles: List[TileCandidate]
message: str
created_at: datetime
```
### UserAnchor
```python
class UserAnchor(BaseModel):
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
confidence: float = 1.0
```
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### ChunkAlignmentResult
```python
class ChunkAlignmentResult(BaseModel):
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or (4,) quaternion
scale: float
```
### RecoveryStatus (Returned by recovery methods)
```python
class RecoveryStatus(BaseModel):
"""Status returned by F11 methods, used by F02 to determine next action."""
success: bool
method: str # "rotation_sweep", "progressive_search", "chunk_matching", "user_input"
gps: Optional[GPSPoint]
chunk_id: Optional[str]
message: Optional[str]
```
_docs/02_components/12_route_chunk_manager/12.01_feature_chunk_lifecycle_management.md
-123
View File
@@ -1,123 +0,0 @@
# Feature: Chunk Lifecycle Management
## Name
Chunk Lifecycle Management
## Description
Core operations for creating, growing, and managing the lifecycle of route chunks. Chunks are first-class entities in the Atlas multi-map architecture, created proactively on tracking loss to continue processing without failure. This feature handles the fundamental CRUD-like operations for chunk management and maintains internal state tracking.
## Component APIs Implemented
### `create_chunk(flight_id: str, start_frame_id: int) -> ChunkHandle`
Creates a new route chunk when tracking is lost or proactively by F11.
- Generates unique chunk_id
- Calls F10.create_new_chunk() to initialize subgraph
- Initializes chunk state (unanchored, active)
- Stores ChunkHandle in internal dictionary
### `add_frame_to_chunk(chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool`
Adds a frame to an active chunk during frame processing.
- Verifies chunk exists and is active
- Appends frame_id to chunk's frames list
- Calls F10.add_relative_factor_to_chunk()
- Updates end_frame_id
### `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]`
Retrieves the currently active chunk for a flight.
- Queries internal state dictionary
- Returns ChunkHandle with is_active=True or None
### `deactivate_chunk(chunk_id: str) -> bool`
Deactivates a chunk after merging or completion.
- Sets is_active=False on ChunkHandle
- Does not delete chunk data
## External Tools and Services
- **F10 Factor Graph Optimizer**: `create_new_chunk()`, `add_relative_factor_to_chunk()`
## Internal Methods
### `_generate_chunk_id() -> str`
Generates unique chunk identifier (UUID or sequential).
### `_validate_chunk_active(chunk_id: str) -> bool`
Checks if chunk exists and is_active=True.
### `_get_chunk_by_id(chunk_id: str) -> Optional[ChunkHandle]`
Internal lookup in _chunks dictionary.
### `_update_chunk_end_frame(chunk_id: str, frame_id: int)`
Updates end_frame_id after adding frame.
## Unit Tests
### Test: create_chunk_returns_active_handle
- Call create_chunk(flight_id, start_frame_id)
- Assert returned ChunkHandle has is_active=True
- Assert chunk_id is non-empty string
- Assert start_frame_id matches input
### Test: create_chunk_calls_f10
- Call create_chunk()
- Assert F10.create_new_chunk() was called with correct parameters
### Test: create_multiple_chunks_same_flight
- Create chunk_1 and chunk_2 for same flight
- Assert both chunks exist with different chunk_ids
### Test: add_frame_to_active_chunk
- Create chunk
- Call add_frame_to_chunk() with valid frame
- Assert returns True
- Assert frame_id in chunk.frames
### Test: add_frame_updates_end_frame_id
- Create chunk with start_frame_id=10
- Add frames 11, 12, 13
- Assert end_frame_id=13
### Test: add_frame_to_inactive_chunk_fails
- Create chunk, then deactivate_chunk()
- Call add_frame_to_chunk()
- Assert returns False
### Test: add_frame_to_nonexistent_chunk_fails
- Call add_frame_to_chunk("invalid_id", ...)
- Assert returns False
### Test: get_active_chunk_returns_correct_chunk
- Create chunk for flight_1
- Assert get_active_chunk(flight_1) returns chunk
- Assert get_active_chunk(flight_2) returns None
### Test: get_active_chunk_none_when_deactivated
- Create chunk, then deactivate
- Assert get_active_chunk() returns None
### Test: deactivate_chunk_success
- Create chunk
- Call deactivate_chunk()
- Assert returns True
- Assert chunk.is_active=False
### Test: deactivate_nonexistent_chunk_fails
- Call deactivate_chunk("invalid_id")
- Assert returns False
## Integration Tests
### Test: chunk_lifecycle_flow
1. create_chunk() → verify ChunkHandle
2. add_frame_to_chunk() × 10 → verify frames list grows
3. get_active_chunk() → returns the chunk
4. deactivate_chunk() → chunk deactivated
5. get_active_chunk() → returns None
### Test: multiple_chunks_isolation
1. Create chunk_1 (flight_A)
2. Create chunk_2 (flight_A)
3. Add frames to chunk_1
4. Add frames to chunk_2
5. Verify frames lists are independent
6. Verify only one can be active at a time per flight
_docs/02_components/12_route_chunk_manager/12.02_feature_chunk_data_retrieval.md
-103
View File
@@ -1,103 +0,0 @@
# Feature: Chunk Data Retrieval
## Name
Chunk Data Retrieval
## Description
Query operations for retrieving chunk data including frame lists, images, estimated bounds, and composite descriptors. These methods provide data to other components (F08, F09, F11) for matching and recovery operations. The feature handles data aggregation and transformation without modifying chunk state.
## Component APIs Implemented
### `get_chunk_frames(chunk_id: str) -> List[int]`
Retrieves ordered list of frame IDs in a chunk.
- Returns frames list from ChunkHandle
- Ordered by sequence (insertion order)
### `get_chunk_images(chunk_id: str) -> List[np.ndarray]`
Retrieves images for all frames in a chunk.
- Calls F05.get_image_by_sequence() for each frame
- Returns images in frame order
### `get_chunk_bounds(chunk_id: str) -> ChunkBounds`
Estimates GPS bounds of a chunk based on VO trajectory.
- Computes estimated center from relative poses
- Calculates radius from trajectory extent
- Returns confidence based on anchor status
### `get_chunk_composite_descriptor(chunk_id: str) -> np.ndarray`
Computes aggregate DINOv2 descriptor for semantic matching.
- Retrieves chunk images
- Calls F08.compute_chunk_descriptor() with aggregation strategy
- Returns 4096-dim or 8192-dim vector
## External Tools and Services
- **F05 Image Input Pipeline**: `get_image_by_sequence()`
- **F08 Global Place Recognition**: `compute_chunk_descriptor()`
## Internal Methods
### `_compute_trajectory_extent(chunk_id: str) -> Tuple[float, float]`
Calculates trajectory spread from VO poses for bounds estimation.
### `_estimate_center_from_poses(chunk_id: str) -> GPSPoint`
Estimates center GPS from accumulated relative poses and last known anchor.
### `_calculate_bounds_confidence(chunk_id: str) -> float`
Returns confidence (0.0-1.0) based on anchor status and trajectory length.
## Unit Tests
### Test: get_chunk_frames_returns_ordered_list
- Create chunk, add frames 10, 11, 12
- Assert get_chunk_frames() returns [10, 11, 12]
### Test: get_chunk_frames_empty_chunk
- Create chunk without adding frames
- Assert get_chunk_frames() returns [start_frame_id] only
### Test: get_chunk_frames_nonexistent_chunk
- Call get_chunk_frames("invalid_id")
- Assert returns empty list or raises exception
### Test: get_chunk_images_returns_correct_count
- Create chunk with 5 frames
- Mock F05.get_image_by_sequence()
- Assert get_chunk_images() returns 5 images
### Test: get_chunk_images_preserves_order
- Create chunk with frames [10, 11, 12]
- Assert images returned in same order as frames
### Test: get_chunk_bounds_unanchored
- Create unanchored chunk with 10 frames
- Assert get_chunk_bounds().confidence < 0.5
- Assert estimated_radius > 0
### Test: get_chunk_bounds_anchored
- Create chunk, mark as anchored
- Assert get_chunk_bounds().confidence > 0.7
### Test: get_chunk_composite_descriptor_shape
- Create chunk with 10 frames
- Mock F08.compute_chunk_descriptor()
- Assert descriptor has expected dimensions (4096 or 8192)
### Test: get_chunk_composite_descriptor_calls_f08
- Create chunk
- Call get_chunk_composite_descriptor()
- Assert F08.compute_chunk_descriptor() called with chunk images
## Integration Tests
### Test: chunk_descriptor_computation
1. Create chunk with 10 frames
2. get_chunk_images() → 10 images
3. get_chunk_composite_descriptor() → aggregated descriptor
4. Verify descriptor shape and non-zero values
### Test: chunk_bounds_accuracy
1. Create chunk with known anchor
2. Add 10 frames with VO results
3. get_chunk_bounds() → verify center near anchor
4. Verify radius reasonable for trajectory length
_docs/02_components/12_route_chunk_manager/12.03_feature_chunk_matching_coordination.md
-153
View File
@@ -1,153 +0,0 @@
# Feature: Chunk Matching Coordination
## Name
Chunk Matching Coordination
## Description
Operations related to the chunk matching workflow including readiness checks, matching status tracking, anchoring, and merging. This feature implements transactional integrity with F10 Factor Graph Optimizer using "Check-Act" pattern: F10 is called first, and only on success is internal state updated. Critical for the Atlas multi-map architecture where chunks are matched and merged into the global trajectory.
## Component APIs Implemented
### `is_chunk_ready_for_matching(chunk_id: str) -> bool`
Checks if chunk meets criteria for satellite matching attempt.
- Min frames: >= 5 (configurable)
- Max frames: <= 20 (configurable)
- Not already matched (status != "anchored" or "merged")
### `get_chunks_for_matching(flight_id: str) -> List[ChunkHandle]`
Retrieves all unanchored chunks ready for matching.
- Filters by flight_id
- Returns chunks with matching_status="unanchored" and is_ready_for_matching=True
### `mark_chunk_matching(chunk_id: str) -> bool`
Marks chunk as currently being matched.
- Updates matching_status to "matching"
- Prevents duplicate matching attempts
### `mark_chunk_anchored(chunk_id: str, frame_id: int, gps: GPSPoint) -> bool`
Anchors chunk to GPS coordinate after successful satellite matching.
- **Transactional**: Calls F10.add_chunk_anchor() first
- On success: Updates has_anchor, anchor_frame_id, anchor_gps, matching_status="anchored"
- On failure: Returns False, no state change
### `merge_chunks(main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool`
Merges new_chunk INTO main_chunk using Sim(3) transformation.
- Resolves flight_id from internal ChunkHandle for main_chunk_id
- **Transactional**: Calls F10.merge_chunk_subgraphs() first
- On success: new_chunk marked "merged" and deactivated, main_chunk extended
- On failure: Returns False, no state change
## External Tools and Services
- **F10 Factor Graph Optimizer**: `add_chunk_anchor()`, `merge_chunk_subgraphs()`
- **F03 Flight Database**: `save_chunk_state()` (after merge)
## Internal Methods
### `_check_matching_criteria(chunk_id: str) -> bool`
Validates chunk meets all matching readiness criteria.
### `_get_matching_status(chunk_id: str) -> str`
Returns current matching_status from ChunkHandle.
### `_update_anchor_state(chunk_id: str, frame_id: int, gps: GPSPoint)`
Updates ChunkHandle with anchor information after successful F10 call.
### `_mark_chunk_merged(chunk_id: str)`
Updates new_chunk state after successful merge operation.
## Unit Tests
### Test: is_chunk_ready_min_frames
- Create chunk with 3 frames
- Assert is_chunk_ready_for_matching() returns False
- Add 2 more frames (total 5)
- Assert is_chunk_ready_for_matching() returns True
### Test: is_chunk_ready_max_frames
- Create chunk with 21 frames
- Assert is_chunk_ready_for_matching() returns False
### Test: is_chunk_ready_already_anchored
- Create chunk, mark as anchored
- Assert is_chunk_ready_for_matching() returns False
### Test: get_chunks_for_matching_filters_correctly
- Create 3 chunks: unanchored+ready, anchored, unanchored+not_ready
- Assert get_chunks_for_matching() returns only first chunk
### Test: get_chunks_for_matching_filters_by_flight
- Create chunks for flight_A and flight_B
- Assert get_chunks_for_matching(flight_A) returns only flight_A chunks
### Test: mark_chunk_matching_updates_status
- Create chunk
- Call mark_chunk_matching()
- Assert matching_status="matching"
### Test: mark_chunk_anchored_transactional_success
- Create chunk
- Mock F10.add_chunk_anchor() returns success
- Call mark_chunk_anchored()
- Assert returns True
- Assert has_anchor=True, anchor_gps set, matching_status="anchored"
### Test: mark_chunk_anchored_transactional_failure
- Create chunk
- Mock F10.add_chunk_anchor() returns failure
- Call mark_chunk_anchored()
- Assert returns False
- Assert has_anchor=False (no state change)
### Test: merge_chunks_transactional_success
- Create main_chunk and new_chunk
- Mock F10.merge_chunk_subgraphs() returns success
- Call merge_chunks()
- Assert returns True
- Assert new_chunk.is_active=False
- Assert new_chunk.matching_status="merged"
### Test: merge_chunks_transactional_failure
- Create main_chunk and new_chunk
- Mock F10.merge_chunk_subgraphs() returns failure
- Call merge_chunks()
- Assert returns False
- Assert new_chunk state unchanged
### Test: merge_chunks_resolves_flight_id
- Create main_chunk for flight_A
- Verify merge_chunks() calls F10 with correct flight_id from ChunkHandle
## Integration Tests
### Test: chunk_matching_workflow
1. create_chunk() with 10 frames
2. is_chunk_ready_for_matching() → True
3. mark_chunk_matching() → status="matching"
4. mark_chunk_anchored() → status="anchored"
5. is_chunk_ready_for_matching() → False
### Test: chunk_merge_workflow
1. Create main_chunk (frames 1-10), new_chunk (frames 20-30)
2. Anchor new_chunk via mark_chunk_anchored()
3. merge_chunks(main_chunk, new_chunk, transform)
4. Verify new_chunk deactivated and marked "merged"
5. Verify main_chunk still active
### Test: transactional_integrity_anchor
1. Create chunk
2. Configure F10.add_chunk_anchor() to fail
3. Call mark_chunk_anchored()
4. Verify chunk state unchanged
5. Configure F10.add_chunk_anchor() to succeed
6. Call mark_chunk_anchored()
7. Verify chunk state updated
### Test: transactional_integrity_merge
1. Create main_chunk and new_chunk
2. Configure F10.merge_chunk_subgraphs() to fail
3. Call merge_chunks()
4. Verify both chunks' state unchanged
5. Configure F10.merge_chunk_subgraphs() to succeed
6. Call merge_chunks()
7. Verify new_chunk state updated
_docs/02_components/12_route_chunk_manager/12.04_feature_chunk_state_persistence.md
-114
View File
@@ -1,114 +0,0 @@
# Feature: Chunk State Persistence
## Name
Chunk State Persistence
## Description
Persistence operations for saving and loading chunk state via F03 Flight Database. Enables system recovery after restarts and maintains chunk state across processing sessions. Serializes internal _chunks dictionary to persistent storage.
## Component APIs Implemented
### `save_chunk_state(flight_id: str) -> bool`
Persists all chunk state for a flight.
- Serializes ChunkHandles for specified flight
- Calls F03.save_chunk_state()
- Returns success status
### `load_chunk_state(flight_id: str) -> bool`
Loads chunk state for a flight from persistent storage.
- Calls F03.load_chunk_states()
- Deserializes and populates internal _chunks dictionary
- Returns success status
## External Tools and Services
- **F03 Flight Database**: `save_chunk_state()`, `load_chunk_states()`
## Internal Methods
### `_serialize_chunks(flight_id: str) -> Dict`
Converts ChunkHandles to serializable dictionary format.
### `_deserialize_chunks(data: Dict) -> Dict[str, ChunkHandle]`
Reconstructs ChunkHandles from serialized data.
### `_filter_chunks_by_flight(flight_id: str) -> List[ChunkHandle]`
Filters internal _chunks dictionary by flight_id.
### `_merge_loaded_chunks(chunks: Dict[str, ChunkHandle])`
Merges loaded chunks into internal state without overwriting existing.
## Unit Tests
### Test: save_chunk_state_calls_f03
- Create chunks for flight_A
- Call save_chunk_state(flight_A)
- Assert F03.save_chunk_state() called with serialized data
### Test: save_chunk_state_filters_by_flight
- Create chunks for flight_A and flight_B
- Call save_chunk_state(flight_A)
- Assert only flight_A chunks serialized
### Test: save_chunk_state_success
- Create chunks
- Mock F03.save_chunk_state() returns success
- Assert save_chunk_state() returns True
### Test: save_chunk_state_failure
- Create chunks
- Mock F03.save_chunk_state() returns failure
- Assert save_chunk_state() returns False
### Test: load_chunk_state_populates_internal_dict
- Mock F03.load_chunk_states() returns chunk data
- Call load_chunk_state(flight_A)
- Assert internal _chunks dictionary populated
### Test: load_chunk_state_success
- Mock F03.load_chunk_states() returns success
- Assert load_chunk_state() returns True
### Test: load_chunk_state_failure
- Mock F03.load_chunk_states() returns failure
- Assert load_chunk_state() returns False
### Test: load_chunk_state_empty_flight
- Mock F03.load_chunk_states() returns empty
- Call load_chunk_state(flight_A)
- Assert returns True (no error)
- Assert no chunks loaded
### Test: serialize_preserves_all_fields
- Create chunk with all fields populated (anchor, frames, status)
- Serialize and deserialize
- Assert all fields match original
### Test: deserialize_handles_missing_optional_fields
- Create serialized data with optional fields missing
- Deserialize
- Assert ChunkHandle created with default values
## Integration Tests
### Test: save_load_roundtrip
1. Create chunks with various states (active, anchored, merged)
2. save_chunk_state()
3. Clear internal state
4. load_chunk_state()
5. Verify all chunks restored with correct state
### Test: persistence_across_restart
1. Create flight with chunks
2. Process frames, anchor some chunks
3. save_chunk_state()
4. Simulate restart (new instance)
5. load_chunk_state()
6. Verify can continue processing
### Test: partial_state_recovery
1. Create chunks for multiple flights
2. save_chunk_state(flight_A)
3. Clear internal state
4. load_chunk_state(flight_A)
5. Verify only flight_A chunks loaded
_docs/02_components/12_route_chunk_manager/12._component_route_chunk_manager.md
-471
View File
@@ -1,471 +0,0 @@
# Route Chunk Manager
## Interface Definition
**Interface Name**: `IRouteChunkManager`
### Interface Methods
```python
class IRouteChunkManager(ABC):
@abstractmethod
def create_chunk(self, flight_id: str, start_frame_id: int) -> ChunkHandle:
pass
@abstractmethod
def add_frame_to_chunk(self, chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool:
pass
@abstractmethod
def get_chunk_frames(self, chunk_id: str) -> List[int]:
pass
@abstractmethod
def get_chunk_images(self, chunk_id: str) -> List[np.ndarray]:
pass
@abstractmethod
def get_chunk_composite_descriptor(self, chunk_id: str) -> np.ndarray:
pass
@abstractmethod
def get_chunk_bounds(self, chunk_id: str) -> ChunkBounds:
pass
@abstractmethod
def is_chunk_ready_for_matching(self, chunk_id: str) -> bool:
pass
@abstractmethod
def mark_chunk_anchored(self, chunk_id: str, frame_id: int, gps: GPSPoint) -> bool:
"""
Transactional update:
1. Calls F10.add_chunk_anchor().
2. IF success: Updates internal state to 'anchored'.
3. Returns success status.
"""
pass
@abstractmethod
def get_chunks_for_matching(self, flight_id: str) -> List[ChunkHandle]:
pass
@abstractmethod
def get_active_chunk(self, flight_id: str) -> Optional[ChunkHandle]:
pass
@abstractmethod
def deactivate_chunk(self, chunk_id: str) -> bool:
pass
@abstractmethod
def merge_chunks(self, main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool:
"""
Merges new_chunk INTO main_chunk. Extends main_chunk with new_chunk's frames.
Transactional update:
1. Calls F10.merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform).
2. IF success: Updates internal state (new_chunk merged/deactivated).
3. Returns success status.
Note: flight_id is obtained from the ChunkHandle stored internally for main_chunk_id.
"""
pass
@abstractmethod
def mark_chunk_matching(self, chunk_id: str) -> bool:
pass
@abstractmethod
def save_chunk_state(self, flight_id: str) -> bool:
pass
@abstractmethod
def load_chunk_state(self, flight_id: str) -> bool:
pass
```
## Component Description
### Responsibilities
- **Source of Truth**: Manages chunk states (Active, Matching, Anchored, Merged).
- **Transactional Integrity**: Ensures internal state updates are atomic with respect to Factor Graph (F10) operations.
- **Implementation**: Uses "Check-Act" pattern. Calls F10 first; if F10 fails, F12 does not update state and returns error.
### Internal State Management
F12 maintains an internal dictionary of `ChunkHandle` objects keyed by `chunk_id`. Each `ChunkHandle` contains `flight_id`, allowing F12 to resolve flight context for F10 calls without requiring `flight_id` as a parameter on every method. This simplifies the API for callers while maintaining flight isolation internally.
```python
# Internal state
_chunks: Dict[str, ChunkHandle] # chunk_id -> ChunkHandle (contains flight_id)
```
### Interaction
- Called by **F02.2 Flight Processing Engine** and **F11 Failure Recovery Coordinator** (via F02.2 or direct delegation).
- Calls **F10 Factor Graph Optimizer**.
### Scope
- Chunk lifecycle management
- Chunk state tracking
- Chunk representation generation (descriptors, bounds)
- Integration point for chunk matching coordination
## API Methods
### `create_chunk(flight_id: str, start_frame_id: int) -> ChunkHandle`
**Description**: Initializes a new route chunk.
**Called By**:
- F02.2 Flight Processing Engine (when tracking lost)
- F11 Failure Recovery Coordinator (proactive chunk creation)
**Input**:
```python
flight_id: str
start_frame_id: int # First frame in chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str
```
**Processing Flow**:
1. Generate unique chunk_id
2. Call F10 Factor Graph Optimizer.create_new_chunk()
3. Initialize chunk state (unanchored, active)
4. Store chunk metadata
5. Return ChunkHandle
**Test Cases**:
1. **Create chunk**: Returns ChunkHandle with is_active=True
2. **Multiple chunks**: Can create multiple chunks for same flight
3. **Chunk initialization**: Chunk initialized in factor graph
---
### `add_frame_to_chunk(chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool`
**Description**: Adds a frame to an active chunk.
**Called By**:
- F02.2 Flight Processing Engine (during frame processing)
**Input**:
```python
chunk_id: str
frame_id: int
vo_result: RelativePose # From F07 Sequential VO
```
**Output**:
```python
bool: True if frame added successfully
```
**Processing Flow**:
1. Verify chunk exists and is active
2. Add frame_id to chunk's frames list
3. Store vo_result for chunk
4. Call F10.add_relative_factor_to_chunk()
5. Update chunk's end_frame_id
6. Check if chunk ready for matching
**Test Cases**:
1. **Add frame to active chunk**: Frame added successfully
2. **Add frame to inactive chunk**: Returns False
3. **Chunk growth**: Chunk frames list updated
---
### `get_chunk_frames(chunk_id: str) -> List[int]`
**Description**: Retrieves list of frame IDs in a chunk.
**Called By**:
- F08 Global Place Recognition (for chunk descriptor computation)
- F09 Metric Refinement (for chunk LiteSAM matching)
- F11 Failure Recovery Coordinator (chunk state queries)
**Input**: `chunk_id: str`
**Output**: `List[int]` # Frame IDs in chunk, ordered by sequence
---
### `get_chunk_images(chunk_id: str) -> List[np.ndarray]`
**Description**: Retrieves images for all frames in a chunk.
**Called By**:
- F08 Global Place Recognition (chunk descriptor computation)
- F09 Metric Refinement (chunk LiteSAM matching)
- F06 Image Rotation Manager (chunk rotation)
**Output**: `List[np.ndarray]` # Images for each frame in chunk
---
### `get_chunk_composite_descriptor(chunk_id: str) -> np.ndarray`
**Description**: Computes aggregate DINOv2 descriptor for chunk (for semantic matching).
**Called By**:
- F08 Global Place Recognition (chunk semantic matching)
**Output**: `np.ndarray`: Aggregated descriptor vector (4096-dim or 8192-dim)
---
### `get_chunk_bounds(chunk_id: str) -> ChunkBounds`
**Description**: Estimates GPS bounds of a chunk based on VO trajectory.
**Called By**:
- F11 Failure Recovery Coordinator (for tile search area)
- F04 Satellite Data Manager (for tile prefetching)
**Output**:
```python
ChunkBounds:
estimated_center: GPSPoint
estimated_radius: float # meters
confidence: float
```
---
### `is_chunk_ready_for_matching(chunk_id: str) -> bool`
**Description**: Checks if a chunk has enough data (frames, spread) to attempt satellite matching.
**Criteria**:
- Min frames: >= 5 frames (configurable)
- Max frames: <= 20 frames (configurable, prevents oversized chunks)
- Internal consistency: VO factors have reasonable inlier counts
- Not already matched: matching_status != "anchored" or "merged"
---
### `mark_chunk_anchored(chunk_id: str, frame_id: int, gps: GPSPoint) -> bool`
**Description**: Anchors a chunk to a specific GPS coordinate (e.g., from successful satellite matching).
**Called By**:
- F11 Failure Recovery Coordinator (after successful chunk matching)
**Input**:
```python
chunk_id: str
frame_id: int # Frame within chunk that was anchored
gps: GPSPoint
```
**Output**: `bool` - True if marked successfully
**Processing Flow**:
1. Verify chunk exists
2. Call F10.add_chunk_anchor()
3. If successful:
- Update chunk state (has_anchor=True, anchor_frame_id, anchor_gps)
- Update matching_status to "anchored"
- Trigger chunk optimization
**Test Cases**:
1. **Mark anchored**: Chunk state updated correctly
2. **Anchor in factor graph**: F10 anchor added
3. **Chunk optimization**: Chunk optimized after anchoring
---
### `get_chunks_for_matching(flight_id: str) -> List[ChunkHandle]`
**Description**: Retrieves all unanchored chunks ready for matching.
**Called By**:
- F11 Failure Recovery Coordinator (background matching task)
**Output**: `List[ChunkHandle]` # Unanchored chunks ready for matching
---
### `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]`
**Description**: Gets the currently active chunk for a flight.
**Called By**:
- F02.2 Flight Processing Engine (before processing frame)
**Output**: `Optional[ChunkHandle]` # Active chunk or None
---
### `deactivate_chunk(chunk_id: str) -> bool`
**Description**: Deactivates a chunk (typically after merging or completion).
**Called By**:
- F11 Failure Recovery Coordinator (after chunk merged)
- F02.2 Flight Processing Engine (chunk lifecycle)
**Output**: `bool` - True if deactivated successfully
---
### `merge_chunks(main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool`
**Description**: Merges new_chunk INTO main_chunk. The resulting merged chunk is main_chunk (extended with new_chunk's frames). new_chunk is deactivated after merge.
**Called By**:
- F11 Failure Recovery Coordinator (after successful chunk matching)
**Input**:
```python
main_chunk_id: str # Main chunk being extended (destination, typically older/established trajectory)
new_chunk_id: str # New chunk being merged in (source, typically newer/recently anchored)
transform: Sim3Transform:
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or quaternion
scale: float
```
**Output**: `bool` - True if merge successful
**flight_id Resolution**: F12 internally stores ChunkHandle objects keyed by chunk_id. The flight_id is extracted from the ChunkHandle for main_chunk_id when calling F10 methods.
**Processing Flow**:
1. Get flight_id from internally stored ChunkHandle for main_chunk_id
2. Call F10.merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform)
3. If successful:
- Update new_chunk_id state:
- Set is_active=False
- Set matching_status="merged"
- Call deactivate_chunk(new_chunk_id)
- main_chunk remains active (now contains merged frames)
- Persist chunk state via F03 Flight Database.save_chunk_state()
4. Return True
**Test Cases**:
1. **Merge anchored chunk**: new_chunk merged into main_chunk
2. **New chunk deactivated**: new_chunk marked as merged and deactivated
3. **Main chunk extended**: main_chunk remains active with additional frames
---
### `mark_chunk_matching(chunk_id: str) -> bool`
**Description**: Explicitly marks chunk as being matched (updates matching_status to "matching").
**Called By**:
- F11 Failure Recovery Coordinator (when chunk matching starts)
**Output**: `bool` - True if marked successfully
## Integration Tests
### Test 1: Chunk Lifecycle
1. create_chunk() → chunk created
2. add_frame_to_chunk() × 10 → 10 frames added
3. is_chunk_ready_for_matching() → True
4. mark_chunk_anchored() → chunk anchored
5. deactivate_chunk() → chunk deactivated
### Test 2: Chunk Descriptor Computation
1. Create chunk with 10 frames
2. get_chunk_images() → 10 images
3. get_chunk_composite_descriptor() → aggregated descriptor
4. Verify descriptor more robust than single-image descriptor
### Test 3: Multiple Chunks
1. Create chunk_1 (frames 1-10)
2. Create chunk_2 (frames 20-30)
3. get_chunks_for_matching() → returns both chunks
4. mark_chunk_anchored(chunk_1) → chunk_1 anchored
5. get_chunks_for_matching() → returns only chunk_2
### Test 4: Chunk Merging
1. Create main_chunk (frames 1-10), new_chunk (frames 20-30)
2. Anchor new_chunk via mark_chunk_anchored()
3. merge_chunks(main_chunk, new_chunk, transform) → new_chunk merged into main_chunk
4. Verify new_chunk marked as merged and deactivated
5. Verify main_chunk extended with new frames
6. Verify F10.merge_chunk_subgraphs() called with correct parameters
### Test 5: Chunk Matching Status
1. Create chunk
2. mark_chunk_matching() → status updated to "matching"
3. mark_chunk_anchored() → status updated to "anchored"
4. Verify explicit state transitions
## Non-Functional Requirements
### Performance
- **create_chunk**: < 10ms
- **add_frame_to_chunk**: < 5ms
- **get_chunk_composite_descriptor**: < 3s for 20 images (async)
- **get_chunk_bounds**: < 10ms
### Reliability
- Chunk state persisted across restarts
- Graceful handling of missing frames
- Thread-safe chunk operations
## Dependencies
### Internal Components
- **F10 Factor Graph Optimizer**: Critical dependency for subgraph operations (`create_chunk_subgraph`, `add_relative_factor_to_chunk`, `merge_chunk_subgraphs`).
- **F03 Flight Database**: Persistence via `save_chunk_state()`, `load_chunk_states()`.
- **F05 Image Input Pipeline**: Image retrieval via `get_image_by_sequence()` for `get_chunk_images()`.
- **F08 Global Place Recognition**: Descriptor computation via `compute_chunk_descriptor()` for `get_chunk_composite_descriptor()`.
## Data Models
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### ChunkBounds
```python
class ChunkBounds(BaseModel):
estimated_center: GPSPoint
estimated_radius: float # meters
confidence: float # 0.0 to 1.0
```
### ChunkConfig
```python
class ChunkConfig(BaseModel):
min_frames_for_matching: int = 5
max_frames_per_chunk: int = 20
descriptor_aggregation: str = "mean" # "mean", "vlad", "max"
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/13_coordinate_transformer/13.01_feature_enu_coordinate_management.md
-115
View File
@@ -1,115 +0,0 @@
# Feature: ENU Coordinate Management
## Description
Manages East-North-Up (ENU) coordinate system origins per flight and provides bidirectional conversions between GPS (WGS84) and ENU coordinates. ENU is a local Cartesian coordinate system used by the Factor Graph Optimizer for better numerical stability in optimization operations.
Each flight has its own ENU origin set from the flight's start_gps. All frames in a flight share the same origin for consistent coordinate conversions.
## Component APIs Implemented
### `set_enu_origin(flight_id: str, origin_gps: GPSPoint) -> None`
Sets the ENU origin for a specific flight during flight creation. Precomputes conversion factors for lat/lon to meters at origin latitude.
### `get_enu_origin(flight_id: str) -> GPSPoint`
Returns the stored ENU origin for a flight. Raises `OriginNotSetError` if called before set_enu_origin.
### `gps_to_enu(flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]`
Converts GPS coordinates to ENU (east, north, up) in meters relative to flight's origin.
Algorithm:
- delta_lat = gps.lat - origin.lat
- delta_lon = gps.lon - origin.lon
- east = delta_lon × cos(origin.lat) × 111319.5
- north = delta_lat × 111319.5
- up = 0
### `enu_to_gps(flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint`
Converts ENU coordinates back to GPS using the flight's ENU origin.
Algorithm:
- delta_lon = east / (cos(origin.lat) × 111319.5)
- delta_lat = north / 111319.5
- lat = origin.lat + delta_lat
- lon = origin.lon + delta_lon
## External Tools and Services
None - pure mathematical conversions.
## Internal Methods
### `_compute_meters_per_degree(latitude: float) -> Tuple[float, float]`
Precomputes conversion factors for a given latitude. Returns (meters_per_degree_lon, meters_per_degree_lat). Used during set_enu_origin for efficiency.
### `_get_origin_or_raise(flight_id: str) -> GPSPoint`
Helper to retrieve origin with error handling. Raises OriginNotSetError if not set.
## Unit Tests
### Test: set_enu_origin stores origin
- Call set_enu_origin with flight_id and origin GPS
- Verify origin is stored
- Verify conversion factors are precomputed
### Test: get_enu_origin returns stored origin
- Set origin via set_enu_origin
- Call get_enu_origin
- Verify returned GPS matches stored origin
### Test: get_enu_origin raises error if not set
- Call get_enu_origin for unknown flight_id
- Verify OriginNotSetError is raised
### Test: multiple flights have independent origins
- Set different origins for flight_1 and flight_2
- Verify each get_enu_origin returns correct independent origin
### Test: gps_to_enu at origin returns zero
- Set origin
- Convert origin GPS to ENU
- Verify result is (0, 0, 0)
### Test: gps_to_enu 1km east
- Set origin at known location
- Convert GPS point ~1km east of origin
- Verify east component ~1000m, north ~0
### Test: gps_to_enu 1km north
- Set origin at known location
- Convert GPS point ~1km north of origin
- Verify north component ~1000m, east ~0
### Test: gps_to_enu diagonal offset
- Convert GPS point with both lat/lon offset
- Verify correct east/north components
### Test: enu_to_gps at zero returns origin
- Set origin
- Convert (0, 0, 0) ENU to GPS
- Verify result matches origin GPS
### Test: round-trip conversion preserves coordinates
- Set origin
- Start with GPS point
- Convert to ENU via gps_to_enu
- Convert back via enu_to_gps
- Verify GPS matches original (within floating-point precision)
### Test: latitude affects east conversion factor
- Test gps_to_enu at different latitudes
- Verify east component accounts for cos(latitude) factor
## Integration Tests
### Test: ENU origin used by Factor Graph
- Create flight with start_gps via F02.1
- Verify F02.1 calls set_enu_origin with start_gps
- Verify Factor Graph can retrieve origin via get_enu_origin
### Test: ENU conversions consistent across components
- Set origin via set_enu_origin
- F10 Factor Graph uses gps_to_enu for absolute factors
- F10 uses enu_to_gps for trajectory output
- Verify GPS coordinates are consistent throughout pipeline
_docs/02_components/13_coordinate_transformer/13.02_feature_pixel_gps_projection.md
-169
View File
@@ -1,169 +0,0 @@
# Feature: Pixel-GPS Projection
## Description
Provides coordinate conversions between image pixel coordinates and GPS coordinates using camera model and pose information. This feature enables:
- Converting frame center pixels to GPS for trajectory output
- Converting detected object pixel locations to GPS coordinates (critical for external object detection integration)
- Inverse projection from GPS to pixels for visualization
- Generic point transformations via homography/affine matrices
All projections rely on the ground plane assumption and require frame pose from Factor Graph, camera parameters, and altitude.
## Component APIs Implemented
### `pixel_to_gps(flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint`
Converts pixel coordinates to GPS using camera pose and ground plane intersection.
Algorithm:
1. Unproject pixel to 3D ray using H01 Camera Model
2. Intersect ray with ground plane at altitude
3. Transform 3D point from camera frame to ENU using frame_pose
4. Convert ENU to WGS84 via enu_to_gps(flight_id, enu_point)
### `gps_to_pixel(flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]`
Inverse projection from GPS to image pixel coordinates.
Algorithm:
1. Convert GPS to ENU via gps_to_enu(flight_id, gps)
2. Transform ENU point to camera frame using frame_pose
3. Project 3D point to image plane using H01 Camera Model
4. Return pixel coordinates
### `image_object_to_gps(flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint`
**Critical method** - Converts object pixel coordinates to GPS for external object detection integration.
Processing:
1. Get frame_pose from F10.get_trajectory(flight_id)[frame_id]
2. Get camera_params from F17.get_flight_config(flight_id)
3. Get altitude from F17.get_flight_config(flight_id).altitude
4. Call pixel_to_gps(flight_id, object_pixel, frame_pose, camera_params, altitude)
5. Return GPS
### `transform_points(points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]`
Applies homography (3×3) or affine (2×3) transformation to list of points.
Algorithm:
1. Convert points to homogeneous coordinates
2. Apply transformation matrix
3. Normalize (for homography) and return
## External Tools and Services
- **H01 Camera Model**: `project()` for 3D-to-pixel, `unproject()` for pixel-to-ray
- **H02 GSD Calculator**: For GSD calculations in coordinate conversions
- **F10 Factor Graph Optimizer**: `get_trajectory(flight_id)` for frame poses
- **F17 Configuration Manager**: `get_flight_config(flight_id)` for camera params and altitude
## Internal Methods
### `_unproject_to_ray(pixel: Tuple[float, float], camera_params: CameraParameters) -> np.ndarray`
Uses H01 Camera Model to convert pixel to 3D ray direction in camera frame.
### `_intersect_ray_ground_plane(ray_origin: np.ndarray, ray_direction: np.ndarray, ground_altitude: float) -> np.ndarray`
Computes intersection point of ray with horizontal ground plane at given altitude.
### `_camera_to_enu(point_camera: np.ndarray, frame_pose: Pose) -> np.ndarray`
Transforms point from camera frame to ENU using pose rotation and translation.
### `_enu_to_camera(point_enu: np.ndarray, frame_pose: Pose) -> np.ndarray`
Inverse transformation from ENU to camera frame.
### `_to_homogeneous(points: List[Tuple[float, float]]) -> np.ndarray`
Converts 2D points to homogeneous coordinates (appends 1).
### `_from_homogeneous(points: np.ndarray) -> List[Tuple[float, float]]`
Converts from homogeneous back to 2D (divides by w).
## Unit Tests
### Test: pixel_to_gps at image center
- Create frame_pose at known ENU location
- Call pixel_to_gps with image center pixel
- Verify GPS matches expected frame center location
### Test: pixel_to_gps at image corner
- Call pixel_to_gps with corner pixel
- Verify GPS has appropriate offset from center
### Test: pixel_to_gps at different altitudes
- Same pixel, different altitudes
- Verify GPS changes appropriately with altitude (higher = wider footprint)
### Test: gps_to_pixel at frame center GPS
- Convert frame center GPS to pixel
- Verify pixel is at image center
### Test: gps_to_pixel out of view
- Convert GPS point outside camera field of view
- Verify pixel coordinates outside image bounds
### Test: pixel_to_gps/gps_to_pixel round trip
- Start with pixel
- Convert to GPS via pixel_to_gps
- Convert back via gps_to_pixel
- Verify pixel matches original (within tolerance)
### Test: image_object_to_gps at image center
- Call image_object_to_gps with center pixel
- Verify GPS matches frame center
### Test: image_object_to_gps with offset
- Call image_object_to_gps with off-center pixel
- Verify GPS has correct offset
### Test: image_object_to_gps multiple objects
- Convert multiple object pixels from same frame
- Verify each gets correct independent GPS
### Test: transform_points identity
- Apply identity matrix
- Verify points unchanged
### Test: transform_points rotation
- Apply 90° rotation matrix
- Verify points rotated correctly
### Test: transform_points translation
- Apply translation matrix
- Verify points translated correctly
### Test: transform_points homography
- Apply perspective homography
- Verify points transformed with perspective correction
### Test: transform_points affine
- Apply 2×3 affine matrix
- Verify correct transformation
## Integration Tests
### Test: Frame center GPS calculation
- Process frame through F02.2 Flight Processing Engine
- Get frame_pose from F10 Factor Graph
- Call pixel_to_gps with image center
- Verify GPS within 50m of expected
### Test: Object localization end-to-end
- External detector finds object at pixel (1500, 2000)
- Call image_object_to_gps(flight_id, frame_id, pixel)
- Verify GPS is correct
- Test with multiple objects
### Test: Object GPS updates after trajectory refinement
- Get object GPS before trajectory refinement
- Factor Graph refines trajectory with new absolute factors
- Get object GPS again
- Verify GPS updated to reflect refined trajectory
### Test: Round-trip GPS-pixel-GPS consistency
- Start with GPS point
- Convert to pixel via gps_to_pixel
- Convert back via pixel_to_gps
- Verify GPS matches original within tolerance
### Test: GSD integration via H02
- Call H02.compute_gsd() with known parameters
- Verify GSD matches expected value
- Test at different altitudes
_docs/02_components/13_coordinate_transformer/13._component_coordinate_transformer.md
-444
View File
@@ -1,444 +0,0 @@
# Coordinate Transformer
## Interface Definition
**Interface Name**: `ICoordinateTransformer`
### Interface Methods
```python
class ICoordinateTransformer(ABC):
# ENU Origin Management
@abstractmethod
def set_enu_origin(self, flight_id: str, origin_gps: GPSPoint) -> None:
pass
@abstractmethod
def get_enu_origin(self, flight_id: str) -> GPSPoint:
pass
@abstractmethod
def gps_to_enu(self, flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]:
pass
@abstractmethod
def enu_to_gps(self, flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint:
pass
# Pixel/GPS Conversions
@abstractmethod
def pixel_to_gps(self, flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint:
pass
@abstractmethod
def gps_to_pixel(self, flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]:
pass
@abstractmethod
def image_object_to_gps(self, flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint:
pass
@abstractmethod
def transform_points(self, points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]:
pass
```
## Component Description
### Responsibilities
- Pixel-to-GPS coordinate conversions
- GPS-to-pixel inverse projections
- **Critical**: Convert object pixel coordinates (from external detection system) to GPS
- Handle multiple coordinate systems: WGS84, Web Mercator, ENU, image pixels, rotated coordinates
- Camera model integration for projection operations
**Note**: GSD calculations are delegated to H02 GSD Calculator helper.
### Scope
- Coordinate system transformations
- Camera projection mathematics
- Integration with Factor Graph poses
- Object localization (pixel → GPS)
- Support for external object detection system
## API Methods
### ENU Origin Management
#### `set_enu_origin(flight_id: str, origin_gps: GPSPoint) -> None`
**Description**: Sets the ENU (East-North-Up) coordinate system origin for a specific flight. Called once during flight creation using the flight's start_gps.
**Why ENU Origin?**: Factor graph optimization works in Cartesian coordinates (meters) for better numerical stability. ENU converts GPS (degrees) to local meters relative to an origin. See `helpers/enu_origin_explanation.md` for details.
**Called By**:
- F02.1 Flight Lifecycle Manager (during create_flight)
**Input**:
```python
flight_id: str # Flight identifier
origin_gps: GPSPoint
lat: float # Origin latitude in WGS84 (flight start_gps)
lon: float # Origin longitude in WGS84 (flight start_gps)
```
**Processing Flow**:
1. Store origin_gps as ENU origin for flight_id
2. Precompute conversion factors for lat/lon to meters at origin latitude
3. All subsequent ENU operations for this flight use this origin
**Important**:
- Each flight has its own ENU origin (the flight's start_gps)
- Origin remains constant for the entire flight duration
- All frames in the flight use the same origin for ENU conversions
**Test Cases**:
1. **Set origin**: Store origin GPS
2. **Subsequent gps_to_enu calls**: Use set origin
3. **Multiple flights**: Each flight has independent origin
---
#### `get_enu_origin(flight_id: str) -> GPSPoint`
**Description**: Returns the ENU origin for a specific flight.
**Called By**:
- F10 Factor Graph Optimizer (for coordinate checks)
- F14 Result Manager (for reference)
**Input**:
```python
flight_id: str
```
**Output**:
```python
GPSPoint: The ENU origin (flight start_gps)
```
**Error Conditions**:
- Raises `OriginNotSetError` if called before set_enu_origin() for this flight_id
**Test Cases**:
1. **After set_enu_origin**: Returns stored origin
2. **Before set_enu_origin**: Raises error
---
#### `gps_to_enu(flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]`
**Description**: Converts GPS coordinates to ENU (East, North, Up) relative to the flight's ENU origin.
**Called By**:
- F10 Factor Graph Optimizer (for absolute factors)
- Internal (for pixel_to_gps)
**Input**:
```python
flight_id: str # Flight identifier
gps: GPSPoint
lat: float
lon: float
```
**Output**:
```python
Tuple[float, float, float]: (east, north, up) in meters relative to flight's origin
```
**Algorithm**:
1. Compute delta_lat = gps.lat - origin.lat
2. Compute delta_lon = gps.lon - origin.lon
3. east = delta_lon × cos(origin.lat) × 111319.5 # meters/degree at equator
4. north = delta_lat × 111319.5
5. up = 0 (or computed from elevation if available)
**Test Cases**:
1. **Origin GPS**: Returns (0, 0, 0)
2. **1km East**: Returns (~1000, 0, 0)
3. **1km North**: Returns (0, ~1000, 0)
4. **Diagonal**: Returns correct east/north components
---
#### `enu_to_gps(flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint`
**Description**: Converts ENU coordinates back to GPS using the flight's ENU origin.
**Called By**:
- F10 Factor Graph Optimizer (for get_trajectory)
- F14 Result Manager (for publishing GPS results)
**Input**:
```python
flight_id: str # Flight identifier
enu: Tuple[float, float, float] # (east, north, up) in meters relative to flight's origin
```
**Output**:
```python
GPSPoint: WGS84 coordinates
```
**Algorithm**:
1. delta_lon = east / (cos(origin.lat) × 111319.5)
2. delta_lat = north / 111319.5
3. lat = origin.lat + delta_lat
4. lon = origin.lon + delta_lon
**Test Cases**:
1. **Origin (0, 0, 0)**: Returns origin GPS
2. **Round-trip**: gps → enu → gps matches original (within precision)
---
### Pixel/GPS Conversions
### `pixel_to_gps(flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint`
**Description**: Converts pixel coordinates to GPS using camera pose and ground plane assumption.
**Called By**:
- F02.2 Flight Processing Engine (for frame center GPS)
- Internal (for image_object_to_gps)
**Input**:
```python
flight_id: str # Flight identifier (required for ENU origin lookup)
pixel: Tuple[float, float] # (x, y) in image coordinates
frame_pose: Pose # From Factor Graph (ENU coordinates)
camera_params: CameraParameters
altitude: float # Ground altitude
```
**Output**:
```python
GPSPoint:
lat: float
lon: float
```
**Algorithm**:
1. Unproject pixel to 3D ray using H01 Camera Model
2. Intersect ray with ground plane at altitude
3. Transform 3D point from camera frame to ENU using frame_pose
4. Convert ENU to WGS84 GPS using enu_to_gps(flight_id, enu_point)
**Assumptions**:
- Ground plane assumption (terrain height negligible)
- Downward-pointing camera
- Known altitude
**Error Conditions**:
- None (always returns GPS, may be inaccurate if assumptions violated)
**Test Cases**:
1. **Image center**: Returns frame center GPS
2. **Image corner**: Returns GPS at corner
3. **Object pixel**: Returns object GPS
4. **Altitude variation**: Correct GPS at different altitudes
---
### `gps_to_pixel(flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]`
**Description**: Inverse projection from GPS to image pixel coordinates.
**Called By**:
- Visualization tools (overlay GPS annotations)
- Testing/validation
**Input**:
```python
flight_id: str # Flight identifier (required for ENU origin lookup)
gps: GPSPoint
frame_pose: Pose
camera_params: CameraParameters
altitude: float
```
**Output**:
```python
Tuple[float, float]: (x, y) pixel coordinates
```
**Algorithm**:
1. Convert GPS to ENU using gps_to_enu(flight_id, gps)
2. Transform ENU point to camera frame using frame_pose
3. Project 3D point to image plane using H01 Camera Model
4. Return pixel coordinates
**Test Cases**:
1. **Frame center GPS**: Returns image center pixel
2. **Out of view GPS**: Returns pixel outside image bounds
---
### `image_object_to_gps(flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint`
**Description**: **Critical method** - Converts object pixel coordinates to GPS. Used for external object detection integration.
**Called By**:
- F02 Flight Processor (via convert_object_to_gps delegation from F01)
- F14 Result Manager (converts objects to GPS for output)
**Input**:
```python
flight_id: str # Flight identifier (needed for ENU origin and factor graph)
frame_id: int # Frame containing object
object_pixel: Tuple[float, float] # Pixel coordinates from object detector
```
**Output**:
```python
GPSPoint: GPS coordinates of object center
```
**Processing Flow**:
1. Get frame_pose from F10 Factor Graph Optimizer.get_trajectory(flight_id)[frame_id]
2. Get camera_params from F17 Configuration Manager.get_flight_config(flight_id)
3. Get altitude from F17 Configuration Manager.get_flight_config(flight_id).altitude
4. Call pixel_to_gps(object_pixel, frame_pose, camera_params, altitude)
5. Use enu_to_gps(flight_id, enu_point) for final GPS conversion
6. Return GPS
**User Story**:
- External system detects object in UAV image at pixel (1024, 768)
- Calls F02.convert_object_to_gps(flight_id="abc", frame_id=237, pixel=(1024, 768))
- F02 delegates to F13.image_object_to_gps(flight_id="abc", frame_id=237, object_pixel=(1024, 768))
- Returns GPSPoint(lat=48.123, lon=37.456)
- Object GPS can be used for navigation, targeting, etc.
**Test Cases**:
1. **Object at image center**: Returns frame center GPS
2. **Object at corner**: Returns GPS with offset
3. **Multiple objects**: Each gets correct GPS
4. **Refined trajectory**: Object GPS updates after refinement
---
### `transform_points(points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]`
**Description**: Applies homography or affine transformation to list of points.
**Called By**:
- F06 Image Rotation Manager (for rotation transforms)
- F09 Metric Refinement (homography application)
**Input**:
```python
points: List[Tuple[float, float]]
transformation: np.ndarray # 3×3 homography or 2×3 affine
```
**Output**:
```python
List[Tuple[float, float]]: Transformed points
```
**Processing Flow**:
1. Convert points to homogeneous coordinates
2. Apply transformation matrix
3. Normalize and return
**Test Cases**:
1. **Identity transform**: Points unchanged
2. **Rotation**: Points rotated correctly
3. **Homography**: Points transformed with perspective
## Integration Tests
### Test 1: Frame Center GPS Calculation
1. Get frame_pose from Factor Graph
2. pixel_to_gps(image_center) → GPS
3. Verify GPS matches expected location
4. Verify accuracy < 50m
### Test 2: Object Localization
1. External detector finds object at pixel (1500, 2000)
2. image_object_to_gps(frame_id, pixel) → GPS
3. Verify GPS correct
4. Multiple objects → all get correct GPS
### Test 3: Round-Trip Conversion
1. Start with GPS point
2. gps_to_pixel() → pixel
3. pixel_to_gps() → GPS
4. Verify GPS matches original (within tolerance)
### Test 4: GSD via H02
1. Call H02.compute_gsd() with known parameters
2. Verify matches expected value
3. Test at different altitudes
## Non-Functional Requirements
### Performance
- **pixel_to_gps**: < 5ms
- **gps_to_pixel**: < 5ms
- **image_object_to_gps**: < 10ms
### Accuracy
- **GPS accuracy**: Inherits from Factor Graph accuracy (~20m)
- **GSD calculation**: ±1% precision
- **Projection accuracy**: < 1 pixel error
### Reliability
- Handle edge cases (points outside image)
- Graceful handling of degenerate configurations
- Numerical stability
## Dependencies
### Internal Components
- **F10 Factor Graph Optimizer**: For frame poses via `get_trajectory(flight_id)` - required for `pixel_to_gps()` and `image_object_to_gps()` to get frame pose estimates.
- **F17 Configuration Manager**: For camera parameters via `get_flight_config(flight_id)`.
- **H01 Camera Model**: For projection/unprojection operations (`project()`, `unproject()`).
- **H02 GSD Calculator**: For GSD calculations in coordinate conversions.
**Note**: F13 uses internal `enu_to_gps()` and `gps_to_enu()` methods that rely on the ENU origin set via `set_enu_origin(flight_id, origin_gps)`. H06 Web Mercator Utils is NOT used for ENU conversions - ENU is a local Cartesian coordinate system.
### External Dependencies
- **numpy**: Matrix operations
- **opencv-python**: Homography operations (optional)
## Data Models
### Pose (from Factor Graph)
```python
class Pose(BaseModel):
position: np.ndarray # (3,) - [x, y, z] in ENU
orientation: np.ndarray # (4,) quaternion or (3,3) rotation
timestamp: datetime
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int # pixels
resolution_height: int # pixels
principal_point: Tuple[float, float] # (cx, cy)
distortion_coefficients: Optional[List[float]]
```
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float
lon: float
```
### CoordinateFrame
```python
class CoordinateFrame(Enum):
WGS84 = "wgs84" # GPS coordinates
ENU = "enu" # East-North-Up local frame
ECEF = "ecef" # Earth-Centered Earth-Fixed
IMAGE = "image" # Image pixel coordinates
CAMERA = "camera" # Camera frame
```
_docs/02_components/14_result_manager/14.01_feature_frame_result_persistence.md
-46
View File
@@ -1,46 +0,0 @@
# Feature: Frame Result Persistence
## Description
Handles atomic persistence and real-time publishing of individual frame processing results. Ensures consistency between `frame_results` and `waypoints` tables through transactional updates, and triggers SSE events for live client updates.
## Component APIs Implemented
### `update_frame_result(flight_id: str, frame_id: int, result: FrameResult) -> bool`
Persists frame result atomically (frame_results + waypoints tables) and publishes via SSE.
### `publish_waypoint_update(flight_id: str, frame_id: int) -> bool`
Internal method to trigger waypoint visualization update after persistence.
## External Services Used
- **F03 Flight Database**: `execute_transaction()` for atomic multi-table updates
- **F15 SSE Event Streamer**: `send_frame_result()` for real-time client notification
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_frame_transaction(flight_id, frame_id, result)` | Constructs DB transaction with frame_results INSERT/UPDATE and waypoints UPDATE |
| `_update_flight_statistics(flight_id)` | Updates flight-level statistics after frame persistence |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_update_frame_result_new_frame` | New frame result stored in frame_results table |
| `test_update_frame_result_updates_waypoint` | Waypoint table updated with latest position |
| `test_update_frame_result_transaction_atomic` | Both tables updated or neither (rollback on failure) |
| `test_update_frame_result_triggers_sse` | F15.send_frame_result() called on success |
| `test_update_frame_result_refined_flag` | Existing result updated with refined=True |
| `test_publish_waypoint_fetches_latest` | Fetches current data before publishing |
| `test_publish_waypoint_handles_transient_error` | Retries on transient F15 errors |
| `test_publish_waypoint_logs_on_db_unavailable` | Logs error and continues on DB failure |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_per_frame_processing_e2e` | Process frame → stored in F03 → SSE event sent via F15 |
| `test_statistics_updated_after_frame` | Flight statistics reflect new frame count and confidence |
_docs/02_components/14_result_manager/14.02_feature_result_retrieval.md
-44
View File
@@ -1,44 +0,0 @@
# Feature: Result Retrieval
## Description
Provides read access to frame results for REST API endpoints and SSE reconnection replay. Supports full flight result retrieval with statistics and incremental change detection for efficient reconnection.
## Component APIs Implemented
### `get_flight_results(flight_id: str) -> FlightResults`
Retrieves all frame results and computed statistics for a flight.
### `get_changed_frames(flight_id: str, since: datetime) -> List[int]`
Returns frame IDs modified since a given timestamp for incremental updates.
## External Services Used
- **F03 Flight Database**: Query frame_results and waypoints tables
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_compute_flight_statistics(frames)` | Calculates total_frames, processed_frames, refined_frames, mean_confidence |
| `_query_frames_by_timestamp(flight_id, since)` | Queries frame_results with updated_at > since filter |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_get_flight_results_returns_all_frames` | All frames for flight_id returned |
| `test_get_flight_results_includes_statistics` | FlightStatistics computed correctly |
| `test_get_flight_results_empty_flight` | Returns empty frames list with zero statistics |
| `test_get_flight_results_performance` | < 200ms for 2000 frames |
| `test_get_changed_frames_returns_modified` | Only frames with updated_at > since returned |
| `test_get_changed_frames_empty_result` | Returns empty list when no changes |
| `test_get_changed_frames_includes_refined` | Refined frames included in changed set |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_incremental_update_after_disconnect` | Client reconnects → get_changed_frames returns frames since disconnect |
| `test_get_results_matches_stored_data` | Retrieved results match what was persisted via update_frame_result |
_docs/02_components/14_result_manager/14.03_feature_batch_refinement_updates.md
-48
View File
@@ -1,48 +0,0 @@
# Feature: Batch Refinement Updates
## Description
Handles batch updates for frames that have been retrospectively improved through factor graph optimization or chunk merging. Receives GPS-converted coordinates from F02.2 (which handles ENU→GPS conversion) and updates both persistence and SSE clients.
## Component APIs Implemented
### `mark_refined(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
Updates results for frames refined by factor graph optimization (loop closure, etc.).
### `update_results_after_chunk_merge(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
Updates results for frames affected by chunk merge into main trajectory.
## External Services Used
- **F03 Flight Database**: Batch update frame_results and waypoints within transaction
- **F15 SSE Event Streamer**: `send_refinement()` for each updated frame
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_batch_refinement_transaction(flight_id, refined_results)` | Constructs batch transaction for multiple frame updates |
| `_publish_refinement_events(flight_id, frame_ids)` | Sends SSE refinement events for all updated frames |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_mark_refined_updates_all_frames` | All frames in refined_results updated |
| `test_mark_refined_sets_refined_flag` | refined=True set on all updated frames |
| `test_mark_refined_updates_gps_coordinates` | GPS coordinates match RefinedFrameResult values |
| `test_mark_refined_triggers_sse_per_frame` | F15.send_refinement() called for each frame |
| `test_mark_refined_updates_waypoints` | Waypoint table updated with new positions |
| `test_chunk_merge_updates_all_frames` | All merged frames updated |
| `test_chunk_merge_sets_refined_flag` | refined=True for merged frames |
| `test_chunk_merge_triggers_sse` | SSE events sent for merged frames |
| `test_batch_transaction_atomic` | All updates succeed or all rollback |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_batch_refinement_e2e` | 100 frames processed → 40 refined → all updated in F03 → SSE events sent |
| `test_chunk_merge_e2e` | Chunk merged → affected frames updated → clients notified |
| `test_refined_frames_in_get_results` | get_flight_results returns refined=True for updated frames |
_docs/02_components/14_result_manager/14._component_result_manager.md
-342
View File
@@ -1,342 +0,0 @@
# Result Manager
## Interface Definition
**Interface Name**: `IResultManager`
### Interface Methods
```python
class IResultManager(ABC):
@abstractmethod
def update_frame_result(self, flight_id: str, frame_id: int, result: FrameResult) -> bool:
"""
Atomic update:
1. Saves result to frame_results table.
2. Updates waypoint in waypoints table.
3. All within a single transaction via F03.
"""
pass
@abstractmethod
def publish_waypoint_update(self, flight_id: str, frame_id: int) -> bool:
pass
@abstractmethod
def get_flight_results(self, flight_id: str) -> FlightResults:
pass
@abstractmethod
def mark_refined(self, flight_id: str, refined_results: List[RefinedFrameResult]) -> bool:
"""
Updates results for frames that have been retrospectively improved.
Args:
flight_id: Flight identifier
refined_results: List of RefinedFrameResult containing frame_id and GPS-converted coordinates
(caller F02.2 converts ENU to GPS before calling this method)
"""
pass
@abstractmethod
def get_changed_frames(self, flight_id: str, since: datetime) -> List[int]:
pass
@abstractmethod
def update_results_after_chunk_merge(self, flight_id: str, refined_results: List[RefinedFrameResult]) -> bool:
"""
Updates results for frames affected by chunk merge.
Args:
flight_id: Flight identifier
refined_results: List of RefinedFrameResult with GPS-converted coordinates
(caller F02.2 converts ENU to GPS before calling this method)
"""
pass
```
## Component Description
### Responsibilities
- Result consistency and publishing.
- **Atomic Updates**: Ensures consistency between normalized `waypoints` and denormalized `frame_results` via transaction requests to F03.
### Scope
- Result state management
- Flight Database integration (waypoint storage)
- SSE event triggering
- Incremental update detection
- Result persistence
## API Methods
### `update_frame_result(flight_id: str, frame_id: int, result: FrameResult) -> bool`
**Description**: Persists and publishes the result of a processed frame.
**Called By**:
- Main processing loop (after each frame)
- F10 Factor Graph (after refinement)
**Input**:
```python
flight_id: str
frame_id: int
result: FrameResult:
gps_center: GPSPoint
altitude: float
heading: float
confidence: float
timestamp: datetime
refined: bool
objects: List[ObjectLocation] # From external detector
```
**Output**: `bool` - True if updated
**Processing Flow**:
1. Construct DB transaction:
- Insert/Update `frame_results`.
- Update `waypoints` (latest position).
2. Call `F03.execute_transaction()`.
3. If success: call `F15.send_frame_result()`.
4. Update flight statistics.
**Test Cases**:
1. New frame result → stored and published
2. Refined result → updates existing, marks refined=True
---
### `publish_waypoint_update(flight_id: str, frame_id: int) -> bool`
**Description**: Specifically triggers an update for the waypoint visualization.
**Called By**:
- Internal (after update_frame_result)
**Input**:
```python
flight_id: str
frame_id: int
```
**Output**: `bool` - True if updated successfully
**Processing Flow**:
1. Fetch latest data.
2. Call `F15.send_frame_result()` (or a specific lightweight event).
3. Handle errors (retry if transient)
**Test Cases**:
1. Successful update → Waypoint stored in database
2. Database unavailable → logs error, continues
---
### `get_flight_results(flight_id: str) -> FlightResults`
**Description**: Retrieves all results for a flight.
**Called By**:
- F01 REST API (results endpoint)
- Testing/validation
**Input**: `flight_id: str`
**Output**:
```python
FlightResults:
flight_id: str
frames: List[FrameResult]
statistics: FlightStatistics
```
**Test Cases**:
1. Get all results → returns complete trajectory
---
### `mark_refined(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
**Description**: Updates results for frames that have been retrospectively improved (e.g., after loop closure or chunk merge).
**Called By**:
- F02.2 Flight Processing Engine (after factor graph optimization)
**Input**:
```python
flight_id: str
refined_results: List[RefinedFrameResult] # GPS-converted results from F02.2
```
**Output**: `bool`
**Important**: F14 does NOT call F10 or F13. The caller (F02.2) performs the following steps before calling mark_refined():
1. F02.2 gets refined poses from F10.get_trajectory(flight_id) (ENU coordinates)
2. F02.2 converts ENU to GPS via F13.enu_to_gps(flight_id, enu_tuple)
3. F02.2 constructs RefinedFrameResult objects with GPS coordinates
4. F02.2 calls F14.mark_refined() with the GPS-converted results
**Processing Flow**:
1. For each refined_result in refined_results:
- Extract frame_id, gps_center, confidence from RefinedFrameResult
- Update result with refined=True via F03 Flight Database (part of transaction)
- Update waypoint via F03 Flight Database (part of transaction)
- Call F15 SSE Event Streamer.send_refinement()
**Test Cases**:
1. Batch refinement → all frames updated and published
2. GPS coordinates match converted values
---
### `get_changed_frames(flight_id: str, since: datetime) -> List[int]`
**Description**: Gets frames changed since timestamp (for incremental updates).
**Called By**:
- F15 SSE Event Streamer (for reconnection replay)
**Input**:
```python
flight_id: str
since: datetime
```
**Output**: `List[int]` - Frame IDs changed since timestamp
**Test Cases**:
1. Get changes → returns only modified frames
2. No changes → returns empty list
---
### `update_results_after_chunk_merge(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
**Description**: Updates results for frames affected by chunk merging into main trajectory.
**Called By**:
- F02.2 Flight Processing Engine (after chunk merge completes)
**Input**:
```python
flight_id: str
refined_results: List[RefinedFrameResult] # GPS-converted results for merged frames
```
**Output**: `bool` - True if updated successfully
**Important**: F14 does NOT call F10, F13, or F11. F02.2 coordinates the chunk merge workflow:
1. F11 returns merge result to F02.2 (direct call return, not event)
2. F02.2 gets updated poses from F10.get_trajectory(flight_id) (ENU coordinates)
3. F02.2 converts ENU to GPS via F13.enu_to_gps(flight_id, enu_tuple)
4. F02.2 constructs RefinedFrameResult objects
5. F02.2 calls F14.update_results_after_chunk_merge() with GPS-converted results
**Processing Flow**:
1. For each refined_result in refined_results:
- Extract frame_id, gps_center, confidence from RefinedFrameResult
- Update result with refined=True via F03 Flight Database (part of transaction)
- Update waypoint via F03 Flight Database (part of transaction)
- Call F15 SSE Event Streamer.send_refinement()
**Test Cases**:
1. **Chunk merge updates**: All merged frames updated and published
2. **GPS accuracy**: Updated GPS matches optimized poses
## Integration Tests
### Test 1: Per-Frame Processing
1. Process frame 237
2. update_frame_result() → stores result
3. Verify publish_waypoint_update() called (F03.update_waypoint())
4. Verify F15 SSE event sent
### Test 2: Batch Refinement
1. Process 100 frames
2. Factor graph refines frames 10-50
3. mark_refined([10-50]) → updates all
4. Verify Flight Database updated (F03.batch_update_waypoints())
5. Verify SSE refinement events sent
### Test 3: Incremental Updates
1. Process frames 1-100
2. Client disconnects at frame 50
3. Client reconnects
4. get_changed_frames(since=frame_50_time)
5. Client receives frames 51-100
## Non-Functional Requirements
### Performance
- **update_frame_result**: < 50ms
- **publish_waypoint_update**: < 100ms (non-blocking)
- **get_flight_results**: < 200ms for 2000 frames
### Reliability
- Result persistence survives crashes
- Guaranteed at-least-once delivery to Flight Database
- Idempotent updates
## Dependencies
### Internal Components
- **F03 Flight Database**: Must support transactional updates.
- **F15 SSE Event Streamer**: For real-time result streaming.
**Note**: F14 does NOT depend on F10, F13, or F11. The caller (F02.2) coordinates with those components and provides GPS-converted results to F14. This ensures unidirectional data flow and eliminates circular dependencies.
### External Dependencies
- None
## Data Models
### FrameResult
```python
class ObjectLocation(BaseModel):
object_id: str
pixel: Tuple[float, float]
gps: GPSPoint
class_name: str
confidence: float
class FrameResult(BaseModel):
frame_id: int
gps_center: GPSPoint
altitude: float
heading: float
confidence: float
timestamp: datetime
refined: bool
objects: List[ObjectLocation]
updated_at: datetime
```
### RefinedFrameResult
```python
class RefinedFrameResult(BaseModel):
"""
GPS-converted frame result provided by F02.2 after coordinate transformation.
F02.2 converts ENU poses from F10 to GPS using F13 before passing to F14.
"""
frame_id: int
gps_center: GPSPoint # GPS coordinates (converted from ENU by F02.2)
confidence: float
heading: Optional[float] = None # Updated heading if available
```
### FlightResults
```python
class FlightStatistics(BaseModel):
total_frames: int
processed_frames: int
refined_frames: int
mean_confidence: float
processing_time: float
class FlightResults(BaseModel):
flight_id: str
frames: List[FrameResult]
statistics: FlightStatistics
```
_docs/02_components/15_sse_event_streamer/15.01_feature_connection_lifecycle_management.md
-61
View File
@@ -1,61 +0,0 @@
# Feature: Connection Lifecycle Management
## Description
Manages the lifecycle of SSE (Server-Sent Events) connections including creation, tracking, health monitoring, and graceful closure. Supports multiple concurrent client connections per flight with proper resource management.
## Component APIs Implemented
### `create_stream(flight_id: str, client_id: str) -> StreamConnection`
Establishes a new SSE connection for a client subscribing to a flight's events.
### `close_stream(flight_id: str, client_id: str) -> bool`
Closes an existing SSE connection and cleans up associated resources.
### `get_active_connections(flight_id: str) -> int`
Returns the count of active SSE connections for a given flight.
## External Tools and Services
- **FastAPI/Starlette**: SSE response streaming support
- **asyncio**: Asynchronous connection handling and concurrent stream management
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_register_connection(flight_id, client_id, stream)` | Adds connection to internal registry |
| `_unregister_connection(flight_id, client_id)` | Removes connection from registry |
| `_get_connections_for_flight(flight_id)` | Returns all active streams for a flight |
| `_generate_stream_id()` | Creates unique identifier for stream |
| `_handle_client_disconnect(flight_id, client_id)` | Cleanup on unexpected disconnect |
## Data Structures
```python
# Connection registry: Dict[flight_id, Dict[client_id, StreamConnection]]
# Allows O(1) lookup by flight and client
```
## Unit Tests
| Test | Description |
|------|-------------|
| `test_create_stream_returns_valid_connection` | Verify StreamConnection has all required fields |
| `test_create_stream_unique_stream_ids` | Multiple streams get unique identifiers |
| `test_close_stream_removes_from_registry` | Connection no longer tracked after close |
| `test_close_stream_nonexistent_returns_false` | Graceful handling of invalid close |
| `test_get_active_connections_empty` | Returns 0 when no connections exist |
| `test_get_active_connections_multiple_clients` | Correctly counts concurrent connections |
| `test_multiple_flights_isolated` | Connections for different flights don't interfere |
| `test_same_client_reconnect` | Client can reconnect with same client_id |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_concurrent_client_connections` | 10 clients connect simultaneously to same flight |
| `test_connection_survives_idle_period` | Connection remains open during inactivity |
| `test_client_disconnect_detection` | System detects when client drops connection |
| `test_connection_cleanup_on_flight_completion` | All connections closed when flight ends |
_docs/02_components/15_sse_event_streamer/15.02_feature_event_broadcasting.md
-83
View File
@@ -1,83 +0,0 @@
# Feature: Event Broadcasting
## Description
Broadcasts various event types to all connected clients for a flight. Handles event formatting per SSE protocol, buffering events for temporarily disconnected clients, and replay of missed events on reconnection. Includes heartbeat mechanism for connection health.
## Component APIs Implemented
### `send_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
Broadcasts `frame_processed` event with GPS coordinates, confidence, and metadata.
### `send_search_progress(flight_id: str, search_status: SearchStatus) -> bool`
Broadcasts `search_expanded` event during failure recovery search operations.
### `send_user_input_request(flight_id: str, request: UserInputRequest) -> bool`
Broadcasts `user_input_needed` event when processing requires user intervention.
### `send_refinement(flight_id: str, frame_id: int, updated_result: FrameResult) -> bool`
Broadcasts `frame_refined` event when factor graph optimization improves a position.
### `send_heartbeat(flight_id: str) -> bool`
Sends keepalive ping to all clients to maintain connection health.
## External Tools and Services
- **FastAPI/Starlette**: SSE response streaming
- **asyncio**: Async event delivery to multiple clients
- **JSON**: Event data serialization
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_format_sse_event(event_type, event_id, data)` | Formats data per SSE protocol spec |
| `_broadcast_to_flight(flight_id, event)` | Sends event to all clients of a flight |
| `_buffer_event(flight_id, event)` | Stores event for replay to reconnecting clients |
| `_get_buffered_events(flight_id, last_event_id)` | Retrieves events after given ID for replay |
| `_prune_event_buffer(flight_id)` | Removes old events beyond retention window |
| `_generate_event_id(frame_id, event_type)` | Creates sequential event ID for ordering |
## Event Types
| Event | Format |
|-------|--------|
| `frame_processed` | `{frame_id, gps, altitude, confidence, heading, timestamp}` |
| `frame_refined` | `{frame_id, gps, refined: true}` |
| `search_expanded` | `{frame_id, grid_size, status}` |
| `user_input_needed` | `{request_id, frame_id, candidate_tiles}` |
| `:heartbeat` | SSE comment (no data payload) |
## Buffering Strategy
- Events buffered per flight with configurable retention (default: 1000 events or 5 minutes)
- Events keyed by sequential ID for replay ordering
- On reconnection with `last_event_id`, replay all events after that ID
## Unit Tests
| Test | Description |
|------|-------------|
| `test_send_frame_result_formats_correctly` | Event matches expected JSON structure |
| `test_send_refinement_includes_refined_flag` | Refined events marked appropriately |
| `test_send_search_progress_valid_status` | Status field correctly populated |
| `test_send_user_input_request_has_request_id` | Request includes unique identifier |
| `test_send_heartbeat_sse_comment_format` | Heartbeat uses `:heartbeat` comment format |
| `test_buffer_event_stores_in_order` | Events retrievable in sequence |
| `test_buffer_pruning_removes_old_events` | Buffer doesn't grow unbounded |
| `test_replay_from_last_event_id` | Correct subset returned for replay |
| `test_broadcast_returns_false_no_connections` | Graceful handling when no clients |
| `test_event_id_generation_sequential` | IDs increase monotonically |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_100_frames_all_received_in_order` | Stream 100 frame results, verify completeness |
| `test_reconnection_replay` | Disconnect after 50 events, reconnect, receive 51-100 |
| `test_multiple_event_types_interleaved` | Mix of frame_processed and frame_refined events |
| `test_user_input_flow_roundtrip` | Send request, verify client receives |
| `test_heartbeat_every_30_seconds` | Verify keepalive timing |
| `test_event_latency_under_500ms` | Measure generation-to-receipt time |
| `test_high_throughput_100_events_per_second` | Sustained event rate handling |
_docs/02_components/15_sse_event_streamer/15._component_sse_event_streamer.md
-291
View File
@@ -1,291 +0,0 @@
# SSE Event Streamer
## Interface Definition
**Interface Name**: `ISSEEventStreamer`
### Interface Methods
```python
class ISSEEventStreamer(ABC):
@abstractmethod
def create_stream(self, flight_id: str, client_id: str) -> StreamConnection:
pass
@abstractmethod
def send_frame_result(self, flight_id: str, frame_result: FrameResult) -> bool:
pass
@abstractmethod
def send_search_progress(self, flight_id: str, search_status: SearchStatus) -> bool:
pass
@abstractmethod
def send_user_input_request(self, flight_id: str, request: UserInputRequest) -> bool:
pass
@abstractmethod
def send_refinement(self, flight_id: str, frame_id: int, updated_result: FrameResult) -> bool:
pass
@abstractmethod
def send_heartbeat(self, flight_id: str) -> bool:
pass
@abstractmethod
def close_stream(self, flight_id: str, client_id: str) -> bool:
pass
@abstractmethod
def get_active_connections(self, flight_id: str) -> int:
pass
```
## Component Description
### Responsibilities
- Real-time communication with clients.
- Buffering events for disconnected clients.
### Callers
- **F02.1 Flight Lifecycle Manager**: Calls `create_stream` (delegated from F01).
- **F02.2 Flight Processing Engine**: Calls `send_user_input_request`, `send_search_progress`.
- **F14 Result Manager**: Calls `send_frame_result`, `send_refinement`.
### Consistency Fix
- Previously listed F11 as caller. **Correction**: F11 returns request objects to F02.2. **F02.2 is the sole caller** for user input requests and search status updates. F11 has no dependencies on F15.
### Scope
- SSE protocol implementation
- Event formatting and sending
- Connection management
- Client reconnection handling
- Multiple concurrent streams per flight
## API Methods
### `create_stream(flight_id: str, client_id: str) -> StreamConnection`
**Description**: Establishes a server-sent events connection.
**Called By**: F01 REST API (GET /stream endpoint)
**Output**:
```python
StreamConnection:
stream_id: str
flight_id: str
client_id: str
last_event_id: Optional[str]
```
**Event Types**:
- `frame_processed`
- `frame_refined`
- `search_expanded`
- `user_input_needed`
- `processing_blocked`
- `route_api_updated`
- `route_completed`
**Test Cases**:
1. Create stream → client receives keepalive pings
2. Multiple clients → each gets own stream
---
### `send_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
**Description**: Sends frame_processed event.
**Called By**: F14 Result Manager
**Event Format**:
```json
{
"event": "frame_processed",
"id": "frame_237",
"data": {
"frame_id": 237,
"gps": {"lat": 48.123, "lon": 37.456},
"altitude": 800.0,
"confidence": 0.95,
"heading": 87.3,
"timestamp": "2025-11-24T10:30:00Z"
}
}
```
**Test Cases**:
1. Send event → all clients receive
2. Client disconnected → event buffered for replay
---
### `send_search_progress(flight_id: str, search_status: SearchStatus) -> bool`
**Description**: Sends search_expanded event.
**Called By**: F02.2 Flight Processing Engine (via F11 status return)
**Event Format**:
```json
{
"event": "search_expanded",
"data": {
"frame_id": 237,
"grid_size": 9,
"status": "searching"
}
}
```
---
### `send_user_input_request(flight_id: str, request: UserInputRequest) -> bool`
**Description**: Sends user_input_needed event.
**Called By**: F02.2 Flight Processing Engine (via F11 request object)
**Event Format**:
```json
{
"event": "user_input_needed",
"data": {
"request_id": "uuid",
"frame_id": 237,
"candidate_tiles": [...]
}
}
```
---
### `send_refinement(flight_id: str, frame_id: int, updated_result: FrameResult) -> bool`
**Description**: Sends frame_refined event.
**Called By**: F14 Result Manager
**Event Format**:
```json
{
"event": "frame_refined",
"data": {
"frame_id": 237,
"gps": {"lat": 48.1235, "lon": 37.4562},
"refined": true
}
}
```
---
### `send_heartbeat(flight_id: str) -> bool`
**Description**: Sends heartbeat/keepalive to all clients subscribed to flight.
**Called By**:
- Background heartbeat task (every 30 seconds)
- F02.2 Flight Processing Engine (periodically during processing)
**Event Format**:
```
:heartbeat
```
**Behavior**:
- Sends SSE comment (`:heartbeat`) which doesn't trigger event handlers
- Keeps TCP connection alive
- Client can use to detect connection health
**Test Cases**:
1. Send heartbeat → all clients receive ping
2. Client timeout → connection marked stale
---
### `close_stream(flight_id: str, client_id: str) -> bool`
**Description**: Closes SSE connection.
**Called By**: F01 REST API (on client disconnect)
---
### `get_active_connections(flight_id: str) -> int`
**Description**: Returns count of active SSE connections for a flight.
**Called By**:
- F02.2 Flight Processing Engine (monitoring)
- Admin tools
**Test Cases**:
1. No connections → returns 0
2. 5 clients connected → returns 5
## Integration Tests
### Test 1: Real-Time Streaming
1. Client connects
2. Process 100 frames
3. Client receives 100 frame_processed events
4. Verify order and completeness
### Test 2: Reconnection with Replay
1. Client connects
2. Process 50 frames
3. Client disconnects
4. Process 50 more frames
5. Client reconnects with last_event_id
6. Client receives frames 51-100
### Test 3: User Input Flow
1. Processing blocks
2. send_user_input_request()
3. Client receives event
4. Client responds with fix
5. Processing resumes
## Non-Functional Requirements
### Performance
- **Event latency**: < 500ms from generation to client
- **Throughput**: 100 events/second
- **Concurrent connections**: 1000+ clients
### Reliability
- Event buffering for disconnected clients
- Automatic reconnection support
- Keepalive pings every 30 seconds
## Dependencies
### Internal Components
- None (receives calls from other components)
### External Dependencies
- **FastAPI** or **Flask** SSE support
- **asyncio**: For async event streaming
## Data Models
### StreamConnection
```python
class StreamConnection(BaseModel):
stream_id: str
flight_id: str
client_id: str
created_at: datetime
last_event_id: Optional[str]
```
### SSEEvent
```python
class SSEEvent(BaseModel):
event: str
id: Optional[str]
data: Dict[str, Any]
```
_docs/02_components/16_model_manager/16.01_feature_model_lifecycle_management.md
-63
View File
@@ -1,63 +0,0 @@
# Feature: Model Lifecycle Management
## Description
Manages the complete lifecycle of ML models including loading, caching, warmup, and unloading. Handles all four models (SuperPoint, LightGlue, DINOv2, LiteSAM) with support for multiple formats (TensorRT, ONNX, PyTorch).
## Component APIs Implemented
### `load_model(model_name: str, model_format: str) -> bool`
- Loads model in specified format
- Checks if model already loaded (cache hit)
- Initializes inference engine
- Triggers warmup
- Caches for reuse
### `warmup_model(model_name: str) -> bool`
- Warms up model with dummy input
- Initializes CUDA kernels
- Pre-allocates GPU memory
- Ensures first real inference is fast
## External Tools and Services
- **TensorRT**: Loading TensorRT engine files
- **ONNX Runtime**: Loading ONNX models
- **PyTorch**: Loading PyTorch model weights (optional)
- **CUDA**: GPU memory allocation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_check_model_cache(model_name)` | Check if model already loaded |
| `_load_tensorrt_engine(path)` | Load TensorRT engine from file |
| `_load_onnx_model(path)` | Load ONNX model from file |
| `_allocate_gpu_memory(model)` | Allocate GPU memory for model |
| `_create_dummy_input(model_name)` | Create appropriate dummy input for warmup |
| `_cache_model(model_name, engine)` | Store loaded model in cache |
## Unit Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| UT-16.01-01 | Load TensorRT model | Model loaded, returns True |
| UT-16.01-02 | Load ONNX model | Model loaded, returns True |
| UT-16.01-03 | Load already cached model | Returns True immediately (no reload) |
| UT-16.01-04 | Load invalid model name | Returns False, logs error |
| UT-16.01-05 | Load invalid model path | Returns False, logs error |
| UT-16.01-06 | Warmup SuperPoint | CUDA kernels initialized |
| UT-16.01-07 | Warmup LightGlue | CUDA kernels initialized |
| UT-16.01-08 | Warmup DINOv2 | CUDA kernels initialized |
| UT-16.01-09 | Warmup LiteSAM | CUDA kernels initialized |
| UT-16.01-10 | Warmup unloaded model | Returns False |
## Integration Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| IT-16.01-01 | Load all 4 models sequentially | All models loaded successfully |
| IT-16.01-02 | Load + warmup cycle for each model | All models ready for inference |
| IT-16.01-03 | GPU memory allocation after loading all models | ~4GB GPU memory used |
| IT-16.01-04 | First inference after warmup | Latency within target range |
_docs/02_components/16_model_manager/16.02_feature_inference_engine_provisioning.md
-68
View File
@@ -1,68 +0,0 @@
# Feature: Inference Engine Provisioning
## Description
Provides inference engines to consuming components and handles TensorRT optimization with automatic ONNX fallback. Ensures consistent inference interface regardless of underlying format and meets <5s processing requirement through acceleration.
## Component APIs Implemented
### `get_inference_engine(model_name: str) -> InferenceEngine`
- Returns inference engine for specified model
- Engine provides unified `infer(input: np.ndarray) -> np.ndarray` interface
- Consumers: F07 Sequential VO (SuperPoint, LightGlue), F08 GPR (DINOv2), F09 Metric Refinement (LiteSAM)
### `optimize_to_tensorrt(model_name: str, onnx_path: str) -> str`
- Converts ONNX model to TensorRT engine
- Applies FP16 precision (2-3x speedup)
- Performs graph fusion and kernel optimization
- One-time conversion, result cached
### `fallback_to_onnx(model_name: str) -> bool`
- Detects TensorRT failure
- Loads ONNX model as fallback
- Logs warning for monitoring
- Ensures system continues functioning
## External Tools and Services
- **TensorRT**: Model optimization and inference
- **ONNX Runtime**: Fallback inference
- **CUDA**: GPU execution
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_get_cached_engine(model_name)` | Retrieve engine from cache |
| `_build_tensorrt_engine(onnx_path)` | Build TensorRT engine from ONNX |
| `_apply_fp16_optimization(builder)` | Enable FP16 precision in TensorRT |
| `_cache_tensorrt_engine(model_name, path)` | Save TensorRT engine to disk |
| `_detect_tensorrt_failure(error)` | Determine if error requires ONNX fallback |
| `_create_inference_wrapper(engine, format)` | Create unified InferenceEngine interface |
## Unit Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| UT-16.02-01 | Get SuperPoint engine | Returns valid InferenceEngine |
| UT-16.02-02 | Get LightGlue engine | Returns valid InferenceEngine |
| UT-16.02-03 | Get DINOv2 engine | Returns valid InferenceEngine |
| UT-16.02-04 | Get LiteSAM engine | Returns valid InferenceEngine |
| UT-16.02-05 | Get unloaded model engine | Raises error or returns None |
| UT-16.02-06 | InferenceEngine.infer() with valid input | Returns features array |
| UT-16.02-07 | Optimize ONNX to TensorRT | TensorRT engine file created |
| UT-16.02-08 | TensorRT optimization with FP16 | Engine uses FP16 precision |
| UT-16.02-09 | Fallback to ONNX on TensorRT failure | ONNX model loaded, returns True |
| UT-16.02-10 | Fallback logs warning | Warning logged |
## Integration Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| IT-16.02-01 | SuperPoint inference 100 iterations | Avg latency ~15ms (TensorRT) or ~50ms (ONNX) |
| IT-16.02-02 | LightGlue inference 100 iterations | Avg latency ~50ms (TensorRT) or ~150ms (ONNX) |
| IT-16.02-03 | DINOv2 inference 100 iterations | Avg latency ~150ms (TensorRT) or ~500ms (ONNX) |
| IT-16.02-04 | LiteSAM inference 100 iterations | Avg latency ~60ms (TensorRT) or ~200ms (ONNX) |
| IT-16.02-05 | Simulate TensorRT failure → ONNX fallback | System continues with ONNX |
| IT-16.02-06 | Full pipeline: optimize → load → infer | End-to-end works |
_docs/02_components/16_model_manager/16._component_model_manager.md
-224
View File
@@ -1,224 +0,0 @@
# Model Manager
## Interface Definition
**Interface Name**: `IModelManager`
### Interface Methods
```python
class IModelManager(ABC):
@abstractmethod
def load_model(self, model_name: str, model_format: str) -> bool:
pass
@abstractmethod
def get_inference_engine(self, model_name: str) -> InferenceEngine:
pass
@abstractmethod
def optimize_to_tensorrt(self, model_name: str, onnx_path: str) -> str:
pass
@abstractmethod
def fallback_to_onnx(self, model_name: str) -> bool:
pass
@abstractmethod
def warmup_model(self, model_name: str) -> bool:
pass
```
## Component Description
### Responsibilities
- Load ML models (TensorRT primary, ONNX fallback)
- Manage model lifecycle (loading, unloading, warmup)
- Provide inference engines for:
- SuperPoint (feature extraction)
- LightGlue (feature matching)
- DINOv2 (global descriptors)
- LiteSAM (cross-view matching)
- Handle TensorRT optimization and ONNX fallback
- Ensure <5s processing requirement through acceleration
### Scope
- Model loading and caching
- TensorRT optimization
- ONNX fallback handling
- Inference engine abstraction
- GPU memory management
## API Methods
### `load_model(model_name: str, model_format: str) -> bool`
**Description**: Loads model in specified format.
**Called By**: F02.1 Flight Lifecycle Manager (during system initialization)
**Input**:
```python
model_name: str # "SuperPoint", "LightGlue", "DINOv2", "LiteSAM"
model_format: str # "tensorrt", "onnx", "pytorch"
```
**Output**: `bool` - True if loaded
**Processing Flow**:
1. Check if model already loaded
2. Load model file
3. Initialize inference engine
4. Warm up model
5. Cache for reuse
**Test Cases**:
1. Load TensorRT model → succeeds
2. TensorRT unavailable → fallback to ONNX
3. Load all 4 models → all succeed
---
### `get_inference_engine(model_name: str) -> InferenceEngine`
**Description**: Gets inference engine for a model.
**Called By**:
- F07 Sequential VO (SuperPoint, LightGlue)
- F08 Global Place Recognition (DINOv2)
- F09 Metric Refinement (LiteSAM)
**Output**:
```python
InferenceEngine:
model_name: str
format: str
infer(input: np.ndarray) -> np.ndarray
```
**Test Cases**:
1. Get SuperPoint engine → returns engine
2. Call infer() → returns features
---
### `optimize_to_tensorrt(model_name: str, onnx_path: str) -> str`
**Description**: Converts ONNX model to TensorRT for acceleration.
**Called By**: System initialization (one-time)
**Input**:
```python
model_name: str
onnx_path: str # Path to ONNX model
```
**Output**: `str` - Path to TensorRT engine
**Processing Details**:
- FP16 precision (2-3x speedup)
- Graph fusion and kernel optimization
- One-time conversion, cached for reuse
**Test Cases**:
1. Convert ONNX to TensorRT → engine created
2. Load TensorRT engine → inference faster than ONNX
---
### `fallback_to_onnx(model_name: str) -> bool`
**Description**: Falls back to ONNX if TensorRT fails.
**Called By**: Internal (during load_model)
**Processing Flow**:
1. Detect TensorRT failure
2. Load ONNX model
3. Log warning
4. Continue with ONNX
**Test Cases**:
1. TensorRT fails → ONNX loaded automatically
2. System continues functioning
---
### `warmup_model(model_name: str) -> bool`
**Description**: Warms up model with dummy input.
**Called By**: Internal (after load_model)
**Purpose**: Initialize CUDA kernels, allocate GPU memory
**Test Cases**:
1. Warmup → first real inference fast
## Integration Tests
### Test 1: Model Loading
1. load_model("SuperPoint", "tensorrt")
2. load_model("LightGlue", "tensorrt")
3. load_model("DINOv2", "tensorrt")
4. load_model("LiteSAM", "tensorrt")
5. Verify all loaded
### Test 2: Inference Performance
1. Get inference engine
2. Run inference 100 times
3. Measure average latency
4. Verify meets performance targets
### Test 3: Fallback Scenario
1. Simulate TensorRT failure
2. Verify fallback to ONNX
3. Verify inference still works
## Non-Functional Requirements
### Performance
- **SuperPoint**: ~15ms (TensorRT), ~50ms (ONNX)
- **LightGlue**: ~50ms (TensorRT), ~150ms (ONNX)
- **DINOv2**: ~150ms (TensorRT), ~500ms (ONNX)
- **LiteSAM**: ~60ms (TensorRT), ~200ms (ONNX)
### Memory
- GPU memory: ~4GB for all 4 models
### Reliability
- Graceful fallback to ONNX
- Automatic retry on transient errors
## Dependencies
### External Dependencies
- **TensorRT**: NVIDIA inference optimization
- **ONNX Runtime**: ONNX inference
- **PyTorch**: Model weights (optional)
- **CUDA**: GPU acceleration
## Data Models
### InferenceEngine
```python
class InferenceEngine(ABC):
model_name: str
format: str
@abstractmethod
def infer(self, input: np.ndarray) -> np.ndarray:
pass
```
### ModelConfig
```python
class ModelConfig(BaseModel):
model_name: str
model_path: str
format: str
precision: str # "fp16", "fp32"
warmup_iterations: int = 3
```
_docs/02_components/17_configuration_manager/17.01_feature_system_configuration.md
-61
View File
@@ -1,61 +0,0 @@
# Feature: System Configuration
## Description
Handles loading, validation, and runtime management of system-wide configuration including camera parameters, operational area bounds, model paths, database connections, and API endpoints.
## Component APIs Implemented
- `load_config(config_path: str) -> SystemConfig`
- `validate_config(config: SystemConfig) -> ValidationResult`
- `get_camera_params(camera_id: Optional[str] = None) -> CameraParameters`
- `update_config(section: str, key: str, value: Any) -> bool`
## External Tools and Services
- **PyYAML**: Configuration file parsing (YAML format)
- **pydantic**: Data model validation and serialization
- **os/pathlib**: File system operations for config file access
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_parse_yaml_file` | Reads and parses YAML configuration file |
| `_apply_defaults` | Applies default values for missing configuration keys |
| `_validate_camera_params` | Validates camera parameters are within sensible bounds |
| `_validate_paths` | Checks that configured paths exist |
| `_validate_operational_area` | Validates operational area bounds are valid coordinates |
| `_build_camera_matrix` | Constructs camera intrinsic matrix from parameters |
| `_get_cached_config` | Returns cached configuration to avoid repeated file reads |
## Unit Tests
| Test Case | Description |
|-----------|-------------|
| `test_load_config_valid_yaml` | Load valid YAML config file, verify SystemConfig returned |
| `test_load_config_missing_file_uses_defaults` | Missing config file returns default configuration |
| `test_load_config_invalid_yaml_raises_error` | Malformed YAML raises appropriate error |
| `test_load_config_partial_uses_defaults` | Partial config merges with defaults |
| `test_validate_config_valid` | Valid configuration passes validation |
| `test_validate_config_invalid_focal_length` | Focal length <= 0 fails validation |
| `test_validate_config_invalid_sensor_size` | Sensor dimensions <= 0 fails validation |
| `test_validate_config_invalid_resolution` | Resolution <= 0 fails validation |
| `test_validate_config_invalid_operational_area` | Invalid lat/lon bounds fail validation |
| `test_validate_config_missing_paths` | Non-existent paths fail validation |
| `test_get_camera_params_default` | Get default camera returns valid parameters |
| `test_get_camera_params_specific_camera` | Get specific camera ID returns correct parameters |
| `test_get_camera_params_unknown_camera` | Unknown camera ID raises error |
| `test_update_config_valid_key` | Update existing key succeeds |
| `test_update_config_invalid_section` | Update non-existent section fails |
| `test_update_config_invalid_key` | Update non-existent key fails |
| `test_update_config_type_mismatch` | Update with wrong value type fails |
## Integration Tests
| Test Case | Description |
|-----------|-------------|
| `test_load_and_validate_full_config` | Load config file, validate, verify all sections accessible |
| `test_config_persistence_after_update` | Update config at runtime, verify changes persist in memory |
| `test_camera_params_consistency` | Camera params from get_camera_params match loaded config |
_docs/02_components/17_configuration_manager/17.02_feature_flight_configuration.md
-50
View File
@@ -1,50 +0,0 @@
# Feature: Flight Configuration
## Description
Manages flight-specific configuration including operational altitude, frame spacing, and camera parameters per flight. Provides persistence and retrieval of flight configurations created during flight initialization.
## Component APIs Implemented
- `get_flight_config(flight_id: str) -> FlightConfig`
- `save_flight_config(flight_id: str, config: FlightConfig) -> bool`
- `get_operational_altitude(flight_id: str) -> float`
- `get_frame_spacing(flight_id: str) -> float`
## External Tools and Services
- **F03 Flight Database**: Persistence layer for flight-specific configuration storage
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_flight_config` | Constructs FlightConfig from stored data |
| `_validate_flight_config` | Validates FlightConfig before persistence |
| `_cache_flight_config` | Caches flight config in memory for fast retrieval |
| `_invalidate_cache` | Clears cached flight config when updated |
## Unit Tests
| Test Case | Description |
|-----------|-------------|
| `test_get_flight_config_existing` | Get config for existing flight returns FlightConfig |
| `test_get_flight_config_nonexistent` | Get config for non-existent flight raises error |
| `test_save_flight_config_valid` | Save valid FlightConfig returns True |
| `test_save_flight_config_invalid_flight_id` | Save with empty flight_id fails |
| `test_save_flight_config_invalid_config` | Save with invalid config data fails |
| `test_get_operational_altitude_existing` | Get altitude for existing flight returns value |
| `test_get_operational_altitude_nonexistent` | Get altitude for non-existent flight raises error |
| `test_get_operational_altitude_range` | Verify altitude within expected range (100-500m) |
| `test_get_frame_spacing_existing` | Get frame spacing returns expected distance |
| `test_get_frame_spacing_default` | Frame spacing returns reasonable default (~100m) |
## Integration Tests
| Test Case | Description |
|-----------|-------------|
| `test_save_and_retrieve_flight_config` | Save flight config via save_flight_config, retrieve via get_flight_config, verify data matches |
| `test_flight_config_persists_to_database` | Save flight config, verify stored in F03 Flight Database |
| `test_altitude_and_spacing_consistency` | Flight config altitude matches get_operational_altitude result |
| `test_multiple_flights_isolation` | Multiple flights have independent configurations |
_docs/02_components/17_configuration_manager/17._component_configuration_manager.md
-283
View File
@@ -1,283 +0,0 @@
# Configuration Manager
## Interface Definition
**Interface Name**: `IConfigurationManager`
### Interface Methods
```python
class IConfigurationManager(ABC):
@abstractmethod
def load_config(self, config_path: str) -> SystemConfig:
pass
@abstractmethod
def get_camera_params(self, camera_id: Optional[str] = None) -> CameraParameters:
pass
@abstractmethod
def validate_config(self, config: SystemConfig) -> ValidationResult:
pass
@abstractmethod
def get_flight_config(self, flight_id: str) -> FlightConfig:
pass
@abstractmethod
def update_config(self, section: str, key: str, value: Any) -> bool:
pass
@abstractmethod
def get_operational_altitude(self, flight_id: str) -> float:
"""Returns predefined operational altitude for the flight (NOT from EXIF)."""
pass
@abstractmethod
def get_frame_spacing(self, flight_id: str) -> float:
"""Returns expected distance between consecutive frames in meters."""
pass
@abstractmethod
def save_flight_config(self, flight_id: str, config: FlightConfig) -> bool:
"""Persists flight-specific configuration."""
pass
```
## Component Description
### Responsibilities
- Load system configuration from files/environment
- Provide camera parameters (focal length, sensor size, resolution)
- Manage flight-specific configurations
- Validate configuration integrity
- Support configuration updates at runtime
### Scope
- System-wide configuration
- Camera parameter management
- Operational area bounds
- Model paths and settings
- Database connections
- API endpoints
## API Methods
### `load_config(config_path: str) -> SystemConfig`
**Description**: Loads system configuration.
**Called By**: System startup
**Input**: `config_path: str` - Path to config file (YAML/JSON)
**Output**: `SystemConfig` - Complete configuration
**Test Cases**:
1. Load valid config → succeeds
2. Missing file → uses defaults
3. Invalid config → raises error
---
### `get_camera_params(camera_id: Optional[str] = None) -> CameraParameters`
**Description**: Gets camera parameters.
**Called By**: All processing components
**Output**:
```python
CameraParameters:
focal_length: float
sensor_width: float
sensor_height: float
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float]
distortion_coefficients: List[float]
```
**Test Cases**:
1. Get default camera → returns params
2. Get specific camera → returns params
---
### `validate_config(config: SystemConfig) -> ValidationResult`
**Description**: Validates configuration.
**Called By**: After load_config
**Validation Rules**:
- Camera parameters sensible
- Paths exist
- Operational area valid
- Database connection string valid
**Test Cases**:
1. Valid config → passes
2. Invalid focal length → fails
---
### `get_flight_config(flight_id: str) -> FlightConfig`
**Description**: Gets flight-specific configuration.
**Called By**: F02.1 Flight Lifecycle Manager
**Output**:
```python
FlightConfig:
camera_params: CameraParameters
altitude: float
operational_area: OperationalArea
```
**Test Cases**:
1. Get flight config → returns params
---
### `update_config(section: str, key: str, value: Any) -> bool`
**Description**: Updates config at runtime.
**Called By**: Admin tools
**Test Cases**:
1. Update value → succeeds
2. Invalid key → fails
---
### `get_operational_altitude(flight_id: str) -> float`
**Description**: Returns predefined operational altitude for the flight in meters. This is the altitude provided during flight creation, NOT extracted from EXIF metadata (images don't have GPS/altitude metadata per problem constraints).
**Called By**:
- F10 Factor Graph Optimizer (for scale resolution)
- H02 GSD Calculator (for GSD computation)
- F09 Metric Refinement (for alignment)
**Input**: `flight_id: str`
**Output**: `float` - Altitude in meters (typically 100-500m)
**Test Cases**:
1. Get existing flight altitude → returns value
2. Non-existent flight → raises error
---
### `get_frame_spacing(flight_id: str) -> float`
**Description**: Returns expected distance between consecutive frames in meters. Used for scale estimation in visual odometry.
**Called By**:
- F10 Factor Graph Optimizer (for expected displacement calculation)
**Input**: `flight_id: str`
**Output**: `float` - Expected frame spacing in meters (typically ~100m)
**Test Cases**:
1. Get frame spacing → returns expected distance
---
### `save_flight_config(flight_id: str, config: FlightConfig) -> bool`
**Description**: Persists flight-specific configuration when a flight is created.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight creation)
**Input**:
```python
flight_id: str
config: FlightConfig
```
**Output**: `bool` - True if saved successfully
**Test Cases**:
1. Save valid config → succeeds
2. Invalid flight_id → fails
## Data Models
### SystemConfig
```python
class SystemConfig(BaseModel):
camera: CameraParameters
operational_area: OperationalArea
models: ModelPaths
database: DatabaseConfig
api: APIConfig
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float]
distortion_coefficients: List[float]
```
### OperationalArea
```python
class OperationalArea(BaseModel):
name: str = "Eastern Ukraine"
min_lat: float = 45.0
max_lat: float = 52.0
min_lon: float = 22.0
max_lon: float = 40.0
```
### RecoveryConfig
```python
class RecoveryConfig(BaseModel):
"""Configuration for failure recovery and progressive search."""
search_grid_sizes: List[int] = [1, 4, 9, 16, 25] # Progressive tile search grid
min_chunk_frames_for_matching: int = 5 # Minimum frames before chunk matching
max_chunk_frames_for_matching: int = 20 # Maximum frames per chunk
user_input_threshold_tiles: int = 25 # Request user input after this many tiles
chunk_matching_interval_seconds: float = 5.0 # Background matching interval
confidence_threshold_good: float = 0.7 # Confidence for good tracking
confidence_threshold_degraded: float = 0.5 # Confidence for degraded tracking
min_inlier_count_good: int = 50 # Inliers for good tracking
min_inlier_count_tracking: int = 20 # Minimum inliers before tracking loss
```
### RotationConfig
```python
class RotationConfig(BaseModel):
"""Configuration for image rotation and heading tracking."""
rotation_step_degrees: float = 30.0 # Degrees per rotation step
litesam_max_rotation_tolerance: float = 45.0 # Max rotation LiteSAM handles
sharp_turn_threshold_degrees: float = 45.0 # Threshold for sharp turn detection
heading_history_size: int = 10 # Number of headings to track
confidence_threshold: float = 0.7 # For accepting rotation match
@property
def rotation_iterations(self) -> int:
"""Number of rotation steps (360 / step_degrees)."""
return int(360 / self.rotation_step_degrees)
```
## Dependencies
### Internal Components
- **F03 Flight Database**: For flight-specific configuration persistence (save_flight_config stores to F03)
### External Dependencies
- **pydantic**: Data model validation
- **PyYAML**: Configuration file parsing
_docs/02_components/astral_next_components_diagram.drawio
-165
View File
@@ -1,165 +0,0 @@
<mxfile host="65bd71144e">
<diagram name="ASTRAL-Next Components" id="astral-next-components">
<mxGraphModel dx="1100" dy="750" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1100" pageHeight="600" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="ASTRAL-Next Component Diagram with Jira Epic Numbers" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=16;fontStyle=1;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="300" y="10" width="500" height="30" as="geometry"/>
</mxCell>
<mxCell id="client" value="Client" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1565C0;strokeColor=#64B5F6;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="20" y="60" width="60" height="35" as="geometry"/>
</mxCell>
<mxCell id="f01" value="F01 API&#10;AZ-112" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#8B1A1A;strokeColor=#EF5350;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f02" value="F02 Flight Processor&#10;AZ-113" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#6A1B9A;strokeColor=#BA68C8;fontColor=#ffffff;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="200" y="55" width="130" height="45" as="geometry"/>
</mxCell>
<mxCell id="f03" value="F03 DB&#10;AZ-114" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="350" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f05" value="F05 Image Pipeline&#10;AZ-116" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="450" y="55" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="satellite" value="Satellite&#10;Provider" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFB300;fontColor=#ffffff;dashed=1;" vertex="1" parent="1">
<mxGeometry x="580" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f15" value="F15 SSE&#10;AZ-126" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFA726;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="20" y="130" width="70" height="45" as="geometry"/>
</mxCell>
<mxCell id="f08" value="F08 Place Recog&#10;AZ-119" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f11" value="F11 Failure Recovery&#10;AZ-122" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="220" y="130" width="120" height="45" as="geometry"/>
</mxCell>
<mxCell id="f12" value="F12 Chunk Mgr&#10;AZ-123" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#7B1FA2;strokeColor=#BA68C8;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="360" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f16" value="F16 Model Mgr&#10;AZ-127" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="480" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f06" value="F06 Rotation&#10;AZ-117" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="210" width="90" height="45" as="geometry"/>
</mxCell>
<mxCell id="f07" value="F07 Sequential VO&#10;AZ-118" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="210" y="210" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="f09" value="F09 Metric Refine&#10;AZ-120" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="340" y="210" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="f17" value="F17 Config Mgr&#10;AZ-128" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="470" y="210" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f10" value="F10 Factor Graph&#10;AZ-121" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="260" y="290" width="120" height="50" as="geometry"/>
</mxCell>
<mxCell id="f04" value="F04 Satellite Mgr&#10;AZ-115" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="290" width="110" height="50" as="geometry"/>
</mxCell>
<mxCell id="f13" value="F13 Coord Transform&#10;AZ-124" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="400" y="290" width="120" height="50" as="geometry"/>
</mxCell>
<mxCell id="f14" value="F14 Result Manager&#10;AZ-125" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFA726;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="250" y="370" width="130" height="45" as="geometry"/>
</mxCell>
<mxCell id="helpers" value="H01-H08 Helpers&#10;AZ-129" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#546E7A;strokeColor=#78909C;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="540" y="290" width="110" height="50" as="geometry"/>
</mxCell>
<mxCell id="c1" value="HTTP" style="strokeColor=#FFFFFF;fontColor=#ffffff;fontSize=8;" edge="1" parent="1" source="client" target="f01">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c2" value="SSE" style="strokeColor=#FFFFFF;fontColor=#ffffff;fontSize=8;dashed=1;" edge="1" parent="1" source="f15" target="client">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c3" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f01" target="f02">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c4" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f03">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c4b" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f05">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c5" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f11">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c6" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f07">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c7" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c10" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f08">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c11" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f09">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c12" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f12">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c13" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f06">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c14" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f07" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c15" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f09" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c16" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f12" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c17" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f10" target="f13">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c18" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f10" target="f14">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c19" style="strokeColor=#FFFFFF;dashed=1;" edge="1" parent="1" source="f14" target="f15">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c20" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f08" target="f04">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c21" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f04" target="satellite">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="155" y="370"/>
<mxPoint x="620" y="370"/>
</Array>
</mxGeometry>
</mxCell>
<mxCell id="c22" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f07" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c23" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f08" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c24" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f09" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c25" style="strokeColor=#FFFFFF;dashed=1;" edge="1" parent="1" source="f17" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c26" style="strokeColor=#78909C;dashed=1;" edge="1" parent="1" source="helpers" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c27" style="strokeColor=#78909C;dashed=1;" edge="1" parent="1" source="helpers" target="f13">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c28" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f06" target="f09">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="legend" value="── sync call - - - event/config" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=10;fontColor=#888888;" vertex="1" parent="1">
<mxGeometry x="20" y="430" width="200" height="20" as="geometry"/>
</mxCell>
<mxCell id="legend2" value="Colors: Red=API | Purple=Orchestration | Blue=Visual Processing | Green=State | Orange=Data/IO | Gray=Infrastructure" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=9;fontColor=#666666;" vertex="1" parent="1">
<mxGeometry x="20" y="450" width="650" height="20" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
_docs/02_components/astral_next_components_diagram.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 333 KiB

_docs/02_components/decomposition_plan.md
-499
View File
@@ -1,499 +0,0 @@
# ASTRAL-Next System Component Decomposition Plan
## Design Principle: Interface-Based Architecture
**CRITICAL REQUIREMENT**: Each component MUST implement a well-defined interface to ensure interchangeability with different implementations.
**Benefits**:
- Swap implementations (e.g., replace LiteSAM with TransFG, GTSAM with Ceres)
- Enable unit testing with mocks
- Support multiple backends (TensorRT vs ONNX, different databases)
- Facilitate future enhancements without breaking contracts
**Interface Specification**: Each component spec must define:
- Interface name (e.g., `ISatelliteDataManager`, `IMetricRefinement`)
- All public methods with strict contracts
- Input/output data structures
- Error conditions and exceptions
- Performance guarantees
---
## System Architecture Overview
**Single unified Flight API:**
- Flight CRUD operations (create, read, update, delete)
- Waypoint management within flights
- Geofence management
- Tri-layer localization (SuperPoint+LightGlue, DINOv2, LiteSAM)
- Calls satellite provider for tiles
- Rotation preprocessing (LiteSAM 45° limit)
- Per-frame waypoint updates
- Progressive tile search (1→4→9→16→25)
- SSE streaming for real-time results
---
## FLIGHT API COMPONENTS (17 components)
### Core API Layer
**F01_flight_api**
**Interface**: `IFlightAPI`
**Endpoints**: `POST /flights`, `GET /flights/{flightId}`, `DELETE /flights/{flightId}`, `PUT /flights/{flightId}/waypoints/{waypointId}`, `POST .../images/batch`, `POST .../user-fix`, `GET .../status`, `GET .../stream`
**F02.1_flight_lifecycle_manager**
**Interface**: `IFlightLifecycleManager`
**API**: `create_flight()`, `get_flight()`, `get_flight_state()`, `delete_flight()`, `update_waypoint()`, `batch_update_waypoints()`, `validate_waypoint()`, `validate_geofence()`, `queue_images()`, `handle_user_fix()`, `create_client_stream()`, `convert_object_to_gps()`, `initialize_system()`
**F02.2_flight_processing_engine**
**Interface**: `IFlightProcessingEngine`
**API**: `start_processing()`, `stop_processing()`, `process_frame()`, `apply_user_fix()`, `handle_tracking_loss()`, `get_active_chunk()`, `create_new_chunk()`
**F03_flight_database**
**Interface**: `IFlightDatabase`
**API**: `insert_flight()`, `update_flight()`, `query_flights()`, `get_flight_by_id()`, `delete_flight()`, `get_waypoints()`, `insert_waypoint()`, `update_waypoint()`, `batch_update_waypoints()`, `save_flight_state()`, `load_flight_state()`, `save_frame_result()`, `get_frame_results()`, `save_heading()`, `get_heading_history()`, `get_latest_heading()`, `save_image_metadata()`, `get_image_path()`, `get_image_metadata()`
### Data Management
**F04_satellite_data_manager**
**Interface**: `ISatelliteDataManager`
**API**: `fetch_tile()`, `fetch_tile_grid()`, `prefetch_route_corridor()`, `progressive_fetch()`, `cache_tile()`, `get_cached_tile()`, `compute_tile_coords()`, `expand_search_grid()`, `compute_tile_bounds()`
**Features**: Progressive retrieval, tile caching, grid calculations
**F05_image_input_pipeline**
**Interface**: `IImageInputPipeline`
**API**: `queue_batch()`, `process_next_batch()`, `validate_batch()`, `store_images()`, `get_next_image()`, `get_image_by_sequence()`
**Features**: FIFO queuing, validation, storage
**F06_image_rotation_manager**
**Interface**: `IImageRotationManager`
**API**: `rotate_image_360()`, `try_rotation_steps()`, `calculate_precise_angle()`, `get_current_heading()`, `update_heading()`, `detect_sharp_turn()`, `requires_rotation_sweep()`
**Features**: 30° rotation sweeps, heading tracking
### Visual Processing
**F07_sequential_visual_odometry**
**Interface**: `ISequentialVisualOdometry`
**API**: `compute_relative_pose()`, `extract_features()`, `match_features()`, `estimate_motion()`
**F08_global_place_recognition**
**Interface**: `IGlobalPlaceRecognition`
**API**: `retrieve_candidate_tiles()`, `compute_location_descriptor()`, `query_database()`, `rank_candidates()`
**F09_metric_refinement**
**Interface**: `IMetricRefinement`
**API**: `align_to_satellite(uav_image, satellite_tile, tile_bounds)`, `compute_homography()`, `extract_gps_from_alignment()`, `compute_match_confidence()`
### State Estimation
**F10_factor_graph_optimizer**
**Interface**: `IFactorGraphOptimizer`
**API**: `add_relative_factor()`, `add_absolute_factor()`, `add_altitude_prior()`, `optimize()`, `get_trajectory()`, `get_marginal_covariance()`
**F11_failure_recovery_coordinator**
**Interface**: `IFailureRecoveryCoordinator`
**API**: `check_confidence()`, `detect_tracking_loss()`, `start_search()`, `expand_search_radius()`, `try_current_grid()`, `create_user_input_request()`, `apply_user_anchor()`
**F12_route_chunk_manager**
**Interface**: `IRouteChunkManager`
**API**: `create_chunk()`, `add_frame_to_chunk()`, `get_chunk_frames()`, `get_chunk_images()`, `get_chunk_composite_descriptor()`, `get_chunk_bounds()`, `is_chunk_ready_for_matching()`, `mark_chunk_anchored()`, `get_chunks_for_matching()`, `get_active_chunk()`, `deactivate_chunk()`
**Features**: Chunk lifecycle management, chunk state tracking, chunk matching coordination
**F13_coordinate_transformer**
**Interface**: `ICoordinateTransformer`
**API**: `set_enu_origin()`, `get_enu_origin()`, `gps_to_enu()`, `enu_to_gps()`, `pixel_to_gps()`, `gps_to_pixel()`, `image_object_to_gps()`, `transform_points()`
**Dependencies**: F10 Factor Graph Optimizer (for frame poses), H02 GSD Calculator (for GSD computation)
### Results & Communication
**F14_result_manager**
**Interface**: `IResultManager`
**API**: `update_frame_result()`, `publish_waypoint_update()`, `get_flight_results()`, `mark_refined()`
**F15_sse_event_streamer**
**Interface**: `ISSEEventStreamer`
**API**: `create_stream()`, `send_frame_result()`, `send_search_progress()`, `send_user_input_request()`, `send_refinement()`
### Infrastructure
**F16_model_manager**
**Interface**: `IModelManager`
**API**: `load_model()`, `get_inference_engine()`, `optimize_to_tensorrt()`, `fallback_to_onnx()`
**F17_configuration_manager**
**Interface**: `IConfigurationManager`
**API**: `load_config()`, `get_camera_params()`, `validate_config()`, `get_flight_config()`
---
## HELPER COMPONENTS (8 components)
**H01_camera_model** - `ICameraModel`
**H02_gsd_calculator** - `IGSDCalculator`
**H03_robust_kernels** - `IRobustKernels`
**H04_faiss_index_manager** - `IFaissIndexManager`
**H05_performance_monitor** - `IPerformanceMonitor`
**H06_web_mercator_utils** - `IWebMercatorUtils`
**H07_image_rotation_utils** - `IImageRotationUtils`
**H08_batch_validator** - `IBatchValidator`
---
## System Startup Initialization Order
**Startup sequence** (blocking, sequential):
| Order | Component | Method | Purpose | Dependencies |
|-------|-----------|--------|---------|--------------|
| 1 | F17 Configuration Manager | `load_config()` | Load system configuration | None |
| 2 | F03 Flight Database | Initialize connections | Establish DB connection pool | F17 |
| 3 | F16 Model Manager | `load_model("SuperPoint")` | Load SuperPoint feature extractor | F17 |
| 4 | F16 Model Manager | `load_model("LightGlue")` | Load LightGlue matcher | F17 |
| 5 | F16 Model Manager | `load_model("DINOv2")` | Load DINOv2 for place recognition | F17 |
| 6 | F16 Model Manager | `load_model("LiteSAM")` | Load LiteSAM for cross-view matching | F17 |
| 7 | F04 Satellite Data Manager | Initialize cache | Initialize tile cache directory | F17 |
| 8 | F08 Global Place Recognition | `load_index()` | Load pre-built Faiss index from satellite provider | F04, F16, H04 |
| 9 | F12 Route Chunk Manager | Initialize | Initialize chunk state tracking | F10 |
| 10 | F02 Flight Processor | Ready | Ready to accept flights | All above |
| 11 | F01 Flight API | Start server | Start FastAPI/Uvicorn | F02 |
**Estimated total startup time**: ~30 seconds (dominated by model loading)
**Shutdown sequence** (reverse order):
1. F01 Flight API - Stop accepting requests
2. F02 Flight Processor - Complete or cancel active flights
3. F11 Failure Recovery Coordinator - Stop background chunk matching
4. F12 Route Chunk Manager - Save chunk state
5. F16 Model Manager - Unload models
6. F03 Flight Database - Close connections
7. F04 Satellite Data Manager - Flush cache
---
## Comprehensive Component Interaction Matrix
### System Initialization
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.1 | F17 | `load_config()` | Load system configuration |
| F02.1 | F16 | `load_model()` × 4 | Load SuperPoint, LightGlue, DINOv2, LiteSAM |
| F04 | F08 | Satellite tiles + index | F08 loads pre-built Faiss index from provider |
| F08 | H04 | `load_index()` | Load satellite descriptor index |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model for descriptor computation |
### Flight Creation
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `POST /flights` | Create flight |
| F01 | F02.1 | `create_flight()` | Initialize flight state |
| F02.1 | F17 | `get_flight_config()` | Get camera params, altitude |
| F02.1 | F13 | `set_enu_origin(flight_id, start_gps)` | Set ENU coordinate origin |
| F02.1 | F04 | `prefetch_route_corridor()` | Prefetch tiles |
| F04 | Satellite Provider | `GET /api/satellite/tiles/batch` | HTTP batch download (includes tile metadata) |
| F04 | H06 | `compute_tile_bounds()` | Tile coordinate calculations |
| F02.1 | F03 | `insert_flight()` | Persist flight data |
### SSE Stream Creation
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `GET .../stream` | Open SSE connection |
| F01 | F02.1 | `create_client_stream()` | Route through lifecycle manager |
| F02.1 | F15 | `create_stream()` | Establish SSE channel |
### Image Upload
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `POST .../images/batch` | Upload 10-50 images |
| F01 | F02.1 | `queue_images()` | Route through lifecycle manager |
| F02.1 | F05 | `queue_batch()` | Queue for processing |
| F05 | H08 | `validate_batch()` | Validate sequence, format |
| F05 | F03 | `save_image_metadata()` | Persist image metadata |
### Per-Frame Processing (First Frame / Sharp Turn)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F05 | `get_next_image()` | Get image for processing |
| F02.2 | F06 | `requires_rotation_sweep()` | Check if sweep needed |
| F06 | H07 | `rotate_image()` × 12 | Rotate in 30° steps |
| F06 | F09 | `align_to_satellite(img, tile, bounds)` × 12 | Try LiteSAM each rotation |
| F02.2 | F04 | `get_cached_tile()` + `compute_tile_bounds()` | Get expected tile with bounds |
| F09 | F16 | `get_inference_engine("LiteSAM")` | Get model |
| F06 | H07 | `calculate_rotation_from_points()` | Precise angle from homography |
| F02.2 | F03 | `save_heading()` | Store UAV heading |
### Per-Frame Processing (Sequential VO)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F12 | `get_active_chunk()` | Get active chunk for frame |
| F02.2 | F07 | `compute_relative_pose()` | Provide image and chunk context |
| F07 | F16 | `get_inference_engine("SuperPoint")` | Get feature extractor |
| F07 | F16 | `get_inference_engine("LightGlue")` | Get matcher |
| F07 | H05 | `start_timer()`, `end_timer()` | Monitor timing |
| F02.2 | F10 | `add_relative_factor_to_chunk()` | Add pose measurement to chunk subgraph |
| F02.2 | F12 | `add_frame_to_chunk()` | Add frame to chunk |
### Tracking Good (Drift Correction)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F11 | `check_confidence()` | Check tracking quality |
| F02.2 | F04 | `fetch_tile()` + `compute_tile_bounds()` | Get single tile with bounds |
| F02.2 | F09 | `align_to_satellite(img, tile, bounds)` | Align to 1 tile |
| F02.2 | F10 | `add_absolute_factor()` | Add GPS measurement |
### Tracking Lost (Progressive Search + Chunk Building)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F11 | `check_confidence()` → FAIL | Low confidence |
| F11 | F12 | `create_chunk_on_tracking_loss()` | **Proactive chunk creation** |
| F12 | F10 | `create_chunk_subgraph()` | Create chunk in factor graph |
| F12 | F03 | `save_chunk_state()` | Persist chunk state for recovery |
| F02.2 | F12 | `get_active_chunk()` | Get new active chunk |
| F11 | F06 | `requires_rotation_sweep()` | Trigger rotation sweep (single-image) |
| F11 | F08 | `retrieve_candidate_tiles()` | Coarse localization (single-image) |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model |
| F08 | H04 | `search()` | Query Faiss index |
| F08 | F04 | `get_tile_by_gps()` × 5 | Get candidate tiles |
| F11 | F04 | `expand_search_grid(4)` | Get 2×2 grid |
| F11 | F09 | `align_to_satellite(img, tile, bounds)` | Try LiteSAM on tiles |
| F11 (fail) | F04 | `expand_search_grid(9)` | Expand to 3×3 |
| F11 (fail) | F04 | `expand_search_grid(16)` | Expand to 4×4 |
| F11 (fail) | F04 | `expand_search_grid(25)` | Expand to 5×5 |
| F11 (fail) | F12 | Continue building chunk | **Chunk building continues** |
| F11 (background) | F12 | `get_chunks_for_matching()` | Get unanchored chunks |
| F11 (background) | F08 | `retrieve_candidate_tiles_for_chunk()` | **Chunk semantic matching** |
| F11 (background) | F06 | `try_chunk_rotation_steps()` | **Chunk rotation sweeps** |
| F11 (background) | F09 | `align_chunk_to_satellite()` | **Chunk LiteSAM matching** |
| F11 (background) | F12 | `mark_chunk_anchored()` + `merge_chunks(main, new)` | **Chunk merging via F12** |
| F11 (fail) | F02.2 | Returns `UserInputRequest` | Request human help (last resort) |
| F02.2 | F15 | `send_user_input_request()` | Send SSE event to client |
### Optimization & Results
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F10 | H03 | `huber_loss()`, `cauchy_loss()` | Apply robust kernels |
| F10 | Internal | `optimize()` | Run iSAM2 optimization |
| F02.2 | F10 | `get_trajectory()` | Get optimized poses |
| F02.2 | F13 | `enu_to_gps()` | Convert ENU to GPS |
| F13 | H01 | `project()`, `unproject()` | Camera operations |
| F13 | H02 | `compute_gsd()` | GSD calculations |
| F13 | H06 | `tile_to_latlon()` | Coordinate transforms |
| F02.2 | F14 | Frame GPS + object coords | Provide results |
| F14 | F03 | `update_waypoint()` | Per-frame waypoint update |
| F14 | F15 | `send_frame_result()` | Publish to client |
| F15 | Client | SSE `frame_processed` | Real-time delivery |
| F14 | F03 | `save_frame_result()` | Persist frame result |
### User Input Recovery
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F15 | Client | SSE `user_input_needed` | Notify client |
| Client | F01 | `POST .../user-fix` | Provide anchor |
| F01 | F02.1 | `handle_user_fix()` | Route through lifecycle manager |
| F02.1 | F02.2 | `apply_user_fix()` | Delegate to processing engine |
| F02.2 | F11 | `apply_user_anchor()` | Apply fix |
| F11 | F10 | `add_absolute_factor()` (high confidence) | Hard constraint |
| F10 | Internal | `optimize()` | Re-optimize |
| F02.2 | F15 | `send_frame_result()` | Publish result via SSE |
### Asynchronous Refinement
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F10 | Internal (background) | `optimize()` | Back-propagate anchors |
| F02.2 | F10 | `get_trajectory()` | Get refined poses (ENU) |
| F02.2 | F13 | `enu_to_gps()` | Convert ENU poses to GPS |
| F02.2 | F14 | `mark_refined(List[RefinedFrameResult])` | Pass GPS-converted results |
| F14 | F03 | `batch_update_waypoints()` | Batch update waypoints |
| F14 | F15 | `send_refinement()` × N | Send updates |
| F15 | Client | SSE `frame_refined` × N | Incremental updates |
**Note**: F14 does NOT call F10 or F13. F02.2 performs coordinate conversion and provides GPS results to F14.
### Chunk Matching (Background)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F11 (background) | F12 | `get_chunks_for_matching()` | Get unanchored chunks ready for matching |
| F11 | F12 | `get_chunk_images()` | Get chunk images |
| F12 | F05 | `get_image_by_sequence()` | **Load images for chunk** (F12 delegates to F05 for actual image retrieval) |
| F11 | F08 | `retrieve_candidate_tiles_for_chunk()` | Chunk semantic matching (aggregate DINOv2) |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model for descriptor computation |
| F08 | H04 | `search()` | Query Faiss with chunk descriptor |
| F11 | F06 | `try_chunk_rotation_steps()` | Chunk rotation sweeps (12 rotations) |
| F06 | F09 | `align_chunk_to_satellite()` × 12 | Try LiteSAM for each rotation |
| F11 | F10 | `add_chunk_anchor()` | Anchor chunk with GPS |
| F11 | F12 | `merge_chunks(main_chunk, new_chunk)` | Merge new_chunk into main_chunk (Sim3 transform) |
| F10 | Internal | `optimize_global()` | Global optimization after merging |
| F11 | F12 | `mark_chunk_anchored()` | Update chunk state |
### Cross-Cutting Concerns
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F17 | ALL | `get_*_config()` | Provide configuration |
| H05 | F07, F08, F09, F10, F11 | `start_timer()`, `end_timer()` | Performance monitoring |
---
## Interaction Coverage Verification
✅ **Initialization**: F02.1→F16, F17; F04→F08→H04
✅ **Flight creation**: Client→F01→F02.1→F04,F12,F16,F17,F14
✅ **Image upload**: Client→F01→F02.1→F05→H08
✅ **Rotation sweep**: F06→H07,F09 (12 iterations)
✅ **Sequential VO**: F02.2→F07→F16,F10(chunk),F12,H05
✅ **Drift correction**: F02.2→F04,F09,F10
✅ **Tracking loss**: F02.2→F11→F12(proactive chunk),F06,F08,F04(progressive),F09
✅ **Chunk building**: F02.2→F12→F10,F07
✅ **Chunk image retrieval**: F12→F05(get_image_by_sequence for chunk images)
✅ **Chunk semantic matching**: F11→F12→F08(chunk descriptor)
✅ **Chunk LiteSAM matching**: F11→F06(chunk rotation)→F09(chunk alignment)
✅ **Chunk merging**: F11→F10(Sim3 transform)
✅ **Global PR**: F08→F16,H04,F04
✅ **Optimization**: F10→H03(chunk optimization, global optimization)
✅ **Coordinate transform**: F13→F10,H01,H02,H06
✅ **Results**: F02.2→F13→F14→F15,F03
✅ **User input**: Client→F01→F02.1→F02.2→F11→F10
✅ **Refinement**: F10→F14→F03,F15
✅ **Configuration**: F17→ALL
✅ **Performance**: H05→processing components
**All major component interactions are covered.**
---
## Deliverables
**Component Count**: 25 total
- Flight API: 17 (F01-F17)
- Helpers: 8 (H01-H08)
**For each component**, create `docs/02_components/[##]_[component_name]/[component_name]_spec.md`:
1. **Interface Definition** (interface name, methods, contracts)
2. **Component Description** (responsibilities, scope)
3. **API Methods** (inputs, outputs, errors, which components call it, test cases)
4. **Integration Tests**
5. **Non-Functional Requirements** (performance, accuracy targets)
6. **Dependencies** (which components it calls)
7. **Data Models**
### Component Specifications Status
- [x] F01 Flight API (merged from R01 Route REST API)
- [x] F02.1 Flight Lifecycle Manager (split from F02 Flight Processor)
- [x] F02.2 Flight Processing Engine (split from F02 Flight Processor)
- [x] F03 Flight Database (merged from R04, G17)
- [x] F04 Satellite Data Manager
- [x] F05 Image Input Pipeline
- [x] F06 Image Rotation Manager
- [x] F07 Sequential Visual Odometry
- [x] F08 Global Place Recognition
- [x] F09 Metric Refinement
- [x] F10 Factor Graph Optimizer
- [x] F11 Failure Recovery Coordinator
- [x] F12 Route Chunk Manager (Atlas multi-map chunk lifecycle)
- [x] F13 Coordinate Transformer (with ENU origin)
- [x] F14 Result Manager
- [x] F15 SSE Event Streamer
- [x] F16 Model Manager
- [x] F17 Configuration Manager
- [x] Helper components (H01-H08)
---
## Architecture Notes
### F02 Split: Lifecycle Manager + Processing Engine
F02 has been split into two components with clear Single Responsibility:
**F02.1 Flight Lifecycle Manager**:
- Flight CRUD operations
- System initialization
- API request routing
- SSE stream creation
- User fix delegation
**F02.2 Flight Processing Engine**:
- Main processing loop
- Frame-by-frame processing orchestration
- Recovery coordination with F11
- Chunk management coordination with F12
- Runs in background thread per active flight
**Rationale**: This split aligns with Single Responsibility Principle. F02.1 handles external-facing operations (API layer), while F02.2 handles internal processing (background engine).
**Decision**: Keep as single component. Clear internal organization with separate methods for state vs processing concerns. Event-based communication with F11 keeps recovery logic separate.
### Error Recovery Strategy
**Per-Component Recovery**:
- **F02**: Persists flight state via F03 on each significant update. On restart, loads last known state.
- **F07**: Stateless - reprocesses frame if VO fails
- **F10**: Factor graph state persisted periodically. On restart, rebuilds from F03 checkpoint.
- **F11**: Chunk state persisted via F12→F03. Recovery continues from last chunk state.
- **F12**: All chunk state persisted to F03. Restores chunk handles on restart.
**System-Wide Recovery**:
1. On crash, F02 loads flight state from F03
2. F12 restores chunk state from F03
3. Processing resumes from last successfully processed frame
4. Incomplete chunks continue building/matching
**Event Recovery**:
- Events are fire-and-forget (no persistence)
- Subscribers rebuild state from F03 on restart
### Error Propagation Strategy
**Principle**: Errors propagate upward through the component hierarchy. Lower-level components throw exceptions or return error results; higher-level coordinators handle recovery.
**Error Propagation Chain**:
```
H01-H08 (Helpers) → F07/F08/F09 (Visual Processing) → F11 (Recovery) → F02 (Coordinator) → F01 (API) → Client
↓
Events → F14 → F15 → SSE → Client
```
**Error Categories**:
1. **Recoverable** (F11 handles):
- Tracking loss → Progressive search
- Low confidence → Rotation sweep
- Chunk matching fails → User input request
2. **Propagated to Client** (via SSE):
- User input needed → `user_input_needed` event
- Processing blocked → `processing_blocked` event
- Flight completed → `flight_completed` event
3. **Fatal** (F02 handles, returns to F01):
- Database connection lost → HTTP 503
- Model loading failed → HTTP 500
- Invalid configuration → HTTP 400
**User Fix Flow**:
```
Client ---> F01 (POST /user-fix) ---> F02.handle_user_fix() ---> F11.apply_user_anchor()
↓
F10.add_chunk_anchor()
↓
F02 receives UserFixApplied event
↓
F14.publish_result() ---> F15 ---> SSE ---> Client
```
_docs/02_components/helpers/h01_camera_model_spec.md
-101
View File
@@ -1,101 +0,0 @@
# Camera Model Helper
## Interface Definition
**Interface Name**: `ICameraModel`
### Interface Methods
```python
class ICameraModel(ABC):
@abstractmethod
def project(self, point_3d: np.ndarray, camera_params: CameraParameters) -> Tuple[float, float]:
pass
@abstractmethod
def unproject(self, pixel: Tuple[float, float], depth: float, camera_params: CameraParameters) -> np.ndarray:
pass
@abstractmethod
def get_focal_length(self, camera_params: CameraParameters) -> Tuple[float, float]:
pass
@abstractmethod
def apply_distortion(self, pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]:
pass
@abstractmethod
def remove_distortion(self, pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]:
pass
```
## Component Description
Pinhole camera projection model with Brown-Conrady distortion handling.
## API Methods
### `project(point_3d: np.ndarray, camera_params: CameraParameters) -> Tuple[float, float]`
**Description**: Projects 3D point to 2D image pixel.
**Formula**:
```
x = fx * X/Z + cx
y = fy * Y/Z + cy
```
---
### `unproject(pixel: Tuple[float, float], depth: float, camera_params: CameraParameters) -> np.ndarray`
**Description**: Unprojects pixel to 3D ray at given depth.
**Formula**:
```
X = (x - cx) * depth / fx
Y = (y - cy) * depth / fy
Z = depth
```
---
### `get_focal_length(camera_params: CameraParameters) -> Tuple[float, float]`
**Description**: Returns (fx, fy) in pixels.
**Formula**:
```
fx = focal_length_mm * image_width / sensor_width_mm
fy = focal_length_mm * image_height / sensor_height_mm
```
---
### `apply_distortion(pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]`
**Description**: Applies radial and tangential distortion (Brown-Conrady model).
---
### `remove_distortion(pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]`
**Description**: Removes distortion from observed pixel.
## Dependencies
**External**: opencv-python, numpy
## Data Models
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float] # (cx, cy) pixels
distortion_coefficients: List[float] # [k1, k2, p1, p2, k3]
```
_docs/02_components/helpers/h02_gsd_calculator_spec.md
-78
View File
@@ -1,78 +0,0 @@
# GSD Calculator Helper
## Interface Definition
**Interface Name**: `IGSDCalculator`
### Interface Methods
```python
class IGSDCalculator(ABC):
@abstractmethod
def compute_gsd(self, altitude: float, camera_params: CameraParameters) -> float:
pass
@abstractmethod
def altitude_to_scale(self, altitude: float, focal_length: float) -> float:
pass
@abstractmethod
def meters_per_pixel(self, lat: float, zoom: int) -> float:
pass
@abstractmethod
def gsd_from_camera(self, altitude: float, focal_length: float, sensor_width: float, image_width: int) -> float:
pass
```
## Component Description
Ground Sampling Distance computations for altitude and coordinate systems.
## API Methods
### `compute_gsd(altitude: float, camera_params: CameraParameters) -> float`
**Description**: Computes GSD from altitude and camera parameters.
**Formula**:
```
GSD = (altitude * sensor_width) / (focal_length * image_width)
```
**Example**: altitude=800m, focal=24mm, sensor=36mm, width=6000px → GSD=0.2 m/pixel
---
### `altitude_to_scale(altitude: float, focal_length: float) -> float`
**Description**: Converts altitude to scale factor for VO.
---
### `meters_per_pixel(lat: float, zoom: int) -> float`
**Description**: Computes GSD for Web Mercator tiles at zoom level.
**Formula**:
```
meters_per_pixel = 156543.03392 * cos(lat * π/180) / 2^zoom
```
**Example**: lat=48°N, zoom=19 → ~0.3 m/pixel
---
### `gsd_from_camera(altitude: float, focal_length: float, sensor_width: float, image_width: int) -> float`
**Description**: Direct GSD calculation from parameters.
## Dependencies
**External**: numpy
## Test Cases
1. Standard camera at 800m → GSD ~0.1-0.3 m/pixel
2. Web Mercator zoom 19 at Ukraine → ~0.3 m/pixel

Some files were not shown because too many files have changed in this diff Show More