fix issues

This commit is contained in:
Oleksandr Bezdieniezhnykh
2025-11-30 01:43:23 +02:00
parent 1082316660
commit 310cf78ee7
17 changed files with 584 additions and 240 deletions
@@ -311,8 +311,9 @@ BatchResponse:
1. Validate flight_id exists 1. Validate flight_id exists
2. Validate batch size (10-50 images) 2. Validate batch size (10-50 images)
3. Validate sequence numbers (strict sequential) 3. Validate sequence numbers (strict sequential)
4. Pass to F05 Image Input Pipeline 4. Call F02 Flight Processor → queue_images(flight_id, batch)
5. Return immediately (processing is async) 5. F02 delegates to F05 Image Input Pipeline
6. Return immediately (processing is async)
**Error Conditions**: **Error Conditions**:
- `400 Bad Request`: Invalid batch size, out-of-sequence images - `400 Bad Request`: Invalid batch size, out-of-sequence images
@@ -355,9 +356,10 @@ UserFixResponse:
**Processing Flow**: **Processing Flow**:
1. Validate flight_id exists and is blocked 1. Validate flight_id exists and is blocked
2. Pass to F11 Failure Recovery Coordinator 2. Call F02 Flight Processor → handle_user_fix(flight_id, fix_data)
3. Coordinator applies anchor to Factor Graph 3. F02 delegates to F11 Failure Recovery Coordinator
4. Resume processing pipeline 4. Coordinator applies anchor to Factor Graph
5. Resume processing pipeline
**Error Conditions**: **Error Conditions**:
- `400 Bad Request`: Invalid fix data - `400 Bad Request`: Invalid fix data
@@ -400,8 +402,9 @@ ObjectGPSResponse:
**Processing Flow**: **Processing Flow**:
1. Validate flight_id and frame_id exist 1. Validate flight_id and frame_id exist
2. Validate frame has been processed (has pose in Factor Graph) 2. Validate frame has been processed (has pose in Factor Graph)
3. Call F13.image_object_to_gps(pixel, frame_id) 3. Call F02 Flight Processor → convert_object_to_gps(flight_id, frame_id, pixel)
4. Return GPS with accuracy estimate 4. F02 delegates to F13.image_object_to_gps(flight_id, frame_id, pixel)
5. Return GPS with accuracy estimate
**Error Conditions**: **Error Conditions**:
- `400 Bad Request`: Invalid pixel coordinates - `400 Bad Request`: Invalid pixel coordinates
@@ -479,6 +482,12 @@ SSE Stream with events:
- flight_completed - flight_completed
``` ```
**Processing Flow**:
1. Validate flight_id exists
2. Call F02 Flight Processor → create_client_stream(flight_id, client_id)
3. F02 delegates to F15 SSE Event Streamer → create_stream()
4. Return SSE stream to client
**Event Format**: **Event Format**:
```json ```json
{ {
@@ -575,11 +584,9 @@ SSE Stream with events:
## Dependencies ## Dependencies
### Internal Components ### Internal Components
- **F02 Flight Processor**: For all flight operations and processing orchestration - **F02 Flight Processor**: For ALL operations (flight CRUD, image batching, user fixes, SSE streams, object-to-GPS conversion). F01 is a thin REST layer that delegates all business logic to F02.
- **F05 Image Input Pipeline**: For batch processing
- **F11 Failure Recovery Coordinator**: For user fixes **Note**: F01 does NOT directly call F05, F11, F13, or F15. All operations are routed through F02 to maintain a single coordinator pattern.
- **F15 SSE Event Streamer**: For real-time streaming
- **F03 Flight Database**: For persistence (via F02)
### External Dependencies ### External Dependencies
- **FastAPI**: Web framework - **FastAPI**: Web framework
@@ -59,6 +59,27 @@ class IFlightProcessor(ABC):
def validate_flight_continuity(self, waypoints: List[Waypoint]) -> ValidationResult: def validate_flight_continuity(self, waypoints: List[Waypoint]) -> ValidationResult:
pass pass
# API Delegation Methods (called by F01)
@abstractmethod
def queue_images(self, flight_id: str, batch: ImageBatch) -> BatchQueueResult:
"""Delegates to F05 Image Input Pipeline."""
pass
@abstractmethod
def handle_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResult:
"""Delegates to F11 Failure Recovery Coordinator."""
pass
@abstractmethod
def create_client_stream(self, flight_id: str, client_id: str) -> StreamConnection:
"""Delegates to F15 SSE Event Streamer."""
pass
@abstractmethod
def convert_object_to_gps(self, flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint:
"""Delegates to F13 Coordinate Transformer."""
pass
# Processing Orchestration # Processing Orchestration
@abstractmethod @abstractmethod
def process_frame(self, flight_id: str, frame_id: int) -> FrameResult: def process_frame(self, flight_id: str, frame_id: int) -> FrameResult:
@@ -576,6 +597,129 @@ ValidationResult:
--- ---
## API Delegation Methods
These methods are called by F01 Flight API and delegate to specialized components. This maintains F02 as the single coordinator for all operations.
### `queue_images(flight_id: str, batch: ImageBatch) -> BatchQueueResult`
**Description**: Queues image batch for processing. Delegates to F05 Image Input Pipeline.
**Called By**:
- F01 Flight API (upload_image_batch endpoint)
**Input**:
```python
flight_id: str
batch: ImageBatch
```
**Output**:
```python
BatchQueueResult:
accepted: bool
sequences: List[int]
next_expected: int
message: Optional[str]
```
**Processing Flow**:
1. Validate flight exists and is in valid state
2. Delegate to F05 Image Input Pipeline → queue_batch()
3. Return result to F01
---
### `handle_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult`
**Description**: Handles user-provided GPS fix. Delegates to F11 Failure Recovery Coordinator.
**Called By**:
- F01 Flight API (submit_user_fix endpoint)
**Input**:
```python
flight_id: str
fix_data: UserFixRequest:
frame_id: int
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**:
```python
UserFixResult:
accepted: bool
processing_resumed: bool
message: Optional[str]
```
**Processing Flow**:
1. Validate flight exists and is blocked
2. Delegate to F11 Failure Recovery Coordinator → apply_user_anchor()
3. F11 emits UserFixApplied event (F02 subscribes and resumes processing)
4. Return result to F01
---
### `create_client_stream(flight_id: str, client_id: str) -> StreamConnection`
**Description**: Creates SSE stream for client. Delegates to F15 SSE Event Streamer.
**Called By**:
- F01 Flight API (create_sse_stream endpoint)
**Input**:
```python
flight_id: str
client_id: str
```
**Output**:
```python
StreamConnection:
stream_id: str
flight_id: str
client_id: str
last_event_id: Optional[str]
```
**Processing Flow**:
1. Validate flight exists
2. Delegate to F15 SSE Event Streamer → create_stream()
3. Return StreamConnection to F01
---
### `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint`
**Description**: Converts object pixel to GPS. Delegates to F13 Coordinate Transformer.
**Called By**:
- F01 Flight API (convert_object_to_gps endpoint)
**Input**:
```python
flight_id: str
frame_id: int
pixel: Tuple[float, float]
```
**Output**:
```python
GPSPoint:
lat: float
lon: float
```
**Processing Flow**:
1. Validate flight and frame exist
2. Validate frame has been processed (has pose)
3. Delegate to F13 Coordinate Transformer → image_object_to_gps(flight_id, frame_id, pixel)
4. Return GPSPoint to F01
---
## Processing Orchestration Methods ## Processing Orchestration Methods
### `process_frame(flight_id: str, frame_id: int) -> FrameResult` ### `process_frame(flight_id: str, frame_id: int) -> FrameResult`
@@ -676,6 +676,13 @@ Optional[Dict]: Metadata dictionary or None
## Chunk State Operations ## Chunk State Operations
**Necessity**: These methods are **required** for crash recovery. Without them:
- Chunk state would be lost on system restart
- Processing would need to start from scratch
- Background chunk matching progress would be lost
F12 Route Chunk Manager delegates persistence to F03 to maintain separation of concerns (F12 manages chunk logic, F03 handles storage).
### `save_chunk_state(flight_id: str, chunk: ChunkHandle) -> bool` ### `save_chunk_state(flight_id: str, chunk: ChunkHandle) -> bool`
**Description**: Saves chunk state to database for crash recovery. **Description**: Saves chunk state to database for crash recovery.
@@ -401,8 +401,8 @@ ProcessingStatus:
## Dependencies ## Dependencies
### Internal Components ### Internal Components
- **H08 Batch Validator**: For validation logic
- **F03 Flight Database**: For metadata persistence and flight state information - **F03 Flight Database**: For metadata persistence and flight state information
- **H08 Batch Validator**: For batch validation (naming convention, sequence continuity, format, dimensions)
### External Dependencies ### External Dependencies
- **opencv-python**: Image I/O - **opencv-python**: Image I/O
@@ -495,6 +495,7 @@ return None # No match found
## Dependencies ## Dependencies
### Internal Components ### Internal Components
- **F04 Satellite Data Manager**: For tile fetching (`fetch_tile`) and tile bounds computation (`compute_tile_bounds`)
- **F09 Metric Refinement**: For matching during rotation sweep (align_to_satellite, align_chunk_to_satellite). F06 rotates images, F09 performs the actual matching. - **F09 Metric Refinement**: For matching during rotation sweep (align_to_satellite, align_chunk_to_satellite). F06 rotates images, F09 performs the actual matching.
- **H07 Image Rotation Utils**: For image rotation and angle calculations - **H07 Image Rotation Utils**: For image rotation and angle calculations
- **F12 Route Chunk Manager**: For chunk image retrieval - **F12 Route Chunk Manager**: For chunk image retrieval
@@ -23,12 +23,10 @@ class ISequentialVO(ABC):
@abstractmethod @abstractmethod
def estimate_motion(self, matches: Matches, camera_params: CameraParameters) -> Optional[Motion]: def estimate_motion(self, matches: Matches, camera_params: CameraParameters) -> Optional[Motion]:
pass pass
@abstractmethod
def compute_relative_pose_in_chunk(self, prev_image: np.ndarray, curr_image: np.ndarray, chunk_id: str) -> Optional[RelativePose]:
pass
``` ```
**Note**: F07 is chunk-agnostic. It only computes relative poses between images. The caller (F02 Flight Processor) determines which chunk the frames belong to and routes factors to the appropriate subgraph via F12 → F10.
## Component Description ## Component Description
### Responsibilities ### Responsibilities
@@ -38,14 +36,13 @@ class ISequentialVO(ABC):
- Estimate relative pose (translation + rotation) between frames - Estimate relative pose (translation + rotation) between frames
- Return relative pose factors for Factor Graph Optimizer - Return relative pose factors for Factor Graph Optimizer
- Detect tracking loss (low inlier count) - Detect tracking loss (low inlier count)
- **Chunk-aware VO operations (factors added to chunk subgraph)**
### Scope ### Scope
- Frame-to-frame visual odometry - Frame-to-frame visual odometry
- Feature-based motion estimation - Feature-based motion estimation
- Handles low overlap and challenging agricultural environments - Handles low overlap and challenging agricultural environments
- Provides relative measurements for trajectory optimization - Provides relative measurements for trajectory optimization
- **Chunk-scoped operations (Atlas multi-map architecture)** - **Chunk-agnostic**: F07 doesn't know about chunks. Caller (F02) routes results to appropriate chunk subgraph.
## API Methods ## API Methods
@@ -224,49 +221,6 @@ Motion:
2. **Low inliers**: May return None 2. **Low inliers**: May return None
3. **Degenerate motion**: Handles pure rotation 3. **Degenerate motion**: Handles pure rotation
---
### `compute_relative_pose_in_chunk(prev_image: np.ndarray, curr_image: np.ndarray, chunk_id: str) -> Optional[RelativePose]`
**Description**: Computes relative camera pose between consecutive frames within a chunk context.
**Called By**:
- F02 Flight Processor (chunk-aware processing)
**Input**:
```python
prev_image: np.ndarray # Previous frame (t-1)
curr_image: np.ndarray # Current frame (t)
chunk_id: str # Chunk identifier for context
```
**Output**:
```python
RelativePose:
translation: np.ndarray # (x, y, z) in meters
rotation: np.ndarray # 3×3 rotation matrix or quaternion
confidence: float # 0.0 to 1.0
inlier_count: int
total_matches: int
tracking_good: bool
chunk_id: str # Chunk context
```
**Processing Flow**:
1. Same as compute_relative_pose() (SuperPoint + LightGlue)
2. Return RelativePose with chunk_id context
3. Factor will be added to chunk's subgraph (not global graph)
**Chunk Context**:
- VO operations are chunk-scoped
- Factors added to chunk's subgraph via F10.add_relative_factor_to_chunk()
- Chunk isolation ensures independent optimization
**Test Cases**:
1. **Chunk-aware VO**: Returns RelativePose with chunk_id
2. **Chunk isolation**: Factors isolated to chunk
3. **Multiple chunks**: VO operations don't interfere between chunks
## Integration Tests ## Integration Tests
### Test 1: Normal Flight Sequence ### Test 1: Normal Flight Sequence
@@ -377,11 +377,11 @@ Optional[np.ndarray]: 3×3 homography matrix or None
## Dependencies ## Dependencies
### Internal Components ### Internal Components
- **F12 Route Chunk Manager**: For chunk image retrieval and chunk operations
- **F16 Model Manager**: For LiteSAM model - **F16 Model Manager**: For LiteSAM model
- **H01 Camera Model**: For projection operations - **H01 Camera Model**: For projection operations
- **H02 GSD Calculator**: For coordinate transformations - **H02 GSD Calculator**: For coordinate transformations
- **H05 Performance Monitor**: For timing - **H05 Performance Monitor**: For timing
- **F12 Route Chunk Manager**: For chunk image retrieval
**Note**: tile_bounds is passed as parameter from caller (F02 Flight Processor gets it from F04 Satellite Data Manager) **Note**: tile_bounds is passed as parameter from caller (F02 Flight Processor gets it from F04 Satellite Data Manager)
@@ -8,64 +8,67 @@
```python ```python
class IFactorGraphOptimizer(ABC): class IFactorGraphOptimizer(ABC):
# All methods take flight_id to support concurrent flights
# F10 maintains Dict[str, FactorGraph] keyed by flight_id internally
@abstractmethod @abstractmethod
def add_relative_factor(self, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool: def add_relative_factor(self, flight_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass pass
@abstractmethod @abstractmethod
def add_absolute_factor(self, frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool: def add_absolute_factor(self, flight_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool:
pass pass
@abstractmethod @abstractmethod
def add_altitude_prior(self, frame_id: int, altitude: float, covariance: float) -> bool: def add_altitude_prior(self, flight_id: str, frame_id: int, altitude: float, covariance: float) -> bool:
pass pass
@abstractmethod @abstractmethod
def optimize(self, iterations: int) -> OptimizationResult: def optimize(self, flight_id: str, iterations: int) -> OptimizationResult:
pass pass
@abstractmethod @abstractmethod
def get_trajectory(self) -> Dict[int, Pose]: def get_trajectory(self, flight_id: str) -> Dict[int, Pose]:
pass pass
@abstractmethod @abstractmethod
def get_marginal_covariance(self, frame_id: int) -> np.ndarray: def get_marginal_covariance(self, flight_id: str, frame_id: int) -> np.ndarray:
pass
# Chunk operations - F10 only manages factor graph subgraphs
# F12 owns chunk metadata (status, is_active, etc.)
@abstractmethod
def create_chunk_subgraph(self, flight_id: str, chunk_id: str, start_frame_id: int) -> bool:
pass pass
@abstractmethod @abstractmethod
def create_new_chunk(self, chunk_id: str, start_frame_id: int) -> ChunkHandle: def add_relative_factor_to_chunk(self, flight_id: str, chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass pass
@abstractmethod @abstractmethod
def get_chunk_for_frame(self, frame_id: int) -> Optional[ChunkHandle]: def add_chunk_anchor(self, flight_id: str, chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool:
pass pass
@abstractmethod @abstractmethod
def add_relative_factor_to_chunk(self, chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool: def merge_chunk_subgraphs(self, flight_id: str, source_chunk_id: str, target_chunk_id: str, transform: Sim3Transform) -> bool:
"""Merges source_chunk INTO target_chunk. Source chunk subgraph is merged into target."""
pass pass
@abstractmethod @abstractmethod
def add_chunk_anchor(self, chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool: def get_chunk_trajectory(self, flight_id: str, chunk_id: str) -> Dict[int, Pose]:
pass pass
@abstractmethod @abstractmethod
def merge_chunks(self, chunk_id_1: str, chunk_id_2: str, transform: Sim3Transform) -> bool: def optimize_chunk(self, flight_id: str, chunk_id: str, iterations: int) -> OptimizationResult:
pass pass
@abstractmethod @abstractmethod
def get_chunk_trajectory(self, chunk_id: str) -> Dict[int, Pose]: def optimize_global(self, flight_id: str, iterations: int) -> OptimizationResult:
pass pass
@abstractmethod @abstractmethod
def get_all_chunks(self) -> List[ChunkHandle]: def delete_flight_graph(self, flight_id: str) -> bool:
pass """Cleanup factor graph when flight is deleted."""
@abstractmethod
def optimize_chunk(self, chunk_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def optimize_global(self, iterations: int) -> OptimizationResult:
pass pass
``` ```
@@ -84,16 +87,20 @@ class IFactorGraphOptimizer(ABC):
### Chunk Responsibility Clarification ### Chunk Responsibility Clarification
**F10 provides low-level factor graph operations only**: **F10 provides low-level factor graph operations only**:
- `create_new_chunk()`: Creates subgraph in factor graph - `create_chunk_subgraph()`: Creates subgraph in factor graph (returns bool, not ChunkHandle)
- `add_relative_factor_to_chunk()`: Adds factors to chunk's subgraph - `add_relative_factor_to_chunk()`: Adds factors to chunk's subgraph
- `add_chunk_anchor()`: Adds GPS anchor to chunk - `add_chunk_anchor()`: Adds GPS anchor to chunk's subgraph
- `merge_chunks()`: Applies Sim(3) transform and merges subgraphs - `merge_chunk_subgraphs()`: Applies Sim(3) transform and merges subgraphs
- `optimize_chunk()`, `optimize_global()`: Runs optimization - `optimize_chunk()`, `optimize_global()`: Runs optimization
**F12 is the source of truth for chunk state** (see F12 spec): **F10 does NOT own chunk metadata** - only factor graph data structures.
- Chunk lifecycle management (active, anchored, merged status)
**F12 is the source of truth for ALL chunk state** (see F12 spec):
- ChunkHandle with all metadata (is_active, has_anchor, matching_status)
- Chunk lifecycle management
- Chunk readiness determination - Chunk readiness determination
- High-level chunk queries - High-level chunk queries
- F12 calls F10 for factor graph operations
**F11 coordinates recovery** (see F11 spec): **F11 coordinates recovery** (see F11 spec):
- Triggers chunk creation via F12 - Triggers chunk creation via F12
@@ -155,16 +162,19 @@ F07 returns unit translation vectors due to monocular scale ambiguity. F10 resol
**Explicit Flow**: **Explicit Flow**:
```python ```python
# In add_relative_factor(): # In add_relative_factor():
# altitude comes from F05 Image Input Pipeline (extracted from EXIF metadata) # altitude comes from F17 Configuration Manager (predefined operational altitude)
# focal_length, sensor_width from F17 Configuration Manager # focal_length, sensor_width from F17 Configuration Manager
gsd = H02.compute_gsd(altitude, focal_length, sensor_width, image_width) config = F17.get_flight_config(flight_id)
altitude = config.altitude # Predefined altitude, NOT from EXIF
gsd = H02.compute_gsd(altitude, config.camera_params.focal_length,
config.camera_params.sensor_width,
config.camera_params.resolution_width)
expected_displacement = frame_spacing * gsd # ~100m typical at 300m altitude expected_displacement = frame_spacing * gsd # ~100m typical at 300m altitude
scaled_translation = relative_pose.translation * expected_displacement scaled_translation = relative_pose.translation * expected_displacement
# Add scaled_translation to factor graph # Add scaled_translation to factor graph
``` ```
**Note**: Altitude is passed through the processing chain: **Note**: Altitude comes from F17 Configuration Manager (predefined operational altitude), NOT from EXIF metadata. The problem statement specifies images don't have GPS metadata.
- F05 extracts altitude from EXIF → F02 includes in FrameData → F10 receives with add_relative_factor()
**Output**: **Output**:
```python ```python
@@ -95,7 +95,11 @@ class IFailureRecoveryCoordinator(ABC):
F11 emits events instead of directly calling F02's status update methods. This decouples recovery logic from flight state management. F11 emits events instead of directly calling F02's status update methods. This decouples recovery logic from flight state management.
**Events Emitted**: ### Internal Events (Component-to-Component)
F11 emits events for internal component communication. These are NOT directly sent to clients.
**Events Emitted (internal)**:
- `RecoveryStarted`: When tracking loss detected and recovery begins - `RecoveryStarted`: When tracking loss detected and recovery begins
- `RecoverySucceeded`: When recovery finds a match (single-image or chunk) - `RecoverySucceeded`: When recovery finds a match (single-image or chunk)
- `RecoveryFailed`: When all recovery strategies exhausted - `RecoveryFailed`: When all recovery strategies exhausted
@@ -104,12 +108,27 @@ F11 emits events instead of directly calling F02's status update methods. This d
- `ChunkCreated`: When new chunk created on tracking loss - `ChunkCreated`: When new chunk created on tracking loss
- `ChunkAnchored`: When chunk successfully matched and anchored - `ChunkAnchored`: When chunk successfully matched and anchored
- `ChunkMerged`: When chunk merged into main trajectory (includes flight_id, chunk_id, merged_frames) - `ChunkMerged`: When chunk merged into main trajectory (includes flight_id, chunk_id, merged_frames)
- `ChunkMatchingFailed`: When chunk matching exhausts all candidates and fails
**Event Listeners** (F02 Flight Processor subscribes to these): **Event Listeners** (F02 Flight Processor subscribes to these):
- On `RecoveryStarted`: Update status to "recovering" - On `RecoveryStarted`: Update status to "recovering"
- On `RecoveryFailed`: Update status to "blocked" - On `RecoveryFailed`: Update status to "blocked"
- On `RecoverySucceeded`: Update status to "processing" - On `RecoverySucceeded`: Update status to "processing"
- On `UserInputNeeded`: Update status to "blocked", blocked=True - On `UserInputNeeded`: Update status to "blocked", blocked=True
- On `ChunkMatchingFailed`: Re-queue chunk for user input or continue building
### External SSE Events (to Clients)
F11 does NOT directly send events to clients. External events are routed through F14 Result Manager which manages SSE streaming via F15:
- When F11 emits `UserInputNeeded` → F02 receives → F02 calls F14.publish_user_input_request() → F14 sends via F15 SSE
- When F11 emits `ChunkMerged` → F02 receives → F02 calls F14.update_results_after_chunk_merge() → F14 sends via F15 SSE
- When F11 emits `RecoveryFailed` → F02 receives → F02 calls F14.publish_processing_blocked() → F14 sends via F15 SSE
This separation ensures:
1. F11 is decoupled from SSE implementation
2. F14 controls all client-facing communication
3. Consistent event format for clients
### Scope ### Scope
- Confidence monitoring - Confidence monitoring
@@ -516,13 +535,13 @@ bool: True if merge successful
1. Get chunk frames via F12.get_chunk_frames(chunk_id) → merged_frames 1. Get chunk frames via F12.get_chunk_frames(chunk_id) → merged_frames
2. Get chunk anchor frame (middle frame or best frame) 2. Get chunk anchor frame (middle frame or best frame)
3. Call F12.mark_chunk_anchored() with GPS (F12 coordinates with F10) 3. Call F12.mark_chunk_anchored() with GPS (F12 coordinates with F10)
4. **Resolve target chunk**: 4. **Determine merge target**:
- Query F12.get_merge_target(chunk_id) → returns target_chunk_id - Target is typically the temporal predecessor (previous chunk by frame_id order)
- Target selection logic (inside F12): - If no predecessor: merge to main trajectory (target_chunk_id="main")
- If chunk has temporal predecessor (previous chunk by frame_id order): merge to predecessor - F11 determines target based on chunk frame_id ordering
- If no predecessor: merge to main trajectory (chunk_id="main") 5. Call F12.merge_chunks(target_chunk_id, chunk_id, transform)
- F12 maintains chunk ordering based on first frame_id in each chunk - Note: `merge_chunks(target, source)` merges source INTO target
5. Call F12.merge_chunks(chunk_id, target_chunk_id, transform) (F12 coordinates with F10) - chunk_id (source) is merged into target_chunk_id
6. F12 handles chunk state updates (deactivation, status updates) 6. F12 handles chunk state updates (deactivation, status updates)
7. F10 optimizes merged graph globally (via F12.merge_chunks()) 7. F10 optimizes merged graph globally (via F12.merge_chunks())
8. **Emit ChunkMerged event** with flight_id and merged_frames 8. **Emit ChunkMerged event** with flight_id and merged_frames
@@ -580,10 +599,25 @@ while flight_active:
- Reduces user input requests - Reduces user input requests
- **Lifecycle**: Starts when flight becomes active, stops when flight completed - **Lifecycle**: Starts when flight becomes active, stops when flight completed
**Chunk Failure Flow**:
When chunk matching fails after trying all candidates:
1. Emit `ChunkMatchingFailed` event with chunk_id and flight_id
2. F02 receives event and decides next action:
- **Option A**: Wait for more frames and retry later (chunk may gain more distinctive features)
- **Option B**: Create user input request for the chunk
3. If user input requested:
- F02 calls F14.publish_user_input_request() with chunk context
- Client receives via SSE
- User provides GPS anchor for one frame in chunk
- F02 receives user fix via handle_user_fix()
- F11.apply_user_anchor() anchors the chunk
- Processing resumes
**Test Cases**: **Test Cases**:
1. **Background matching**: Unanchored chunks matched asynchronously 1. **Background matching**: Unanchored chunks matched asynchronously
2. **Chunk merging**: Chunks merged when matches found 2. **Chunk merging**: Chunks merged when matches found
3. **Non-blocking**: Frame processing continues during matching 3. **Non-blocking**: Frame processing continues during matching
4. **Chunk matching fails**: ChunkMatchingFailed emitted, user input requested if needed
## Integration Tests ## Integration Tests
@@ -53,12 +53,8 @@ class IRouteChunkManager(ABC):
pass pass
@abstractmethod @abstractmethod
def merge_chunks(self, chunk_id_1: str, chunk_id_2: str, transform: Sim3Transform) -> bool: def merge_chunks(self, target_chunk_id: str, source_chunk_id: str, transform: Sim3Transform) -> bool:
pass """Merges source_chunk INTO target_chunk. Result is stored in target_chunk."""
@abstractmethod
def get_merge_target(self, chunk_id: str) -> str:
"""Returns target chunk_id for merging. Returns 'main' for main trajectory."""
pass pass
@abstractmethod @abstractmethod
@@ -468,17 +464,17 @@ bool: True if deactivated successfully
--- ---
### `merge_chunks(chunk_id_1: str, chunk_id_2: str, transform: Sim3Transform) -> bool` ### `merge_chunks(target_chunk_id: str, source_chunk_id: str, transform: Sim3Transform) -> bool`
**Description**: Coordinates chunk merging by validating chunks, calling F10 for factor graph merge, and updating chunk states. **Description**: Merges source_chunk INTO target_chunk. The resulting merged chunk is target_chunk. Source chunk is deactivated after merge.
**Called By**: **Called By**:
- F11 Failure Recovery Coordinator (after successful chunk matching) - F11 Failure Recovery Coordinator (after successful chunk matching)
**Input**: **Input**:
```python ```python
chunk_id_1: str # Source chunk (typically newer, to be merged) target_chunk_id: str # Target chunk (receives the merge, typically older/main)
chunk_id_2: str # Target chunk (typically older, merged into) source_chunk_id: str # Source chunk (being merged in, typically newer)
transform: Sim3Transform: transform: Sim3Transform:
translation: np.ndarray # (3,) translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or quaternion rotation: np.ndarray # (3, 3) or quaternion
@@ -492,34 +488,33 @@ bool: True if merge successful
**Processing Flow**: **Processing Flow**:
1. Verify both chunks exist 1. Verify both chunks exist
2. Verify chunk_id_1 is anchored (has_anchor=True) 2. Verify source_chunk_id is anchored (has_anchor=True)
3. Validate chunks can be merged (not already merged, not same chunk) 3. Validate chunks can be merged (not already merged, not same chunk)
4. **Merge direction**: chunk_id_1 (newer, source) merges INTO chunk_id_2 (older, target) 4. Call F10.merge_chunk_subgraphs(flight_id, source_chunk_id, target_chunk_id, transform)
5. Call F10.merge_chunks(chunk_id_1, chunk_id_2, transform) 5. Update source_chunk_id state:
6. Update chunk_id_1 state:
- Set is_active=False - Set is_active=False
- Set matching_status="merged" - Set matching_status="merged"
- Call deactivate_chunk(chunk_id_1) - Call deactivate_chunk(source_chunk_id)
7. Update chunk_id_2 state (if needed) 6. target_chunk remains active (now contains merged frames)
8. Persist chunk state via F03 Flight Database.save_chunk_state() 7. Persist chunk state via F03 Flight Database.save_chunk_state()
9. Return True 8. Return True
**Merge Convention**:
- `merge_chunks(target, source)` → source is merged INTO target
- Result is stored in target_chunk
- source_chunk is deactivated after merge
- Example: `merge_chunks("main", "chunk_3")` merges chunk_3 into main trajectory
**Validation**: **Validation**:
- Both chunks must exist - Both chunks must exist
- chunk_id_1 must be anchored - source_chunk must be anchored
- chunk_id_1 must not already be merged - source_chunk must not already be merged
- chunk_id_1 and chunk_id_2 must be different - target_chunk and source_chunk must be different
**Merge Direction**:
- **chunk_id_1**: Source chunk (newer, recently anchored)
- **chunk_id_2**: Target chunk (older, main trajectory or previous chunk)
- Newer chunks merge INTO older chunks to maintain chronological consistency
**Test Cases**: **Test Cases**:
1. **Merge anchored chunks**: Chunks merged successfully, chunk_id_1 deactivated 1. **Merge anchored chunk**: source_chunk merged into target_chunk
2. **Merge unanchored chunk**: Returns False (validation fails) 2. **Source deactivated**: source_chunk marked as merged and deactivated
3. **Merge already merged chunk**: Returns False (validation fails) 3. **Target unchanged**: target_chunk remains active with new frames
4. **State updates**: chunk_id_1 marked as merged and deactivated
--- ---
@@ -35,7 +35,7 @@ class ICoordinateTransformer(ABC):
pass pass
@abstractmethod @abstractmethod
def image_object_to_gps(self, object_pixel: Tuple[float, float], frame_id: int) -> GPSPoint: def image_object_to_gps(self, flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint:
pass pass
@abstractmethod @abstractmethod
@@ -273,18 +273,19 @@ Tuple[float, float]: (x, y) pixel coordinates
--- ---
### `image_object_to_gps(object_pixel: Tuple[float, float], frame_id: int) -> GPSPoint` ### `image_object_to_gps(flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint`
**Description**: **Critical method** - Converts object pixel coordinates to GPS. Used for external object detection integration. **Description**: **Critical method** - Converts object pixel coordinates to GPS. Used for external object detection integration.
**Called By**: **Called By**:
- F01 Flight API (via `POST /flights/{flightId}/frames/{frameId}/object-to-gps` endpoint) - F02 Flight Processor (via convert_object_to_gps delegation from F01)
- F14 Result Manager (converts objects to GPS for output) - F14 Result Manager (converts objects to GPS for output)
**Input**: **Input**:
```python ```python
object_pixel: Tuple[float, float] # Pixel coordinates from object detector flight_id: str # Flight identifier (needed for ENU origin and factor graph)
frame_id: int # Frame containing object frame_id: int # Frame containing object
object_pixel: Tuple[float, float] # Pixel coordinates from object detector
``` ```
**Output**: **Output**:
@@ -293,15 +294,17 @@ GPSPoint: GPS coordinates of object center
``` ```
**Processing Flow**: **Processing Flow**:
1. Get frame_pose from F10 Factor Graph 1. Get frame_pose from F10 Factor Graph Optimizer.get_trajectory(flight_id)[frame_id]
2. Get camera_params from F17 Configuration Manager 2. Get camera_params from F17 Configuration Manager.get_flight_config(flight_id)
3. Get altitude from configuration 3. Get altitude from F17 Configuration Manager.get_flight_config(flight_id).altitude
4. Call pixel_to_gps(object_pixel, frame_pose, camera_params, altitude) 4. Call pixel_to_gps(object_pixel, frame_pose, camera_params, altitude)
5. Return GPS 5. Use enu_to_gps(flight_id, enu_point) for final GPS conversion
6. Return GPS
**User Story**: **User Story**:
- External system detects object in UAV image at pixel (1024, 768) - External system detects object in UAV image at pixel (1024, 768)
- Calls image_object_to_gps(frame_id=237, object_pixel=(1024, 768)) - Calls F02.convert_object_to_gps(flight_id="abc", frame_id=237, pixel=(1024, 768))
- F02 delegates to F13.image_object_to_gps(flight_id="abc", frame_id=237, object_pixel=(1024, 768))
- Returns GPSPoint(lat=48.123, lon=37.456) - Returns GPSPoint(lat=48.123, lon=37.456)
- Object GPS can be used for navigation, targeting, etc. - Object GPS can be used for navigation, targeting, etc.
@@ -155,8 +155,9 @@ frame_ids: List[int] # Frames with updated poses
**Processing Flow**: **Processing Flow**:
1. For each frame_id: 1. For each frame_id:
- Get refined pose from F10 Factor Graph Optimizer ENU pose - Get refined pose from F10 Factor Graph Optimizer.get_trajectory(flight_id) → Pose object with ENU position
- Convert ENU pose to GPS via F13 Coordinate Transformer.enu_to_gps(flight_id, enu_pose) - Extract ENU tuple: `enu_tuple = (pose.position[0], pose.position[1], pose.position[2])`
- Convert ENU to GPS via F13 Coordinate Transformer.enu_to_gps(flight_id, enu_tuple) → GPSPoint
- Update result with refined=True via F03 Flight Database.save_frame_result() - Update result with refined=True via F03 Flight Database.save_frame_result()
- Update waypoint via F03 Flight Database.update_waypoint() - Update waypoint via F03 Flight Database.update_waypoint()
- Call F15 SSE Event Streamer.send_refinement() - Call F15 SSE Event Streamer.send_refinement()
@@ -205,8 +206,9 @@ merged_frames: List[int] # Frames whose poses changed due to chunk merge
**Processing Flow**: **Processing Flow**:
1. For each frame_id in merged_frames: 1. For each frame_id in merged_frames:
- Get updated pose from F10 Factor Graph Optimizer.get_trajectory() → ENU pose - Get updated pose from F10 Factor Graph Optimizer.get_trajectory(flight_id) → Pose object with ENU position
- Convert ENU pose to GPS via F13 Coordinate Transformer.enu_to_gps(flight_id, enu_pose) - Extract ENU tuple: `enu_tuple = (pose.position[0], pose.position[1], pose.position[2])`
- Convert ENU to GPS via F13 Coordinate Transformer.enu_to_gps(flight_id, enu_tuple) → GPSPoint
- Update frame result via F03 Flight Database.save_frame_result() - Update frame result via F03 Flight Database.save_frame_result()
- Update waypoint via F03 Flight Database.update_waypoint() - Update waypoint via F03 Flight Database.update_waypoint()
- Send refinement event via F15 SSE Event Streamer.send_refinement() - Send refinement event via F15 SSE Event Streamer.send_refinement()
@@ -28,9 +28,19 @@ class ISSEEventStreamer(ABC):
def send_refinement(self, flight_id: str, frame_id: int, updated_result: FrameResult) -> bool: def send_refinement(self, flight_id: str, frame_id: int, updated_result: FrameResult) -> bool:
pass pass
@abstractmethod
def send_heartbeat(self, flight_id: str) -> bool:
"""Sends heartbeat/keepalive to all clients subscribed to flight."""
pass
@abstractmethod @abstractmethod
def close_stream(self, flight_id: str, client_id: str) -> bool: def close_stream(self, flight_id: str, client_id: str) -> bool:
pass pass
@abstractmethod
def get_active_connections(self, flight_id: str) -> int:
"""Returns count of active SSE connections for a flight."""
pass
``` ```
## Component Description ## Component Description
@@ -170,12 +180,50 @@ StreamConnection:
--- ---
### `send_heartbeat(flight_id: str) -> bool`
**Description**: Sends heartbeat/keepalive ping to all clients subscribed to a flight. Keeps connections alive and helps detect stale connections.
**Called By**:
- Background heartbeat task (every 30 seconds)
- F02 Flight Processor (periodically during processing)
**Event Format**:
```
:heartbeat
```
**Behavior**:
- Sends SSE comment (`:heartbeat`) which doesn't trigger event handlers
- Keeps TCP connection alive
- Client can use to detect connection health
**Test Cases**:
1. Send heartbeat → all clients receive ping
2. Client timeout → connection marked stale
---
### `close_stream(flight_id: str, client_id: str) -> bool` ### `close_stream(flight_id: str, client_id: str) -> bool`
**Description**: Closes SSE connection. **Description**: Closes SSE connection.
**Called By**: F01 REST API (on client disconnect) **Called By**: F01 REST API (on client disconnect)
---
### `get_active_connections(flight_id: str) -> int`
**Description**: Returns count of active SSE connections for a flight.
**Called By**:
- F02 Flight Processor (monitoring)
- Admin tools
**Test Cases**:
1. No connections → returns 0
2. 5 clients connected → returns 5
## Integration Tests ## Integration Tests
### Test 1: Real-Time Streaming ### Test 1: Real-Time Streaming
@@ -27,6 +27,21 @@ class IConfigurationManager(ABC):
@abstractmethod @abstractmethod
def update_config(self, section: str, key: str, value: Any) -> bool: def update_config(self, section: str, key: str, value: Any) -> bool:
pass pass
@abstractmethod
def get_operational_altitude(self, flight_id: str) -> float:
"""Returns predefined operational altitude for the flight (NOT from EXIF)."""
pass
@abstractmethod
def get_frame_spacing(self, flight_id: str) -> float:
"""Returns expected distance between consecutive frames in meters."""
pass
@abstractmethod
def save_flight_config(self, flight_id: str, config: FlightConfig) -> bool:
"""Persists flight-specific configuration."""
pass
``` ```
## Component Description ## Component Description
@@ -136,6 +151,62 @@ FlightConfig:
1. Update value → succeeds 1. Update value → succeeds
2. Invalid key → fails 2. Invalid key → fails
---
### `get_operational_altitude(flight_id: str) -> float`
**Description**: Returns predefined operational altitude for the flight in meters. This is the altitude provided during flight creation, NOT extracted from EXIF metadata (images don't have GPS/altitude metadata per problem constraints).
**Called By**:
- F10 Factor Graph Optimizer (for scale resolution)
- H02 GSD Calculator (for GSD computation)
- F09 Metric Refinement (for alignment)
**Input**: `flight_id: str`
**Output**: `float` - Altitude in meters (typically 100-500m)
**Test Cases**:
1. Get existing flight altitude → returns value
2. Non-existent flight → raises error
---
### `get_frame_spacing(flight_id: str) -> float`
**Description**: Returns expected distance between consecutive frames in meters. Used for scale estimation in visual odometry.
**Called By**:
- F10 Factor Graph Optimizer (for expected displacement calculation)
**Input**: `flight_id: str`
**Output**: `float` - Expected frame spacing in meters (typically ~100m)
**Test Cases**:
1. Get frame spacing → returns expected distance
---
### `save_flight_config(flight_id: str, config: FlightConfig) -> bool`
**Description**: Persists flight-specific configuration when a flight is created.
**Called By**:
- F02 Flight Processor (during flight creation)
**Input**:
```python
flight_id: str
config: FlightConfig
```
**Output**: `bool` - True if saved successfully
**Test Cases**:
1. Save valid config → succeeds
2. Invalid flight_id → fails
## Data Models ## Data Models
### SystemConfig ### SystemConfig
+48 -3
View File
@@ -189,20 +189,27 @@
| Client | F01 | `POST /flights` | Create flight | | Client | F01 | `POST /flights` | Create flight |
| F01 | F02 | `create_flight()` | Initialize flight state | | F01 | F02 | `create_flight()` | Initialize flight state |
| F02 | F16 | `get_flight_config()` | Get camera params, altitude | | F02 | F16 | `get_flight_config()` | Get camera params, altitude |
| F02 | F12 | `set_enu_origin(start_gps)` | Set ENU coordinate origin | | F02 | F13 | `set_enu_origin(flight_id, start_gps)` | Set ENU coordinate origin |
| F02 | F04 | `prefetch_route_corridor()` | Prefetch tiles | | F02 | F04 | `prefetch_route_corridor()` | Prefetch tiles |
| F04 | Satellite Provider | `GET /api/satellite/tiles/batch` | HTTP batch download | | F04 | Satellite Provider | `GET /api/satellite/tiles/batch` | HTTP batch download |
| F04 | H06 | `compute_tile_bounds()` | Tile coordinate calculations | | F04 | H06 | `compute_tile_bounds()` | Tile coordinate calculations |
| F02 | F03 | `insert_flight()` | Persist flight data | | F02 | F03 | `insert_flight()` | Persist flight data |
### SSE Stream Creation
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `GET .../stream` | Open SSE connection | | Client | F01 | `GET .../stream` | Open SSE connection |
| F01 | F14 | `create_stream()` | Establish SSE channel | | F01 | F02 | `create_client_stream()` | Route through coordinator |
| F02 | F15 | `create_stream()` | Establish SSE channel |
### Image Upload ### Image Upload
| Source | Target | Method | Purpose | | Source | Target | Method | Purpose |
|--------|--------|--------|---------| |--------|--------|--------|---------|
| Client | F01 | `POST .../images/batch` | Upload 10-50 images | | Client | F01 | `POST .../images/batch` | Upload 10-50 images |
| F01 | F05 | `queue_batch()` | Queue for processing | | F01 | F02 | `queue_images()` | Route through coordinator |
| F02 | F05 | `queue_batch()` | Queue for processing |
| F05 | H08 | `validate_batch()` | Validate sequence, format | | F05 | H08 | `validate_batch()` | Validate sequence, format |
| F05 | F03 | `save_image_metadata()` | Persist image metadata | | F05 | F03 | `save_image_metadata()` | Persist image metadata |
@@ -433,3 +440,41 @@ F02 Flight Processor handles multiple concerns:
**Event Recovery**: **Event Recovery**:
- Events are fire-and-forget (no persistence) - Events are fire-and-forget (no persistence)
- Subscribers rebuild state from F03 on restart - Subscribers rebuild state from F03 on restart
### Error Propagation Strategy
**Principle**: Errors propagate upward through the component hierarchy. Lower-level components throw exceptions or return error results; higher-level coordinators handle recovery.
**Error Propagation Chain**:
```
H01-H08 (Helpers) → F07/F08/F09 (Visual Processing) → F11 (Recovery) → F02 (Coordinator) → F01 (API) → Client
Events → F14 → F15 → SSE → Client
```
**Error Categories**:
1. **Recoverable** (F11 handles):
- Tracking loss → Progressive search
- Low confidence → Rotation sweep
- Chunk matching fails → User input request
2. **Propagated to Client** (via SSE):
- User input needed → `user_input_needed` event
- Processing blocked → `processing_blocked` event
- Flight completed → `flight_completed` event
3. **Fatal** (F02 handles, returns to F01):
- Database connection lost → HTTP 503
- Model loading failed → HTTP 500
- Invalid configuration → HTTP 400
**User Fix Flow**:
```
Client ---> F01 (POST /user-fix) ---> F02.handle_user_fix() ---> F11.apply_user_anchor()
F10.add_chunk_anchor()
F02 receives UserFixApplied event
F14.publish_result() ---> F15 ---> SSE ---> Client
```
@@ -27,11 +27,25 @@ class IFaissIndexManager(ABC):
@abstractmethod @abstractmethod
def load_index(self, path: str) -> FaissIndex: def load_index(self, path: str) -> FaissIndex:
pass pass
@abstractmethod
def is_gpu_available(self) -> bool:
pass
@abstractmethod
def set_device(self, device: str) -> bool:
"""Set device: 'gpu' or 'cpu'."""
pass
``` ```
## Component Description ## Component Description
Manages Faiss indices for AnyLoc retrieval (IVF, HNSW options). Manages Faiss indices for DINOv2 descriptor similarity search. H04 builds indexes from UAV image descriptors for:
1. **Loop closure detection**: Find when UAV revisits previously seen areas within the same flight
2. **Chunk-to-chunk matching**: Match disconnected chunks to each other
3. **Flight-to-flight matching**: Match current flight to previous flights in same area
**Index Source**: Descriptors are computed from UAV images using F08's DINOv2 encoder, NOT from satellite images. The index enables finding similar UAV viewpoints.
## API Methods ## API Methods
@@ -76,9 +90,18 @@ Manages Faiss indices for AnyLoc retrieval (IVF, HNSW options).
**External**: faiss-gpu or faiss-cpu **External**: faiss-gpu or faiss-cpu
## GPU/CPU Fallback
H04 supports automatic fallback from GPU to CPU:
- `is_gpu_available()`: Returns True if faiss-gpu is available and CUDA works
- `set_device("gpu")`: Use GPU acceleration (faster for large indexes)
- `set_device("cpu")`: Use CPU (fallback when GPU unavailable)
## Test Cases ## Test Cases
1. Build index with 10,000 descriptors → succeeds 1. Build index with 10,000 UAV image descriptors → succeeds
2. Search query → returns top-k matches 2. Search query UAV descriptor → returns top-k similar UAV frames
3. Save/load index → index restored correctly 3. Save/load index → index restored correctly
4. GPU unavailable → automatically falls back to CPU
5. Add descriptors incrementally → index grows correctly
+25 -25
View File
@@ -185,7 +185,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
│ F02 Flight Processor │ │ F02 Flight Processor │
│ │ │ │
│ ┌─────────────┐ │ │ ┌─────────────┐ │
@@ -265,14 +265,14 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌───────────────────────────────────────────────────────────────────────────
│ F06 Image Rotation Manager │ │ F06 Image Rotation Manager │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
│ │ try_rotation_steps(image, satellite_tile, tile_bounds) │ │ │ │ try_rotation_steps(image, satellite_tile, tile_bounds) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │ │ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │ │ │ │
│ ┌───────────────────────┴───────────────────────────┐ │ │ ┌───────────────────────┴───────────────────────────┐ │
│ │ For angle in [0°, 30°, 60°, ... 330°]: │ │ │ │ For angle in [0°, 30°, 60°, ... 330°]: │ │
│ │ │ │ │ │ │ │
│ │ ┌─────────────────────────────────────┐ │ │ │ │ ┌─────────────────────────────────────┐ │ │
@@ -304,7 +304,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ │ Return RotationResult │ │ │ │ Return RotationResult │ │
│ │ or None │ │ │ │ or None │ │
│ └─────────────────────────┘ │ │ └─────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────────────────────
``` ```
**Output**: RotationResult with precise heading angle **Output**: RotationResult with precise heading angle
@@ -319,7 +319,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
│ F11 Failure Recovery Coordinator │ │ F11 Failure Recovery Coordinator │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -353,12 +353,12 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ ┌────────────────────┴────────────────────┐ │ │ ┌────────────────────┴────────────────────┐ │
│ │ Single-image match found? │ │ │ │ Single-image match found? │ │
│ ▼ ▼ │ │ ▼ ▼ │
│ ┌─────────────────────┐ ┌────────────────────────────┐ │ │ ┌─────────────────────┐ ┌────────────────────────────┐
│ │ EMIT RecoverySucceeded│ │ Continue chunk building │ │ │ │ EMIT RecoverySucceeded│ │ Continue chunk building │ │
│ │ Resume normal flow │ │ → Flow 7 (Chunk Building) │ │ │ │ Resume normal flow │ │ → Flow 7 (Chunk Building) │ │
│ └─────────────────────┘ │ → Flow 8 (Chunk Matching) │ │ │ └─────────────────────┘ │ → Flow 8 (Chunk Matching) │ │
│ │ (Background) │ │ │ │ (Background) │ │
└─────────────────────────────┘ │ ────────────────────────────┘
│ │ │ │ │ │
│ ▼ │ │ ▼ │
│ ┌─────────────────────────────────────┐ │ │ ┌─────────────────────────────────────┐ │
@@ -376,7 +376,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌───────────────────────────────────────────────────────────────────────────
│ F12 Route Chunk Manager │ │ F12 Route Chunk Manager │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -386,17 +386,17 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ │ └─ F03 save_chunk_state() │ │ │ │ └─ F03 save_chunk_state() │ │
│ └──────────────────────────────┬──────────────────────────────┘ │ │ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │ │ │ │
│ ┌───────────────────────┴───────────────────────────┐ │ │ ┌───────────────────────┴───────────────────────────┐ │
│ │ For each frame in chunk: │ │ │ │ For each frame in chunk: │ │
│ │ │ │ │ │ │ │
│ │ ┌─────────────────────────────────────┐ │ │ │ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F07 compute_relative_pose_in_chunk()│ │ │ │ │ │ F07 compute_relative_pose_in_chunk()│ │ │
│ │ └───────────────────┬─────────────────┘ │ │ │ │ └───────────────────┬─────────────────┘ │ │
│ │ ▼ │ │ │ │ ▼ │ │
│ │ ┌─────────────────────────────────────┐ │ │ │ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F12 add_frame_to_chunk() │ │ │ │ │ │ F12 add_frame_to_chunk() │ │ │
│ │ │ └─ F10 add_relative_factor_to_chunk│ │ │ │ │ │ └─ F10 add_relative_factor_to_chunk│ │ │
│ │ └───────────────────┬─────────────────┘ │ │ │ │ └───────────────────┬─────────────────┘ │ │
│ │ ▼ │ │ │ │ ▼ │ │
│ │ ┌─────────────────────────────────────┐ │ │ │ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F10 optimize_chunk() (local) │ │ │ │ │ │ F10 optimize_chunk() (local) │ │ │
@@ -411,7 +411,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ │ ├─ Max 20 frames │ │ │ │ ├─ Max 20 frames │ │
│ │ └─ Internal consistency (good VO inlier counts) │ │ │ │ └─ Internal consistency (good VO inlier counts) │ │
│ └─────────────────────────────────────────────────────────────┘ │ │ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────────────────────
``` ```
--- ---
@@ -424,7 +424,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
│ F11 process_unanchored_chunks() (Background Task) │ │ F11 process_unanchored_chunks() (Background Task) │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -472,7 +472,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌───────────────────────────────────────────────────────────────────────-──┐
│ F11 merge_chunk_to_trajectory() │ │ F11 merge_chunk_to_trajectory() │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -485,7 +485,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ └──────────────────────────────┬──────────────────────────────┘ │ │ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │ │ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 3. Resolve target: F12 get_merge_target(chunk_id) │ │ │ │ 3. Determine target chunk (predecessor or "main") │ │
│ │ └─ Returns target_chunk_id (predecessor or "main") │ │ │ │ └─ Returns target_chunk_id (predecessor or "main") │ │
│ └──────────────────────────────┬──────────────────────────────┘ │ │ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │ │ ▼ │
@@ -523,7 +523,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
│ F11 create_user_input_request() │ │ F11 create_user_input_request() │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -537,7 +537,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
└──────────────────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────────────────────┘
▼ Client receives SSE event ▼ Client receives SSE event
┌──────────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────────────
│ Client UI │ │ Client UI │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -550,7 +550,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
│ │ POST /flights/{flightId}/user-fix │ │ │ │ POST /flights/{flightId}/user-fix │ │
│ │ body: { frame_id, uav_pixel, satellite_gps } │ │ │ │ body: { frame_id, uav_pixel, satellite_gps } │ │
│ └─────────────────────────────────────────────────────────────┘ │ │ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────────────────────
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────┐
@@ -577,7 +577,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────-────────────┐
│ F10 Background Optimization │ │ F10 Background Optimization │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -650,7 +650,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
**Sequence**: **Sequence**:
``` ```
┌─────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────
│ F02 Flight Processor │ │ F02 Flight Processor │
│ │ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │ │ ┌─────────────────────────────────────────────────────────────┐ │
@@ -797,7 +797,7 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
┌─────────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────────┐
│ STATE ESTIMATION LAYER │ │ STATE ESTIMATION LAYER │
│ ┌────────────────────────────────────────────────────────────────────┐ │ │ ┌──────────────────────────────────────────────────────────────────k──┐ │
│ │ F10 Factor Graph Optimizer │ │ │ │ F10 Factor Graph Optimizer │ │
│ │ (GTSAM, iSAM2, Robust kernels, Chunk subgraphs, Sim(3) merging) │ │ │ │ (GTSAM, iSAM2, Robust kernels, Chunk subgraphs, Sim(3) merging) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │ └─────────────────────────────────────────────────────────────────────┘ │
@@ -815,14 +815,14 @@ ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer match
└─────────────────────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────────-
│ INFRASTRUCTURE LAYER │ │ INFRASTRUCTURE LAYER │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────────────────┐│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────────────────-┐│
│ │ F16 │ │ F17 │ │ HELPERS ││ │ │ F16 │ │ F17 │ │ HELPERS ││
│ │ Model │ │Configuration │ │ H01-H08 (Camera, GSD, Kernels, ││ │ │ Model │ │Configuration │ │ H01-H08 (Camera, GSD, Kernels, ││
│ │ Manager │ │ Manager │ │ Faiss, Monitor, Mercator, etc) ││ │ │ Manager │ │ Manager │ │ Faiss, Monitor, Mercator, etc) ││
│ └───────────────┘ └───────────────┘ └───────────────────────────────────┘│ │ └───────────────┘ └───────────────┘ └───────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────────────────────
``` ```
--- ---