azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-04-22 22:46:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

initial structure implemented

Browse Source 
docs -> _docs
This commit is contained in:
Oleksandr Bezdieniezhnykh
2025-12-01 14:20:56 +02:00
parent 9134c5db06
commit abc26d5c20
360 changed files with 3881 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_components/01_flight_api/01.01_feature_flight_management.md
+80
View File
@@ -0,0 +1,80 @@
# Feature: Flight Management
## Description
Core REST endpoints for flight lifecycle management including CRUD operations, status retrieval, and waypoint management. This feature provides the fundamental data operations for flight entities.
## Component APIs Implemented
- `create_flight(flight_data: FlightCreateRequest) -> FlightResponse`
- `get_flight(flight_id: str) -> FlightDetailResponse`
- `delete_flight(flight_id: str) -> DeleteResponse`
- `get_flight_status(flight_id: str) -> FlightStatusResponse`
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse`
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights` | Create new flight |
| GET | `/flights/{flightId}` | Get flight details |
| DELETE | `/flights/{flightId}` | Delete flight |
| GET | `/flights/{flightId}/status` | Get processing status |
| PUT | `/flights/{flightId}/waypoints/{waypointId}` | Update single waypoint |
| PUT | `/flights/{flightId}/waypoints/batch` | Batch update waypoints |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **Pydantic**: Request/response validation and serialization
- **Uvicorn**: ASGI server
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_gps_coordinates(lat, lon)` | Validate GPS coordinate ranges |
| `_validate_camera_params(params)` | Validate camera parameter values |
| `_validate_geofences(geofences)` | Validate geofence polygon data |
| `_build_flight_response(flight_data)` | Build response from F02 result |
| `_build_status_response(status_data)` | Build status response |
## Unit Tests
1. **create_flight validation**
- Valid request → calls F02.create_flight() and returns 201
- Missing required field (name) → returns 400
- Invalid GPS (lat > 90) → returns 400
- Invalid camera params → returns 400
2. **get_flight**
- Valid flight_id → returns 200 with flight data
- Non-existent flight_id → returns 404
3. **delete_flight**
- Valid flight_id → returns 200
- Non-existent flight_id → returns 404
- Processing flight → returns 409
4. **get_flight_status**
- Valid flight_id → returns 200 with status
- Non-existent flight_id → returns 404
5. **update_waypoint**
- Valid update → returns 200
- Invalid waypoint_id → returns 404
- Invalid coordinates → returns 400
6. **batch_update_waypoints**
- Valid batch → returns success with updated_count
- Empty batch → returns success, updated_count=0
- Partial failures → returns failed_ids
## Integration Tests
1. **Flight lifecycle**
- POST /flights → GET /flights/{id} → DELETE /flights/{id}
- Verify data consistency across operations
2. **Waypoint updates during processing**
- Create flight → Process frames → Verify waypoints updated
- Simulate refinement updates → Verify refined=true
3. **Concurrent operations**
- Create 10 flights concurrently → All succeed
- Batch update 500 waypoints → All succeed
_docs/02_components/01_flight_api/01.02_feature_image_upload.md
+70
View File
@@ -0,0 +1,70 @@
# Feature: Image Batch Upload
## Description
REST endpoint for uploading batches of UAV images for processing. Handles multipart form data with 10-50 images per request, validates sequence numbers, and queues images for async processing via F02.
## Component APIs Implemented
- `upload_image_batch(flight_id: str, batch: ImageBatch) -> BatchResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights/{flightId}/images/batch` | Upload batch of 10-50 images |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **python-multipart**: Multipart form data handling
- **Pydantic**: Validation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_batch_size(images)` | Validate batch contains 10-50 images |
| `_validate_sequence_numbers(metadata)` | Validate start/end sequence are valid |
| `_validate_sequence_continuity(flight_id, start_seq)` | Validate sequence continues from last batch |
| `_parse_multipart_images(request)` | Parse multipart form data into image list |
| `_validate_image_format(image)` | Validate image file is valid JPEG/PNG |
| `_build_batch_response(result)` | Build response with accepted sequences |
## Unit Tests
1. **Batch size validation**
- 20 images → accepted
- 9 images → returns 400 (too few)
- 51 images → returns 400 (too many)
2. **Sequence validation**
- Valid sequence (start=100, end=119) → accepted
- Invalid sequence (start > end) → returns 400
- Gap in sequence → returns 400
3. **Flight validation**
- Valid flight_id → accepted
- Non-existent flight_id → returns 404
4. **Image validation**
- Valid JPEG images → accepted
- Corrupted image → returns 400
- Non-image file → returns 400
5. **Size limits**
- 50 × 2MB images → accepted
- Batch > 500MB → returns 413
## Integration Tests
1. **Sequential batch uploads**
- Upload batch 1 (seq 0-49) → success
- Upload batch 2 (seq 50-99) → success
- Verify next_expected increments correctly
2. **Large image handling**
- Upload 50 × 8MB images → success within timeout
- Verify all images queued for processing
3. **Rate limiting**
- Rapid consecutive uploads → eventually returns 429
- Wait and retry → succeeds
4. **Concurrent uploads to same flight**
- Two clients upload different batches → both succeed
- Verify sequence integrity maintained
_docs/02_components/01_flight_api/01.03_feature_user_interaction.md
+68
View File
@@ -0,0 +1,68 @@
# Feature: User Interaction
## Description
REST endpoints for user-triggered operations: submitting GPS fixes for blocked flights and converting detected object pixel coordinates to GPS. These endpoints support the human-in-the-loop workflow when automated localization fails.
## Component APIs Implemented
- `submit_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResponse`
- `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/flights/{flightId}/user-fix` | Submit user-provided GPS anchor |
| POST | `/flights/{flightId}/frames/{frameId}/object-to-gps` | Convert pixel to GPS |
## External Tools and Services
- **FastAPI**: Web framework for REST endpoints
- **Pydantic**: Request/response validation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_user_fix_request(fix_data)` | Validate pixel and GPS coordinates |
| `_validate_flight_blocked(flight_id)` | Verify flight is in blocked state |
| `_validate_frame_processed(flight_id, frame_id)` | Verify frame has pose in Factor Graph |
| `_validate_pixel_coordinates(pixel, resolution)` | Validate pixel within image bounds |
| `_build_user_fix_response(result)` | Build response with processing status |
| `_build_object_gps_response(result)` | Build GPS response with accuracy |
## Unit Tests
1. **submit_user_fix validation**
- Valid request for blocked flight → returns 200, processing_resumed=true
- Flight not blocked → returns 409
- Invalid GPS coordinates → returns 400
- Non-existent flight_id → returns 404
2. **submit_user_fix pixel validation**
- Pixel within image bounds → accepted
- Negative pixel coordinates → returns 400
- Pixel outside image bounds → returns 400
3. **convert_object_to_gps validation**
- Valid processed frame → returns GPS with accuracy
- Frame not yet processed → returns 409
- Non-existent frame_id → returns 404
- Invalid pixel coordinates → returns 400
4. **convert_object_to_gps accuracy**
- High confidence frame → low accuracy_meters
- Low confidence frame → high accuracy_meters
## Integration Tests
1. **User fix unblocks processing**
- Process until blocked → Submit user fix → Verify processing resumes
- Verify SSE `processing_resumed` event sent
2. **Object-to-GPS workflow**
- Process flight → Call object-to-gps for multiple pixels
- Verify GPS coordinates are spatially consistent
3. **User fix with invalid anchor**
- Submit fix with GPS far outside geofence
- Verify appropriate error handling
4. **Concurrent object-to-gps calls**
- Multiple clients request conversion simultaneously
- All receive correct responses
_docs/02_components/01_flight_api/01.04_feature_sse_streaming.md
+88
View File
@@ -0,0 +1,88 @@
# Feature: SSE Streaming
## Description
Server-Sent Events (SSE) endpoint for real-time streaming of flight processing results to clients. Manages SSE connections, handles keepalive, and supports client reconnection with event replay.
## Component APIs Implemented
- `create_sse_stream(flight_id: str) -> SSEStream`
## REST Endpoints
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/flights/{flightId}/stream` | Open SSE connection |
## SSE Events
| Event Type | Description |
|------------|-------------|
| `frame_processed` | New frame GPS calculated |
| `frame_refined` | Previous frame GPS refined |
| `search_expanded` | Search grid expanded |
| `user_input_needed` | User input required |
| `processing_blocked` | Processing halted |
| `flight_completed` | Flight processing finished |
## External Tools and Services
- **FastAPI**: Web framework with SSE support
- **sse-starlette**: SSE event streaming library
- **asyncio**: Async event handling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_validate_flight_exists(flight_id)` | Verify flight exists before connecting |
| `_create_sse_response(flight_id)` | Create SSE StreamingResponse |
| `_event_generator(flight_id, client_id)` | Async generator yielding events |
| `_handle_client_disconnect(client_id)` | Cleanup on client disconnect |
| `_send_keepalive()` | Send ping every 30 seconds |
| `_replay_missed_events(last_event_id)` | Replay events since last_event_id |
| `_format_sse_event(event_type, data)` | Format event for SSE protocol |
## Unit Tests
1. **Connection establishment**
- Valid flight_id → SSE connection opens
- Non-existent flight_id → returns 404
- Invalid flight_id format → returns 400
2. **Event formatting**
- frame_processed event → correct JSON structure
- All event types → valid SSE format with id, event, data
3. **Keepalive**
- No events for 30s → ping sent
- Connection stays alive during idle periods
4. **Client disconnect handling**
- Client closes connection → resources cleaned up
- No memory leaks on disconnect
## Integration Tests
1. **Full event flow**
- Connect to stream → Upload images → Receive all frame_processed events
- Verify event count matches frames processed
2. **Event ordering**
- Process 100 frames → events received in order
- Event IDs are sequential
3. **Reconnection with replay**
- Connect → Process 50 frames → Disconnect → Reconnect with last_event_id
- Receive missed events since last_event_id
4. **user_input_needed flow**
- Process until blocked → Receive user_input_needed event
- Submit user fix → Receive processing_resumed confirmation
5. **Multiple concurrent clients**
- 10 clients connect to same flight
- All receive same events
- No cross-contamination between flights
6. **Long-running stream**
- Stream 2000 frames over extended period
- Connection remains stable
- All events delivered
7. **Flight completion**
- Process all frames → Receive flight_completed event
- Stream closes gracefully
_docs/02_components/01_flight_api/01._component_light_api.md
+704
View File
@@ -0,0 +1,704 @@
# Flight API
## Interface Definition
**Interface Name**: `IFlightAPI`
### Interface Methods
```python
class IFlightAPI(ABC):
@abstractmethod
def create_flight(self, flight_data: FlightCreateRequest) -> FlightResponse:
pass
@abstractmethod
def get_flight(self, flight_id: str) -> FlightDetailResponse:
pass
@abstractmethod
def delete_flight(self, flight_id: str) -> DeleteResponse:
pass
@abstractmethod
def update_waypoint(self, flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse:
pass
@abstractmethod
def batch_update_waypoints(self, flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse:
pass
@abstractmethod
def upload_image_batch(self, flight_id: str, batch: ImageBatch) -> BatchResponse:
pass
@abstractmethod
def submit_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResponse:
pass
@abstractmethod
def get_flight_status(self, flight_id: str) -> FlightStatusResponse:
pass
@abstractmethod
def create_sse_stream(self, flight_id: str) -> SSEStream:
pass
@abstractmethod
def convert_object_to_gps(self, flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse:
pass
```
## Component Description
### Responsibilities
- Expose REST API endpoints for complete flight lifecycle management
- Handle flight CRUD operations (create, read, update, delete)
- Manage waypoints and geofences within flights
- Handle satellite data prefetching on flight creation
- Accept batch image uploads (10-50 images per request)
- Accept user-provided GPS fixes for blocked flights
- Provide real-time status updates
- Stream results via Server-Sent Events (SSE)
### Scope
- FastAPI-based REST endpoints
- Request/response validation
- Coordinate with Flight Processor for all operations
- Multipart form data handling for image uploads
- SSE connection management
- Authentication and rate limiting
---
## Flight Management Endpoints
### `create_flight(flight_data: FlightCreateRequest) -> FlightResponse`
**REST Endpoint**: `POST /flights`
**Description**: Creates a new flight with initial waypoints, geofences, camera parameters, and triggers satellite data prefetching.
**Called By**:
- Client applications (Flight UI, Mission Planner UI)
**Input**:
```python
FlightCreateRequest:
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
**Output**:
```python
FlightResponse:
flight_id: str
status: str # "prefetching", "ready", "error"
message: Optional[str]
created_at: datetime
```
**Processing Flow**:
1. Validate request data
2. Call F02 Flight Processor → create_flight()
3. Flight Processor triggers satellite prefetch
4. Return flight_id immediately (prefetch is async)
**Error Conditions**:
- `400 Bad Request`: Invalid input data (missing required fields, invalid GPS coordinates)
- `409 Conflict`: Flight with same ID already exists
- `500 Internal Server Error`: Database or internal error
**Test Cases**:
1. **Valid flight creation**: Provide valid flight data → returns 201 with flight_id
2. **Missing required field**: Omit name → returns 400 with error message
3. **Invalid GPS coordinates**: Provide lat > 90 → returns 400
4. **Concurrent flight creation**: Multiple flights → all succeed
---
### `get_flight(flight_id: str) -> FlightDetailResponse`
**REST Endpoint**: `GET /flights/{flightId}`
**Description**: Retrieves complete flight information including all waypoints, geofences, and processing status.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightDetailResponse:
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: Flight ID does not exist
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Existing flight**: Valid flightId → returns 200 with complete flight data
2. **Non-existent flight**: Invalid flightId → returns 404
3. **Flight with many waypoints**: Flight with 2000+ waypoints → returns 200 with all data
---
### `delete_flight(flight_id: str) -> DeleteResponse`
**REST Endpoint**: `DELETE /flights/{flightId}`
**Description**: Deletes a flight and all associated waypoints, images, and processing data.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
DeleteResponse:
deleted: bool
flight_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight does not exist
- `409 Conflict`: Flight is currently being processed
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Delete existing flight**: Valid flightId → returns 200
2. **Delete non-existent flight**: Invalid flightId → returns 404
3. **Delete processing flight**: Active processing → returns 409
---
### `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/{waypointId}`
**Description**: Updates a specific waypoint within a flight. Used for per-frame GPS refinement.
**Called By**:
- Internal (F13 Result Manager for per-frame updates)
- Client applications (manual corrections)
**Input**:
```python
flight_id: str
waypoint_id: str
waypoint: Waypoint:
lat: float
lon: float
altitude: Optional[float]
confidence: float
timestamp: datetime
refined: bool
```
**Output**:
```python
UpdateResponse:
updated: bool
waypoint_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight or waypoint not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Update existing waypoint**: Valid data → returns 200
2. **Refinement update**: Refined coordinates → updates successfully
3. **Invalid coordinates**: lat > 90 → returns 400
4. **Non-existent waypoint**: Invalid waypoint_id → returns 404
---
### `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/batch`
**Description**: Updates multiple waypoints in a single request. Used for trajectory refinements.
**Called By**:
- Internal (F13 Result Manager for asynchronous refinement updates)
**Input**:
```python
flight_id: str
waypoints: List[Waypoint]
```
**Output**:
```python
BatchUpdateResponse:
success: bool
updated_count: int
failed_ids: List[str]
```
**Error Conditions**:
- `404 Not Found`: Flight not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Batch update 100 waypoints**: All succeed
2. **Partial failure**: 5 waypoints fail → returns failed_ids
3. **Empty batch**: Returns success=True, updated_count=0
4. **Large batch**: 500 waypoints → succeeds
---
## Image Processing Endpoints
### `upload_image_batch(flight_id: str, batch: ImageBatch) -> BatchResponse`
**REST Endpoint**: `POST /flights/{flightId}/images/batch`
**Description**: Uploads a batch of 10-50 UAV images for processing.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
ImageBatch: multipart/form-data
images: List[UploadFile]
metadata: BatchMetadata
start_sequence: int
end_sequence: int
```
**Output**:
```python
BatchResponse:
accepted: bool
sequences: List[int]
next_expected: int
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists
2. Validate batch size (10-50 images)
3. Validate sequence numbers (strict sequential)
4. Call F02 Flight Processor → queue_images(flight_id, batch)
5. F02 delegates to F05 Image Input Pipeline
6. Return immediately (processing is async)
**Error Conditions**:
- `400 Bad Request`: Invalid batch size, out-of-sequence images
- `404 Not Found`: flight_id doesn't exist
- `413 Payload Too Large`: Batch exceeds size limit
- `429 Too Many Requests`: Rate limit exceeded
**Test Cases**:
1. **Valid batch upload**: 20 images → returns 202 Accepted
2. **Out-of-sequence batch**: Sequence gap detected → returns 400
3. **Too many images**: 60 images → returns 400
4. **Large images**: 50 × 8MB images → successfully uploads
---
### `submit_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResponse`
**REST Endpoint**: `POST /flights/{flightId}/user-fix`
**Description**: Submits user-provided GPS anchor point to unblock failed localization.
**Called By**:
- Client applications (when user responds to `user_input_needed` event)
**Input**:
```python
UserFixRequest:
frame_id: int
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**:
```python
UserFixResponse:
accepted: bool
processing_resumed: bool
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists and is blocked
2. Call F02 Flight Processor → handle_user_fix(flight_id, fix_data)
3. F02 delegates to F11 Failure Recovery Coordinator
4. Coordinator applies anchor to Factor Graph
5. Resume processing pipeline
**Error Conditions**:
- `400 Bad Request`: Invalid fix data
- `404 Not Found`: flight_id or frame_id not found
- `409 Conflict`: Flight not in blocked state
**Test Cases**:
1. **Valid user fix**: Blocked flight → returns 200, processing resumes
2. **Fix for non-blocked flight**: Returns 409
3. **Invalid GPS coordinates**: Returns 400
---
### `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> ObjectGPSResponse`
**REST Endpoint**: `POST /flights/{flightId}/frames/{frameId}/object-to-gps`
**Description**: Converts object pixel coordinates to GPS. Used by external object detection systems (e.g., Azaion.Inference) to get GPS coordinates for detected objects.
**Called By**:
- External object detection systems (Azaion.Inference)
- Any system needing pixel-to-GPS conversion for a specific frame
**Input**:
```python
ObjectToGPSRequest:
pixel_x: float # X coordinate in image
pixel_y: float # Y coordinate in image
```
**Output**:
```python
ObjectGPSResponse:
gps: GPSPoint
accuracy_meters: float # Estimated accuracy
frame_id: int
pixel: Tuple[float, float]
```
**Processing Flow**:
1. Validate flight_id and frame_id exist
2. Validate frame has been processed (has pose in Factor Graph)
3. Call F02 Flight Processor → convert_object_to_gps(flight_id, frame_id, pixel)
4. F02 delegates to F13.image_object_to_gps(flight_id, frame_id, pixel)
5. Return GPS with accuracy estimate
**Error Conditions**:
- `400 Bad Request`: Invalid pixel coordinates
- `404 Not Found`: flight_id or frame_id not found
- `409 Conflict`: Frame not yet processed (no pose available)
**Test Cases**:
1. **Valid conversion**: Object at (1024, 768) → returns GPS
2. **Unprocessed frame**: Frame not in Factor Graph → returns 409
3. **Invalid pixel**: Negative coordinates → returns 400
---
### `get_flight_status(flight_id: str) -> FlightStatusResponse`
**REST Endpoint**: `GET /flights/{flightId}/status`
**Description**: Retrieves current processing status of a flight.
**Called By**:
- Client applications (polling for status)
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightStatusResponse:
status: str # "prefetching", "ready", "processing", "blocked", "completed", "failed"
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
**Test Cases**:
1. **Processing flight**: Returns current progress
2. **Blocked flight**: Returns blocked=true with search_grid_size
3. **Completed flight**: Returns status="completed" with final counts
---
### `create_sse_stream(flight_id: str) -> SSEStream`
**REST Endpoint**: `GET /flights/{flightId}/stream`
**Description**: Opens Server-Sent Events connection for real-time result streaming.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
SSE Stream with events:
- frame_processed
- frame_refined
- search_expanded
- user_input_needed
- processing_blocked
- flight_completed
```
**Processing Flow**:
1. Validate flight_id exists
2. Call F02 Flight Processor → create_client_stream(flight_id, client_id)
3. F02 delegates to F15 SSE Event Streamer → create_stream()
4. Return SSE stream to client
**Event Format**:
```json
{
"event": "frame_processed",
"data": {
"frame_id": 237,
"gps": {"lat": 48.123, "lon": 37.456},
"altitude": 800.0,
"confidence": 0.95,
"heading": 87.3,
"timestamp": "2025-11-24T10:30:00Z"
}
}
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
- Connection closed on client disconnect
**Test Cases**:
1. **Connect to stream**: Opens SSE connection successfully
2. **Receive frame events**: Process 100 frames → receive 100 events
3. **Receive user_input_needed**: Blocked frame → event sent
4. **Client reconnect**: Replay missed events from last_event_id
---
## Integration Tests
### Test 1: Complete Flight Lifecycle
1. POST /flights with valid data
2. GET /flights/{flightId} → verify data
3. GET /flights/{flightId}/stream (open SSE)
4. POST /flights/{flightId}/images/batch × 40
5. Receive frame_processed events via SSE
6. Receive flight_completed event
7. GET /flights/{flightId} → verify waypoints updated
8. DELETE /flights/{flightId}
### Test 2: User Fix Flow
1. Create flight and process images
2. Receive user_input_needed event
3. POST /flights/{flightId}/user-fix
4. Receive processing_resumed event
5. Continue receiving frame_processed events
### Test 3: Concurrent Flights
1. Create 10 flights concurrently
2. Upload batches to all flights in parallel
3. Stream results from all flights simultaneously
4. Verify no cross-contamination
### Test 4: Waypoint Updates
1. Create flight
2. Simulate per-frame updates via PUT /flights/{flightId}/waypoints/{waypointId} × 100
3. GET flight and verify all waypoints updated
4. Verify refined=true flag set
---
## Non-Functional Requirements
### Performance
- **create_flight**: < 500ms response (prefetch is async)
- **get_flight**: < 200ms for flights with < 2000 waypoints
- **update_waypoint**: < 100ms (critical for real-time updates)
- **upload_image_batch**: < 2 seconds for 50 × 2MB images
- **submit_user_fix**: < 200ms response
- **get_flight_status**: < 100ms
- **SSE latency**: < 500ms from event generation to client receipt
### Scalability
- Support 100 concurrent flight processing sessions
- Handle 1000+ concurrent SSE connections
- Handle flights with up to 3000 waypoints
- Support 10,000 requests per minute
### Reliability
- Request timeout: 30 seconds for batch uploads
- SSE keepalive: Ping every 30 seconds
- Automatic SSE reconnection with event replay
- Graceful handling of client disconnects
### Security
- API key authentication
- Rate limiting: 100 requests/minute per client
- Max upload size: 500MB per batch
- CORS configuration for web clients
- Input validation on all endpoints
- SQL injection prevention
---
## Dependencies
### Internal Components
- **F02 Flight Processor**: For ALL operations (flight CRUD, image batching, user fixes, SSE streams, object-to-GPS conversion). F01 is a thin REST layer that delegates all business logic to F02.
**Note**: F01 does NOT directly call F05, F11, F13, or F15. All operations are routed through F02 to maintain a single coordinator pattern.
### External Dependencies
- **FastAPI**: Web framework
- **Uvicorn**: ASGI server
- **Pydantic**: Validation
- **python-multipart**: Multipart form handling
---
## Data Models
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float # Latitude -90 to 90
lon: float # Longitude -180 to 180
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int # pixels
resolution_height: int # pixels
distortion_coefficients: Optional[List[float]] = None
```
### Polygon
```python
class Polygon(BaseModel):
north_west: GPSPoint
south_east: GPSPoint
```
### Geofences
```python
class Geofences(BaseModel):
polygons: List[Polygon]
```
### FlightCreateRequest
```python
class FlightCreateRequest(BaseModel):
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
### Waypoint
```python
class Waypoint(BaseModel):
id: str
lat: float
lon: float
altitude: Optional[float] = None
confidence: float
timestamp: datetime
refined: bool = False
```
### FlightDetailResponse
```python
class FlightDetailResponse(BaseModel):
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
### FlightStatusResponse
```python
class FlightStatusResponse(BaseModel):
status: str
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
### BatchMetadata
```python
class BatchMetadata(BaseModel):
start_sequence: int
end_sequence: int
batch_number: int
```
### BatchUpdateResponse
```python
class BatchUpdateResponse(BaseModel):
success: bool
updated_count: int
failed_ids: List[str]
errors: Optional[Dict[str, str]]
```
_docs/02_components/02_flight_processor/02.1.01_feature_flight_waypoint_management.md
+67
View File
@@ -0,0 +1,67 @@
# Feature: Flight & Waypoint Management
## Description
Core CRUD operations for flights and waypoints. Handles flight creation with satellite prefetching trigger, flight state management, waypoint updates, and data validation. This is the primary data management feature of the Flight Lifecycle Manager.
## Component APIs Implemented
### Flight Lifecycle
- `create_flight(flight_data: FlightData) -> str` - Creates flight, validates data, sets ENU origin, triggers satellite prefetching
- `get_flight(flight_id: str) -> Optional[Flight]` - Retrieves flight by ID
- `get_flight_state(flight_id: str) -> Optional[FlightState]` - Retrieves current flight state
- `delete_flight(flight_id: str) -> bool` - Deletes flight and associated resources
- `update_flight_status(flight_id: str, status: FlightStatusUpdate) -> bool` - Updates flight status
### Waypoint Management
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool` - Updates single waypoint
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResult` - Batch waypoint update
- `get_flight_metadata(flight_id: str) -> Optional[FlightMetadata]` - Retrieves flight metadata
### Validation
- `validate_waypoint(waypoint: Waypoint) -> ValidationResult` - Validates single waypoint
- `validate_geofence(geofence: Geofences) -> ValidationResult` - Validates geofence boundaries
- `validate_flight_continuity(waypoints: List[Waypoint]) -> ValidationResult` - Validates waypoint sequence continuity
## External Dependencies
- **F03 Flight Database** - Persistence layer for flights and waypoints
- **F04 Satellite Data Manager** - Triggered for prefetch_route_corridor on flight creation
- **F13 Coordinate Transformer** - Called for set_enu_origin on flight creation
## Internal Methods
### Flight Operations
- `_generate_flight_id()` - Generates unique flight identifier
- `_persist_flight(flight: Flight)` - Saves flight to F03
- `_load_flight(flight_id: str)` - Loads flight from F03
- `_delete_flight_resources(flight_id: str)` - Cleans up flight-related resources
### Validation Helpers
- `_validate_gps_bounds(lat: float, lon: float)` - Validates GPS coordinates within valid range
- `_validate_waypoint_sequence(waypoints: List[Waypoint])` - Checks waypoint ordering and gaps
- `_validate_geofence_polygon(geofence: Geofences)` - Validates geofence geometry
### Coordinate Setup
- `_setup_enu_origin(flight_id: str, start_gps: GPSPoint)` - Delegates to F13 for ENU origin setup
## Unit Tests
- Test flight creation with valid data returns flight_id
- Test flight creation with invalid geofence returns validation error
- Test flight creation triggers F04.prefetch_route_corridor
- Test flight creation calls F13.set_enu_origin
- Test get_flight returns None for non-existent flight
- Test delete_flight removes flight from database
- Test update_flight_status transitions state correctly
- Test validate_waypoint rejects out-of-bounds GPS
- Test validate_geofence rejects invalid polygon
- Test validate_flight_continuity detects excessive gaps
- Test batch_update_waypoints handles partial failures
- Test waypoint update validates before persisting
## Integration Tests
- Test flight creation end-to-end with F03 persistence
- Test flight creation triggers F04 satellite prefetching
- Test flight creation sets ENU origin via F13
- Test flight deletion cleans up all related data in F03
- Test concurrent flight creation handles ID generation correctly
- Test waypoint batch update maintains data consistency
_docs/02_components/02_flight_processor/02.1.02_feature_processing_delegation.md
+59
View File
@@ -0,0 +1,59 @@
# Feature: Processing Delegation
## Description
Delegation methods that route API calls from F01 Flight API to appropriate subsystems. Manages F02.2 Processing Engine instances, coordinates image queuing, handles user fixes, and provides access to SSE streaming and coordinate transformation services.
## Component APIs Implemented
- `queue_images(flight_id: str, batch: ImageBatch) -> BatchQueueResult` - Queues images via F05, ensures F02.2 engine is active
- `handle_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Forwards user fix to active F02.2 engine
- `create_client_stream(flight_id: str, client_id: str) -> StreamConnection` - Creates SSE stream via F15
- `convert_object_to_gps(flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint` - Converts pixel to GPS via F13
## External Dependencies
- **F02.2 Flight Processing Engine** - Managed child component, created/retrieved per flight
- **F05 Image Input Pipeline** - Image batch queuing
- **F13 Coordinate Transformer** - Pixel-to-GPS conversion
- **F15 SSE Event Streamer** - Client stream creation
## Internal Methods
### Engine Management
- `_get_or_create_engine(flight_id: str) -> FlightProcessingEngine` - Retrieves existing or creates new F02.2 instance
- `_get_active_engine(flight_id: str) -> Optional[FlightProcessingEngine]` - Gets active engine or None
- `_engine_registry: Dict[str, FlightProcessingEngine]` - Registry of active engines per flight
### Image Queuing
- `_delegate_queue_batch(flight_id: str, batch: ImageBatch)` - Delegates to F05.queue_batch
- `_trigger_processing(engine: FlightProcessingEngine)` - Triggers async engine.start_processing
### User Fix Handling
- `_validate_fix_request(fix_data: UserFixRequest)` - Validates fix data before delegation
- `_apply_fix_to_engine(engine: FlightProcessingEngine, fix_data: UserFixRequest)` - Calls engine.apply_user_fix
### Stream Management
- `_delegate_stream_creation(flight_id: str, client_id: str)` - Delegates to F15
### Coordinate Conversion
- `_delegate_coordinate_transform(flight_id: str, frame_id: int, pixel: Tuple)` - Delegates to F13
## Unit Tests
- Test queue_images delegates to F05.queue_batch
- Test queue_images creates new engine if none exists
- Test queue_images retrieves existing engine for active flight
- Test queue_images triggers engine.start_processing
- Test handle_user_fix returns error for non-existent flight
- Test handle_user_fix delegates to correct engine
- Test create_client_stream delegates to F15
- Test convert_object_to_gps delegates to F13
- Test engine registry tracks active engines correctly
- Test multiple queue_images calls reuse same engine instance
## Integration Tests
- Test queue_images end-to-end with F05 and F02.2
- Test image batch flows through F05 to F02.2 processing
- Test user fix applied correctly through F02.2
- Test SSE stream creation and connection via F15
- Test coordinate conversion accuracy via F13
- Test engine cleanup on flight completion
- Test concurrent requests to same flight share engine
_docs/02_components/02_flight_processor/02.1.03_feature_system_initialization.md
+55
View File
@@ -0,0 +1,55 @@
# Feature: System Initialization
## Description
System startup routine that initializes all dependent components in correct order. Loads configuration, initializes ML models, prepares database connections, sets up satellite cache, and loads place recognition indexes. This is called once at service startup.
## Component APIs Implemented
- `initialize_system() -> bool` - Executes full system initialization sequence
## External Dependencies
- **F17 Configuration Manager** - Configuration loading
- **F16 Model Manager** - ML model initialization (SuperPoint, LightGlue, LiteSAM, DINOv2)
- **F03 Flight Database** - Database connection initialization
- **F04 Satellite Data Manager** - Satellite cache initialization
- **F08 Global Place Recognition** - Faiss index loading
## Internal Methods
### Initialization Sequence
- `_load_configuration()` - Loads config via F17, validates required settings
- `_initialize_models()` - Initializes ML models via F16 with TensorRT optimization
- `_initialize_database()` - Sets up F03 database connections and schema
- `_initialize_satellite_cache()` - Prepares F04 cache with operational region data
- `_load_place_recognition_indexes()` - Loads F08 Faiss indexes into memory
### Health Checks
- `_verify_gpu_availability()` - Checks CUDA/TensorRT availability
- `_verify_model_loading()` - Validates all models loaded correctly
- `_verify_database_connection()` - Tests database connectivity
- `_verify_index_integrity()` - Validates Faiss indexes are loadable
### Error Handling
- `_handle_initialization_failure(component: str, error: Exception)` - Logs and handles component init failures
- `_rollback_partial_initialization()` - Cleans up on partial initialization failure
## Unit Tests
- Test initialize_system calls F17 config loading
- Test initialize_system calls F16 model initialization
- Test initialize_system calls F03 database initialization
- Test initialize_system calls F04 cache initialization
- Test initialize_system calls F08 index loading
- Test initialize_system returns False on config load failure
- Test initialize_system returns False on model init failure
- Test initialize_system returns False on database init failure
- Test initialization order is correct (config first, then models, etc.)
- Test partial initialization failure triggers rollback
## Integration Tests
- Test full system initialization with all real components
- Test system startup on cold start (no cached data)
- Test system startup with existing database
- Test initialization with GPU unavailable falls back gracefully
- Test initialization timeout handling
- Test system ready state after successful initialization
- Test re-initialization after failure recovery
_docs/02_components/02_flight_processor/02.1._component_flight_lifecycle_manager.md
+146
View File
@@ -0,0 +1,146 @@
# Flight Lifecycle Manager
## Interface Definition
**Interface Name**: `IFlightLifecycleManager`
### Interface Methods
```python
class IFlightLifecycleManager(ABC):
# Flight Lifecycle
@abstractmethod
def create_flight(self, flight_data: FlightData) -> str:
pass
@abstractmethod
def get_flight(self, flight_id: str) -> Optional[Flight]:
pass
@abstractmethod
def get_flight_state(self, flight_id: str) -> Optional[FlightState]:
pass
@abstractmethod
def delete_flight(self, flight_id: str) -> bool:
pass
@abstractmethod
def update_flight_status(self, flight_id: str, status: FlightStatusUpdate) -> bool:
pass
# Waypoint Management
@abstractmethod
def update_waypoint(self, flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool:
pass
@abstractmethod
def batch_update_waypoints(self, flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResult:
pass
@abstractmethod
def get_flight_metadata(self, flight_id: str) -> Optional[FlightMetadata]:
pass
# Validation
@abstractmethod
def validate_waypoint(self, waypoint: Waypoint) -> ValidationResult:
pass
@abstractmethod
def validate_geofence(self, geofence: Geofences) -> ValidationResult:
pass
@abstractmethod
def validate_flight_continuity(self, waypoints: List[Waypoint]) -> ValidationResult:
pass
# API Delegation Methods (called by F01)
@abstractmethod
def queue_images(self, flight_id: str, batch: ImageBatch) -> BatchQueueResult:
"""Delegates to F05 Image Input Pipeline and triggers F02.2 Processing Engine."""
pass
@abstractmethod
def handle_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResult:
"""Delegates to F02.2 Processing Engine to apply fix."""
pass
@abstractmethod
def create_client_stream(self, flight_id: str, client_id: str) -> StreamConnection:
"""Delegates to F15 SSE Event Streamer."""
pass
@abstractmethod
def convert_object_to_gps(self, flight_id: str, frame_id: int, pixel: Tuple[float, float]) -> GPSPoint:
"""Delegates to F13 Coordinate Transformer."""
pass
# System Initialization
@abstractmethod
def initialize_system(self) -> bool:
pass
```
## Component Description
### Responsibilities
- **Component ID**: F02.1
- **Flight Lifecycle**: Manage flight creation, persistence, and deletion.
- **System Entry Point**: Main interface for the REST API (F01).
- **Resource Management**: Initializes system components and manages `F02.2 Flight Processing Engine` instances.
- **Data Validation**: Validates inputs before processing.
- **Satellite Prefetching**: Triggers F04 prefetching on flight creation.
### Scope
- CRUD operations for Flights and Waypoints.
- Coordination of system startup.
- Hand-off of active processing tasks to F02.2.
- Does **NOT** contain the processing loop or recovery logic.
## API Methods
### `create_flight(flight_data: FlightData) -> str`
**Description**: Creates a new flight, validates data, sets ENU origin, and triggers satellite prefetching.
**Called By**: F01 Flight API
**Processing Flow**:
1. Generate flight_id.
2. Validate geofences and waypoints.
3. Call F13.set_enu_origin(flight_id, start_gps).
4. Trigger F04.prefetch_route_corridor().
5. Save to F03 Flight Database.
6. Return flight_id.
### `queue_images(flight_id: str, batch: ImageBatch) -> BatchQueueResult`
**Description**: Queues images and ensures a Processing Engine is active for this flight.
**Processing Flow**:
1. Delegate to F05.queue_batch().
2. Retrieve or create instance of `F02.2 Flight Processing Engine` for this flight.
3. Trigger `engine.start_processing()` (async).
4. Return result.
### `handle_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult`
**Description**: Forwards user fix to the active processing engine.
**Processing Flow**:
1. Get active engine for flight_id.
2. Call `engine.apply_user_fix(fix_data)`.
3. Return result.
### `initialize_system() -> bool`
**Description**: startup routine.
**Processing Flow**:
1. Load config (F17).
2. Initialize models (F16).
3. Initialize DB (F03).
4. Initialize Cache (F04).
5. Load Indexes (F08).
## Dependencies
- **F02.2 Flight Processing Engine**: Managed child component.
- **F03 Flight Database**: Persistence.
- **F04 Satellite Data Manager**: Prefetching.
- **F05 Image Input Pipeline**: Image queuing.
- **F13 Coordinate Transformer**: ENU origin setting.
- **F15 SSE Event Streamer**: Stream creation.
- **F17 Configuration Manager**: Config loading.
_docs/02_components/02_flight_processor/02.2.01_feature_frame_processing_loop.md
+43
View File
@@ -0,0 +1,43 @@
# Feature: Frame Processing Loop
## Description
Core frame-by-frame processing orchestration that runs the main visual odometry pipeline. Manages the continuous loop of fetching images, computing poses, updating the factor graph, and publishing results. Includes flight state machine management (Processing, Blocked, Recovering, Completed).
## Component APIs Implemented
- `start_processing(flight_id: str) -> None` - Starts the main processing loop in a background thread/task
- `stop_processing(flight_id: str) -> None` - Stops the processing loop gracefully
- `process_frame(flight_id: str, frame_id: int) -> FrameResult` - Processes a single frame through the pipeline
## External Dependencies
- **F05 Image Input Pipeline**: `get_next_image()` - Image source
- **F06 Image Rotation Manager**: `requires_rotation_sweep()` - Pre-processing checks
- **F07 Sequential Visual Odometry**: `compute_relative_pose()` - Motion estimation
- **F09 Metric Refinement**: `align_to_satellite()` - Drift correction
- **F10 Factor Graph Optimizer**: `add_relative_factor()`, `optimize_chunk()` - State estimation
- **F14 Result Manager**: `update_frame_result()` - Saving results
## Internal Methods
- `run_processing_loop(flight_id: str)` - Main loop: while images available, process each frame
- `_process_single_frame(flight_id: str, image: Image) -> FrameResult` - Single frame processing pipeline
- `_check_tracking_status(vo_result: VOResult) -> bool` - Determines if tracking is good or lost
- `_update_flight_status(flight_id: str, status: FlightStatus)` - Updates state machine
- `_get_flight_status(flight_id: str) -> FlightStatus` - Gets current flight status
- `_is_processing_active(flight_id: str) -> bool` - Checks if processing should continue
## Unit Tests
- Test `start_processing` initiates background loop
- Test `stop_processing` gracefully terminates active processing
- Test `process_frame` returns valid FrameResult with pose
- Test flight status transitions: Processing -> Completed on last frame
- Test flight status transitions: Processing -> Blocked on tracking loss
- Test processing stops when `stop_processing` called mid-flight
- Test processing handles empty image queue gracefully
- Test state machine rejects invalid transitions
## Integration Tests
- Test full pipeline: F05 -> F06 -> F07 -> F10 -> F14 flow
- Test processing 10 consecutive frames with good tracking
- Test real-time result publishing to F14/F15
- Test concurrent access to flight status from multiple threads
- Test processing resumes after Blocked -> Processing transition
_docs/02_components/02_flight_processor/02.2.02_feature_tracking_loss_recovery.md
+42
View File
@@ -0,0 +1,42 @@
# Feature: Tracking Loss Recovery
## Description
Handles recovery when visual odometry loses tracking. Implements progressive search strategy through F11 and manages user input workflow when automatic recovery fails. Coordinates the transition to BLOCKED status and back to PROCESSING upon successful recovery.
## Component APIs Implemented
- `handle_tracking_loss(flight_id: str, frame_id: int) -> RecoveryStatus` - Initiates and manages tracking loss recovery
- `apply_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Applies user-provided GPS anchor
## External Dependencies
- **F11 Failure Recovery Coordinator**:
- `start_search()` - Initiates progressive search
- `try_current_grid()` - Attempts matching on current tile grid
- `expand_search_radius()` - Expands search area
- `mark_found()` - Marks successful recovery
- `create_user_input_request()` - Creates request for user intervention
- `apply_user_anchor()` - Applies user-provided anchor
- **F15 SSE Event Streamer**: `send_user_input_request()` - Notifies client of required input
## Internal Methods
- `_run_progressive_search(flight_id: str, frame_id: int) -> Optional[RecoveryResult]` - Executes 1->25 tile progressive search
- `_request_user_input(flight_id: str, frame_id: int, request: UserInputRequest)` - Transitions to BLOCKED and notifies client
- `_validate_user_fix(fix_data: UserFixRequest) -> bool` - Validates user input data
- `_apply_fix_and_resume(flight_id: str, fix_data: UserFixRequest) -> UserFixResult` - Applies fix and resumes processing
## Unit Tests
- Test `handle_tracking_loss` starts progressive search via F11
- Test progressive search expands from 1 to 25 tiles
- Test successful recovery returns RecoveryStatus.FOUND
- Test failed recovery transitions status to BLOCKED
- Test `apply_user_fix` validates input coordinates
- Test `apply_user_fix` returns success on valid anchor
- Test `apply_user_fix` transitions status to PROCESSING on success
- Test `apply_user_fix` rejects fix when not in BLOCKED status
## Integration Tests
- Test full recovery flow: tracking loss -> progressive search -> found
- Test full blocked flow: tracking loss -> search fails -> user input -> resume
- Test SSE notification sent when user input required
- Test recovery integrates with F11 search grid expansion
- Test user fix properly anchors subsequent frame processing
_docs/02_components/02_flight_processor/02.2.03_feature_chunk_lifecycle_orchestration.md
+37
View File
@@ -0,0 +1,37 @@
# Feature: Chunk Lifecycle Orchestration
## Description
Manages route chunk boundaries and creation during processing. Detects when new chunks should be created (tracking loss, sharp turns) and orchestrates chunk lifecycle through F12. Implements proactive chunk creation strategy where new chunks are created immediately on tracking loss rather than waiting for matching failures.
## Component APIs Implemented
- `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]` - Returns current active chunk for the flight
- `create_new_chunk(flight_id: str, frame_id: int) -> ChunkHandle` - Creates new chunk starting at specified frame
## External Dependencies
- **F12 Route Chunk Manager**:
- `get_active_chunk()` - Retrieves current chunk
- `create_chunk()` - Creates new chunk
- `add_frame_to_chunk()` - Adds processed frame to chunk
## Internal Methods
- `_detect_chunk_boundary(flight_id: str, frame_id: int, tracking_status: bool) -> bool` - Determines if chunk boundary detected
- `_should_create_chunk_on_tracking_loss(flight_id: str) -> bool` - Checks if proactive chunk creation needed
- `_create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle` - Creates chunk proactively on tracking loss
- `_add_frame_to_active_chunk(flight_id: str, frame_id: int, frame_result: FrameResult)` - Adds frame to current chunk
## Unit Tests
- Test `get_active_chunk` returns current chunk handle
- Test `get_active_chunk` returns None when no active chunk
- Test `create_new_chunk` delegates to F12 and returns handle
- Test chunk boundary detection on tracking loss
- Test proactive chunk creation triggers on tracking loss
- Test `_add_frame_to_active_chunk` delegates to F12
- Test multiple chunks can exist for same flight
## Integration Tests
- Test chunk creation flow: tracking loss -> new chunk -> frames added
- Test chunk boundary detection integrates with VO tracking status
- Test chunk handles are properly propagated to F10 factor graph
- Test chunk lifecycle across multiple tracking loss events
- Test chunk state consistency between F02.2 and F12
_docs/02_components/02_flight_processor/02.2._spec_flight_processing_engine_spec.md
+108
View File
@@ -0,0 +1,108 @@
# Flight Processing Engine
## Interface Definition
**Interface Name**: `IFlightProcessingEngine`
### Interface Methods
```python
class IFlightProcessingEngine(ABC):
@abstractmethod
def start_processing(self, flight_id: str) -> None:
"""Starts the main processing loop in a background thread/task."""
pass
@abstractmethod
def stop_processing(self, flight_id: str) -> None:
pass
@abstractmethod
def process_frame(self, flight_id: str, frame_id: int) -> FrameResult:
pass
@abstractmethod
def apply_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResult:
pass
@abstractmethod
def handle_tracking_loss(self, flight_id: str, frame_id: int) -> RecoveryStatus:
pass
# Chunk Lifecycle (Delegates to F12 but orchestrated here)
@abstractmethod
def get_active_chunk(self, flight_id: str) -> Optional[ChunkHandle]:
pass
@abstractmethod
def create_new_chunk(self, flight_id: str, frame_id: int) -> ChunkHandle:
pass
```
## Component Description
### Responsibilities
- **Component ID**: F02.2
- **Processing Orchestration**: Runs the main loop: Image -> VO -> Graph -> Result.
- **State Machine**: Manages flight status (Processing, Blocked, Recovering, Completed).
- **Recovery Coordination**: Calls F11 methods and acts on return values.
- **Chunk Management**: Calls F12 to manage chunk lifecycle based on tracking status.
- **Background Tasks**: Manages background chunk matching tasks.
### Scope
- Per-flight processing logic.
- Interaction with Visual Pipeline (F06, F07, F09).
- Interaction with State Estimation (F10).
- Interaction with Recovery (F11).
- Result Publishing (F14, F15).
## Processing Flow
### `run_processing_loop(flight_id: str)`
1. **Loop**: While `F05.get_next_image()` returns image:
2. **Chunk Check**:
* `active_chunk = F12.get_active_chunk()`
* If `detect_chunk_boundary()`: `active_chunk = F12.create_chunk()`.
3. **Visual Odometry**:
* Call `F06.requires_rotation_sweep()`.
* Call `F07.compute_relative_pose()`.
4. **Tracking Check**:
* If tracking good:
* `F12.add_frame_to_chunk()`.
* `F09.align_to_satellite()` (optional drift correction).
* `F10.add_relative_factor()`.
* `F10.optimize_chunk()`.
* `F14.update_frame_result()`.
* If tracking lost:
* **Proactive Chunking**: `F11.create_chunk_on_tracking_loss()`.
* **Recovery**: Call `handle_tracking_loss()`.
### `handle_tracking_loss(flight_id, frame_id)`
1. Call `F11.start_search()`.
2. Loop progressive search (1..25 tiles):
* `result = F11.try_current_grid()`.
* If found: `F11.mark_found()`, break.
* If not found: `F11.expand_search_radius()`.
3. If still not found:
* `req = F11.create_user_input_request()`.
* `F15.send_user_input_request(req)`.
* Set status to **BLOCKED**.
* Wait for `apply_user_fix`.
### `apply_user_fix(fix_data)`
1. Call `F11.apply_user_anchor(fix_data)`.
2. If success:
* Set status to **PROCESSING**.
* Resume loop.
## Dependencies
- **F05 Image Input Pipeline**: Image source.
- **F06 Image Rotation Manager**: Pre-processing.
- **F07 Sequential Visual Odometry**: Motion estimation.
- **F09 Metric Refinement**: Satellite alignment.
- **F10 Factor Graph Optimizer**: State estimation.
- **F11 Failure Recovery Coordinator**: Recovery logic.
- **F12 Route Chunk Manager**: Chunk state.
- **F14 Result Manager**: Saving results.
- **F15 SSE Event Streamer**: Real-time updates.
_docs/02_components/03_flight_database/03.01_feature_flight_crud_operations.md
+114
View File
@@ -0,0 +1,114 @@
# Feature: Flight & Waypoint CRUD Operations
## Description
Core database infrastructure and CRUD operations for flights, waypoints, and geofences. Provides connection pooling, transaction support, and all primary data operations. This feature handles the main entities that define a flight and its route.
## Component APIs Implemented
- `execute_transaction(operations: List[Callable]) -> bool`
- `insert_flight(flight: Flight) -> str`
- `update_flight(flight: Flight) -> bool`
- `query_flights(filters: Dict[str, Any], limit: int, offset: int) -> List[Flight]`
- `get_flight_by_id(flight_id: str) -> Optional[Flight]`
- `delete_flight(flight_id: str) -> bool`
- `get_waypoints(flight_id: str, limit: Optional[int] = None) -> List[Waypoint]`
- `insert_waypoint(flight_id: str, waypoint: Waypoint) -> str`
- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool`
- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchResult`
## External Tools and Services
- **PostgreSQL**: Primary database
- **SQLAlchemy**: ORM and connection pooling
- **Alembic**: Schema migrations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_get_connection()` | Acquire connection from pool |
| `_release_connection(conn)` | Return connection to pool |
| `_execute_with_retry(query, params, retries=3)` | Execute query with automatic retry on transient errors |
| `_build_flight_from_row(row)` | Map database row to Flight object |
| `_build_waypoint_from_row(row)` | Map database row to Waypoint object |
| `_serialize_camera_params(params)` | Serialize CameraParameters to JSONB |
| `_deserialize_camera_params(jsonb)` | Deserialize JSONB to CameraParameters |
| `_build_filter_query(filters)` | Build WHERE clause from filter dict |
## Unit Tests
1. **Connection management**
- Pool returns connections correctly
- Connection recycling after timeout
- Health check on stale connections
2. **execute_transaction**
- All operations succeed → commit
- One operation fails → rollback all
- Connection error → retry and succeed
- Nested transactions handled correctly
3. **insert_flight**
- Valid flight with 100 waypoints → all persisted
- Duplicate flight_id → raises IntegrityError
- Partial failure mid-insert → complete rollback
- Empty waypoints list → flight created with no waypoints
4. **update_flight**
- Existing flight → returns True, fields updated
- Non-existent flight → returns False
- Only specified fields updated, others unchanged
5. **query_flights**
- Filter by name → returns matching flights
- Filter by status → returns matching flights
- Pagination (offset=100, limit=50) → returns correct slice
- No matches → returns empty list
6. **get_flight_by_id**
- Existing flight → returns complete Flight with waypoints
- Non-existent flight → returns None
- Large flight (3000 waypoints) → returns within 150ms
7. **delete_flight**
- Existing flight → returns True, cascade to all tables
- Non-existent flight → returns False
- Verify cascade deletes waypoints, geofences, etc.
8. **get_waypoints**
- All waypoints (limit=None) → returns complete list
- Limited (limit=100) → returns first 100
- Non-existent flight → returns empty list
9. **insert_waypoint**
- Valid insertion → returns waypoint_id
- Non-existent flight → raises ForeignKeyError
10. **update_waypoint**
- Existing waypoint → returns True
- Non-existent waypoint → returns False
- Concurrent updates → no data corruption
11. **batch_update_waypoints**
- Batch of 100 → all succeed
- Partial failure → returns failed_ids
- Empty batch → returns success with updated_count=0
## Integration Tests
1. **Complete flight lifecycle**
- insert_flight() → update_waypoint() × 100 → get_flight_by_id() → delete_flight()
- Verify all data persisted and cascade delete works
2. **High-frequency waypoint updates**
- Insert flight with 2000 waypoints
- Concurrent update_waypoint() calls (100/sec)
- Verify throughput > 200 ops/sec
- Verify no data corruption
3. **Transaction atomicity**
- Begin transaction with 5 operations
- Fail on operation 3
- Verify operations 1-2 rolled back
4. **Connection pool behavior**
- Exhaust pool → new requests wait
- Pool recovery after connection failures
- Connection reuse efficiency
_docs/02_components/03_flight_database/03.02_feature_processing_state_persistence.md
+103
View File
@@ -0,0 +1,103 @@
# Feature: Processing State Persistence
## Description
Persistence layer for flight processing state, frame results, and heading history. Supports crash recovery by persisting processing progress and enables temporal smoothing through heading history tracking. Critical for maintaining processing continuity and data integrity during frame refinement operations.
## Component APIs Implemented
- `save_flight_state(flight_state: FlightState) -> bool`
- `load_flight_state(flight_id: str) -> Optional[FlightState]`
- `query_processing_history(filters: Dict[str, Any]) -> List[FlightState]`
- `save_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
- `get_frame_results(flight_id: str) -> List[FrameResult]`
- `save_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
- `get_heading_history(flight_id: str, last_n: Optional[int] = None) -> List[HeadingRecord]`
- `get_latest_heading(flight_id: str) -> Optional[float]`
## External Tools and Services
- **PostgreSQL**: Primary database
- **SQLAlchemy**: ORM and connection pooling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_flight_state_from_row(row)` | Map database row to FlightState object |
| `_build_frame_result_from_row(row)` | Map database row to FrameResult object |
| `_build_heading_record_from_row(row)` | Map database row to HeadingRecord object |
| `_upsert_flight_state(state)` | Insert or update flight state (single source of truth) |
| `_upsert_frame_result(flight_id, result)` | Insert or update frame result (handles refinement updates) |
## Unit Tests
1. **save_flight_state**
- New state → created successfully
- Update existing state → overwrites previous
- Verify all fields persisted correctly
- Verify heading NOT stored in flight_state (use heading_history)
2. **load_flight_state**
- Existing state → returns FlightState object
- Non-existent flight → returns None
- Verify all fields deserialized correctly
3. **query_processing_history**
- Filter by date range → returns flights in range
- Filter by status → returns matching flights
- Combined filters → returns intersection
- No matches → returns empty list
4. **save_frame_result**
- New frame → persisted successfully
- Update on refinement → overwrites with refined=True
- Verify GPS, altitude, heading, confidence persisted
- Verify timestamp and updated_at set correctly
5. **get_frame_results**
- Flight with 500 frames → returns all results ordered by frame_id
- No results → returns empty list
- Performance: 2000 frames returned within 100ms
6. **save_heading**
- New heading → persisted correctly
- Overwrite same frame_id → updates value
- Verify timestamp persisted
- Heading range validation (0-360)
7. **get_heading_history**
- All headings (last_n=None) → returns complete history
- Last 10 headings → returns 10 most recent by timestamp
- Non-existent flight → returns empty list
- Ordered by frame_id descending
8. **get_latest_heading**
- Has history → returns latest heading value
- No history → returns None
- After multiple saves → returns most recent
## Integration Tests
1. **Processing state recovery**
- Save state with frames_processed=250
- Simulate crash (close connection)
- Reconnect and load_flight_state()
- Verify state intact, processing can resume
2. **Frame result refinement flow**
- save_frame_result() with refined=False
- Later, save_frame_result() with refined=True
- get_frame_results() → shows refined=True
- Verify GPS coordinates updated
3. **Heading history for smoothing**
- save_heading() × 100 frames
- get_heading_history(last_n=10)
- Verify correct 10 headings returned for smoothing calculation
- Verify ordering correct for temporal analysis
4. **High-frequency persistence**
- Concurrent save_frame_result() and save_heading() calls
- Measure throughput > 200 ops/sec
- Verify no data loss or corruption
5. **Transactional consistency**
- Transaction: update frame_result + update waypoint
- Verify atomic update or complete rollback
_docs/02_components/03_flight_database/03.03_feature_auxiliary_data_persistence.md
+99
View File
@@ -0,0 +1,99 @@
# Feature: Auxiliary Data Persistence
## Description
Persistence layer for supporting data including image metadata and chunk state. Image metadata enables image pipeline to track stored files, while chunk state persistence is critical for crash recovery of route chunk processing. These operations support other components' specific data requirements.
## Component APIs Implemented
- `save_image_metadata(flight_id: str, frame_id: int, file_path: str, metadata: Dict) -> bool`
- `get_image_path(flight_id: str, frame_id: int) -> Optional[str]`
- `get_image_metadata(flight_id: str, frame_id: int) -> Optional[Dict]`
- `save_chunk_state(flight_id: str, chunk: ChunkHandle) -> bool`
- `load_chunk_states(flight_id: str) -> List[ChunkHandle]`
- `delete_chunk_state(flight_id: str, chunk_id: str) -> bool`
## External Tools and Services
- **PostgreSQL**: Primary database with JSONB support
- **SQLAlchemy**: ORM and connection pooling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_chunk_handle_from_row(row)` | Map database row to ChunkHandle object |
| `_serialize_chunk_frames(frames)` | Serialize frame list to JSONB array |
| `_deserialize_chunk_frames(jsonb)` | Deserialize JSONB array to frame list |
| `_serialize_metadata(metadata)` | Serialize metadata dict to JSONB |
| `_deserialize_metadata(jsonb)` | Deserialize JSONB to metadata dict |
| `_upsert_chunk_state(flight_id, chunk)` | Insert or update chunk state |
## Unit Tests
1. **save_image_metadata**
- New image → metadata persisted with file_path
- Overwrite same frame_id → updates metadata
- Verify JSONB serialization of metadata dict
- Verify uploaded_at timestamp set
2. **get_image_path**
- Existing image → returns file path string
- Non-existent frame → returns None
- Non-existent flight → returns None
3. **get_image_metadata**
- Existing image → returns metadata dict
- Verify deserialization of original_name, width, height, file_size
- Non-existent → returns None
4. **save_chunk_state**
- New chunk → persisted successfully
- Update existing chunk → state updated
- Verify all fields: chunk_id, start_frame_id, end_frame_id, frames
- Verify anchor fields: has_anchor, anchor_frame_id, anchor_gps
- Verify matching_status persisted
- Verify frames JSONB array serialization
5. **load_chunk_states**
- Flight with 3 chunks → returns all chunk handles
- No chunks → returns empty list
- Verify correct deserialization of ChunkHandle fields
- Verify frames list deserialized from JSONB
6. **delete_chunk_state**
- Existing chunk → deleted, returns True
- Non-existent chunk → returns False
- Verify other chunks unaffected
## Integration Tests
1. **Image metadata lifecycle**
- save_image_metadata() for 100 frames
- get_image_path() for each → all paths returned
- Delete flight → cascade deletes image metadata
- Verify no orphan records
2. **Chunk state crash recovery**
- Create flight, save 5 chunk states
- Simulate crash (close connection)
- Reconnect, load_chunk_states()
- Verify all 5 chunks restored with correct state
- Verify frames lists intact
3. **Chunk lifecycle operations**
- save_chunk_state() → active chunk
- Update chunk: add frames, set has_anchor=True
- save_chunk_state() → verify update
- delete_chunk_state() after merge → verify removed
4. **Concurrent chunk operations**
- Multiple chunks saved concurrently
- Verify no data corruption
- Verify unique chunk_ids enforced
5. **JSONB query performance**
- Save chunks with large frames arrays (500+ frames)
- load_chunk_states() performance within 50ms
- Verify JSONB indexing effectiveness
6. **Foreign key constraints**
- save_chunk_state with invalid anchor_frame_id → proper error handling
- Verify FK constraint fk_anchor_frame enforced
- Delete referenced image → anchor_frame_id set to NULL
_docs/02_components/03_flight_database/03._component_flight_database.md
+1059
View File
File diff suppressed because it is too large Load Diff
_docs/02_components/04_satellite_data_manager/04.01_feature_tile_cache_management.md
+41
View File
@@ -0,0 +1,41 @@
# Feature: Tile Cache Management
## Description
Manages persistent disk-based caching of satellite tiles with flight-specific organization. Provides storage, retrieval, and cleanup of cached tiles to minimize redundant API calls and enable offline access to prefetched data.
## Component APIs Implemented
- `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
- `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
- `clear_flight_cache(flight_id: str) -> bool`
## External Tools and Services
- **diskcache**: Persistent cache library for disk storage management
- **opencv-python**: Image serialization (PNG encoding/decoding)
- **numpy**: Image array handling
## Internal Methods
- `_generate_cache_path(flight_id: str, tile_coords: TileCoords) -> Path`: Generates cache file path following pattern `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
- `_ensure_cache_directory(flight_id: str, zoom: int) -> bool`: Creates cache directory structure if not exists
- `_serialize_tile(tile_data: np.ndarray) -> bytes`: Encodes tile array to PNG bytes
- `_deserialize_tile(data: bytes) -> Optional[np.ndarray]`: Decodes PNG bytes to tile array
- `_update_cache_index(flight_id: str, tile_coords: TileCoords, action: str) -> None`: Updates cache index for tracking
- `_check_global_cache(tile_coords: TileCoords) -> Optional[np.ndarray]`: Fallback lookup in shared cache
## Unit Tests
1. **cache_tile_success**: Cache new tile → file created at correct path
2. **cache_tile_overwrite**: Cache existing tile → file updated
3. **cache_tile_disk_error**: Simulate disk full → returns False
4. **get_cached_tile_hit**: Tile exists → returns np.ndarray
5. **get_cached_tile_miss**: Tile not exists → returns None
6. **get_cached_tile_corrupted**: Invalid file → returns None, logs warning
7. **get_cached_tile_global_fallback**: Not in flight cache, found in global → returns tile
8. **clear_flight_cache_success**: Flight with tiles → all files removed
9. **clear_flight_cache_nonexistent**: No such flight → returns True (no-op)
10. **cache_path_generation**: Various tile coords → correct paths generated
## Integration Tests
1. **cache_round_trip**: cache_tile() then get_cached_tile() → returns identical data
2. **multi_flight_isolation**: Cache tiles for flight A and B → each retrieves only own tiles
3. **clear_does_not_affect_others**: Clear flight A → flight B cache intact
4. **large_cache_handling**: Cache 1000 tiles → all retrievable
_docs/02_components/04_satellite_data_manager/04.02_feature_tile_coordinate_operations.md
+44
View File
@@ -0,0 +1,44 @@
# Feature: Tile Coordinate Operations
## Description
Handles all tile coordinate calculations including GPS-to-tile conversion, tile grid computation, and grid expansion for progressive search. Delegates core Web Mercator projection math to H06 Web Mercator Utils to maintain single source of truth.
## Component APIs Implemented
- `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
- `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
- `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
- `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
## External Tools and Services
None (pure computation, delegates to H06)
## Internal Dependencies
- **H06 Web Mercator Utils**: Core projection calculations
- `H06.latlon_to_tile()` for coordinate conversion
- `H06.compute_tile_bounds()` for bounding box calculation
## Internal Methods
- `_compute_grid_offset(grid_size: int) -> int`: Calculates offset from center for symmetric grid (e.g., 3×3 → offset 1)
- `_grid_size_to_dimensions(grid_size: int) -> Tuple[int, int]`: Maps grid_size (1,4,9,16,25) to (rows, cols)
- `_generate_grid_tiles(center: TileCoords, rows: int, cols: int) -> List[TileCoords]`: Generates all tile coords in grid
## Unit Tests
1. **compute_tile_coords_ukraine**: Ukraine GPS coords at zoom 19 → valid tile coords
2. **compute_tile_coords_origin**: lat=0, lon=0 → correct center tile
3. **compute_tile_coords_edge_cases**: lat=90, lon=180, lon=-180 → handled correctly
4. **compute_tile_bounds_zoom19**: Zoom 19 tile → GSD ≈ 0.3 m/pixel
5. **compute_tile_bounds_corners**: Returns valid GPS for all 4 corners
6. **get_tile_grid_1**: grid_size=1 → returns [center]
7. **get_tile_grid_4**: grid_size=4 → returns 4 tiles (2×2)
8. **get_tile_grid_9**: grid_size=9 → returns 9 tiles (3×3) centered
9. **get_tile_grid_25**: grid_size=25 → returns 25 tiles (5×5)
10. **expand_search_grid_1_to_4**: Returns 3 new tiles only
11. **expand_search_grid_4_to_9**: Returns 5 new tiles only
12. **expand_search_grid_9_to_16**: Returns 7 new tiles only
13. **expand_search_grid_no_duplicates**: Expanded tiles not in original set
## Integration Tests
1. **h06_delegation_verify**: compute_tile_coords() result matches direct H06.latlon_to_tile()
2. **grid_bounds_coverage**: get_tile_grid(9) → all 9 tile bounds form contiguous area
3. **expand_completes_grid**: get_tile_grid(4) + expand_search_grid(4,9) == get_tile_grid(9)
_docs/02_components/04_satellite_data_manager/04.03_feature_tile_fetching.md
+51
View File
@@ -0,0 +1,51 @@
# Feature: Tile Fetching
## Description
Handles HTTP-based satellite tile retrieval from external provider API with multiple fetching patterns: single tile, grid, progressive expansion, and route corridor prefetching. Integrates with cache for performance optimization and supports parallel fetching for throughput.
## Component APIs Implemented
- `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
- `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
- `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
- `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
## External Tools and Services
- **Satellite Provider API**: HTTP tile source (`GET /api/satellite/tiles/latlon`)
- **httpx** or **requests**: HTTP client with async support
- **numpy**: Image array handling
## Internal Dependencies
- **01_feature_tile_cache_management**: cache_tile, get_cached_tile
- **02_feature_tile_coordinate_operations**: compute_tile_coords, get_tile_grid
## Internal Methods
- `_fetch_from_api(tile_coords: TileCoords) -> Optional[np.ndarray]`: HTTP GET to satellite provider, handles response parsing
- `_fetch_with_retry(tile_coords: TileCoords, max_retries: int = 3) -> Optional[np.ndarray]`: Wraps _fetch_from_api with retry logic
- `_fetch_tiles_parallel(tiles: List[TileCoords], max_concurrent: int = 20) -> Dict[str, np.ndarray]`: Parallel fetching with connection pooling
- `_compute_corridor_tiles(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> List[TileCoords]`: Calculates tiles covering route corridor polygon
- `_generate_tile_id(tile_coords: TileCoords) -> str`: Creates unique tile identifier string
## Unit Tests
1. **fetch_tile_cache_hit**: Tile in cache → returns immediately, no HTTP call
2. **fetch_tile_cache_miss**: Not cached → HTTP fetch, cache, return
3. **fetch_tile_api_error**: HTTP 500 → returns None
4. **fetch_tile_invalid_coords**: Invalid GPS → returns None
5. **fetch_tile_retry_success**: First attempt fails, second succeeds → returns tile
6. **fetch_tile_retry_exhausted**: All 3 attempts fail → returns None
7. **fetch_tile_grid_2x2**: grid_size=4 → returns dict with 4 tiles
8. **fetch_tile_grid_3x3**: grid_size=9 → returns dict with 9 tiles
9. **fetch_tile_grid_partial_failure**: 2 of 9 tiles fail → returns 7 tiles
10. **fetch_tile_grid_all_cached**: All tiles cached → no HTTP calls
11. **prefetch_route_corridor_success**: 10 waypoints → prefetches tiles, returns True
12. **prefetch_route_corridor_partial_failure**: Some tiles fail → continues, returns True
13. **prefetch_route_corridor_complete_failure**: All tiles fail → returns False
14. **progressive_fetch_yields_sequence**: [1,4,9] → yields 3 dicts in order
15. **progressive_fetch_early_termination**: Break after 4 → doesn't fetch 9,16,25
## Integration Tests
1. **fetch_and_cache_verify**: fetch_tile() → get_cached_tile() returns same data
2. **progressive_search_simulation**: progressive_fetch with simulated match on grid 9
3. **grid_expansion_no_refetch**: fetch_tile_grid(4) then expand → no duplicate fetches
4. **corridor_prefetch_coverage**: prefetch_route_corridor → all corridor tiles cached
5. **concurrent_fetch_stress**: Fetch 100 tiles in parallel → all complete within timeout
_docs/02_components/04_satellite_data_manager/04._component_satellite_data_manager.md
+562
View File
@@ -0,0 +1,562 @@
# Satellite Data Manager
## Interface Definition
**Interface Name**: `ISatelliteDataManager`
### Interface Methods
```python
class ISatelliteDataManager(ABC):
@abstractmethod
def fetch_tile(self, lat: float, lon: float, zoom: int) -> Optional[np.ndarray]:
pass
@abstractmethod
def fetch_tile_grid(self, center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]:
pass
@abstractmethod
def prefetch_route_corridor(self, waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool:
pass
@abstractmethod
def progressive_fetch(self, center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]:
pass
@abstractmethod
def cache_tile(self, flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool:
pass
@abstractmethod
def get_cached_tile(self, flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]:
pass
@abstractmethod
def get_tile_grid(self, center: TileCoords, grid_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_coords(self, lat: float, lon: float, zoom: int) -> TileCoords:
pass
@abstractmethod
def expand_search_grid(self, center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_bounds(self, tile_coords: TileCoords) -> TileBounds:
pass
@abstractmethod
def clear_flight_cache(self, flight_id: str) -> bool:
pass
```
## Component Description
### Responsibilities
- Fetch satellite tiles from external provider API
- Manage local tile cache per flight
- Calculate tile coordinates and grid layouts
- Support progressive tile grid expansion (1→4→9→16→25)
- Handle Web Mercator projection calculations
- Coordinate corridor prefetching for flight routes
### Scope
- **HTTP client** for Satellite Provider API
- **Local caching** with disk storage
- **Grid calculations** for search patterns
- **Tile coordinate transformations** (GPS↔Tile coordinates)
- **Progressive retrieval** for "kidnapped robot" recovery
## API Methods
### `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
**Description**: Fetches a single satellite tile by GPS coordinates.
**Called By**:
- F09 Metric Refinement (single tile for drift correction)
- Internal (during prefetching)
**Input**:
```python
lat: float # Latitude
lon: float # Longitude
zoom: int # Zoom level (19 for 0.3m/pixel at Ukraine latitude)
```
**Output**:
```python
np.ndarray: Tile image (H×W×3 RGB) or None if failed
```
**HTTP Request**:
```
GET /api/satellite/tiles/latlon?lat={lat}&lon={lon}&zoom={zoom}
```
**Processing Flow**:
1. Convert GPS to tile coordinates
2. Check cache
3. If not cached, fetch from satellite provider
4. Cache tile
5. Return tile image
**Error Conditions**:
- Returns `None`: Tile unavailable, HTTP error
- Logs errors for monitoring
**Test Cases**:
1. **Cache hit**: Tile in cache → returns immediately
2. **Cache miss**: Fetches from API → caches → returns
3. **API error**: Returns None
4. **Invalid coordinates**: Returns None
---
### `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
**Description**: Fetches NxN grid of tiles centered on GPS coordinates.
**Called By**:
- F09 Metric Refinement (for progressive search)
- F11 Failure Recovery Coordinator
**Input**:
```python
center_lat: float
center_lon: float
grid_size: int # 1, 4 (2×2), 9 (3×3), 16 (4×4), or 25 (5×5)
zoom: int
```
**Output**:
```python
Dict[str, np.ndarray] # tile_id -> tile_image
```
**Processing Flow**:
1. Compute tile grid centered on coordinates
2. For each tile in grid:
- Check cache
- If not cached, fetch from API
3. Return dict of tiles
**HTTP Request** (if using batch endpoint):
```
GET /api/satellite/tiles/batch?tiles=[...]
```
**Error Conditions**:
- Returns partial dict if some tiles fail
- Empty dict if all tiles fail
**Test Cases**:
1. **2×2 grid**: Returns 4 tiles
2. **3×3 grid**: Returns 9 tiles
3. **5×5 grid**: Returns 25 tiles
4. **Partial failure**: Some tiles unavailable → returns available tiles
5. **All cached**: Fast retrieval without HTTP requests
---
### `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
**Description**: Prefetches satellite tiles along route corridor for a flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight creation)
**Input**:
```python
waypoints: List[GPSPoint] # Rough route waypoints
corridor_width_m: float # Corridor width in meters (e.g., 1000m)
zoom: int
```
**Output**:
```python
bool: True if prefetch completed, False on error
```
**Processing Flow**:
1. For each waypoint pair:
- Calculate corridor polygon
- Determine tiles covering corridor
2. Fetch tiles (async, parallel)
3. Cache all tiles with flight_id reference
**Algorithm**:
- Use H06 Web Mercator Utils for tile calculations
- Parallel fetching (10-20 concurrent requests)
- Progress tracking for monitoring
**Error Conditions**:
- Returns `False`: Major error preventing prefetch
- Logs warnings for individual tile failures
**Test Cases**:
1. **Simple route**: 10 waypoints → prefetches 50-100 tiles
2. **Long route**: 50 waypoints → prefetches 200-500 tiles
3. **Partial failure**: Some tiles fail → continues, returns True
4. **Complete failure**: All tiles fail → returns False
---
### `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
**Description**: Progressively fetches expanding tile grids for "kidnapped robot" recovery.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search)
**Input**:
```python
center_lat: float
center_lon: float
grid_sizes: List[int] # e.g., [1, 4, 9, 16, 25]
zoom: int
```
**Output**:
```python
Iterator yielding Dict[str, np.ndarray] for each grid size
```
**Processing Flow**:
1. For each grid_size in sequence:
- Fetch tile grid
- Yield tiles
- If match found by caller, iterator can be stopped
**Usage Pattern**:
```python
for tiles in progressive_fetch(lat, lon, [1, 4, 9, 16, 25], 19):
if litesam_match_found(tiles):
break # Stop expanding search
```
**Test Cases**:
1. **Progressive search**: Yields 1, then 4, then 9 tiles
2. **Early termination**: Match on 4 tiles → doesn't fetch 9, 16, 25
3. **Full search**: No match → fetches all grid sizes
---
### `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
**Description**: Caches a satellite tile to disk with flight_id association.
**Called By**:
- Internal (after fetching tiles)
**Input**:
```python
flight_id: str # Flight this tile belongs to
tile_coords: TileCoords:
x: int
y: int
zoom: int
tile_data: np.ndarray
```
**Output**:
```python
bool: True if cached successfully
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Create flight cache directory if not exists
3. Serialize tile_data (PNG format)
4. Write to disk cache directory
5. Update cache index with flight_id association
**Error Conditions**:
- Returns `False`: Disk write error, space full
**Test Cases**:
1. **Cache new tile**: Writes successfully
2. **Overwrite existing**: Updates tile
3. **Disk full**: Returns False
---
### `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
**Description**: Retrieves a cached tile from disk, checking flight-specific cache first.
**Called By**:
- Internal (before fetching from API)
- F09 Metric Refinement (direct cache lookup)
**Input**:
```python
flight_id: str # Flight to check cache for
tile_coords: TileCoords
```
**Output**:
```python
Optional[np.ndarray]: Tile image or None if not cached
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Check flight-specific cache first
3. If not found, check global cache (shared tiles)
4. If file exists, load and deserialize
5. Return tile_data or None
**Error Conditions**:
- Returns `None`: Not cached, corrupted file
**Test Cases**:
1. **Cache hit**: Returns tile quickly
2. **Cache miss**: Returns None
3. **Corrupted cache**: Returns None, logs warning
---
### `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
**Description**: Calculates tile coordinates for NxN grid centered on a tile.
**Called By**:
- Internal (for grid fetching)
- F11 Failure Recovery Coordinator
**Input**:
```python
center: TileCoords
grid_size: int # 1, 4, 9, 16, 25
```
**Output**:
```python
List[TileCoords] # List of tile coordinates in grid
```
**Algorithm**:
- For grid_size=9 (3×3): tiles from center-1 to center+1 in both x and y
- For grid_size=16 (4×4): asymmetric grid with center slightly off-center
**Test Cases**:
1. **1-tile grid**: Returns [center]
2. **4-tile grid (2×2)**: Returns 4 tiles
3. **9-tile grid (3×3)**: Returns 9 tiles centered
4. **25-tile grid (5×5)**: Returns 25 tiles
---
### `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
**Description**: Converts GPS coordinates to tile coordinates.
**Called By**:
- All methods that need tile coordinates from GPS
**Input**:
```python
lat: float
lon: float
zoom: int
```
**Output**:
```python
TileCoords:
x: int
y: int
zoom: int
```
**Algorithm** (Web Mercator):
```python
n = 2 ** zoom
x = int((lon + 180) / 360 * n)
lat_rad = lat * π / 180
y = int((1 - log(tan(lat_rad) + sec(lat_rad)) / π) / 2 * n)
```
**Test Cases**:
1. **Ukraine coordinates**: Produces valid tile coords
2. **Edge cases**: lat=0, lon=0, lat=90, lon=180
---
### `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
**Description**: Returns only NEW tiles when expanding from current grid to larger grid.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search optimization)
**Input**:
```python
center: TileCoords
current_size: int # e.g., 4
new_size: int # e.g., 9
```
**Output**:
```python
List[TileCoords] # Only tiles not in current_size grid
```
**Purpose**: Avoid re-fetching tiles already tried in smaller grid.
**Test Cases**:
1. **4→9 expansion**: Returns 5 new tiles (9-4)
2. **9→16 expansion**: Returns 7 new tiles
3. **1→4 expansion**: Returns 3 new tiles
---
### `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
**Description**: Computes GPS bounding box of a tile.
**Called By**:
- F09 Metric Refinement (for homography calculations)
- H06 Web Mercator Utils (shared calculation)
**Input**:
```python
tile_coords: TileCoords
```
**Output**:
```python
TileBounds:
nw: GPSPoint # North-West corner
ne: GPSPoint # North-East corner
sw: GPSPoint # South-West corner
se: GPSPoint # South-East corner
center: GPSPoint
gsd: float # Ground Sampling Distance (meters/pixel)
```
**Algorithm**:
- Inverse Web Mercator projection
- GSD calculation: `156543.03392 * cos(lat * π/180) / 2^zoom`
**Test Cases**:
1. **Zoom 19 at Ukraine**: GSD ≈ 0.3 m/pixel
2. **Tile bounds**: Valid GPS coordinates
---
### `clear_flight_cache(flight_id: str) -> bool`
**Description**: Clears cached tiles for a completed flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (cleanup after flight completion)
**Input**:
```python
flight_id: str
```
**Output**:
```python
bool: True if cleared successfully
```
**Processing Flow**:
1. Find all tiles associated with flight_id
2. Delete tile files
3. Update cache index
**Test Cases**:
1. **Clear flight cache**: Removes all associated tiles
2. **Non-existent flight**: Returns True (no-op)
## Integration Tests
### Test 1: Prefetch and Retrieval
1. prefetch_route_corridor() with 20 waypoints
2. Verify tiles cached
3. get_cached_tile() for each tile → all hit cache
4. clear_flight_cache() → cache cleared
### Test 2: Progressive Search Simulation
1. progressive_fetch() with [1, 4, 9, 16, 25]
2. Simulate match on 9 tiles
3. Verify only 1, 4, 9 fetched (not 16, 25)
### Test 3: Grid Expansion
1. fetch_tile_grid(4) → 4 tiles
2. expand_search_grid(4, 9) → 5 new tiles
3. Verify no duplicate fetches
## Non-Functional Requirements
### Performance
- **fetch_tile**: < 200ms (cached: < 10ms)
- **fetch_tile_grid(9)**: < 1 second
- **prefetch_route_corridor**: < 30 seconds for 500 tiles
- **Cache lookup**: < 5ms
### Scalability
- Cache 10,000+ tiles per flight
- Support 100 concurrent tile fetches
- Handle 10GB+ cache size
### Reliability
- Retry failed HTTP requests (3 attempts)
- Graceful degradation on partial failures
- Cache corruption recovery
## Dependencies
### Internal Components
- **H06 Web Mercator Utils**: Tile coordinate calculations
**Note on Tile Coordinate Calculations**: F04 delegates ALL tile coordinate calculations to H06 Web Mercator Utils:
- `compute_tile_coords()` → internally calls `H06.latlon_to_tile()`
- `compute_tile_bounds()` → internally calls `H06.compute_tile_bounds()`
- `get_tile_grid()` → uses H06 for coordinate math
This ensures single source of truth for Web Mercator projection logic and avoids duplication with H06.
### External Dependencies
- **Satellite Provider API**: HTTP tile source
- **requests** or **httpx**: HTTP client
- **numpy**: Image handling
- **opencv-python**: Image I/O
- **diskcache**: Persistent cache
## Data Models
### TileCoords
```python
class TileCoords(BaseModel):
x: int
y: int
zoom: int
```
### TileBounds
```python
class TileBounds(BaseModel):
nw: GPSPoint
ne: GPSPoint
sw: GPSPoint
se: GPSPoint
center: GPSPoint
gsd: float # meters/pixel
```
### CacheConfig
```python
class CacheConfig(BaseModel):
cache_dir: str = "./satellite_cache"
max_size_gb: int = 50
eviction_policy: str = "lru"
ttl_days: int = 30
```
_docs/02_components/05_image_input_pipeline/05.01_feature_batch_queue_management.md
+61
View File
@@ -0,0 +1,61 @@
# Feature: Batch Queue Management
## Description
Handles ingestion, validation, and FIFO queuing of image batches. This feature manages the entry point for all images into the system, ensuring sequence integrity and proper queuing for downstream processing.
## Component APIs Implemented
- `queue_batch(flight_id: str, batch: ImageBatch) -> bool`
- `validate_batch(batch: ImageBatch) -> ValidationResult`
- `process_next_batch(flight_id: str) -> Optional[ProcessedBatch]`
## External Tools and Services
- **H08 Batch Validator**: Delegated validation for naming convention, sequence continuity, format, dimensions
- **Pillow**: Image decoding and metadata extraction
- **opencv-python**: Image I/O operations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_add_to_queue(flight_id, batch)` | Adds validated batch to flight's FIFO queue |
| `_dequeue_batch(flight_id)` | Removes and returns next batch from queue |
| `_check_sequence_continuity(flight_id, batch)` | Validates batch continues from last processed sequence |
| `_decode_images(batch)` | Decompresses/decodes raw image bytes to ImageData |
| `_extract_metadata(image_bytes)` | Extracts EXIF, dimensions from raw image |
| `_get_queue_capacity(flight_id)` | Returns remaining queue capacity for backpressure |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_queue_batch_valid` | Valid batch queued successfully, returns True |
| `test_queue_batch_sequence_gap` | Batch with sequence gap from last processed → ValidationError |
| `test_queue_batch_invalid_naming` | Non-consecutive filenames → ValidationError |
| `test_queue_batch_queue_full` | Queue at capacity → QueueFullError |
| `test_validate_batch_size_min` | 9 images → invalid (min 10) |
| `test_validate_batch_size_max` | 51 images → invalid (max 50) |
| `test_validate_batch_naming_convention` | ADxxxxxx.jpg format validated |
| `test_validate_batch_invalid_format` | IMG_0001.jpg → invalid |
| `test_validate_batch_non_consecutive` | AD000101, AD000103 → invalid |
| `test_validate_batch_file_format` | JPEG/PNG accepted, others rejected |
| `test_validate_batch_dimensions` | Within 640x480 to 6252x4168 |
| `test_validate_batch_file_size` | < 10MB per image |
| `test_process_next_batch_dequeue` | Returns ProcessedBatch with decoded images |
| `test_process_next_batch_empty_queue` | Empty queue → returns None |
| `test_process_next_batch_corrupted_image` | Corrupted image skipped, others processed |
| `test_process_next_batch_metadata_extraction` | EXIF and dimensions extracted correctly |
| `test_fifo_order` | Multiple batches processed in queue order |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_batch_flow_queue_to_process` | queue_batch → process_next_batch → verify ImageData list |
| `test_multiple_batches_fifo` | Queue 5 batches, process in order, verify sequence maintained |
| `test_batch_validation_with_h08` | Integration with H08 Batch Validator |
| `test_concurrent_queue_access` | Multiple flights queuing simultaneously |
| `test_backpressure_handling` | Queue fills up, backpressure signal returned |
_docs/02_components/05_image_input_pipeline/05.02_feature_image_storage_retrieval.md
+70
View File
@@ -0,0 +1,70 @@
# Feature: Image Storage and Retrieval
## Description
Handles persistent storage of processed images and provides retrieval mechanisms for sequential processing, random access, and metadata queries. Manages disk storage structure, maintains sequence tracking per flight, and provides processing status information.
## Component APIs Implemented
- `store_images(flight_id: str, images: List[ImageData]) -> bool`
- `get_next_image(flight_id: str) -> Optional[ImageData]`
- `get_image_by_sequence(flight_id: str, sequence: int) -> Optional[ImageData]`
- `get_image_metadata(flight_id: str, sequence: int) -> Optional[ImageMetadata]`
- `get_processing_status(flight_id: str) -> ProcessingStatus`
## External Tools and Services
- **F03 Flight Database**: Metadata persistence, flight state queries
- **opencv-python**: Image I/O (cv2.imread, cv2.imwrite)
- **numpy**: Image array handling
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_create_flight_directory(flight_id)` | Creates storage directory structure for flight |
| `_write_image(flight_id, filename, image_data)` | Writes single image to disk |
| `_update_metadata_index(flight_id, metadata_list)` | Updates metadata.json with new image metadata |
| `_load_image_from_disk(flight_id, filename)` | Reads image file and returns np.ndarray |
| `_construct_filename(sequence)` | Converts sequence number to ADxxxxxx.jpg format |
| `_get_sequence_tracker(flight_id)` | Gets/initializes current sequence position for flight |
| `_increment_sequence(flight_id)` | Advances sequence counter after get_next_image |
| `_load_metadata_from_index(flight_id, sequence)` | Reads metadata from index without loading image |
| `_calculate_processing_rate(flight_id)` | Computes images/second processing rate |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_store_images_success` | All images written to correct paths |
| `test_store_images_creates_directory` | Flight directory created if not exists |
| `test_store_images_updates_metadata` | metadata.json updated with image info |
| `test_store_images_disk_full` | Storage error returns False |
| `test_get_next_image_sequential` | Returns images in sequence order |
| `test_get_next_image_increments_counter` | Sequence counter advances after each call |
| `test_get_next_image_end_of_sequence` | Returns None when no more images |
| `test_get_next_image_missing_file` | Handles missing image gracefully |
| `test_get_image_by_sequence_valid` | Returns correct image for sequence number |
| `test_get_image_by_sequence_invalid` | Invalid sequence returns None |
| `test_get_image_by_sequence_constructs_filename` | Sequence 101 → AD000101.jpg |
| `test_get_image_metadata_fast` | Returns metadata without loading full image |
| `test_get_image_metadata_missing` | Missing image returns None |
| `test_get_image_metadata_contains_fields` | Returns sequence, filename, dimensions, file_size, timestamp, exif |
| `test_get_processing_status_counts` | Accurate total_images, processed_images counts |
| `test_get_processing_status_current_sequence` | Reflects current processing position |
| `test_get_processing_status_queued_batches` | Includes queue depth |
| `test_get_processing_status_rate` | Processing rate calculation |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_store_then_retrieve_sequential` | store_images → get_next_image × N → all images retrieved |
| `test_store_then_retrieve_by_sequence` | store_images → get_image_by_sequence → correct image |
| `test_metadata_persistence_f03` | store_images → metadata persisted to F03 Flight Database |
| `test_crash_recovery_resume` | Restart processing from last stored sequence |
| `test_concurrent_retrieval` | Multiple consumers retrieving images simultaneously |
| `test_storage_large_batch` | Store and retrieve 3000 images for single flight |
| `test_multiple_flights_isolation` | Multiple flights don't interfere with each other's storage |
| `test_status_updates_realtime` | Status reflects current state during active processing |
_docs/02_components/05_image_input_pipeline/05._component_image_input_pipeline.md
+455
View File
@@ -0,0 +1,455 @@
# Image Input Pipeline
## Interface Definition
**Interface Name**: `IImageInputPipeline`
### Interface Methods
```python
class IImageInputPipeline(ABC):
@abstractmethod
def queue_batch(self, flight_id: str, batch: ImageBatch) -> bool:
pass
@abstractmethod
def process_next_batch(self, flight_id: str) -> Optional[ProcessedBatch]:
pass
@abstractmethod
def validate_batch(self, batch: ImageBatch) -> ValidationResult:
pass
@abstractmethod
def store_images(self, flight_id: str, images: List[ImageData]) -> bool:
pass
@abstractmethod
def get_next_image(self, flight_id: str) -> Optional[ImageData]:
pass
@abstractmethod
def get_image_by_sequence(self, flight_id: str, sequence: int) -> Optional[ImageData]:
pass
@abstractmethod
def get_image_metadata(self, flight_id: str, sequence: int) -> Optional[ImageMetadata]:
pass
@abstractmethod
def get_processing_status(self, flight_id: str) -> ProcessingStatus:
pass
```
## Component Description
### Responsibilities
- Unified image ingestion, validation, storage, and retrieval
- FIFO batch queuing for processing
- Validate consecutive naming (AD000001, AD000002, etc.)
- Validate sequence integrity (strict sequential ordering)
- Image persistence with indexed retrieval
- Metadata extraction (EXIF, dimensions)
### Scope
- Batch queue management
- Image validation
- Disk storage management
- Sequential processing coordination
- Metadata management
## API Methods
### `queue_batch(flight_id: str, batch: ImageBatch) -> bool`
**Description**: Queues a batch of images for processing (FIFO).
**Called By**:
- F02.1 Flight Lifecycle Manager (via F01 Flight API image upload route)
**Input**:
```python
flight_id: str
batch: ImageBatch:
images: List[bytes] # Raw image data
filenames: List[str] # e.g., ["AD000101.jpg", "AD000102.jpg", ...]
start_sequence: int # 101
end_sequence: int # 150
```
**Output**:
```python
bool: True if queued successfully
```
**Processing Flow**:
1. Validate batch using H08 Batch Validator
2. Check sequence continuity (no gaps)
3. Add to FIFO queue for flight_id
4. Return immediately (async processing)
**Error Conditions**:
- `ValidationError`: Sequence gap, invalid naming
- `QueueFullError`: Queue capacity exceeded
**Test Cases**:
1. **Valid batch**: Queued successfully
2. **Sequence gap**: Batch 101-150, expecting 51-100 → error
3. **Invalid naming**: Non-consecutive names → error
4. **Queue full**: Returns error with backpressure signal
---
### `process_next_batch(flight_id: str) -> Optional[ProcessedBatch]`
**Description**: Dequeues and processes the next batch from FIFO queue.
**Called By**:
- Internal processing loop (background worker)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ProcessedBatch:
images: List[ImageData]
batch_id: str
start_sequence: int
end_sequence: int
```
**Processing Flow**:
1. Dequeue next batch
2. Decompress/decode images
3. Extract metadata (EXIF, dimensions)
4. Store images to disk
5. Return ProcessedBatch for pipeline
**Error Conditions**:
- Returns `None`: Queue empty
- `ImageCorruptionError`: Invalid image data
**Test Cases**:
1. **Process batch**: Dequeues, returns ImageData list
2. **Empty queue**: Returns None
3. **Corrupted image**: Logs error, skips image
---
### `validate_batch(batch: ImageBatch) -> ValidationResult`
**Description**: Validates batch integrity and sequence continuity.
**Called By**:
- Internal (before queuing)
- H08 Batch Validator (delegated validation)
**Input**:
```python
batch: ImageBatch
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
1. **Batch size**: 10 <= len(images) <= 50
2. **Naming convention**: ADxxxxxx.jpg (6 digits)
3. **Sequence continuity**: Consecutive numbers
4. **File format**: JPEG or PNG
5. **Image dimensions**: 640x480 to 6252x4168
6. **File size**: < 10MB per image
**Test Cases**:
1. **Valid batch**: Returns valid=True
2. **Too few images**: 5 images → invalid
3. **Too many images**: 60 images → invalid
4. **Non-consecutive**: AD000101, AD000103 → invalid
5. **Invalid naming**: IMG_0001.jpg → invalid
---
### `store_images(flight_id: str, images: List[ImageData]) -> bool`
**Description**: Persists images to disk with indexed storage.
**Called By**:
- Internal (after processing batch)
**Input**:
```python
flight_id: str
images: List[ImageData]
```
**Output**:
```python
bool: True if stored successfully
```
**Storage Structure**:
```
/image_storage/
{flight_id}/
AD000001.jpg
AD000002.jpg
metadata.json
```
**Processing Flow**:
1. Create flight directory if not exists
2. Write each image to disk
3. Update metadata index
4. Persist to F03 Database Layer (metadata only)
**Error Conditions**:
- `StorageError`: Disk full, permission error
**Test Cases**:
1. **Store batch**: All images written successfully
2. **Disk full**: Returns False
3. **Verify storage**: Images retrievable after storage
---
### `get_next_image(flight_id: str) -> Optional[ImageData]`
**Description**: Gets the next image in sequence for processing.
**Called By**:
- F02.2 Flight Processing Engine (main processing loop)
- F06 Image Rotation Manager (via F02.2)
- F07 Sequential VO (via F02.2)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ImageData:
flight_id: str
sequence: int
filename: str
image: np.ndarray # Loaded image
metadata: ImageMetadata
```
**Processing Flow**:
1. Track current sequence number for flight
2. Load next image from disk
3. Increment sequence counter
4. Return ImageData
**Error Conditions**:
- Returns `None`: No more images
- `ImageNotFoundError`: Expected image missing
**Test Cases**:
1. **Get sequential images**: Returns images in order
2. **End of sequence**: Returns None
3. **Missing image**: Handles gracefully
---
### `get_image_by_sequence(flight_id: str, sequence: int) -> Optional[ImageData]`
**Description**: Retrieves a specific image by sequence number.
**Called By**:
- F11 Failure Recovery Coordinator (for user fix)
- F13 Result Manager (for refinement)
**Input**:
```python
flight_id: str
sequence: int
```
**Output**:
```python
Optional[ImageData]
```
**Processing Flow**:
1. Construct filename from sequence (ADxxxxxx.jpg)
2. Load from disk
3. Load metadata
4. Return ImageData
**Error Conditions**:
- Returns `None`: Image not found
**Test Cases**:
1. **Get specific image**: Returns correct image
2. **Invalid sequence**: Returns None
---
### `get_image_metadata(flight_id: str, sequence: int) -> Optional[ImageMetadata]`
**Description**: Retrieves metadata without loading full image (lightweight).
**Called By**:
- F02.1 Flight Lifecycle Manager (status checks)
- F13 Result Manager (metadata-only queries)
**Input**:
```python
flight_id: str
sequence: int
```
**Output**:
```python
ImageMetadata:
sequence: int
filename: str
dimensions: Tuple[int, int] # (width, height)
file_size: int # bytes
timestamp: datetime
exif_data: Optional[Dict]
```
**Test Cases**:
1. **Get metadata**: Returns quickly without loading image
2. **Missing image**: Returns None
---
### `get_processing_status(flight_id: str) -> ProcessingStatus`
**Description**: Gets current processing status for a flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (status queries via F01 Flight API)
- F02.2 Flight Processing Engine (processing loop status)
**Input**:
```python
flight_id: str
```
**Output**:
```python
ProcessingStatus:
flight_id: str
total_images: int
processed_images: int
current_sequence: int
queued_batches: int
processing_rate: float # images/second
```
**Processing Flow**:
1. Get flight state via F03 Flight Database.get_flight(flight_id).status
2. Combine with internal queue status
3. Return ProcessingStatus
**Test Cases**:
1. **Get status**: Returns accurate counts
2. **During processing**: Updates in real-time
## Integration Tests
### Test 1: Batch Processing Flow
1. queue_batch() with 50 images
2. process_next_batch() → returns batch
3. store_images() → persists to disk
4. get_next_image() × 50 → retrieves all sequentially
5. Verify metadata
### Test 2: Multiple Batches
1. queue_batch() × 5 (250 images total)
2. process_next_batch() × 5
3. Verify FIFO order maintained
4. Verify sequence continuity
### Test 3: Error Handling
1. Queue batch with sequence gap
2. Verify validation error
3. Queue valid batch → succeeds
4. Simulate disk full → storage fails gracefully
## Non-Functional Requirements
### Performance
- **queue_batch**: < 100ms
- **process_next_batch**: < 2 seconds for 50 images
- **get_next_image**: < 50ms
- **get_image_by_sequence**: < 50ms
- **Processing throughput**: 10-20 images/second
### Scalability
- Support 3000 images per flight
- Handle 10 concurrent flights
- Manage 100GB+ image storage
### Reliability
- Crash recovery (resume processing from last sequence)
- Atomic batch operations
- Data integrity validation
## Dependencies
### Internal Components
- **F03 Flight Database**: For metadata persistence and flight state information
- **H08 Batch Validator**: For batch validation (naming convention, sequence continuity, format, dimensions)
### External Dependencies
- **opencv-python**: Image I/O
- **Pillow**: Image processing
- **numpy**: Image arrays
## Data Models
### ImageBatch
```python
class ImageBatch(BaseModel):
images: List[bytes]
filenames: List[str]
start_sequence: int
end_sequence: int
batch_number: int
```
### ImageData
```python
class ImageData(BaseModel):
flight_id: str
sequence: int
filename: str
image: np.ndarray
metadata: ImageMetadata
```
### ImageMetadata
```python
class ImageMetadata(BaseModel):
sequence: int
filename: str
dimensions: Tuple[int, int]
file_size: int
timestamp: datetime
exif_data: Optional[Dict]
```
### ProcessingStatus
```python
class ProcessingStatus(BaseModel):
flight_id: str
total_images: int
processed_images: int
current_sequence: int
queued_batches: int
processing_rate: float
```
_docs/02_components/06_image_rotation_manager/06.01_feature_image_rotation_core.md
+39
View File
@@ -0,0 +1,39 @@
# Feature: Image Rotation Core
## Description
Pure image rotation operations without state. Provides utility functions to rotate single images and batches of images by specified angles around their center. This is the foundation for rotation sweeps and pre-rotation before matching.
## Component APIs Implemented
- `rotate_image_360(image: np.ndarray, angle: float) -> np.ndarray`
- `rotate_chunk_360(chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]`
## External Tools and Services
- **opencv-python**: `cv2.warpAffine` for rotation transformation
- **numpy**: Matrix operations for rotation matrix construction
## Internal Methods
- `_build_rotation_matrix(center: Tuple[float, float], angle: float) -> np.ndarray`: Constructs 2x3 affine rotation matrix
- `_get_image_center(image: np.ndarray) -> Tuple[float, float]`: Calculates image center coordinates
## Unit Tests
### rotate_image_360
1. **Rotate 0°**: Input image equals output image (identity)
2. **Rotate 90°**: Image rotated 90° clockwise, dimensions preserved
3. **Rotate 180°**: Image inverted correctly
4. **Rotate 270°**: Image rotated 270° clockwise
5. **Rotate 45°**: Diagonal rotation with black fill at corners
6. **Rotate 360°**: Equivalent to 0° rotation
7. **Negative angle**: -90° equivalent to 270°
8. **Large angle normalization**: 450° equivalent to 90°
### rotate_chunk_360
1. **Empty chunk**: Returns empty list
2. **Single image chunk**: Equivalent to rotate_image_360
3. **Multiple images**: All images rotated by same angle
4. **Image independence**: Original chunk images unchanged
5. **Consistent dimensions**: All output images have same dimensions as input
## Integration Tests
None - this feature is stateless and has no external dependencies beyond opencv/numpy.
_docs/02_components/06_image_rotation_manager/06.02_feature_heading_management.md
+70
View File
@@ -0,0 +1,70 @@
# Feature: Heading Management
## Description
Manages UAV heading state per flight. Tracks current heading, maintains heading history, detects sharp turns, and determines when rotation sweeps are required. This is the stateful core of the rotation manager.
## Component APIs Implemented
- `get_current_heading(flight_id: str) -> Optional[float]`
- `update_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
- `detect_sharp_turn(flight_id: str, new_heading: float) -> bool`
- `requires_rotation_sweep(flight_id: str) -> bool`
## External Tools and Services
None - pure Python state management.
## Internal Methods
- `_normalize_angle(angle: float) -> float`: Normalizes angle to 0-360 range
- `_calculate_angle_delta(angle1: float, angle2: float) -> float`: Calculates smallest delta between two angles (handles wraparound)
- `_get_flight_state(flight_id: str) -> HeadingHistory`: Gets or creates heading state for flight
- `_add_to_history(flight_id: str, heading: float)`: Adds heading to circular history buffer
- `_set_sweep_required(flight_id: str, required: bool)`: Sets sweep required flag (used after tracking loss)
## Unit Tests
### get_current_heading
1. **New flight**: Returns None (no heading set)
2. **After update**: Returns last updated heading
3. **Multiple flights**: Each flight has independent heading
### update_heading
1. **First heading**: Sets initial heading, returns True
2. **Update heading**: Overwrites previous heading
3. **Angle normalization**: 370° stored as 10°
4. **Negative normalization**: -30° stored as 330°
5. **History tracking**: Heading added to history list
6. **History limit**: Only last 10 headings kept
### detect_sharp_turn
1. **No current heading**: Returns False (can't detect turn)
2. **Small turn (15°)**: 60° → 75° returns False
3. **Sharp turn (60°)**: 60° → 120° returns True
4. **Exactly 45°**: Returns False (threshold is >45)
5. **Exactly 46°**: Returns True
6. **Wraparound small**: 350° → 20° returns False (30° delta)
7. **Wraparound sharp**: 350° → 60° returns True (70° delta)
8. **180° turn**: 0° → 180° returns True
### requires_rotation_sweep
1. **First frame (no heading)**: Returns True
2. **Heading known, no flags**: Returns False
3. **Tracking loss flag set**: Returns True
4. **Sharp turn detected recently**: Returns True
5. **After successful match**: Returns False
## Integration Tests
### Test 1: Heading Lifecycle
1. Create new flight
2. get_current_heading → None
3. requires_rotation_sweep → True
4. update_heading(heading=45°)
5. get_current_heading → 45°
6. requires_rotation_sweep → False
### Test 2: Sharp Turn Flow
1. update_heading(heading=90°)
2. detect_sharp_turn(new_heading=100°) → False
3. detect_sharp_turn(new_heading=180°) → True
4. Set sweep required flag
5. requires_rotation_sweep → True
_docs/02_components/06_image_rotation_manager/06.03_feature_rotation_sweep_orchestration.md
+75
View File
@@ -0,0 +1,75 @@
# Feature: Rotation Sweep Orchestration
## Description
Coordinates rotation sweeps by rotating images at 30° steps and delegating matching to an injected matcher (F09 Metric Refinement). Calculates precise angles from homography matrices after successful matches. This feature ties together image rotation and heading management with external matching.
## Component APIs Implemented
- `try_rotation_steps(flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]`
- `try_chunk_rotation_steps(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]`
- `calculate_precise_angle(homography: np.ndarray, initial_angle: float) -> float`
## External Tools and Services
- **H07 Image Rotation Utils**: Angle extraction from homography
- **IImageMatcher (injected)**: F09 Metric Refinement for align_to_satellite and align_chunk_to_satellite
## Internal Methods
- `_get_rotation_steps() -> List[float]`: Returns [0, 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330]
- `_extract_rotation_from_homography(homography: np.ndarray) -> float`: Extracts rotation component from 3x3 homography
- `_combine_angles(initial_angle: float, delta_angle: float) -> float`: Combines step angle with homography delta, normalizes result
- `_select_best_result(results: List[Tuple[float, AlignmentResult]]) -> Tuple[float, AlignmentResult]`: Selects highest confidence match if multiple found
## Unit Tests
### calculate_precise_angle
1. **Identity homography**: Returns initial_angle unchanged
2. **Small rotation delta**: initial=60°, homography shows +2.5° → returns 62.5°
3. **Negative delta**: initial=30°, homography shows -3° → returns 27°
4. **Large delta normalization**: initial=350°, delta=+20° → returns 10°
5. **Invalid homography (singular)**: Returns initial_angle as fallback
6. **Near-zero homography**: Returns initial_angle as fallback
### try_rotation_steps (with mock matcher)
1. **Match at 0°**: First rotation matches, returns RotationResult with initial_angle=0
2. **Match at 60°**: Third rotation matches, returns RotationResult with initial_angle=60
3. **Match at 330°**: Last rotation matches, returns RotationResult with initial_angle=330
4. **No match**: All 12 rotations fail, returns None
5. **Multiple matches**: Returns highest confidence result
6. **Heading updated**: After match, flight heading is updated
7. **Confidence threshold**: Match below threshold rejected
### try_chunk_rotation_steps (with mock matcher)
1. **Match at 0°**: First rotation matches chunk
2. **Match at 120°**: Returns RotationResult with initial_angle=120
3. **No match**: All 12 rotations fail, returns None
4. **Chunk consistency**: All images rotated by same angle before matching
5. **Does not update heading**: Chunk matching doesn't affect flight state
## Integration Tests
### Test 1: First Frame Rotation Sweep
1. Create flight with no heading
2. Call try_rotation_steps with image, satellite tile, mock matcher
3. Mock matcher returns match at 60° rotation
4. Verify RotationResult.initial_angle = 60
5. Verify RotationResult.precise_angle refined from homography
6. Verify flight heading updated to precise_angle
### Test 2: Full Sweep No Match
1. Call try_rotation_steps with mock matcher that never matches
2. Verify all 12 rotations attempted (0°, 30°, ..., 330°)
3. Verify returns None
4. Verify flight heading unchanged
### Test 3: Chunk Rotation Sweep
1. Create chunk with 10 images
2. Call try_chunk_rotation_steps with mock matcher
3. Mock matcher returns match at 90° rotation
4. Verify all 10 images were rotated before matching call
5. Verify RotationResult returned with correct angles
### Test 4: Precise Angle Calculation
1. Perform rotation sweep, match at 60° step
2. Homography indicates +2.3° additional rotation
3. Verify precise_angle = 62.3°
4. Verify heading updated to 62.3° (not 60°)
_docs/02_components/06_image_rotation_manager/06._component_image_rotation_manager.md
+554
View File
@@ -0,0 +1,554 @@
# Image Rotation Manager
## Interface Definition
**Interface Name**: `IImageRotationManager`
### Interface Methods
```python
class IImageRotationManager(ABC):
@abstractmethod
def rotate_image_360(self, image: np.ndarray, angle: float) -> np.ndarray:
pass
@abstractmethod
def try_rotation_steps(self, flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]:
"""
Performs rotation sweep.
'matcher' is an injected dependency (usually F09) to avoid direct coupling.
"""
pass
@abstractmethod
def calculate_precise_angle(self, homography: np.ndarray, initial_angle: float) -> float:
pass
@abstractmethod
def get_current_heading(self, flight_id: str) -> Optional[float]:
pass
@abstractmethod
def update_heading(self, flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool:
pass
@abstractmethod
def detect_sharp_turn(self, flight_id: str, new_heading: float) -> bool:
pass
@abstractmethod
def requires_rotation_sweep(self, flight_id: str) -> bool:
pass
@abstractmethod
def rotate_chunk_360(self, chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]:
pass
@abstractmethod
def try_chunk_rotation_steps(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]:
pass
```
## Component Description
### Responsibilities
- Image rotation utility.
- Heading tracking.
- Coordination of rotation sweeps.
### Decoupling Fix
- **Problem**: F06 previously depended directly on `F09 Metric Refinement`.
- **Fix**: Methods `try_rotation_steps` and `try_chunk_rotation_steps` now accept a `matcher` argument conforming to `IImageMatcher`.
- **IImageMatcher Interface**:
```python
class IImageMatcher(ABC):
def align_to_satellite(self, uav_image, satellite_tile, tile_bounds) -> AlignmentResult: pass
def align_chunk_to_satellite(self, chunk_images, satellite_tile, tile_bounds) -> ChunkAlignmentResult: pass
```
- **Runtime**: F02.2 injects F09 instance when calling F06 methods.
### Scope
- Image rotation operations (pure rotation, no matching)
- UAV heading tracking and history
- Sharp turn detection
- Rotation sweep coordination (rotates images, delegates matching to F09 Metric Refinement)
- Precise angle calculation from homography (extracted from F09 results)
- **Chunk-level rotation (all images rotated by same angle)**
## API Methods
### `rotate_image_360(image: np.ndarray, angle: float) -> np.ndarray`
**Description**: Rotates an image by specified angle around center.
**Called By**:
- Internal (during rotation sweep)
- H07 Image Rotation Utils (may delegate to)
**Input**:
```python
image: np.ndarray # Input image (H×W×3)
angle: float # Rotation angle in degrees (0-360)
```
**Output**:
```python
np.ndarray # Rotated image (same dimensions)
```
**Processing Details**:
- Rotation around image center
- Preserves image dimensions
- Fills borders with black or extrapolation
**Error Conditions**:
- None (always returns rotated image)
**Test Cases**:
1. **Rotate 90°**: Image rotated correctly
2. **Rotate 0°**: Image unchanged
3. **Rotate 180°**: Image inverted
4. **Rotate 45°**: Diagonal rotation
---
### `try_rotation_steps(flight_id: str, frame_id: int, image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds, timestamp: datetime, matcher: IImageMatcher) -> Optional[RotationResult]`
**Description**: Performs 30° rotation sweep, rotating image at each step and delegating matching to F09 Metric Refinement.
**Called By**:
- Internal (when requires_rotation_sweep() returns True)
- Main processing loop (first frame or sharp turn)
**Input**:
```python
flight_id: str
frame_id: int # Frame identifier for heading persistence
image: np.ndarray # UAV image
satellite_tile: np.ndarray # Satellite reference tile
tile_bounds: TileBounds # GPS bounds and GSD of satellite tile (passed to F09)
timestamp: datetime # Timestamp for heading persistence
matcher: IImageMatcher # Injected matcher (F09)
```
**About tile_bounds**: `TileBounds` contains the GPS bounding box of the satellite tile:
- `nw`, `ne`, `sw`, `se`: GPS coordinates of tile corners
- `center`: GPS coordinate of tile center
- `gsd`: Ground Sampling Distance (meters/pixel)
The caller (F02 Flight Processor) obtains tile_bounds by calling `F04.compute_tile_bounds(tile_coords)` before calling this method. F06 passes tile_bounds to F09.align_to_satellite() which uses it to convert pixel coordinates to GPS.
**Output**:
```python
RotationResult:
matched: bool
initial_angle: float # Best matching step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
```
**Algorithm**:
```
For angle in [0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°]:
rotated_image = rotate_image_360(image, angle)
result = matcher.align_to_satellite(rotated_image, satellite_tile, tile_bounds)
if result.matched and result.confidence > threshold:
precise_angle = calculate_precise_angle(result.homography, angle)
update_heading(flight_id, frame_id, precise_angle, timestamp)
return RotationResult(matched=True, initial_angle=angle, precise_angle=precise_angle, ...)
return None # No match found
```
**Processing Flow**:
1. For each 30° step:
- Rotate image via rotate_image_360()
- Call matcher.align_to_satellite(rotated_image, satellite_tile, tile_bounds)
- Check if match found
2. If match found:
- Calculate precise angle from homography via calculate_precise_angle()
- Update UAV heading via update_heading()
- Return RotationResult
3. If no match:
- Return None (triggers progressive search expansion)
**Error Conditions**:
- Returns `None`: No match found in any rotation
- This is expected behavior (leads to progressive search)
**Test Cases**:
1. **Match at 60°**: Finds match, returns result
2. **Match at 0°**: No rotation needed, finds match
3. **No match**: All 12 rotations tried, returns None
4. **Multiple matches**: Returns best confidence
---
### `calculate_precise_angle(homography: np.ndarray, initial_angle: float) -> float`
**Description**: Calculates precise rotation angle from homography matrix point shifts.
**Called By**:
- Internal (after LiteSAM match in rotation sweep)
**Input**:
```python
homography: np.ndarray # 3×3 homography matrix from LiteSAM
initial_angle: float # 30° step angle that matched
```
**Output**:
```python
float: Precise rotation angle (e.g., 62.3° refined from 60° step)
```
**Algorithm**:
1. Extract rotation component from homography
2. Calculate angle from rotation matrix
3. Refine initial_angle with delta from homography
**Uses**: H07 Image Rotation Utils for angle calculation
**Error Conditions**:
- Falls back to initial_angle if calculation fails
**Test Cases**:
1. **Refine 60°**: Returns 62.5° (small delta)
2. **Refine 0°**: Returns 3.2° (small rotation)
3. **Invalid homography**: Returns initial_angle
---
### `get_current_heading(flight_id: str) -> Optional[float]`
**Description**: Gets current UAV heading angle for a flight.
**Called By**:
- F06 Internal (to check if pre-rotation needed)
- Main processing loop (before LiteSAM)
- F11 Failure Recovery Coordinator (logging)
**Input**:
```python
flight_id: str
```
**Output**:
```python
Optional[float]: Heading angle in degrees (0-360), or None if not initialized
```
**Error Conditions**:
- Returns `None`: First frame, heading not yet determined
**Test Cases**:
1. **After first frame**: Returns heading angle
2. **Before first frame**: Returns None
3. **During flight**: Returns current heading
---
### `update_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
**Description**: Updates UAV heading angle after successful match.
**Called By**:
- Internal (after rotation sweep match)
- Internal (after normal LiteSAM match with small rotation delta)
**Input**:
```python
flight_id: str
frame_id: int # Frame identifier for database persistence
heading: float # New heading angle (0-360)
timestamp: datetime # Timestamp for database persistence
```
**Output**:
```python
bool: True if updated successfully
```
**Processing Flow**:
1. Normalize angle to 0-360 range
2. Add to heading history (last 10 headings)
3. Update current_heading for flight
4. Return True (caller F02 is responsible for persistence via F03)
**Note**: Heading persistence is the caller's responsibility (F02 Flight Processor calls F03.save_heading() after receiving the updated heading).
**Test Cases**:
1. **Update heading**: Sets new heading
2. **Angle normalization**: 370° → 10°
3. **History tracking**: Maintains last 10 headings
---
### `detect_sharp_turn(flight_id: str, new_heading: float) -> bool`
**Description**: Detects if UAV made a sharp turn (>45° heading change).
**Called By**:
- Internal (before deciding if rotation sweep needed)
- Main processing loop
**Input**:
```python
flight_id: str
new_heading: float # Proposed new heading
```
**Output**:
```python
bool: True if sharp turn detected (>45° change)
```
**Algorithm**:
```python
current = get_current_heading(flight_id)
if current is None:
return False
delta = abs(new_heading - current)
if delta > 180: # Handle wraparound
delta = 360 - delta
return delta > 45
```
**Test Cases**:
1. **Small turn**: 60° → 75° → False (15° delta)
2. **Sharp turn**: 60° → 120° → True (60° delta)
3. **Wraparound**: 350° → 20° → False (30° delta)
4. **180° turn**: 0° → 180° → True
---
### `requires_rotation_sweep(flight_id: str) -> bool`
**Description**: Determines if rotation sweep is needed for current frame.
**Called By**:
- Main processing loop (before each frame)
- F11 Failure Recovery Coordinator (after tracking loss)
**Input**:
```python
flight_id: str
```
**Output**:
```python
bool: True if rotation sweep required
```
**Conditions for sweep**:
1. **First frame**: heading not initialized
2. **Sharp turn detected**: >45° heading change from VO
3. **Tracking loss**: LiteSAM failed to match in previous frame
4. **User flag**: Manual trigger (rare)
**Test Cases**:
1. **First frame**: Returns True
2. **Second frame, no turn**: Returns False
3. **Sharp turn detected**: Returns True
4. **Tracking loss**: Returns True
---
### `rotate_chunk_360(chunk_images: List[np.ndarray], angle: float) -> List[np.ndarray]`
**Description**: Rotates all images in a chunk by the same angle.
**Called By**:
- Internal (during try_chunk_rotation_steps)
- F11 Failure Recovery Coordinator (chunk rotation sweeps)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
angle: float # Rotation angle in degrees (0-360)
```
**Output**:
```python
List[np.ndarray] # Rotated images (same dimensions)
```
**Processing Flow**:
1. For each image in chunk:
- rotate_image_360(image, angle) → rotated_image
2. Return list of rotated images
**Performance**:
- Rotation time: ~20ms × N images
- For 10 images: ~200ms total
**Test Cases**:
1. **Rotate chunk**: All images rotated correctly
2. **Angle consistency**: All images rotated by same angle
3. **Image preservation**: Original images unchanged
---
### `try_chunk_rotation_steps(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds, matcher: IImageMatcher) -> Optional[RotationResult]`
**Description**: Performs 30° rotation sweep on entire chunk, rotating all images at each step and delegating matching to F09 Metric Refinement.
**Called By**:
- F11 Failure Recovery Coordinator (chunk matching with rotation)
**Input**:
```python
chunk_images: List[np.ndarray] # Chunk images
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of satellite tile (for F09)
matcher: IImageMatcher # Injected matcher
```
**Output**:
```python
RotationResult:
matched: bool
initial_angle: float # Best matching step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
```
**Algorithm**:
```
For angle in [0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, 330°]:
rotated_chunk = rotate_chunk_360(chunk_images, angle)
result = matcher.align_chunk_to_satellite(rotated_chunk, satellite_tile, tile_bounds)
if result.matched and result.confidence > threshold:
precise_angle = calculate_precise_angle(result.homography, angle)
return RotationResult(matched=True, initial_angle=angle, precise_angle=precise_angle, ...)
return None # No match found
```
**Processing Flow**:
1. For each 30° step:
- Rotate all chunk images via rotate_chunk_360()
- Call matcher.align_chunk_to_satellite(rotated_chunk, satellite_tile, tile_bounds)
- Check if match found
2. If match found:
- Calculate precise angle from homography via calculate_precise_angle()
- Return RotationResult
3. If no match:
- Return None
**Performance**:
- 12 rotations × chunk matching via F09 (~60ms) = ~720ms
- Acceptable for chunk matching (async operation)
**Test Cases**:
1. **Match at 60°**: Finds match, returns result
2. **Match at 0°**: No rotation needed, finds match
3. **No match**: All 12 rotations tried, returns None
4. **Multiple matches**: Returns best confidence
## Integration Tests
### Test 1: First Frame Rotation Sweep
1. First frame arrives (no heading set)
2. requires_rotation_sweep() → True
3. try_rotation_steps(flight_id, frame_id=1, image, satellite_tile, tile_bounds, timestamp=now()) → rotates 12 times
4. F09 Metric Refinement called for each rotation
5. Match found at 60° step
6. calculate_precise_angle() → 62.3°
7. update_heading(flight_id, frame_id=1, heading=62.3°, timestamp=now())
8. Subsequent frames use 62.3° heading
### Test 2: Normal Frame Processing
1. Heading known (90°)
2. requires_rotation_sweep() → False
3. Pre-rotate image to 90°
4. LiteSAM match succeeds with small delta (+2.5°)
5. update_heading(flight_id, frame_id=237, heading=92.5°, timestamp=now())
### Test 3: Sharp Turn Detection
1. UAV heading 45°
2. Next frame shows 120° heading (from VO estimate)
3. detect_sharp_turn() → True (75° delta)
4. requires_rotation_sweep() → True
5. Perform rotation sweep → find match at 120° step
### Test 4: Tracking Loss Recovery
1. F09 Metric Refinement fails to match (no overlap after turn)
2. requires_rotation_sweep() → True
3. try_rotation_steps(flight_id, frame_id, image, satellite_tile, tile_bounds, timestamp) with all 12 rotations
4. F09 called for each rotation step
5. Match found → heading updated
### Test 5: Chunk Rotation Sweeps
1. Build chunk with 10 images (unknown orientation)
2. try_chunk_rotation_steps(chunk_images, satellite_tile, tile_bounds) with all 12 rotations
3. F09 Metric Refinement called for each rotation
4. Match found at 120° step
5. Precise angle calculated (122.5°)
6. Verify all images rotated consistently
## Non-Functional Requirements
### Performance
- **rotate_image_360**: < 20ms per rotation
- **try_rotation_steps**: < 1.2 seconds (12 rotations × 100ms LiteSAM)
- **calculate_precise_angle**: < 10ms
- **get_current_heading**: < 1ms
- **update_heading**: < 5ms
### Accuracy
- **Angle precision**: ±0.5° for precise angle calculation
- **Sharp turn detection**: 100% accuracy for >45° turns
### Reliability
- Rotation sweep always completes all 12 steps
- Graceful handling of no-match scenarios
- Heading history preserved across failures
## Dependencies
### Internal Components
- **H07 Image Rotation Utils**: For image rotation and angle calculations
- **Injected Matcher (F09)**.
**Note**:
- `TileBounds` data model is imported from F09 Metric Refinement.
- F06 does NOT call F04 directly. The caller (F02 or F11) provides satellite tiles and tile_bounds.
- F06 does NOT persist heading to database. The caller (F02) is responsible for calling F03.save_heading().
- Chunk rotation orchestration (calling try_chunk_rotation_steps in recovery flow) is done by F11 Failure Recovery Coordinator.
### External Dependencies
- **opencv-python**: Image rotation (`cv2.warpAffine`)
- **numpy**: Matrix operations
## Data Models
### RotationResult
```python
class RotationResult(BaseModel):
matched: bool
initial_angle: float # 30° step angle (0, 30, 60, ...)
precise_angle: float # Refined angle from homography
confidence: float
homography: np.ndarray
inlier_count: int
```
### HeadingHistory
```python
class HeadingHistory(BaseModel):
flight_id: str
current_heading: float
heading_history: List[float] # Last 10 headings
last_update: datetime
sharp_turns: int # Count of sharp turns detected
```
### RotationConfig
```python
class RotationConfig(BaseModel):
step_angle: float = 30.0 # Degrees
sharp_turn_threshold: float = 45.0 # Degrees
confidence_threshold: float = 0.7 # For accepting match
history_size: int = 10 # Number of headings to track
```
_docs/02_components/07_sequential_visual_odometry/07.01_feature_combined_neural_inference.md
+129
View File
@@ -0,0 +1,129 @@
# Feature: Combined Neural Inference
## Description
Single-pass SuperPoint+LightGlue TensorRT inference for feature extraction and matching. Takes two images as input and outputs matched keypoints directly, eliminating intermediate feature transfer overhead.
## Component APIs Implemented
- `extract_and_match(image1: np.ndarray, image2: np.ndarray) -> Matches`
## External Tools and Services
- **Combined SuperPoint+LightGlue TensorRT Engine**: Single model combining extraction and matching
- **F16 Model Manager**: Provides pre-loaded TensorRT engine instance
- **Reference**: [D_VINS](https://github.com/kajo-kurisu/D_VINS/) for TensorRT optimization patterns
## Internal Methods
### `_preprocess_images(image1: np.ndarray, image2: np.ndarray) -> Tuple[np.ndarray, np.ndarray]`
Converts images to grayscale if needed, normalizes pixel values, resizes to model input dimensions.
### `_run_combined_inference(img1_tensor: np.ndarray, img2_tensor: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]`
Executes combined TensorRT engine. Returns matched keypoints from both images and match confidence scores.
**Internal Pipeline** (within single inference):
1. SuperPoint extracts keypoints + descriptors from both images
2. LightGlue performs attention-based matching with adaptive depth
3. Dustbin mechanism filters unmatched features
4. Returns only matched keypoint pairs
### `_filter_matches_by_confidence(keypoints1: np.ndarray, keypoints2: np.ndarray, scores: np.ndarray, threshold: float) -> Matches`
Filters low-confidence matches, constructs final Matches object.
## Architecture Notes
### Combined Model Benefits
- Single GPU memory transfer (both images together)
- No intermediate descriptor serialization
- Optimized attention layers for batch processing
- Adaptive depth exits early for easy (high-overlap) pairs
### TensorRT Engine Configuration
```
Input shapes:
image1: (1, 1, H, W) - grayscale
image2: (1, 1, H, W) - grayscale
Output shapes:
keypoints1: (1, M, 2) - matched keypoints from image1
keypoints2: (1, M, 2) - matched keypoints from image2
scores: (1, M) - match confidence scores
```
### Model Export (reference from D_VINS)
```bash
trtexec --onnx='superpoint_lightglue_combined.onnx' \
--fp16 \
--minShapes=image1:1x1x480x752,image2:1x1x480x752 \
--optShapes=image1:1x1x480x752,image2:1x1x480x752 \
--maxShapes=image1:1x1x480x752,image2:1x1x480x752 \
--saveEngine=sp_lg_combined.engine \
--warmUp=500 --duration=10
```
## Unit Tests
### Test: Grayscale Conversion
- Input: Two RGB images (H×W×3)
- Verify: _preprocess_images returns grayscale tensors
### Test: Grayscale Passthrough
- Input: Two grayscale images (H×W)
- Verify: _preprocess_images returns unchanged
### Test: High Overlap Matching
- Input: Two images with >50% overlap
- Verify: Returns 500+ matches
- Verify: Inference time ~35-50ms (adaptive depth fast path)
### Test: Low Overlap Matching
- Input: Two images with 5-10% overlap
- Verify: Returns 20-50 matches
- Verify: Inference time ~80ms (full depth)
### Test: No Overlap Handling
- Input: Two non-overlapping images
- Verify: Returns <10 matches
- Verify: No exception raised
### Test: Confidence Score Range
- Input: Any valid image pair
- Verify: All scores in [0, 1] range
### Test: Empty/Invalid Image Handling
- Input: Black/invalid image pair
- Verify: Returns empty Matches (never raises exception)
### Test: High Resolution Images
- Input: Two 6252×4168 images
- Verify: Preprocessing resizes appropriately
- Verify: Completes within performance budget
### Test: Output Shape Consistency
- Input: Any valid image pair
- Verify: keypoints1.shape[0] == keypoints2.shape[0] == scores.shape[0]
## Integration Tests
### Test: Model Manager Integration
- Verify: Successfully retrieves combined SP+LG engine from F16
- Verify: Engine loaded with correct TensorRT backend
### Test: Performance Budget
- Input: Two FullHD images
- Verify: Combined inference completes in <80ms on RTX 2060
### Test: Adaptive Depth Behavior
- Input: High overlap pair, then low overlap pair
- Verify: High overlap completes faster than low overlap
### Test: Agricultural Texture Handling
- Input: Two wheat field images with repetitive patterns
- Verify: Produces valid matches despite repetitive textures
### Test: Memory Efficiency
- Verify: Single GPU memory allocation vs two separate models
- Verify: No intermediate descriptor buffer allocation
### Test: Batch Consistency
- Input: Same image pair multiple times
- Verify: Consistent match results (deterministic)
_docs/02_components/07_sequential_visual_odometry/07.02_feature_geometric_pose_estimation.md
+133
View File
@@ -0,0 +1,133 @@
# Feature: Geometric Pose Estimation
## Description
Estimates camera motion from matched keypoints using Essential Matrix decomposition. Orchestrates the full visual odometry pipeline and provides tracking quality assessment. Pure geometric computation (non-ML).
## Component APIs Implemented
- `compute_relative_pose(prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]`
- `estimate_motion(matches: Matches, camera_params: CameraParameters) -> Optional[Motion]`
## External Tools and Services
- **opencv-python**: Essential Matrix estimation via RANSAC, matrix decomposition
- **numpy**: Matrix operations, coordinate normalization
- **F17 Configuration Manager**: Camera parameters (focal length, principal point)
- **H01 Camera Model**: Coordinate normalization utilities
## Internal Methods
### `_normalize_keypoints(keypoints: np.ndarray, camera_params: CameraParameters) -> np.ndarray`
Normalizes pixel coordinates to camera-centered coordinates using intrinsic matrix K.
```
normalized = K^(-1) @ [x, y, 1]^T
```
### `_estimate_essential_matrix(points1: np.ndarray, points2: np.ndarray) -> Tuple[Optional[np.ndarray], np.ndarray]`
RANSAC-based Essential Matrix estimation. Returns E matrix and inlier mask.
- Uses cv2.findEssentialMat with RANSAC
- Requires minimum 8 point correspondences
- Returns None if insufficient inliers
### `_decompose_essential_matrix(E: np.ndarray, points1: np.ndarray, points2: np.ndarray, camera_params: CameraParameters) -> Tuple[np.ndarray, np.ndarray]`
Decomposes Essential Matrix into rotation R and translation t.
- Uses cv2.recoverPose
- Selects correct solution via cheirality check
- Translation is unit vector (scale ambiguous)
### `_compute_tracking_quality(inlier_count: int, total_matches: int) -> Tuple[float, bool]`
Computes confidence score and tracking_good flag:
- **Good**: inlier_count > 50, inlier_ratio > 0.5 → confidence > 0.8
- **Degraded**: inlier_count 20-50 → confidence 0.4-0.8
- **Lost**: inlier_count < 20 → tracking_good = False
### `_build_relative_pose(motion: Motion, matches: Matches) -> RelativePose`
Constructs RelativePose dataclass from motion estimate and match statistics.
## Unit Tests
### Test: Keypoint Normalization
- Input: Pixel coordinates and camera params (fx=1000, cx=640, cy=360)
- Verify: Output centered at principal point, scaled by focal length
### Test: Essential Matrix Estimation - Good Data
- Input: 100+ inlier correspondences from known motion
- Verify: Returns valid Essential Matrix
- Verify: det(E) ≈ 0
- Verify: Singular values satisfy 2σ₁ ≈ σ₂, σ₃ ≈ 0
### Test: Essential Matrix Estimation - Insufficient Points
- Input: <8 point correspondences
- Verify: Returns None
### Test: Essential Matrix Decomposition
- Input: Valid Essential Matrix from known motion
- Verify: Returns valid rotation (det(R) = 1, R^T R = I)
- Verify: Translation is unit vector (||t|| = 1)
### Test: Tracking Quality - Good
- Input: inlier_count=100, total_matches=150
- Verify: tracking_good=True
- Verify: confidence > 0.8
### Test: Tracking Quality - Degraded
- Input: inlier_count=30, total_matches=50
- Verify: tracking_good=True
- Verify: 0.4 < confidence < 0.8
### Test: Tracking Quality - Lost
- Input: inlier_count=10, total_matches=20
- Verify: tracking_good=False
### Test: Scale Ambiguity Marker
- Input: Any valid motion estimate
- Verify: translation vector has unit norm (||t|| = 1)
- Verify: scale_ambiguous flag is True
### Test: Pure Rotation Handling
- Input: Matches from pure rotational motion (no translation)
- Verify: Returns valid pose
- Verify: translation ≈ [0, 0, 0] or arbitrary unit vector
### Test: Forward Motion
- Input: Matches from forward camera motion
- Verify: translation z-component is positive
## Integration Tests
### Test: Full Pipeline - Normal Flight
- Input: Consecutive frames with 50% overlap
- Verify: Returns valid RelativePose
- Verify: inlier_count > 100
- Verify: Total time < 150ms
### Test: Full Pipeline - Low Overlap
- Input: Frames with 5% overlap
- Verify: Returns valid RelativePose
- Verify: inlier_count > 20
### Test: Full Pipeline - Tracking Loss
- Input: Non-overlapping frames (sharp turn)
- Verify: Returns None
- Verify: No exception raised
### Test: Configuration Manager Integration
- Verify: Successfully retrieves camera_params from F17
- Verify: Parameters match expected resolution and focal length
### Test: Camera Model Integration
- Verify: H01 normalization produces correct coordinates
- Verify: Consistent with opencv undistortion
### Test: Pipeline Orchestration
- Verify: extract_and_match called once (combined inference)
- Verify: estimate_motion called with correct params
- Verify: Returns RelativePose with all fields populated
### Test: Agricultural Environment
- Input: Wheat field images with repetitive texture
- Verify: Pipeline succeeds with reasonable inlier count
### Test: Known Motion Validation
- Input: Synthetic image pair with known ground truth motion
- Verify: Estimated rotation within ±2° of ground truth
- Verify: Estimated translation direction within ±5° of ground truth
_docs/02_components/07_sequential_visual_odometry/07._component_sequential_visual_odometry.md
+301
View File
@@ -0,0 +1,301 @@
# Sequential Visual Odometry
## Interface Definition
**Interface Name**: `ISequentialVisualOdometry`
### Interface Methods
```python
class ISequentialVisualOdometry(ABC):
@abstractmethod
def compute_relative_pose(self, prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]:
pass
@abstractmethod
def extract_and_match(self, image1: np.ndarray, image2: np.ndarray) -> Matches:
pass
@abstractmethod
def estimate_motion(self, matches: Matches, camera_params: CameraParameters) -> Optional[Motion]:
pass
```
**Note**: F07 is chunk-agnostic. It only computes relative poses between images. The caller (F02.2 Flight Processing Engine) determines which chunk the frames belong to and routes factors to the appropriate subgraph via F12 → F10.
## Component Description
### Responsibilities
- Combined SuperPoint+LightGlue neural network inference for extraction and matching
- Handle <5% overlap scenarios via LightGlue attention mechanism
- Estimate relative pose (translation + rotation) between frames
- Return relative pose factors for Factor Graph Optimizer
- Detect tracking loss (low inlier count)
### Scope
- Frame-to-frame visual odometry
- Feature-based motion estimation using combined neural network
- Handles low overlap and challenging agricultural environments
- Provides relative measurements for trajectory optimization
- **Chunk-agnostic**: F07 doesn't know about chunks. Caller (F02.2) routes results to appropriate chunk subgraph.
### Architecture Notes
- Uses combined SuperPoint+LightGlue TensorRT model (single inference pass)
- Reference: [D_VINS](https://github.com/kajo-kurisu/D_VINS/) for TensorRT optimization patterns
- Model outputs matched keypoints directly from two input images
## API Methods
### `compute_relative_pose(prev_image: np.ndarray, curr_image: np.ndarray) -> Optional[RelativePose]`
**Description**: Computes relative camera pose between consecutive frames.
**Called By**:
- Main processing loop (per-frame)
**Input**:
```python
prev_image: np.ndarray # Previous frame (t-1)
curr_image: np.ndarray # Current frame (t)
```
**Output**:
```python
RelativePose:
translation: np.ndarray # (x, y, z) in meters
rotation: np.ndarray # 3×3 rotation matrix or quaternion
confidence: float # 0.0 to 1.0
inlier_count: int
total_matches: int
tracking_good: bool
```
**Processing Flow**:
1. extract_and_match(prev_image, curr_image) → matches
2. estimate_motion(matches, camera_params) → motion
3. Return RelativePose
**Tracking Quality Indicators**:
- **Good tracking**: inlier_count > 50, inlier_ratio > 0.5
- **Degraded tracking**: inlier_count 20-50
- **Tracking loss**: inlier_count < 20
**Error Conditions**:
- Returns `None`: Tracking lost (insufficient matches)
**Test Cases**:
1. **Good overlap (>50%)**: Returns reliable pose
2. **Low overlap (5-10%)**: Still succeeds with LightGlue
3. **<5% overlap**: May return None (tracking loss)
4. **Agricultural texture**: Handles repetitive patterns
---
### `extract_and_match(image1: np.ndarray, image2: np.ndarray) -> Matches`
**Description**: Single-pass neural network inference combining SuperPoint feature extraction and LightGlue matching.
**Called By**:
- Internal (during compute_relative_pose)
**Input**:
```python
image1: np.ndarray # First image (H×W×3 or H×W)
image2: np.ndarray # Second image (H×W×3 or H×W)
```
**Output**:
```python
Matches:
matches: np.ndarray # (M, 2) - indices [idx1, idx2]
scores: np.ndarray # (M,) - match confidence scores
keypoints1: np.ndarray # (M, 2) - matched keypoints from image 1
keypoints2: np.ndarray # (M, 2) - matched keypoints from image 2
```
**Processing Details**:
- Uses F16 Model Manager to get combined SuperPoint+LightGlue TensorRT engine
- Single inference pass processes both images
- Converts to grayscale internally if needed
- SuperPoint extracts keypoints + 256-dim descriptors
- LightGlue performs attention-based matching with adaptive depth
- "Dustbin" mechanism handles unmatched features
**Performance** (TensorRT on RTX 2060):
- Combined inference: ~50-80ms (vs ~65-115ms separate)
- Faster for high-overlap pairs (adaptive depth exits early)
**Test Cases**:
1. **High overlap**: ~35ms, 500+ matches
2. **Low overlap (<5%)**: ~80ms, 20-50 matches
3. **No overlap**: Few or no matches (< 10)
---
### `estimate_motion(matches: Matches, camera_params: CameraParameters) -> Optional[Motion]`
**Description**: Estimates camera motion from matched keypoints using Essential Matrix.
**Called By**:
- Internal (during compute_relative_pose)
**Input**:
```python
matches: Matches
camera_params: CameraParameters:
focal_length: float
principal_point: Tuple[float, float]
resolution: Tuple[int, int]
```
**Output**:
```python
Motion:
translation: np.ndarray # (x, y, z) - unit vector (scale ambiguous)
rotation: np.ndarray # 3×3 rotation matrix
inliers: np.ndarray # Boolean mask of inlier matches
inlier_count: int
```
**Algorithm**:
1. Normalize keypoint coordinates using camera intrinsics
2. Estimate Essential Matrix using RANSAC
3. Decompose Essential Matrix → [R, t]
4. Return motion with inlier mask
**Scale Ambiguity**:
- Monocular VO has inherent scale ambiguity
- Translation is unit vector (direction only, magnitude = 1)
- **F07 does NOT resolve scale** - it only outputs unit translation vectors
- Scale resolution is handled by F10 Factor Graph Optimizer, which uses:
- Altitude priors (soft constraints)
- GSD-based expected displacement calculations (via H02)
- Absolute GPS anchors from F09 Metric Refinement
**Critical Handoff to F10**:
The caller (F02.2) must pass the unit translation to F10 for scale resolution:
```python
vo_result = F07.compute_relative_pose(prev_image, curr_image)
# vo_result.translation is a UNIT VECTOR (||t|| = 1)
# F02.2 passes to F10 which scales using:
# 1. altitude = F17.get_operational_altitude(flight_id)
# 2. gsd = H02.compute_gsd(altitude, camera_params)
# 3. expected_displacement = frame_spacing * gsd
# 4. scaled_translation = vo_result.translation * expected_displacement
F10.add_relative_factor(flight_id, frame_i, frame_j, vo_result, covariance)
```
**Error Conditions**:
- Returns `None`: Insufficient inliers (< 8 points for Essential Matrix)
**Test Cases**:
1. **Good matches**: Returns motion with high inlier count
2. **Low inliers**: May return None
3. **Degenerate motion**: Handles pure rotation
## Integration Tests
### Test 1: Normal Flight Sequence
1. Load consecutive frames with 50% overlap
2. compute_relative_pose() → returns valid pose
3. Verify translation direction reasonable
4. Verify inlier_count > 100
### Test 2: Low Overlap Scenario
1. Load frames with 5% overlap
2. compute_relative_pose() → still succeeds
3. Verify inlier_count > 20
4. Verify combined model finds matches despite low overlap
### Test 3: Tracking Loss
1. Load frames with 0% overlap (sharp turn)
2. compute_relative_pose() → returns None
3. Verify tracking_good = False
4. Trigger global place recognition
### Test 4: Agricultural Texture
1. Load images of wheat fields (repetitive texture)
2. compute_relative_pose() → SuperPoint handles better than SIFT
3. Verify match quality
### Test 5: VO with Chunk Routing
1. Create chunk_1 and chunk_2 via F12
2. compute_relative_pose() for frames in chunk_1, F02.2 routes to chunk_1
3. compute_relative_pose() for frames in chunk_2, F02.2 routes to chunk_2
4. Verify F02.2 calls F12.add_frame_to_chunk() with correct chunk_id
5. Verify chunks optimized independently via F10
## Non-Functional Requirements
### Performance
- **compute_relative_pose**: < 150ms total
- Combined SP+LG inference: ~50-80ms
- Motion estimation: ~10ms
- **Frame rate**: 5-10 FPS processing (meets <5s requirement)
### Accuracy
- **Relative rotation**: ±2° error
- **Relative translation direction**: ±5° error
- **Inlier ratio**: >50% for good tracking
### Reliability
- Handle 100m spacing between frames
- Survive temporary tracking degradation
- Recover from brief occlusions
## Dependencies
### Internal Components
- **F16 Model Manager**: For combined SuperPoint+LightGlue TensorRT model
- **F17 Configuration Manager**: For camera parameters
- **H01 Camera Model**: For coordinate normalization
- **H05 Performance Monitor**: For timing measurements
**Note**: F07 is chunk-agnostic and does NOT depend on F10 Factor Graph Optimizer. F07 only computes relative poses between images and returns them to the caller (F02.2). The caller (F02.2) determines which chunk the frames belong to and routes factors to the appropriate subgraph via F12 → F10.
### External Dependencies
- **Combined SuperPoint+LightGlue**: Single TensorRT engine for extraction + matching
- **opencv-python**: Essential Matrix estimation
- **numpy**: Matrix operations
## Data Models
### Matches
```python
class Matches(BaseModel):
matches: np.ndarray # (M, 2) - pairs of indices
scores: np.ndarray # (M,) - match confidence
keypoints1: np.ndarray # (M, 2)
keypoints2: np.ndarray # (M, 2)
```
### RelativePose
```python
class RelativePose(BaseModel):
translation: np.ndarray # (3,) - unit vector
rotation: np.ndarray # (3, 3) or (4,) quaternion
confidence: float
inlier_count: int
total_matches: int
tracking_good: bool
scale_ambiguous: bool = True
chunk_id: Optional[str] = None
```
### Motion
```python
class Motion(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3)
inliers: np.ndarray # Boolean mask
inlier_count: int
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float
principal_point: Tuple[float, float]
resolution: Tuple[int, int]
```
_docs/02_components/08_global_place_recognition/08.01_feature_index_management.md
+64
View File
@@ -0,0 +1,64 @@
# Feature: Index Management
## Description
Load and manage pre-built satellite descriptor database (Faiss index). The semantic index is built by the satellite data provider offline using DINOv2 + VLAD - F08 only loads and validates the index. This is the foundation for all place recognition queries.
## Component APIs Implemented
- `load_index(flight_id: str, index_path: str) -> bool`
## External Tools and Services
- **H04 Faiss Index Manager**: For `load_index()`, `validate_index()` operations
- **Faiss**: Facebook similarity search library
## Internal Methods
### `_validate_index_integrity(index) -> bool`
Validates loaded Faiss index: checks descriptor dimensions (4096 or 8192), verifies tile count matches metadata.
### `_load_tile_metadata(metadata_path: str) -> Dict[int, TileMetadata]`
Loads tile_id → gps_center, bounds mapping from JSON file provided by satellite provider.
### `_verify_metadata_alignment(index, metadata: Dict) -> bool`
Ensures metadata entries match index size (same number of tiles).
## Unit Tests
### Test: Load Valid Index
- Input: Valid index file path from satellite provider
- Verify: Returns True, index operational
### Test: Index Not Found
- Input: Non-existent path
- Verify: Raises `IndexNotFoundError`
### Test: Corrupted Index
- Input: Corrupted/truncated index file
- Verify: Raises `IndexCorruptedError`
### Test: Dimension Validation
- Input: Index with wrong descriptor dimensions
- Verify: Raises `IndexCorruptedError` with descriptive message
### Test: Metadata Mismatch
- Input: Index with 1000 entries, metadata with 500 entries
- Verify: Raises `MetadataMismatchError`
### Test: Empty Metadata File
- Input: Valid index, empty metadata JSON
- Verify: Raises `MetadataMismatchError`
### Test: Load Performance
- Input: Index with 10,000 tiles
- Verify: Load completes in <10 seconds
## Integration Tests
### Test: Faiss Manager Integration
- Verify: Successfully delegates index loading to H04 Faiss Index Manager
- Verify: Index accessible for subsequent queries
### Test: Query After Load
- Setup: Load valid index
- Action: Query with random descriptor
- Verify: Returns valid matches (not empty, valid indices)
_docs/02_components/08_global_place_recognition/08.02_feature_descriptor_computation.md
+95
View File
@@ -0,0 +1,95 @@
# Feature: Descriptor Computation
## Description
Compute global location descriptors using DINOv2 + VLAD aggregation. Supports both single-image descriptors and aggregate chunk descriptors for robust matching. DINOv2 features are semantic and invariant to season/texture changes, critical for UAV-to-satellite domain gap.
## Component APIs Implemented
- `compute_location_descriptor(image: np.ndarray) -> np.ndarray`
- `compute_chunk_descriptor(chunk_images: List[np.ndarray]) -> np.ndarray`
## External Tools and Services
- **F16 Model Manager**: Provides DINOv2 inference engine via `get_inference_engine("DINOv2")`
- **DINOv2**: Meta's foundation vision model for semantic feature extraction
- **numpy**: Array operations and L2 normalization
## Internal Methods
### `_preprocess_image(image: np.ndarray) -> np.ndarray`
Resizes and normalizes image for DINOv2 input (typically 224x224 or 518x518).
### `_extract_dense_features(preprocessed: np.ndarray) -> np.ndarray`
Runs DINOv2 inference, extracts dense feature map from multiple spatial locations.
### `_vlad_aggregate(dense_features: np.ndarray, codebook: np.ndarray) -> np.ndarray`
Applies VLAD (Vector of Locally Aggregated Descriptors) aggregation using pre-trained cluster centers.
### `_l2_normalize(descriptor: np.ndarray) -> np.ndarray`
L2-normalizes descriptor vector for cosine similarity search.
### `_aggregate_chunk_descriptors(descriptors: List[np.ndarray], strategy: str) -> np.ndarray`
Aggregates multiple descriptors into one using strategy: "mean" (default), "vlad", or "max".
## Unit Tests
### compute_location_descriptor
#### Test: Output Dimensions
- Input: UAV image (any resolution)
- Verify: Returns 4096-dim or 8192-dim vector
#### Test: Normalization
- Input: Any valid image
- Verify: Output L2 norm equals 1.0
#### Test: Deterministic Output
- Input: Same image twice
- Verify: Identical descriptors (no randomness)
#### Test: Season Invariance
- Input: Two images of same location, different seasons
- Verify: Cosine similarity > 0.7
#### Test: Location Discrimination
- Input: Two images of different locations
- Verify: Cosine similarity < 0.5
#### Test: Domain Invariance
- Input: UAV image and satellite image of same location
- Verify: Cosine similarity > 0.6 (cross-domain)
### compute_chunk_descriptor
#### Test: Empty Chunk
- Input: Empty list
- Verify: Raises ValueError
#### Test: Single Image Chunk
- Input: List with one image
- Verify: Equivalent to compute_location_descriptor
#### Test: Multiple Images Mean Aggregation
- Input: 5 images from chunk
- Verify: Descriptor is mean of individual descriptors
#### Test: Aggregated Normalization
- Input: Any chunk
- Verify: Output L2 norm equals 1.0
#### Test: Chunk More Robust Than Single
- Input: Featureless terrain chunk (10 images)
- Verify: Chunk descriptor has lower variance than individual descriptors
## Integration Tests
### Test: Model Manager Integration
- Verify: Successfully retrieves DINOv2 from F16 Model Manager
- Verify: Model loaded with correct TensorRT/ONNX backend
### Test: Performance Budget
- Input: FullHD image
- Verify: Single descriptor computed in ~150ms
### Test: Chunk Performance
- Input: 10 images
- Verify: Chunk descriptor in <2s (parallelizable)
_docs/02_components/08_global_place_recognition/08.03_feature_candidate_retrieval.md
+130
View File
@@ -0,0 +1,130 @@
# Feature: Candidate Retrieval
## Description
Query satellite database and retrieve ranked tile candidates. Orchestrates the full place recognition pipeline: descriptor computation → database query → candidate ranking. Supports both single-image and chunk-based retrieval for "kidnapped robot" recovery.
## Component APIs Implemented
- `query_database(descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]`
- `rank_candidates(candidates: List[TileCandidate]) -> List[TileCandidate]`
- `retrieve_candidate_tiles(image: np.ndarray, top_k: int) -> List[TileCandidate]`
- `retrieve_candidate_tiles_for_chunk(chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]`
## External Tools and Services
- **H04 Faiss Index Manager**: For `search()` operation
- **F04 Satellite Data Manager**: For tile metadata retrieval (gps_center, bounds) after Faiss returns indices
- **F12 Route Chunk Manager**: Provides chunk images for chunk-based retrieval
## Internal Methods
### `_distance_to_similarity(distance: float) -> float`
Converts L2 distance to normalized similarity score [0, 1].
### `_retrieve_tile_metadata(indices: List[int]) -> List[TileMetadata]`
Fetches GPS center and bounds for tile indices from F04 Satellite Data Manager.
### `_apply_spatial_reranking(candidates: List[TileCandidate], dead_reckoning_estimate: Optional[GPSPoint]) -> List[TileCandidate]`
Re-ranks candidates based on proximity to dead-reckoning estimate if available.
### `_apply_trajectory_reranking(candidates: List[TileCandidate], previous_trajectory: Optional[List[GPSPoint]]) -> List[TileCandidate]`
Favors tiles that continue the previous trajectory direction.
### `_filter_by_geofence(candidates: List[TileCandidate], geofence: Optional[BoundingBox]) -> List[TileCandidate]`
Removes candidates outside operational geofence.
## Unit Tests
### query_database
#### Test: Returns Top-K Matches
- Input: Valid descriptor, top_k=5
- Verify: Returns exactly 5 DatabaseMatch objects
#### Test: Ordered by Distance
- Input: Any descriptor
- Verify: Matches sorted by ascending distance
#### Test: Similarity Score Range
- Input: Any query
- Verify: All similarity_score values in [0, 1]
#### Test: Empty Database
- Input: Query when no index loaded
- Verify: Returns empty list (not exception)
#### Test: Query Performance
- Input: Large database (10,000 tiles)
- Verify: Query completes in <50ms
### rank_candidates
#### Test: Preserves Order Without Heuristics
- Input: Candidates without dead-reckoning estimate
- Verify: Order unchanged (similarity-based)
#### Test: Spatial Reranking Applied
- Input: Candidates + dead-reckoning estimate
- Verify: Closer tile promoted in ranking
#### Test: Tie Breaking
- Input: Two candidates with similar similarity scores
- Verify: Spatial proximity breaks tie
#### Test: Geofence Filtering
- Input: Candidates with some outside geofence
- Verify: Out-of-bounds candidates removed
### retrieve_candidate_tiles
#### Test: End-to-End Single Image
- Input: UAV image, top_k=5
- Verify: Returns 5 TileCandidate with valid gps_center
#### Test: Correct Tile in Top-5
- Input: UAV image with known location
- Verify: Correct tile appears in top-5 (Recall@5 test)
#### Test: Performance Budget
- Input: FullHD UAV image
- Verify: Total time <200ms (descriptor ~150ms + query ~50ms)
### retrieve_candidate_tiles_for_chunk
#### Test: End-to-End Chunk
- Input: 10 chunk images, top_k=5
- Verify: Returns 5 TileCandidate
#### Test: Chunk More Accurate Than Single
- Input: Featureless terrain images
- Verify: Chunk retrieval finds correct tile where single-image fails
#### Test: Recall@5 > 90%
- Input: Various chunk scenarios
- Verify: Correct tile in top-5 at least 90% of test cases
## Integration Tests
### Test: Faiss Manager Integration
- Verify: query_database correctly delegates to H04 Faiss Index Manager
### Test: Satellite Data Manager Integration
- Verify: Tile metadata correctly retrieved from F04 after Faiss query
### Test: Full Pipeline Single Image
- Setup: Load index, prepare UAV image
- Action: retrieve_candidate_tiles()
- Verify: Returns valid candidates with GPS coordinates
### Test: Full Pipeline Chunk
- Setup: Load index, prepare chunk images
- Action: retrieve_candidate_tiles_for_chunk()
- Verify: Returns valid candidates, more robust than single-image
### Test: Season Invariance
- Setup: Satellite tiles from summer, UAV image from autumn
- Action: retrieve_candidate_tiles()
- Verify: Correct match despite appearance change
### Test: Recall@5 Benchmark
- Input: Test dataset of 100 UAV images with ground truth
- Verify: Recall@5 > 85% for single-image, > 90% for chunk
_docs/02_components/08_global_place_recognition/08._component_global_place_recognition.md
+424
View File
@@ -0,0 +1,424 @@
# Global Place Recognition
## Interface Definition
**Interface Name**: `IGlobalPlaceRecognition`
### Interface Methods
```python
class IGlobalPlaceRecognition(ABC):
@abstractmethod
def retrieve_candidate_tiles(self, image: np.ndarray, top_k: int) -> List[TileCandidate]:
pass
@abstractmethod
def compute_location_descriptor(self, image: np.ndarray) -> np.ndarray:
pass
@abstractmethod
def query_database(self, descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]:
pass
@abstractmethod
def rank_candidates(self, candidates: List[TileCandidate]) -> List[TileCandidate]:
pass
@abstractmethod
def load_index(self, flight_id: str, index_path: str) -> bool:
pass
@abstractmethod
def retrieve_candidate_tiles_for_chunk(self, chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]:
pass
@abstractmethod
def compute_chunk_descriptor(self, chunk_images: List[np.ndarray]) -> np.ndarray:
pass
```
## Component Description
### Responsibilities
- AnyLoc (DINOv2 + VLAD) for coarse localization after tracking loss
- "Kidnapped robot" recovery after sharp turns
- Compute image descriptors robust to season/appearance changes
- Query Faiss index of satellite tile descriptors
- Return top-k candidate tile regions for progressive refinement
- **Load pre-built satellite descriptor index** (index is built by satellite provider, NOT by F08)
- **Chunk semantic matching (aggregate DINOv2 features)**
- **Chunk descriptor computation for robust matching**
### Scope
- Global localization (not frame-to-frame)
- Appearance-based place recognition
- Handles domain gap (UAV vs satellite imagery)
- Semantic feature extraction (DINOv2)
- Efficient similarity search (Faiss)
- **Chunk-level matching (more robust than single-image)**
## API Methods
### `retrieve_candidate_tiles(image: np.ndarray, top_k: int) -> List[TileCandidate]`
**Description**: Retrieves top-k candidate satellite tiles for a UAV image.
**Called By**:
- F11 Failure Recovery Coordinator (after tracking loss)
**Input**:
```python
image: np.ndarray # UAV image
top_k: int # Number of candidates (typically 5)
```
**Output**:
```python
List[TileCandidate]:
tile_id: str
gps_center: GPSPoint
similarity_score: float
rank: int
```
**Processing Flow**:
1. compute_location_descriptor(image) → descriptor
2. query_database(descriptor, top_k) → database_matches
3. Retrieve tile metadata for matches
4. rank_candidates() → sorted by similarity
5. Return top-k candidates
**Error Conditions**:
- Returns empty list: Database not initialized, query failed
**Test Cases**:
1. **UAV image over Ukraine**: Returns relevant tiles
2. **Different season**: DINOv2 handles appearance change
3. **Top-1 accuracy**: Correct tile in top-5 > 85%
---
### `compute_location_descriptor(image: np.ndarray) -> np.ndarray`
**Description**: Computes global descriptor using DINOv2 + VLAD aggregation.
**Called By**:
- Internal (during retrieve_candidate_tiles)
- System initialization (for satellite database)
**Input**:
```python
image: np.ndarray # UAV or satellite image
```
**Output**:
```python
np.ndarray: Descriptor vector (4096-dim or 8192-dim)
```
**Algorithm (AnyLoc)**:
1. Extract DINOv2 features (dense feature map)
2. Apply VLAD (Vector of Locally Aggregated Descriptors) aggregation
3. L2-normalize descriptor
4. Return compact global descriptor
**Processing Details**:
- Uses F16 Model Manager to get DINOv2 model
- Dense features: extracts from multiple spatial locations
- VLAD codebook: pre-trained cluster centers
- Semantic features: invariant to texture/color changes
**Performance**:
- Inference time: ~150ms for DINOv2 + VLAD
**Test Cases**:
1. **Same location, different season**: Similar descriptors
2. **Different locations**: Dissimilar descriptors
3. **UAV vs satellite**: Domain-invariant features
---
### `query_database(descriptor: np.ndarray, top_k: int) -> List[DatabaseMatch]`
**Description**: Queries Faiss index for most similar satellite tiles.
**Called By**:
- Internal (during retrieve_candidate_tiles)
**Input**:
```python
descriptor: np.ndarray # Query descriptor
top_k: int
```
**Output**:
```python
List[DatabaseMatch]:
index: int # Tile index in database
distance: float # L2 distance
similarity_score: float # Normalized score
```
**Processing Details**:
- Uses H04 Faiss Index Manager
- Index type: IVF (Inverted File) or HNSW for fast search
- Distance metric: L2 (Euclidean)
- Query time: ~10-50ms for 10,000+ tiles
**Error Conditions**:
- Returns empty list: Query failed
**Test Cases**:
1. **Query satellite database**: Returns top-5 matches
2. **Large database (10,000 tiles)**: Fast retrieval (<50ms)
---
### `rank_candidates(candidates: List[TileCandidate]) -> List[TileCandidate]`
**Description**: Re-ranks candidates based on additional heuristics.
**Called By**:
- Internal (during retrieve_candidate_tiles)
**Input**:
```python
candidates: List[TileCandidate] # Initial ranking by similarity
```
**Output**:
```python
List[TileCandidate] # Re-ranked list
```
**Re-ranking Factors**:
1. **Similarity score**: Primary factor
2. **Spatial proximity**: Prefer tiles near dead-reckoning estimate
3. **Previous trajectory**: Favor continuation of route
4. **Geofence constraints**: Within operational area
**Test Cases**:
1. **Spatial re-ranking**: Closer tile promoted
2. **Similar scores**: Spatial proximity breaks tie
---
### `load_index(flight_id: str, index_path: str) -> bool`
**Description**: Loads pre-built satellite descriptor database from file. **Note**: The semantic index (DINOv2 descriptors + Faiss index) MUST be provided by the satellite data provider. F08 does NOT build the index - it only loads it.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight initialization, index_path from F04 Satellite Data Manager)
**Input**:
```python
index_path: str # Path to pre-built Faiss index file from satellite provider
```
**Output**:
```python
bool: True if database loaded successfully
```
**Processing Flow**:
1. Load pre-built Faiss index from index_path
2. Load tile metadata (tile_id → gps_center, bounds mapping)
3. Validate index integrity (check descriptor dimensions, tile count)
4. Return True if loaded successfully
**Satellite Provider Responsibility**:
- Satellite provider builds the semantic index offline using DINOv2 + VLAD
- Provider delivers index file along with satellite tiles
- **Index format**: Faiss IVF1000 (Inverted File with 1000 clusters) + tile metadata JSON
- Provider is responsible for index updates when satellite data changes
- Index is rebuilt by provider whenever new satellite tiles are fetched on demand
- Supported providers: Maxar, Google Maps, Copernicus, etc.
**Error Conditions**:
- Raises `IndexNotFoundError`: Index file not found
- Raises `IndexCorruptedError`: Index file corrupted or invalid format
- Raises `MetadataMismatchError`: Metadata doesn't match index
**Performance**:
- **Load time**: <10 seconds for 10,000+ tiles
**Test Cases**:
1. **Load valid index**: Completes successfully, index operational
2. **Index not found**: Raises IndexNotFoundError
3. **Corrupted index**: Raises IndexCorruptedError
4. **Index query after load**: Works correctly
---
### `retrieve_candidate_tiles_for_chunk(chunk_images: List[np.ndarray], top_k: int) -> List[TileCandidate]`
**Description**: Retrieves top-k candidate satellite tiles for a chunk using aggregate descriptor.
**Called By**:
- F11 Failure Recovery Coordinator (chunk semantic matching)
- F12 Route Chunk Manager (chunk matching coordination)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
top_k: int # Number of candidates (typically 5)
```
**Output**:
```python
List[TileCandidate]:
tile_id: str
gps_center: GPSPoint
similarity_score: float
rank: int
```
**Processing Flow**:
1. compute_chunk_descriptor(chunk_images) → aggregate descriptor
2. query_database(descriptor, top_k) → database_matches
3. Retrieve tile metadata for matches
4. rank_candidates() → sorted by similarity
5. Return top-k candidates
**Advantages over Single-Image Matching**:
- Aggregate descriptor more robust to featureless terrain
- Multiple images provide more context
- Better handles plain fields where single-image matching fails
**Test Cases**:
1. **Chunk matching**: Returns relevant tiles
2. **Featureless terrain**: Succeeds where single-image fails
3. **Top-1 accuracy**: Correct tile in top-5 > 90% (better than single-image)
---
### `compute_chunk_descriptor(chunk_images: List[np.ndarray]) -> np.ndarray`
**Description**: Computes aggregate DINOv2 descriptor from multiple chunk images.
**Called By**:
- Internal (during retrieve_candidate_tiles_for_chunk)
- F12 Route Chunk Manager (chunk descriptor computation - delegates to F08)
**Input**:
```python
chunk_images: List[np.ndarray] # 5-20 images from chunk
```
**Output**:
```python
np.ndarray: Aggregated descriptor vector (4096-dim or 8192-dim)
```
**Algorithm**:
1. For each image in chunk:
- compute_location_descriptor(image) → descriptor (DINOv2 + VLAD)
2. Aggregate descriptors:
- **Mean aggregation**: Average all descriptors
- **VLAD aggregation**: Use VLAD codebook for aggregation
- **Max aggregation**: Element-wise maximum
3. L2-normalize aggregated descriptor
4. Return composite descriptor
**Aggregation Strategy**:
- **Mean**: Simple average (default)
- **VLAD**: More sophisticated, preserves spatial information
- **Max**: Emphasizes strongest features
**Performance**:
- Descriptor computation: ~150ms × N images (can be parallelized)
- Aggregation: ~10ms
**Test Cases**:
1. **Compute descriptor**: Returns aggregated descriptor
2. **Multiple images**: Descriptor aggregates correctly
3. **Descriptor quality**: More robust than single-image descriptor
## Integration Tests
### Test 1: Place Recognition Flow
1. Load UAV image from sharp turn
2. retrieve_candidate_tiles(top_k=5)
3. Verify correct tile in top-5
4. Pass candidates to F11 Failure Recovery
### Test 2: Season Invariance
1. Satellite tiles from summer
2. UAV images from autumn
3. retrieve_candidate_tiles() → correct match despite appearance change
### Test 3: Index Loading
1. Prepare pre-built index file from satellite provider
2. load_index(index_path)
3. Verify Faiss index loaded correctly
4. Query with test image → returns matches
### Test 4: Chunk Semantic Matching
1. Build chunk with 10 images (plain field scenario)
2. compute_chunk_descriptor() → aggregate descriptor
3. retrieve_candidate_tiles_for_chunk() → returns candidates
4. Verify correct tile in top-5 (where single-image matching failed)
5. Verify chunk matching more robust than single-image
## Non-Functional Requirements
### Performance
- **retrieve_candidate_tiles**: < 200ms total
- Descriptor computation: ~150ms
- Database query: ~50ms
- **compute_location_descriptor**: ~150ms
- **query_database**: ~10-50ms
### Accuracy
- **Recall@5**: > 85% (correct tile in top-5)
- **Recall@1**: > 60% (correct tile is top-1)
### Scalability
- Support 10,000+ satellite tiles in database
- Fast query even with large database
## Dependencies
### Internal Components
- **F16 Model Manager**: For DINOv2 inference engine via `get_inference_engine("DINOv2")`.
- **H04 Faiss Index Manager**: For similarity search via `load_index()`, `search()`. Critical for `query_database()`.
- **F04 Satellite Data Manager**: For tile metadata retrieval after Faiss search returns tile indices.
- **F12 Route Chunk Manager**: For chunk image retrieval during chunk descriptor computation.
### External Dependencies
- **DINOv2**: Foundation vision model
- **Faiss**: Similarity search library
- **numpy**: Array operations
## Data Models
### TileCandidate
```python
class TileCandidate(BaseModel):
tile_id: str
gps_center: GPSPoint
bounds: TileBounds
similarity_score: float
rank: int
spatial_score: Optional[float]
```
### DatabaseMatch
```python
class DatabaseMatch(BaseModel):
index: int
tile_id: str
distance: float
similarity_score: float
```
### SatelliteTile
```python
class SatelliteTile(BaseModel):
tile_id: str
image: np.ndarray
gps_center: GPSPoint
bounds: TileBounds
descriptor: Optional[np.ndarray]
```
_docs/02_components/09_metric_refinement/09.01_feature_single_image_alignment.md
+55
View File
@@ -0,0 +1,55 @@
# Feature: Single Image Alignment
## Description
Core UAV-to-satellite cross-view matching for individual frames using LiteSAM. Computes precise GPS coordinates by aligning a pre-rotated UAV image to a georeferenced satellite tile through homography estimation.
## Component APIs Implemented
- `align_to_satellite(uav_image, satellite_tile, tile_bounds) -> AlignmentResult`
- `compute_homography(uav_image, satellite_tile) -> Optional[np.ndarray]`
- `extract_gps_from_alignment(homography, tile_bounds, image_center) -> GPSPoint`
- `compute_match_confidence(alignment) -> float`
## External Tools and Services
- **LiteSAM**: Cross-view matching model (TAIFormer encoder, CTM correlation)
- **opencv-python**: RANSAC homography estimation, image operations
- **numpy**: Matrix operations, coordinate transformations
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_extract_features(image)` | Extract multi-scale features using LiteSAM TAIFormer encoder |
| `_compute_correspondences(uav_features, sat_features)` | Compute dense correspondence field via CTM |
| `_estimate_homography_ransac(correspondences)` | Estimate 3×3 homography using RANSAC |
| `_refine_homography(homography, correspondences)` | Non-linear refinement of homography |
| `_validate_match(homography, inliers)` | Check inlier count/ratio thresholds |
| `_pixel_to_gps(pixel, tile_bounds)` | Convert satellite pixel coordinates to GPS |
| `_compute_inlier_ratio(inliers, total)` | Calculate inlier ratio for confidence |
| `_compute_spatial_distribution(inliers)` | Assess inlier spatial distribution quality |
| `_compute_reprojection_error(homography, correspondences)` | Calculate mean reprojection error |
## Unit Tests
1. **Feature extraction**: LiteSAM encoder produces valid feature tensors
2. **Correspondence computation**: CTM produces dense correspondence field
3. **Homography estimation**: RANSAC returns valid 3×3 matrix for good correspondences
4. **Homography estimation failure**: Returns None for insufficient correspondences (<15 inliers)
5. **GPS extraction accuracy**: Pixel-to-GPS conversion within expected tolerance
6. **Confidence high**: Returns >0.8 for inlier_ratio >0.6, inlier_count >50, MRE <0.5px
7. **Confidence medium**: Returns 0.5-0.8 for moderate match quality
8. **Confidence low**: Returns <0.5 for poor matches
9. **Reprojection error calculation**: Correctly computes mean pixel error
10. **Spatial distribution scoring**: Penalizes clustered inliers
## Integration Tests
1. **Single tile drift correction**: Load UAV image + satellite tile → align_to_satellite() returns GPS within 20m of ground truth
2. **Progressive search (4 tiles)**: align_to_satellite() on 2×2 grid, first 3 fail, 4th succeeds
3. **Rotation sensitivity**: Unrotated image (>45°) fails; pre-rotated image succeeds
4. **Multi-scale robustness**: Different GSD (UAV 0.1m/px, satellite 0.3m/px) → match succeeds
5. **Altitude variation**: UAV at various altitudes (<1km) → consistent GPS accuracy
6. **Performance benchmark**: align_to_satellite() completes in ~60ms (TensorRT)
_docs/02_components/09_metric_refinement/09.02_feature_chunk_alignment.md
+51
View File
@@ -0,0 +1,51 @@
# Feature: Chunk Alignment
## Description
Batch UAV-to-satellite matching that aggregates correspondences from multiple images in a chunk for more robust geo-localization. Handles scenarios where single-image matching fails (featureless terrain, partial occlusions). Returns Sim(3) transform for the entire chunk.
## Component APIs Implemented
- `align_chunk_to_satellite(chunk_images, satellite_tile, tile_bounds) -> ChunkAlignmentResult`
- `match_chunk_homography(chunk_images, satellite_tile) -> Optional[np.ndarray]`
## External Tools and Services
- **LiteSAM**: Cross-view matching model (TAIFormer encoder, CTM correlation)
- **opencv-python**: RANSAC homography estimation
- **numpy**: Matrix operations, feature aggregation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_extract_chunk_features(chunk_images)` | Extract features from all chunk images |
| `_aggregate_features(features_list)` | Combine features via mean/max pooling |
| `_aggregate_correspondences(correspondences_list)` | Merge correspondences from multiple images |
| `_estimate_chunk_homography(aggregated_correspondences)` | Estimate homography from aggregate data |
| `_compute_sim3_transform(homography, tile_bounds)` | Extract translation, rotation, scale |
| `_get_chunk_center_gps(homography, tile_bounds, chunk_images)` | GPS of middle frame center |
| `_validate_chunk_match(inliers, confidence)` | Check chunk-specific thresholds (>30 inliers) |
## Unit Tests
1. **Feature aggregation**: Mean pooling produces valid combined features
2. **Correspondence aggregation**: Merges correspondences from N images correctly
3. **Chunk homography estimation**: Returns valid 3×3 matrix for aggregate correspondences
4. **Chunk homography failure**: Returns None for insufficient aggregate correspondences
5. **Sim(3) extraction**: Correctly decomposes homography into translation, rotation, scale
6. **Chunk center GPS**: Returns GPS of middle frame's center pixel
7. **Chunk confidence high**: Returns >0.7 for >50 inliers
8. **Chunk confidence medium**: Returns 0.5-0.7 for 30-50 inliers
9. **Chunk validation**: Rejects matches with <30 inliers
## Integration Tests
1. **Chunk LiteSAM matching**: 10 images from plain field → align_chunk_to_satellite() returns GPS within 20m
2. **Chunk vs single-image robustness**: Featureless terrain where single-image fails, chunk succeeds
3. **Chunk rotation sweeps**: Unknown orientation → try rotations (0°, 30°, ..., 330°) → match at correct angle
4. **Sim(3) transform correctness**: Verify transform aligns chunk trajectory to satellite coordinates
5. **Multi-scale chunk matching**: GSD mismatch handled correctly
6. **Performance benchmark**: 10-image chunk alignment completes within acceptable time
7. **Partial occlusion handling**: Some images occluded → chunk still matches successfully
_docs/02_components/09_metric_refinement/09._component_metric_refinement.md
+462
View File
@@ -0,0 +1,462 @@
# Metric Refinement
## Interface Definition
**Interface Name**: `IMetricRefinement`
### Interface Methods
```python
class IMetricRefinement(ABC):
@abstractmethod
def align_to_satellite(self, uav_image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[AlignmentResult]:
pass
@abstractmethod
def compute_homography(self, uav_image: np.ndarray, satellite_tile: np.ndarray) -> Optional[np.ndarray]:
pass
@abstractmethod
def extract_gps_from_alignment(self, homography: np.ndarray, tile_bounds: TileBounds, image_center: Tuple[int, int]) -> GPSPoint:
pass
@abstractmethod
def compute_match_confidence(self, alignment: AlignmentResult) -> float:
pass
@abstractmethod
def align_chunk_to_satellite(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[ChunkAlignmentResult]:
pass
@abstractmethod
def match_chunk_homography(self, chunk_images: List[np.ndarray], satellite_tile: np.ndarray) -> Optional[np.ndarray]:
pass
```
## Component Description
### Responsibilities
- LiteSAM for precise UAV-to-satellite cross-view matching
- **Requires pre-rotated images** from Image Rotation Manager
- Compute homography mapping UAV image to satellite tile
- Extract absolute GPS coordinates from alignment
- Process against single tile (drift correction) or tile grid (progressive search)
- Achieve <20m accuracy requirement
- **Chunk-to-satellite matching (more robust than single-image)**
- **Chunk homography computation**
### Scope
- Cross-view geo-localization (UAV↔satellite)
- Handles altitude variations (<1km)
- Multi-scale processing for different GSDs
- Domain gap (UAV downward vs satellite nadir view)
- **Critical**: Fails if rotation >45° (handled by F06)
- **Chunk-level matching (aggregate correspondences from multiple images)**
## API Methods
### `align_to_satellite(uav_image: np.ndarray, satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[AlignmentResult]`
**Description**: Aligns UAV image to satellite tile, returning GPS location.
**Called By**:
- F06 Image Rotation Manager (during rotation sweep)
- F11 Failure Recovery Coordinator (progressive search)
- F02.2 Flight Processing Engine (drift correction with single tile)
**Input**:
```python
uav_image: np.ndarray # Pre-rotated UAV image
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of the satellite tile
```
**Output**:
```python
AlignmentResult:
matched: bool
homography: np.ndarray # 3×3 transformation matrix
gps_center: GPSPoint # UAV image center GPS
confidence: float
inlier_count: int
total_correspondences: int
```
**Processing Flow**:
1. Extract features from both images using LiteSAM encoder
2. Compute dense correspondence field
3. Estimate homography from correspondences
4. Validate match quality (inlier count, reprojection error)
5. If valid match:
- Extract GPS from homography using tile_bounds
- Return AlignmentResult
6. If no match:
- Return None
**Match Criteria**:
- **Good match**: inlier_count > 30, confidence > 0.7
- **Weak match**: inlier_count 15-30, confidence 0.5-0.7
- **No match**: inlier_count < 15
**Error Conditions**:
- Returns `None`: No match found, rotation >45° (should be pre-rotated)
**Test Cases**:
1. **Good alignment**: Returns GPS within 20m of ground truth
2. **Altitude variation**: Handles GSD mismatch
3. **Rotation >45°**: Fails (by design, requires pre-rotation)
4. **Multi-scale**: Processes at multiple scales
---
### `compute_homography(uav_image: np.ndarray, satellite_tile: np.ndarray) -> Optional[np.ndarray]`
**Description**: Computes homography transformation from UAV to satellite.
**Called By**:
- Internal (during align_to_satellite)
**Input**:
```python
uav_image: np.ndarray
satellite_tile: np.ndarray
```
**Output**:
```python
Optional[np.ndarray]: 3×3 homography matrix or None
```
**Algorithm (LiteSAM)**:
1. Extract multi-scale features using TAIFormer
2. Compute correlation via Convolutional Token Mixer (CTM)
3. Generate dense correspondences
4. Estimate homography using RANSAC
5. Refine with non-linear optimization
**Homography Properties**:
- Maps pixels from UAV image to satellite image
- Accounts for: scale, rotation, perspective
- 8 DoF (degrees of freedom)
**Error Conditions**:
- Returns `None`: Insufficient correspondences
**Test Cases**:
1. **Valid correspondence**: Returns 3×3 matrix
2. **Insufficient features**: Returns None
---
### `extract_gps_from_alignment(homography: np.ndarray, tile_bounds: TileBounds, image_center: Tuple[int, int]) -> GPSPoint`
**Description**: Extracts GPS coordinates from homography and tile georeferencing.
**Called By**:
- Internal (during align_to_satellite)
- F06 Image Rotation Manager (for precise angle calculation)
**Input**:
```python
homography: np.ndarray # 3×3 matrix
tile_bounds: TileBounds # GPS bounds of satellite tile
image_center: Tuple[int, int] # Center pixel of UAV image
```
**Output**:
```python
GPSPoint:
lat: float
lon: float
```
**Algorithm**:
1. Apply homography to UAV image center point
2. Get pixel coordinates in satellite tile
3. Convert satellite pixel to GPS using tile_bounds and GSD
4. Return GPS coordinates
**Uses**: tile_bounds parameter, H02 GSD Calculator
**Test Cases**:
1. **Center alignment**: UAV center → correct GPS
2. **Corner alignment**: UAV corner → correct GPS
3. **Multiple points**: All points consistent
---
### `compute_match_confidence(alignment: AlignmentResult) -> float`
**Description**: Computes match confidence score from alignment quality.
**Called By**:
- Internal (during align_to_satellite)
- F11 Failure Recovery Coordinator (to decide if match acceptable)
**Input**:
```python
alignment: AlignmentResult
```
**Output**:
```python
float: Confidence score (0.0 to 1.0)
```
**Confidence Factors**:
1. **Inlier ratio**: inliers / total_correspondences
2. **Inlier count**: Absolute number of inliers
3. **Reprojection error**: Mean error of inliers (in pixels)
4. **Spatial distribution**: Inliers well-distributed vs clustered
**Thresholds**:
- **High confidence (>0.8)**: inlier_ratio > 0.6, inlier_count > 50, MRE < 0.5px
- **Medium confidence (0.5-0.8)**: inlier_ratio > 0.4, inlier_count > 30
- **Low confidence (<0.5)**: Reject match
**Test Cases**:
1. **Good match**: confidence > 0.8
2. **Weak match**: confidence 0.5-0.7
3. **Poor match**: confidence < 0.5
---
### `align_chunk_to_satellite(chunk_images: List[np.ndarray], satellite_tile: np.ndarray, tile_bounds: TileBounds) -> Optional[ChunkAlignmentResult]`
**Description**: Aligns entire chunk to satellite tile, returning GPS location.
**Called By**:
- F06 Image Rotation Manager (during chunk rotation sweep)
- F11 Failure Recovery Coordinator (chunk LiteSAM matching)
**Input**:
```python
chunk_images: List[np.ndarray] # Pre-rotated chunk images (5-20 images)
satellite_tile: np.ndarray # Reference satellite tile
tile_bounds: TileBounds # GPS bounds and GSD of the satellite tile
```
**Output**:
```python
ChunkAlignmentResult:
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint # GPS of chunk center (middle frame)
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform
```
**Processing Flow**:
1. For each image in chunk:
- Extract features using LiteSAM encoder
- Compute correspondences with satellite tile
2. Aggregate correspondences from all images
3. Estimate homography from aggregate correspondences
4. Validate match quality (inlier count, reprojection error)
5. If valid match:
- Extract GPS from chunk center using tile_bounds
- Compute Sim(3) transform (translation, rotation, scale)
- Return ChunkAlignmentResult
6. If no match:
- Return None
**Match Criteria**:
- **Good match**: inlier_count > 50, confidence > 0.7
- **Weak match**: inlier_count 30-50, confidence 0.5-0.7
- **No match**: inlier_count < 30
**Advantages over Single-Image Matching**:
- More correspondences (aggregate from multiple images)
- More robust to featureless terrain
- Better handles partial occlusions
- Higher confidence scores
**Test Cases**:
1. **Chunk alignment**: Returns GPS within 20m of ground truth
2. **Featureless terrain**: Succeeds where single-image fails
3. **Rotation >45°**: Fails (requires pre-rotation via F06)
4. **Multi-scale**: Handles GSD mismatch
---
### `match_chunk_homography(chunk_images: List[np.ndarray], satellite_tile: np.ndarray) -> Optional[np.ndarray]`
**Description**: Computes homography transformation from chunk to satellite.
**Called By**:
- Internal (during align_chunk_to_satellite)
**Input**:
```python
chunk_images: List[np.ndarray]
satellite_tile: np.ndarray
```
**Output**:
```python
Optional[np.ndarray]: 3×3 homography matrix or None
```
**Algorithm (LiteSAM)**:
1. Extract multi-scale features from all chunk images using TAIFormer
2. Aggregate features (mean or max pooling)
3. Compute correlation via Convolutional Token Mixer (CTM)
4. Generate dense correspondences
5. Estimate homography using RANSAC
6. Refine with non-linear optimization
**Homography Properties**:
- Maps pixels from chunk center to satellite image
- Accounts for: scale, rotation, perspective
- 8 DoF (degrees of freedom)
**Test Cases**:
1. **Valid correspondence**: Returns 3×3 matrix
2. **Insufficient features**: Returns None
3. **Aggregate correspondences**: More robust than single-image
## Integration Tests
### Test 1: Single Tile Drift Correction
1. Load UAV image and expected satellite tile
2. Pre-rotate UAV image to known heading
3. align_to_satellite() → returns GPS
4. Verify GPS within 20m of ground truth
### Test 2: Progressive Search (4 tiles)
1. Load UAV image from sharp turn
2. Get 2×2 tile grid from F04
3. align_to_satellite() for each tile (with tile_bounds)
4. First 3 tiles: No match
5. 4th tile: Match found → GPS extracted
### Test 3: Rotation Sensitivity
1. Rotate UAV image by 60° (not pre-rotated)
2. align_to_satellite() → returns None (fails as expected)
3. Pre-rotate to 60°
4. align_to_satellite() → succeeds
### Test 4: Multi-Scale Robustness
1. UAV at 500m altitude (GSD=0.1m/pixel)
2. Satellite at zoom 19 (GSD=0.3m/pixel)
3. LiteSAM handles scale difference → match succeeds
### Test 5: Chunk LiteSAM Matching
1. Build chunk with 10 images (plain field scenario)
2. Pre-rotate chunk to known heading
3. align_chunk_to_satellite() → returns GPS
4. Verify GPS within 20m of ground truth
5. Verify chunk matching more robust than single-image
### Test 6: Chunk Rotation Sweeps
1. Build chunk with unknown orientation
2. Try chunk rotation steps (0°, 30°, ..., 330°)
3. align_chunk_to_satellite() for each rotation
4. Match found at 120° → GPS extracted
5. Verify Sim(3) transform computed correctly
## Non-Functional Requirements
### Performance
- **align_to_satellite**: ~60ms per tile (TensorRT optimized)
- **Progressive search 25 tiles**: ~1.5 seconds total (25 × 60ms)
- Meets <5s per frame requirement
### Accuracy
- **GPS accuracy**: 60% of frames < 20m error, 80% < 50m error
- **Mean Reprojection Error (MRE)**: < 1.0 pixels
- **Alignment success rate**: > 95% when rotation correct
### Reliability
- Graceful failure when no match
- Robust to altitude variations (<1km)
- Handles seasonal appearance changes (to extent possible)
## Dependencies
### Internal Components
- **F12 Route Chunk Manager**: For chunk image retrieval and chunk operations
- **F16 Model Manager**: For LiteSAM model
- **H01 Camera Model**: For projection operations
- **H02 GSD Calculator**: For coordinate transformations
- **H05 Performance Monitor**: For timing
**Critical Dependency on F06 Image Rotation Manager**:
- F09 requires pre-rotated images (rotation <45° from north)
- Caller (F06 or F11) must pre-rotate images using F06.rotate_image_360() before calling F09.align_to_satellite()
- If rotation >45°, F09 will fail to match (by design)
- F06 handles the rotation sweep (trying 0°, 30°, 60°, etc.) and calls F09 for each rotation
**Note**: tile_bounds is passed as parameter from caller (F02.2 Flight Processing Engine gets it from F04 Satellite Data Manager)
### External Dependencies
- **LiteSAM**: Cross-view matching model
- **opencv-python**: Homography estimation
- **numpy**: Matrix operations
## Data Models
### AlignmentResult
```python
class AlignmentResult(BaseModel):
matched: bool
homography: np.ndarray # (3, 3)
gps_center: GPSPoint
confidence: float
inlier_count: int
total_correspondences: int
reprojection_error: float # Mean error in pixels
```
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float
lon: float
```
### TileBounds
```python
class TileBounds(BaseModel):
nw: GPSPoint
ne: GPSPoint
sw: GPSPoint
se: GPSPoint
center: GPSPoint
gsd: float # Ground Sampling Distance (m/pixel)
```
### LiteSAMConfig
```python
class LiteSAMConfig(BaseModel):
model_path: str
confidence_threshold: float = 0.7
min_inliers: int = 15
max_reprojection_error: float = 2.0 # pixels
multi_scale_levels: int = 3
chunk_min_inliers: int = 30 # Higher threshold for chunk matching
```
### ChunkAlignmentResult
```python
class ChunkAlignmentResult(BaseModel):
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform # Translation, rotation, scale
reprojection_error: float # Mean error in pixels
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/10_factor_graph_optimizer/10.01_feature_core_factor_management.md
+125
View File
@@ -0,0 +1,125 @@
# Feature: Core Factor Management
## Name
Core Factor Management
## Description
Provides the fundamental building blocks for factor graph construction by adding relative pose measurements (from VO), absolute GPS measurements (from LiteSAM or user anchors), and altitude priors. Handles scale resolution for monocular VO via GSD computation and applies robust kernels (Huber/Cauchy) for outlier handling.
## Component APIs Implemented
- `add_relative_factor(flight_id, frame_i, frame_j, relative_pose, covariance) -> bool`
- `add_absolute_factor(flight_id, frame_id, gps, covariance, is_user_anchor) -> bool`
- `add_altitude_prior(flight_id, frame_id, altitude, covariance) -> bool`
## External Tools and Services
- **GTSAM**: BetweenFactor, PriorFactor, UnaryFactor creation
- **H02 GSD Calculator**: Scale resolution for monocular VO (GSD computation)
- **H03 Robust Kernels**: Huber/Cauchy loss functions for outlier handling
- **F17 Configuration Manager**: Camera parameters, altitude, frame spacing
## Internal Methods
### `_scale_relative_translation(flight_id, translation, frame_i, frame_j) -> np.ndarray`
Scales unit translation vector from VO using GSD and expected displacement.
- Retrieves altitude and camera params from F17
- Calls H02.compute_gsd() to get ground sampling distance
- Computes expected_displacement = frame_spacing × GSD
- Returns scaled_translation = unit_translation × expected_displacement
### `_create_between_factor(frame_i, frame_j, pose, covariance) -> gtsam.BetweenFactor`
Creates GTSAM BetweenFactor for relative pose measurement.
- Creates Pose3 from translation and rotation
- Wraps with robust kernel (Huber) for outlier handling
- Returns configured factor
### `_create_prior_factor(frame_id, position, covariance, is_hard_constraint) -> gtsam.PriorFactor`
Creates GTSAM PriorFactor for absolute position constraint.
- Converts GPS (lat/lon) to ENU coordinates
- Sets covariance based on source (5m for user anchor, 20-50m for LiteSAM)
- Returns configured factor
### `_create_altitude_factor(frame_id, altitude, covariance) -> gtsam.UnaryFactor`
Creates GTSAM UnaryFactor for Z-coordinate constraint.
- Creates soft constraint on altitude
- Prevents scale drift in monocular VO
### `_apply_robust_kernel(factor, kernel_type, threshold) -> gtsam.Factor`
Wraps factor with robust kernel for outlier handling.
- Supports Huber (default) and Cauchy kernels
- Critical for 350m outlier handling
### `_gps_to_enu(gps, reference_origin) -> np.ndarray`
Converts GPS coordinates to local ENU (East-North-Up) frame.
- Uses reference origin (first frame or flight start GPS)
- Returns (x, y, z) in meters
### `_ensure_graph_initialized(flight_id) -> bool`
Ensures factor graph exists for flight, creates if needed.
- Creates new GTSAM iSAM2 instance if not exists
- Initializes reference origin for ENU conversion
## Unit Tests
### Test: add_relative_factor_scales_translation
- Arrange: Mock H02, F17 with known GSD values
- Act: Call add_relative_factor with unit translation
- Assert: Factor added with correctly scaled translation
### Test: add_relative_factor_applies_huber_kernel
- Arrange: Create factor graph
- Act: Add relative factor
- Assert: Factor has Huber kernel applied
### Test: add_absolute_factor_converts_gps_to_enu
- Arrange: Set reference origin
- Act: Add absolute factor with known GPS
- Assert: Factor position in correct ENU coordinates
### Test: add_absolute_factor_sets_covariance_by_source
- Arrange: Create factor graph
- Act: Add user anchor (is_user_anchor=True)
- Assert: Covariance is 5m (high confidence)
### Test: add_absolute_factor_litesam_covariance
- Arrange: Create factor graph
- Act: Add LiteSAM match (is_user_anchor=False)
- Assert: Covariance is 20-50m
### Test: add_altitude_prior_creates_soft_constraint
- Arrange: Create factor graph
- Act: Add altitude prior
- Assert: UnaryFactor created with correct covariance
### Test: add_relative_factor_returns_false_on_invalid_flight
- Arrange: No graph for flight_id
- Act: Call add_relative_factor
- Assert: Returns False (or creates graph per implementation)
### Test: scale_resolution_uses_f17_config
- Arrange: Mock F17 with specific altitude, camera params
- Act: Add relative factor
- Assert: H02 called with correct parameters from F17
## Integration Tests
### Test: incremental_trajectory_building
1. Initialize graph with first frame
2. Add 100 relative factors from VO
3. Add 100 altitude priors
4. Verify all factors added successfully
5. Verify graph structure correct
### Test: outlier_handling_350m
1. Add normal relative factors (10 frames)
2. Add 350m outlier factor (simulating tilt error)
3. Add more normal factors
4. Verify outlier factor weight reduced by Huber kernel
5. Verify graph remains stable
### Test: mixed_factor_types
1. Add relative factors
2. Add absolute GPS factors
3. Add altitude priors
4. Verify all factor types coexist correctly
5. Verify graph ready for optimization
_docs/02_components/10_factor_graph_optimizer/10.02_feature_trajectory_optimization.md
+125
View File
@@ -0,0 +1,125 @@
# Feature: Trajectory Optimization & Retrieval
## Name
Trajectory Optimization & Retrieval
## Description
Provides core optimization functionality using GTSAM's iSAM2 for incremental updates and Levenberg-Marquardt for batch optimization. Retrieves optimized trajectory poses and marginal covariances for uncertainty quantification. Supports both real-time incremental optimization (after each frame) and batch refinement (when new absolute factors added).
## Component APIs Implemented
- `optimize(flight_id, iterations) -> OptimizationResult`
- `get_trajectory(flight_id) -> Dict[int, Pose]`
- `get_marginal_covariance(flight_id, frame_id) -> np.ndarray`
## External Tools and Services
- **GTSAM**: iSAM2 incremental optimizer, Levenberg-Marquardt batch solver
- **numpy**: Matrix operations for covariance extraction
## Internal Methods
### `_run_isam2_update(flight_id, iterations) -> OptimizationResult`
Runs incremental iSAM2 optimization on flight's factor graph.
- Updates only affected nodes (fast, <100ms)
- Used for real-time frame-by-frame optimization
- Returns optimization statistics
### `_run_batch_optimization(flight_id, iterations) -> OptimizationResult`
Runs full Levenberg-Marquardt optimization on entire graph.
- Re-optimizes all poses (slower, ~500ms for 100 frames)
- Used when new absolute factors added for back-propagation
- Returns optimization statistics
### `_check_convergence(prev_error, curr_error, threshold) -> bool`
Checks if optimization has converged.
- Compares error reduction against threshold
- Returns True if error reduction < threshold
### `_extract_poses_from_graph(flight_id) -> Dict[int, Pose]`
Extracts all optimized poses from GTSAM Values.
- Converts GTSAM Pose3 to internal Pose model
- Includes position, orientation, timestamp
### `_compute_marginal_covariance(flight_id, frame_id) -> np.ndarray`
Computes marginal covariance for specific frame.
- Uses GTSAM Marginals class
- Returns 6x6 covariance matrix [x, y, z, roll, pitch, yaw]
### `_get_optimized_frames(flight_id, prev_values, curr_values) -> List[int]`
Determines which frames had pose updates.
- Compares previous and current values
- Returns list of frame IDs with significant changes
### `_compute_mean_reprojection_error(flight_id) -> float`
Computes mean reprojection error across all factors.
- Used for quality assessment
- Should be < 1.0 pixels per AC
## Unit Tests
### Test: optimize_incremental_updates_affected_nodes
- Arrange: Create graph with 10 frames, add new factor to frame 8
- Act: Call optimize with few iterations
- Assert: Only frames 8-10 significantly updated
### Test: optimize_batch_updates_all_frames
- Arrange: Create graph with drift, add absolute GPS at frame 50
- Act: Call optimize with batch mode
- Assert: All frames updated (back-propagation)
### Test: optimize_returns_convergence_status
- Arrange: Create well-constrained graph
- Act: Call optimize
- Assert: OptimizationResult.converged == True
### Test: get_trajectory_returns_all_poses
- Arrange: Create graph with 100 frames, optimize
- Act: Call get_trajectory
- Assert: Returns dict with 100 poses
### Test: get_trajectory_poses_in_enu
- Arrange: Create graph with known reference
- Act: Get trajectory
- Assert: Positions in ENU coordinates
### Test: get_marginal_covariance_returns_6x6
- Arrange: Create and optimize graph
- Act: Call get_marginal_covariance for frame
- Assert: Returns (6, 6) numpy array
### Test: get_marginal_covariance_reduces_after_absolute_factor
- Arrange: Create graph, get covariance for frame 50
- Act: Add absolute GPS factor at frame 50, re-optimize
- Assert: New covariance smaller than before
### Test: optimize_respects_iteration_limit
- Arrange: Create complex graph
- Act: Call optimize with iterations=5
- Assert: iterations_used <= 5
## Integration Tests
### Test: drift_correction_with_absolute_gps
1. Build trajectory with VO only (100 frames) - will drift
2. Add absolute GPS factor at frame 50
3. Optimize (batch mode)
4. Verify trajectory corrects
5. Verify frames 1-49 also corrected (back-propagation)
### Test: user_anchor_immediate_refinement
1. Build trajectory to frame 237
2. Add user anchor (high confidence, is_user_anchor=True)
3. Optimize
4. Verify trajectory snaps to anchor
5. Verify low covariance at anchor frame
### Test: performance_incremental_under_100ms
1. Create graph with 50 frames
2. Add new relative factor
3. Time incremental optimization
4. Assert: < 100ms
### Test: performance_batch_under_500ms
1. Create graph with 100 frames
2. Time batch optimization
3. Assert: < 500ms
_docs/02_components/10_factor_graph_optimizer/10.03_feature_chunk_subgraph_operations.md
+139
View File
@@ -0,0 +1,139 @@
# Feature: Chunk Subgraph Operations
## Name
Chunk Subgraph Operations
## Description
Manages chunk-level factor graph subgraphs for handling tracking loss recovery. Creates independent subgraphs for route chunks (map fragments), adds factors to chunk-specific subgraphs, anchors chunks with GPS measurements, and optimizes chunks independently. F10 provides low-level factor graph operations only; F12 owns chunk metadata (status, is_active, etc.).
## Component APIs Implemented
- `create_chunk_subgraph(flight_id, chunk_id, start_frame_id) -> bool`
- `add_relative_factor_to_chunk(flight_id, chunk_id, frame_i, frame_j, relative_pose, covariance) -> bool`
- `add_chunk_anchor(flight_id, chunk_id, frame_id, gps, covariance) -> bool`
- `get_chunk_trajectory(flight_id, chunk_id) -> Dict[int, Pose]`
- `optimize_chunk(flight_id, chunk_id, iterations) -> OptimizationResult`
## External Tools and Services
- **GTSAM**: Subgraph management, BetweenFactor, PriorFactor
- **H03 Robust Kernels**: Huber kernel for chunk factors
## Internal Methods
### `_create_subgraph(flight_id, chunk_id) -> gtsam.NonlinearFactorGraph`
Creates new GTSAM subgraph for chunk.
- Initializes empty factor graph
- Stores in chunk subgraph dictionary
### `_initialize_chunk_origin(flight_id, chunk_id, start_frame_id) -> bool`
Initializes first frame pose in chunk's local coordinate system.
- Sets origin pose at identity (or relative to previous chunk)
- Adds prior factor for origin frame
### `_add_factor_to_subgraph(flight_id, chunk_id, factor) -> bool`
Adds factor to chunk's specific subgraph.
- Verifies chunk exists
- Appends factor to chunk's graph
- Marks chunk as needing optimization
### `_get_chunk_subgraph(flight_id, chunk_id) -> Optional[gtsam.NonlinearFactorGraph]`
Retrieves chunk's subgraph if exists.
- Returns None if chunk not found
### `_extract_chunk_poses(flight_id, chunk_id) -> Dict[int, Pose]`
Extracts all optimized poses from chunk's subgraph.
- Poses in chunk's local coordinate system
- Converts GTSAM Pose3 to internal Pose model
### `_optimize_subgraph(flight_id, chunk_id, iterations) -> OptimizationResult`
Runs Levenberg-Marquardt on chunk's isolated subgraph.
- Optimizes only chunk's factors
- Does not affect other chunks
- Returns optimization result
### `_anchor_chunk_to_global(flight_id, chunk_id, frame_id, enu_position) -> bool`
Adds GPS anchor to chunk's subgraph as PriorFactor.
- Converts GPS to ENU (may need separate reference or global reference)
- Adds prior factor for anchor frame
- Returns success status
### `_track_chunk_frames(flight_id, chunk_id, frame_id) -> bool`
Tracks which frames belong to which chunk.
- Updates frame-to-chunk mapping
- Used for factor routing
## Unit Tests
### Test: create_chunk_subgraph_returns_true
- Arrange: Valid flight_id
- Act: Call create_chunk_subgraph
- Assert: Returns True, subgraph created
### Test: create_chunk_subgraph_initializes_origin
- Arrange: Create chunk
- Act: Check subgraph
- Assert: Has prior factor for start_frame_id
### Test: add_relative_factor_to_chunk_success
- Arrange: Create chunk
- Act: Add relative factor to chunk
- Assert: Returns True, factor in chunk's subgraph
### Test: add_relative_factor_to_chunk_nonexistent_chunk
- Arrange: No chunk created
- Act: Add relative factor to nonexistent chunk
- Assert: Returns False
### Test: add_chunk_anchor_success
- Arrange: Create chunk with frames
- Act: Add GPS anchor
- Assert: Returns True, prior factor added
### Test: get_chunk_trajectory_returns_poses
- Arrange: Create chunk, add factors, optimize
- Act: Call get_chunk_trajectory
- Assert: Returns dict with chunk poses
### Test: get_chunk_trajectory_empty_chunk
- Arrange: Create chunk without factors
- Act: Call get_chunk_trajectory
- Assert: Returns dict with only origin pose
### Test: optimize_chunk_success
- Arrange: Create chunk with factors
- Act: Call optimize_chunk
- Assert: Returns OptimizationResult with converged=True
### Test: optimize_chunk_isolation
- Arrange: Create chunk_1 and chunk_2 with factors
- Act: Optimize chunk_1
- Assert: chunk_2 poses unchanged
### Test: multiple_chunks_simultaneous
- Arrange: Create 3 chunks
- Act: Add factors to each
- Assert: All chunks maintain independent state
## Integration Tests
### Test: multi_chunk_creation_and_isolation
1. Create chunk_1 with frames 1-10
2. Create chunk_2 with frames 20-30 (disconnected)
3. Add relative factors to each chunk
4. Verify chunks optimized independently
5. Verify factors isolated to respective chunks
### Test: chunk_anchoring
1. Create chunk with frames 1-10
2. Add relative factors
3. Add chunk_anchor at frame 5
4. Optimize chunk
5. Verify trajectory consistent with anchor
6. Verify has_anchor reflected (via F12 query)
### Test: chunk_trajectory_local_coordinates
1. Create chunk
2. Add relative factors
3. Get chunk trajectory
4. Verify poses in chunk's local coordinate system
5. Verify origin at identity or start frame
_docs/02_components/10_factor_graph_optimizer/10.04_feature_chunk_merging_global_optimization.md
+142
View File
@@ -0,0 +1,142 @@
# Feature: Chunk Merging & Global Optimization
## Name
Chunk Merging & Global Optimization
## Description
Provides Sim(3) similarity transformation for merging chunk subgraphs and global optimization across all chunks. Handles scale, rotation, and translation differences between chunks (critical for monocular VO with scale ambiguity). Called by F12 Route Chunk Manager after chunk matching succeeds.
## Component APIs Implemented
- `merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform: Sim3Transform) -> bool`
- `optimize_global(flight_id, iterations) -> OptimizationResult`
## External Tools and Services
- **GTSAM**: Graph merging, global optimization
- **numpy/scipy**: Sim(3) transformation computation
## Internal Methods
### `_apply_sim3_transform(poses: Dict[int, Pose], transform: Sim3Transform) -> Dict[int, Pose]`
Applies Sim(3) similarity transformation to all poses.
- Translation: p' = s * R * p + t
- Rotation: R' = R_transform * R_original
- Scale: All positions scaled by transform.scale
- Critical for merging chunks with different scales
### `_merge_subgraphs(flight_id, source_chunk_id, dest_chunk_id) -> bool`
Merges source chunk's subgraph into destination chunk's subgraph.
- Copies all factors from source to destination
- Updates node keys to avoid conflicts
- Preserves factor graph structure
### `_update_frame_to_chunk_mapping(flight_id, source_chunk_id, dest_chunk_id) -> bool`
Updates frame-to-chunk mapping after merge.
- All frames from source_chunk now belong to dest_chunk
- Used for routing future factors
### `_transform_chunk_factors(flight_id, chunk_id, transform: Sim3Transform) -> bool`
Transforms all factors in chunk's subgraph by Sim(3).
- Updates BetweenFactor translations with scale
- Updates PriorFactor positions
- Preserves covariances (may need adjustment for scale)
### `_validate_merge_preconditions(flight_id, new_chunk_id, main_chunk_id) -> bool`
Validates that merge can proceed.
- Verifies both chunks exist
- Verifies new_chunk has anchor (required for merge)
- Returns False if preconditions not met
### `_collect_all_chunk_subgraphs(flight_id) -> List[gtsam.NonlinearFactorGraph]`
Collects all chunk subgraphs for global optimization.
- Returns list of all subgraphs
- Filters out merged/inactive chunks (via F12)
### `_run_global_optimization(flight_id, iterations) -> OptimizationResult`
Runs optimization across all anchored chunks.
- Transforms chunks to global coordinate system
- Runs Levenberg-Marquardt on combined graph
- Updates all chunk trajectories
### `_compute_sim3_from_correspondences(source_poses, dest_poses) -> Sim3Transform`
Computes Sim(3) transformation from pose correspondences.
- Uses Horn's method or similar for rigid + scale estimation
- Returns translation, rotation, scale
### `_update_all_chunk_trajectories(flight_id) -> bool`
Updates all chunk trajectories after global optimization.
- Extracts poses from global graph
- Distributes to chunk-specific storage
## Unit Tests
### Test: merge_chunk_subgraphs_success
- Arrange: Create chunk_1 (anchored), chunk_2 (with anchor)
- Act: Call merge_chunk_subgraphs with Sim3 transform
- Assert: Returns True, chunks merged
### Test: merge_chunk_subgraphs_unanchored_fails
- Arrange: Create chunk_1, chunk_2 (no anchor)
- Act: Call merge_chunk_subgraphs
- Assert: Returns False
### Test: merge_applies_sim3_transform
- Arrange: Create chunks with known poses, define transform
- Act: Merge chunks
- Assert: new_chunk poses transformed correctly
### Test: merge_updates_frame_mapping
- Arrange: Create chunks, merge
- Act: Query frame-to-chunk mapping
- Assert: new_chunk frames now belong to main_chunk
### Test: optimize_global_returns_result
- Arrange: Create multiple anchored chunks
- Act: Call optimize_global
- Assert: Returns OptimizationResult with all frames
### Test: optimize_global_achieves_consistency
- Arrange: Create chunks with overlapping anchors
- Act: Optimize global
- Assert: Anchor frames align within tolerance
### Test: sim3_transform_handles_scale
- Arrange: Poses with scale=2.0 difference
- Act: Apply Sim(3) transform
- Assert: Scaled positions match expected
### Test: sim3_transform_handles_rotation
- Arrange: Poses with 90° rotation difference
- Act: Apply Sim(3) transform
- Assert: Rotated poses match expected
## Integration Tests
### Test: chunk_anchoring_and_merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Add relative factors to each
3. Add chunk_anchor to chunk_2 (frame 25)
4. Optimize chunk_2 → local consistency improved
5. Merge chunk_2 into chunk_1 with Sim(3) transform
6. Optimize global → both chunks globally consistent
7. Verify final trajectory coherent across chunks
### Test: simultaneous_multi_chunk_processing
1. Create 3 chunks simultaneously (disconnected segments)
2. Process frames in each chunk independently
3. Anchor each chunk asynchronously
4. Merge chunks as anchors become available
5. Verify final global trajectory consistent
### Test: global_optimization_performance
1. Create 5 chunks with 20 frames each
2. Anchor all chunks
3. Time optimize_global
4. Assert: < 500ms for 100 total frames
### Test: scale_resolution_across_chunks
1. Create chunk_1 with scale drift (monocular VO)
2. Create chunk_2 with different scale drift
3. Anchor both chunks
4. Merge with Sim(3) accounting for scale
5. Verify merged trajectory has consistent scale
_docs/02_components/10_factor_graph_optimizer/10.05_feature_multi_flight_graph_lifecycle.md
+152
View File
@@ -0,0 +1,152 @@
# Feature: Multi-Flight Graph Lifecycle
## Name
Multi-Flight Graph Lifecycle
## Description
Manages flight-scoped factor graph state for concurrent multi-flight processing. Each flight maintains an independent factor graph (Dict[str, FactorGraph] keyed by flight_id), enabling parallel processing without cross-contamination. Provides cleanup when flights are deleted.
## Component APIs Implemented
- `delete_flight_graph(flight_id) -> bool`
## External Tools and Services
- **GTSAM**: Factor graph instance management
## Internal Methods
### `_get_or_create_flight_graph(flight_id) -> FlightGraphState`
Gets existing flight graph state or creates new one.
- Manages Dict[str, FlightGraphState] internally
- Initializes iSAM2 instance, reference origin, chunk subgraphs
- Thread-safe access
### `_initialize_flight_graph(flight_id) -> FlightGraphState`
Initializes new factor graph state for flight.
- Creates new iSAM2 instance
- Initializes empty chunk subgraph dictionary
- Sets reference origin to None (set on first absolute factor)
### `_cleanup_flight_resources(flight_id) -> bool`
Cleans up all resources associated with flight.
- Removes factor graph from memory
- Cleans up chunk subgraphs
- Clears frame-to-chunk mappings
- Releases GTSAM objects
### `_validate_flight_exists(flight_id) -> bool`
Validates that flight graph exists.
- Returns True if flight_id in graph dictionary
- Used as guard for all operations
### `_get_all_flight_ids() -> List[str]`
Returns list of all active flight IDs.
- Used for administrative/monitoring purposes
### `_get_flight_statistics(flight_id) -> FlightGraphStats`
Returns statistics about flight's factor graph.
- Number of frames, factors, chunks
- Memory usage estimate
- Last optimization time
## Data Structures
### `FlightGraphState`
Internal state container for a flight's factor graph.
```python
class FlightGraphState:
flight_id: str
isam2: gtsam.ISAM2
values: gtsam.Values
reference_origin: Optional[GPSPoint] # ENU reference
chunk_subgraphs: Dict[str, gtsam.NonlinearFactorGraph]
chunk_values: Dict[str, gtsam.Values]
frame_to_chunk: Dict[int, str]
created_at: datetime
last_optimized: Optional[datetime]
```
### `FlightGraphStats`
Statistics for monitoring.
```python
class FlightGraphStats:
flight_id: str
num_frames: int
num_factors: int
num_chunks: int
num_active_chunks: int
estimated_memory_mb: float
last_optimization_time_ms: float
```
## Unit Tests
### Test: delete_flight_graph_success
- Arrange: Create flight graph with factors
- Act: Call delete_flight_graph
- Assert: Returns True, graph removed
### Test: delete_flight_graph_nonexistent
- Arrange: No graph for flight_id
- Act: Call delete_flight_graph
- Assert: Returns False (graceful handling)
### Test: delete_flight_graph_cleans_chunks
- Arrange: Create flight with multiple chunks
- Act: Delete flight graph
- Assert: All chunk subgraphs cleaned up
### Test: flight_isolation_no_cross_contamination
- Arrange: Create flight_1, flight_2
- Act: Add factors to flight_1
- Assert: flight_2 graph unchanged
### Test: concurrent_flight_processing
- Arrange: Create flight_1, flight_2
- Act: Add factors to both concurrently
- Assert: Both maintain independent state
### Test: get_or_create_creates_new
- Arrange: No existing graph
- Act: Call internal _get_or_create_flight_graph
- Assert: New graph created with default state
### Test: get_or_create_returns_existing
- Arrange: Graph exists for flight_id
- Act: Call internal _get_or_create_flight_graph
- Assert: Same graph instance returned
### Test: flight_statistics_accurate
- Arrange: Create flight with known factors
- Act: Get flight statistics
- Assert: Counts match expected
## Integration Tests
### Test: multi_flight_concurrent_processing
1. Create flight_1, flight_2, flight_3 concurrently
2. Add factors to each flight in parallel
3. Optimize each flight independently
4. Verify each flight has correct trajectory
5. Delete flight_2
6. Verify flight_1, flight_3 unaffected
### Test: flight_cleanup_memory_release
1. Create flight with 1000 frames
2. Optimize trajectory
3. Note memory usage
4. Delete flight graph
5. Verify memory released (within tolerance)
### Test: reference_origin_per_flight
1. Create flight_1 with start GPS at point A
2. Create flight_2 with start GPS at point B
3. Add absolute factors to each
4. Verify ENU coordinates relative to respective origins
### Test: chunk_cleanup_on_flight_delete
1. Create flight with 5 chunks
2. Add factors and anchors to chunks
3. Delete flight
4. Verify all chunk subgraphs cleaned up
5. Verify no memory leaks
_docs/02_components/10_factor_graph_optimizer/10._component_factor_graph_optimizer.md
+830
View File
@@ -0,0 +1,830 @@
# Factor Graph Optimizer
## Interface Definition
**Interface Name**: `IFactorGraphOptimizer`
### Interface Methods
```python
class IFactorGraphOptimizer(ABC):
"""
GTSAM-based factor graph optimizer for trajectory estimation.
## Multi-Flight Support
All state-modifying methods require `flight_id` parameter.
Each flight maintains an independent factor graph state.
F10 internally manages Dict[str, FactorGraph] keyed by flight_id.
This enables:
- Concurrent processing of multiple flights
- Flight-scoped optimization without cross-contamination
- Independent cleanup via delete_flight_graph()
"""
@abstractmethod
def add_relative_factor(self, flight_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def add_absolute_factor(self, flight_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool:
pass
@abstractmethod
def add_altitude_prior(self, flight_id: str, frame_id: int, altitude: float, covariance: float) -> bool:
pass
@abstractmethod
def optimize(self, flight_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def get_trajectory(self, flight_id: str) -> Dict[int, Pose]:
pass
@abstractmethod
def get_marginal_covariance(self, flight_id: str, frame_id: int) -> np.ndarray:
pass
# Chunk operations - F10 only manages factor graph subgraphs
# F12 owns chunk metadata (status, is_active, etc.)
@abstractmethod
def create_chunk_subgraph(self, flight_id: str, chunk_id: str, start_frame_id: int) -> bool:
pass
@abstractmethod
def add_relative_factor_to_chunk(self, flight_id: str, chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def add_chunk_anchor(self, flight_id: str, chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool:
pass
@abstractmethod
def merge_chunk_subgraphs(self, flight_id: str, new_chunk_id: str, main_chunk_id: str, transform: Sim3Transform) -> bool:
"""Merges new_chunk INTO main_chunk. Extends main_chunk with new_chunk's subgraph."""
pass
@abstractmethod
def get_chunk_trajectory(self, flight_id: str, chunk_id: str) -> Dict[int, Pose]:
pass
@abstractmethod
def optimize_chunk(self, flight_id: str, chunk_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def optimize_global(self, flight_id: str, iterations: int) -> OptimizationResult:
pass
@abstractmethod
def delete_flight_graph(self, flight_id: str) -> bool:
"""Cleanup factor graph when flight is deleted."""
pass
```
## Component Description
### Responsibilities
- GTSAM-based fusion of relative and absolute measurements
- Incremental optimization (iSAM2) for real-time performance
- Robust kernels (Huber/Cauchy) for 350m outlier handling
- Scale resolution through altitude priors and absolute GPS
- Trajectory smoothing and global consistency
- Back-propagation of refinements to previous frames
- **Low-level factor graph chunk operations** (subgraph creation, factor addition, optimization)
- **Sim(3) transformation for chunk merging**
### Chunk Responsibility Clarification
**F10 provides low-level factor graph operations only**:
- `create_chunk_subgraph()`: Creates subgraph in factor graph (returns bool, not ChunkHandle)
- `add_relative_factor_to_chunk()`: Adds factors to chunk's subgraph
- `add_chunk_anchor()`: Adds GPS anchor to chunk's subgraph
- `merge_chunk_subgraphs()`: Applies Sim(3) transform and merges subgraphs
- `optimize_chunk()`, `optimize_global()`: Runs optimization
**F10 does NOT own chunk metadata** - only factor graph data structures.
**F12 is the source of truth for ALL chunk state** (see F12 spec):
- ChunkHandle with all metadata (is_active, has_anchor, matching_status)
- Chunk lifecycle management
- Chunk readiness determination
- High-level chunk queries
- F12 calls F10 for factor graph operations
**F11 coordinates recovery** (see F11 spec):
- Triggers chunk creation via F12
- Coordinates matching workflows
- Emits chunk-related events
### Scope
- Non-linear least squares optimization
- Factor graph representation of SLAM problem
- Handles monocular scale ambiguity
- Real-time incremental updates
- Asynchronous batch refinement
- **Multi-chunk factor graph with independent subgraphs**
- **Chunk-level optimization and global merging**
- **Sim(3) similarity transformation for chunk alignment**
### Design Pattern: Composition Over Complex Interface
F10 uses **composition** to keep the interface manageable. Rather than exposing 20+ methods in a monolithic interface, complex operations are composed from simpler primitives:
**Primitive Operations**:
- `add_relative_factor()`, `add_absolute_factor()`, `add_altitude_prior()` - Factor management
- `optimize()`, `get_trajectory()`, `get_marginal_covariance()` - Core optimization
**Chunk Operations** (composed from primitives):
- `create_new_chunk()`, `add_relative_factor_to_chunk()`, `add_chunk_anchor()` - Chunk factor management
- `merge_chunks()`, `optimize_chunk()`, `optimize_global()` - Chunk optimization
**Callers compose these primitives** for complex workflows (e.g., F11 composes anchor + merge + optimize_global).
## API Methods
### `add_relative_factor(frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool`
**Description**: Adds relative pose measurement between consecutive frames.
**Called By**:
- F07 Sequential VO (frame-to-frame odometry)
**Input**:
```python
frame_i: int # Previous frame ID
frame_j: int # Current frame ID (typically frame_i + 1)
relative_pose: RelativePose:
translation: np.ndarray # (3,) - unit vector (scale ambiguous from VO)
rotation: np.ndarray # (3, 3) or quaternion
covariance: np.ndarray # (6, 6) - uncertainty
```
**Scale Resolution**:
F07 returns unit translation vectors due to monocular scale ambiguity. F10 resolves scale by:
1. Using altitude prior to constrain Z-axis
2. **Computing expected displacement**: Call H02 GSD Calculator.compute_gsd() to get GSD
- GSD = (sensor_width × altitude) / (focal_length × resolution_width)
- expected_displacement ≈ frame_spacing × GSD (typically ~100m)
3. Scaling: scaled_translation = unit_translation × expected_displacement
4. Global refinement using absolute GPS factors from F09 LiteSAM
**Explicit Flow**:
```python
# In add_relative_factor():
# altitude comes from F17 Configuration Manager (predefined operational altitude)
# focal_length, sensor_width from F17 Configuration Manager
config = F17.get_flight_config(flight_id)
altitude = config.altitude # Predefined altitude, NOT from EXIF
gsd = H02.compute_gsd(altitude, config.camera_params.focal_length,
config.camera_params.sensor_width,
config.camera_params.resolution_width)
expected_displacement = frame_spacing * gsd # ~100m typical at 300m altitude
scaled_translation = relative_pose.translation * expected_displacement
# Add scaled_translation to factor graph
```
**Note**: Altitude comes from F17 Configuration Manager (predefined operational altitude), NOT from EXIF metadata. The problem statement specifies images don't have GPS metadata.
**Output**:
```python
bool: True if factor added successfully
```
**Processing Flow**:
1. Create BetweenFactor in GTSAM
2. Apply robust kernel (Huber) to handle outliers
3. Add to factor graph
4. Mark graph as needing optimization
**Robust Kernel**:
- **Huber loss**: Downweights large errors (>threshold)
- **Critical** for 350m outlier handling from tilt
**Test Cases**:
1. **Normal motion**: Factor added, contributes to optimization
2. **Large displacement** (350m outlier): Huber kernel reduces weight
3. **Consecutive factors**: Chain of relative factors builds trajectory
---
### `add_absolute_factor(frame_id: int, gps: GPSPoint, covariance: np.ndarray, is_user_anchor: bool) -> bool`
**Description**: Adds absolute GPS measurement for drift correction or user anchor.
**Called By**:
- F09 Metric Refinement (after LiteSAM alignment)
- F11 Failure Recovery Coordinator (user-provided anchors)
**Input**:
```python
frame_id: int
gps: GPSPoint:
lat: float
lon: float
covariance: np.ndarray # (2, 2) or (3, 3) - GPS uncertainty
is_user_anchor: bool # True for user-provided fixes (high confidence)
```
**Output**:
```python
bool: True if factor added
```
**Processing Flow**:
1. Convert GPS to local ENU coordinates (East-North-Up)
2. Create PriorFactor or UnaryFactor
3. Set covariance (low for user anchors, higher for LiteSAM)
4. Add to factor graph
5. Trigger optimization (immediate for user anchors)
**Covariance Settings**:
- **User anchor**: σ = 5m (high confidence)
- **LiteSAM match**: σ = 20-50m (depends on confidence)
**Test Cases**:
1. **LiteSAM GPS**: Adds absolute factor, corrects drift
2. **User anchor**: High confidence, immediately refines trajectory
3. **Multiple absolute factors**: Graph optimizes to balance all
---
### `add_altitude_prior(frame_id: int, altitude: float, covariance: float) -> bool`
**Description**: Adds altitude constraint to resolve monocular scale ambiguity.
**Called By**:
- Main processing loop (for each frame)
**Input**:
```python
frame_id: int
altitude: float # Predefined altitude in meters
covariance: float # Altitude uncertainty (e.g., 50m)
```
**Output**:
```python
bool: True if prior added
```
**Processing Flow**:
1. Create UnaryFactor for Z-coordinate
2. Set as soft constraint (not hard constraint)
3. Add to factor graph
**Purpose**:
- Resolves scale ambiguity in monocular VO
- Prevents scale drift (trajectory collapsing or exploding)
- Soft constraint allows adjustment based on absolute GPS
**Test Cases**:
1. **Without altitude prior**: Scale drifts over time
2. **With altitude prior**: Scale stabilizes
3. **Conflicting measurements**: Optimizer balances VO and altitude
---
### `optimize(iterations: int) -> OptimizationResult`
**Description**: Runs optimization to refine trajectory.
**Called By**:
- Main processing loop (incremental after each frame)
- Asynchronous refinement thread (batch optimization)
**Input**:
```python
iterations: int # Max iterations (typically 5-10 for incremental, 50-100 for batch)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int] # Frames with updated poses
```
**Processing Details**:
- **Incremental** (iSAM2): Updates only affected nodes
- **Batch**: Re-optimizes entire trajectory when new absolute factors added
- **Robust M-estimation**: Automatically downweights outliers
**Optimization Algorithm** (Levenberg-Marquardt):
1. Linearize factor graph around current estimate
2. Solve linear system
3. Update pose estimates
4. Check convergence (error reduction < threshold)
**Test Cases**:
1. **Incremental optimization**: Fast (<100ms), local update
2. **Batch optimization**: Slower (~500ms), refines entire trajectory
3. **Convergence**: Error reduces, converges within iterations
---
### `get_trajectory() -> Dict[int, Pose]`
**Description**: Retrieves complete optimized trajectory.
**Called By**:
- F13 Result Manager (for publishing results)
- F12 Coordinate Transformer (for GPS conversion)
**Input**: None
**Output**:
```python
Dict[int, Pose]:
frame_id -> Pose:
position: np.ndarray # (x, y, z) in ENU
orientation: np.ndarray # Quaternion or rotation matrix
timestamp: datetime
```
**Processing Flow**:
1. Extract all pose estimates from graph
2. Convert to appropriate coordinate system
3. Return dictionary
**Test Cases**:
1. **After optimization**: Returns all frame poses
2. **Refined trajectory**: Poses updated after batch optimization
---
### `get_marginal_covariance(frame_id: int) -> np.ndarray`
**Description**: Gets uncertainty (covariance) of a pose estimate.
**Called By**:
- F11 Failure Recovery Coordinator (to detect high uncertainty)
**Input**:
```python
frame_id: int
```
**Output**:
```python
np.ndarray: (6, 6) covariance matrix [x, y, z, roll, pitch, yaw]
```
**Purpose**:
- Uncertainty quantification
- Trigger user input when uncertainty too high (> 50m radius)
**Test Cases**:
1. **Well-constrained pose**: Small covariance
2. **Unconstrained pose**: Large covariance
3. **After absolute factor**: Covariance reduces
---
### `create_new_chunk(chunk_id: str, start_frame_id: int) -> ChunkHandle`
**Description**: Creates a new map fragment/chunk with its own subgraph.
**Called By**:
- F02 Flight Processor (when tracking lost)
- F12 Route Chunk Manager (chunk lifecycle)
**Input**:
```python
chunk_id: str # Unique chunk identifier
start_frame_id: int # First frame in chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
**Processing Flow**:
1. Create new subgraph for chunk
2. Initialize first frame pose in chunk's local coordinate system
3. Mark chunk as active
4. Return ChunkHandle
**Test Cases**:
1. **Create chunk**: Returns ChunkHandle with is_active=True
2. **Multiple chunks**: Can create multiple chunks simultaneously
3. **Chunk isolation**: Factors added to chunk don't affect other chunks
---
### `get_chunk_for_frame(frame_id: int) -> Optional[ChunkHandle]`
**Description**: Gets the chunk containing the specified frame (low-level factor graph query).
**Called By**:
- F12 Route Chunk Manager (for internal queries)
- F07 Sequential VO (to determine chunk context for factor graph operations)
**Note**: This is a low-level method for factor graph operations. For high-level chunk queries, use F12.get_active_chunk(flight_id).
**Input**:
```python
frame_id: int
```
**Output**:
```python
Optional[ChunkHandle] # Active chunk or None if frame not in any chunk
```
**Test Cases**:
1. **Frame in active chunk**: Returns ChunkHandle
2. **Frame not in chunk**: Returns None
3. **Multiple chunks**: Returns correct chunk for frame
---
### `add_relative_factor_to_chunk(chunk_id: str, frame_i: int, frame_j: int, relative_pose: RelativePose, covariance: np.ndarray) -> bool`
**Description**: Adds relative pose measurement to a specific chunk's subgraph.
**Called By**:
- F07 Sequential VO (chunk-scoped operations)
**Input**:
```python
chunk_id: str
frame_i: int # Previous frame ID
frame_j: int # Current frame ID
relative_pose: RelativePose
covariance: np.ndarray # (6, 6)
```
**Output**:
```python
bool: True if factor added successfully
```
**Processing Flow**:
1. Verify chunk exists and is active
2. Create BetweenFactor in chunk's subgraph
3. Apply robust kernel (Huber)
4. Add to chunk's factor graph
5. Mark chunk as needing optimization
**Test Cases**:
1. **Add to active chunk**: Factor added successfully
2. **Add to inactive chunk**: Returns False
3. **Multiple chunks**: Factors isolated to respective chunks
---
### `add_chunk_anchor(chunk_id: str, frame_id: int, gps: GPSPoint, covariance: np.ndarray) -> bool`
**Description**: Adds absolute GPS anchor to a chunk, enabling global localization.
**Called By**:
- F09 Metric Refinement (after chunk LiteSAM matching)
- F11 Failure Recovery Coordinator (chunk matching)
**Input**:
```python
chunk_id: str
frame_id: int # Frame within chunk to anchor
gps: GPSPoint
covariance: np.ndarray # (2, 2) or (3, 3)
```
**Output**:
```python
bool: True if anchor added
```
**Processing Flow**:
1. Convert GPS to ENU coordinates
2. Create PriorFactor in chunk's subgraph
3. Mark chunk as anchored
4. Trigger chunk optimization
5. Enable chunk merging
**Test Cases**:
1. **Anchor unanchored chunk**: Anchor added, has_anchor=True
2. **Anchor already anchored chunk**: Updates anchor
3. **Chunk optimization**: Chunk optimized after anchor
---
### `merge_chunk_subgraphs(flight_id: str, new_chunk_id: str, main_chunk_id: str, transform: Sim3Transform) -> bool`
**Description**: Merges new_chunk INTO main_chunk using Sim(3) similarity transformation. Extends main_chunk with new_chunk's frames.
**Called By**:
- F12 Route Chunk Manager (via merge_chunks method)
**Input**:
```python
flight_id: str # Flight identifier
new_chunk_id: str # New chunk being merged (source, typically newer/recently anchored)
main_chunk_id: str # Main chunk being extended (destination, typically older/established)
transform: Sim3Transform:
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or quaternion
scale: float
```
**Output**:
```python
bool: True if merge successful
```
**Processing Flow**:
1. Verify both chunks exist and new_chunk is anchored
2. Apply Sim(3) transform to all poses in new_chunk
3. Merge new_chunk's subgraph into main_chunk's subgraph
4. Update frame-to-chunk mapping (new_chunk frames now belong to main_chunk)
5. Mark new_chunk subgraph as merged (F12 handles state updates)
6. Optimize merged graph globally
**Sim(3) Transformation**:
- Accounts for translation, rotation, and scale differences
- Critical for merging chunks with different scales (monocular VO)
- Preserves internal consistency of both chunks
**Test Cases**:
1. **Merge anchored chunks**: new_chunk merged into main_chunk successfully
2. **Merge unanchored chunk**: Returns False
3. **Global consistency**: Merged trajectory is globally consistent
---
### `get_chunk_trajectory(chunk_id: str) -> Dict[int, Pose]`
**Description**: Retrieves optimized trajectory for a specific chunk.
**Called By**:
- F12 Route Chunk Manager (chunk state queries)
- F14 Result Manager (result publishing)
**Input**:
```python
chunk_id: str
```
**Output**:
```python
Dict[int, Pose] # Frame ID -> Pose in chunk's local coordinate system
```
**Test Cases**:
1. **Get chunk trajectory**: Returns all poses in chunk
2. **Empty chunk**: Returns empty dict
3. **After optimization**: Returns optimized poses
---
### `get_all_chunks() -> List[ChunkHandle]`
**Description**: Retrieves all chunks in the factor graph.
**Called By**:
- F11 Failure Recovery Coordinator (chunk matching coordination)
- F12 Route Chunk Manager (chunk state queries)
**Input**: None
**Output**:
```python
List[ChunkHandle] # All chunks (active and inactive)
```
**Test Cases**:
1. **Multiple chunks**: Returns all chunks
2. **No chunks**: Returns empty list
3. **Mixed states**: Returns chunks in various states (active, anchored, merged)
---
### `optimize_chunk(chunk_id: str, iterations: int) -> OptimizationResult`
**Description**: Optimizes a specific chunk's subgraph independently.
**Called By**:
- F02 Flight Processor (after chunk anchor added)
- F12 Route Chunk Manager (periodic chunk optimization)
**Input**:
```python
chunk_id: str
iterations: int # Max iterations (typically 5-10 for incremental)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int]
mean_reprojection_error: float
```
**Processing Flow**:
1. Extract chunk's subgraph
2. Run Levenberg-Marquardt optimization
3. Update poses in chunk's local coordinate system
4. Return optimization result
**Test Cases**:
1. **Optimize active chunk**: Chunk optimized successfully
2. **Optimize anchored chunk**: Optimization improves consistency
3. **Chunk isolation**: Other chunks unaffected
---
### `optimize_global(iterations: int) -> OptimizationResult`
**Description**: Optimizes all chunks and performs global merging.
**Called By**:
- Background optimization task (periodic)
- F11 Failure Recovery Coordinator (after chunk matching)
**Input**:
```python
iterations: int # Max iterations (typically 50-100 for global)
```
**Output**:
```python
OptimizationResult:
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int] # All frames across all chunks
mean_reprojection_error: float
```
**Processing Flow**:
1. Collect all chunks
2. For anchored chunks, transform to global coordinate system
3. Optimize merged global graph
4. Update all chunk trajectories
5. Return global optimization result
**Test Cases**:
1. **Global optimization**: All chunks optimized together
2. **Multiple anchored chunks**: Global consistency achieved
3. **Performance**: Completes within acceptable time (<500ms)
## Integration Tests
### Test 1: Incremental Trajectory Building
1. Initialize graph with first frame
2. Add relative factors from VO × 100
3. Add altitude priors × 100
4. Optimize incrementally after each frame
5. Verify smooth trajectory
### Test 2: Drift Correction with Absolute GPS
1. Build trajectory with VO only (will drift)
2. Add absolute GPS factor at frame 50
3. Optimize → trajectory corrects
4. Verify frames 1-49 also corrected (back-propagation)
### Test 3: Outlier Handling
1. Add normal relative factors
2. Add 350m outlier factor (tilt error)
3. Optimize with robust kernel
4. Verify outlier downweighted, trajectory smooth
### Test 4: User Anchor Integration
1. Processing blocked at frame 237
2. User provides anchor (high confidence)
3. add_absolute_factor(is_user_anchor=True)
4. Optimize → trajectory snaps to anchor
### Test 5: Multi-Chunk Creation and Isolation
1. Create chunk_1 with frames 1-10
2. Create chunk_2 with frames 20-30 (disconnected)
3. Add relative factors to each chunk
4. Verify chunks optimized independently
5. Verify factors isolated to respective chunks
### Test 6: Chunk Anchoring and Merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Add chunk_anchor to chunk_2 (frame 25)
3. Optimize chunk_2 → local consistency improved
4. Merge chunk_2 into chunk_1 with Sim(3) transform
5. Optimize global → both chunks globally consistent
6. Verify final trajectory coherent across chunks
### Test 7: Simultaneous Multi-Chunk Processing
1. Create 3 chunks simultaneously (disconnected segments)
2. Process frames in each chunk independently
3. Anchor each chunk asynchronously
4. Merge chunks as anchors become available
5. Verify final global trajectory consistent
## Non-Functional Requirements
### Performance
- **Incremental optimize**: < 100ms per frame (iSAM2)
- **Batch optimize**: < 500ms for 100 frames
- **get_trajectory**: < 10ms
- Real-time capable: 10 FPS processing
### Accuracy
- **Mean Reprojection Error (MRE)**: < 1.0 pixels
- **GPS accuracy**: Meet 80% < 50m, 60% < 20m criteria
- **Trajectory smoothness**: No sudden jumps (except user anchors)
### Reliability
- Numerical stability for 2000+ frame trajectories
- Graceful handling of degenerate configurations
- Robust to missing/corrupted measurements
## Dependencies
### Internal Components
- **H03 Robust Kernels**: For Huber/Cauchy loss functions
- **H02 GSD Calculator**: For GSD computation and scale resolution
### External Dependencies
- **GTSAM**: Graph optimization library
- **numpy**: Matrix operations
- **scipy**: Sparse matrix operations (optional)
## Data Models
### Pose
```python
class Pose(BaseModel):
frame_id: int
position: np.ndarray # (3,) - [x, y, z] in ENU
orientation: np.ndarray # (4,) quaternion or (3,3) rotation matrix
timestamp: datetime
covariance: Optional[np.ndarray] # (6, 6)
```
### RelativePose
```python
class RelativePose(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or (4,)
covariance: np.ndarray # (6, 6)
```
### OptimizationResult
```python
class OptimizationResult(BaseModel):
converged: bool
final_error: float
iterations_used: int
optimized_frames: List[int]
mean_reprojection_error: float
```
### FactorGraphConfig
```python
class FactorGraphConfig(BaseModel):
robust_kernel_type: str = "Huber" # or "Cauchy"
huber_threshold: float = 1.0 # pixels
cauchy_k: float = 0.1
isam2_relinearize_threshold: float = 0.1
isam2_relinearize_skip: int = 1
max_chunks: int = 100 # Maximum number of simultaneous chunks
chunk_merge_threshold: float = 0.1 # Error threshold for chunk merging
```
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/11_failure_recovery_coordinator/11.01_feature_confidence_assessment.md
+45
View File
@@ -0,0 +1,45 @@
# Feature: Confidence Assessment
## Description
Assesses tracking confidence from VO and LiteSAM results and determines when tracking is lost. This is the foundation for all recovery decisions - every frame's results pass through confidence assessment to detect degradation.
## Component APIs Implemented
- `check_confidence(vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment`
- `detect_tracking_loss(confidence: ConfidenceAssessment) -> bool`
## External Tools and Services
None - pure computation based on input metrics.
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_compute_vo_confidence(vo_result)` | Calculates confidence from VO inlier count and ratio |
| `_compute_litesam_confidence(litesam_result)` | Calculates confidence from LiteSAM match score |
| `_compute_overall_confidence(vo_conf, litesam_conf)` | Weighted combination of confidence scores |
| `_determine_tracking_status(overall_conf, inlier_count)` | Maps confidence to "good"/"degraded"/"lost" |
## Thresholds
- **Good**: VO inliers > 50, LiteSAM confidence > 0.7
- **Degraded**: VO inliers 20-50
- **Lost**: VO inliers < 20
## Unit Tests
1. **Good tracking metrics** → Returns "good" status with high confidence
2. **Low VO inliers (25)** → Returns "degraded" status
3. **Very low VO inliers (10)** → Returns "lost" status
4. **High VO, low LiteSAM** → Returns appropriate degraded assessment
5. **LiteSAM result None** → Confidence based on VO only
6. **Edge case at threshold boundary (20 inliers)** → Correct status determination
## Integration Tests
1. **Normal flight sequence** → Confidence remains "good" throughout
2. **Low overlap frame** → Confidence drops to "degraded", recovers
3. **Sharp turn sequence** → Confidence drops to "lost", triggers recovery flow
_docs/02_components/11_failure_recovery_coordinator/11.02_feature_progressive_search.md
+59
View File
@@ -0,0 +1,59 @@
# Feature: Progressive Search
## Description
Coordinates progressive tile search when tracking is lost. Expands search grid from 1→4→9→16→25 tiles, trying LiteSAM matching at each level. Manages search session state and tracks progress.
## Component APIs Implemented
- `start_search(flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession`
- `expand_search_radius(session: SearchSession) -> List[TileCoords]`
- `try_current_grid(session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]`
- `mark_found(session: SearchSession, result: AlignmentResult) -> bool`
- `get_search_status(session: SearchSession) -> SearchStatus`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F04 Satellite Data Manager | `expand_search_grid()`, `compute_tile_bounds()` |
| F09 Metric Refinement | `align_to_satellite()` for tile matching |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_create_search_session(flight_id, frame_id, gps)` | Initializes SearchSession with grid_size=1 |
| `_get_next_grid_size(current_size)` | Returns next size in sequence: 1→4→9→16→25 |
| `_compute_tile_sequence(session)` | Returns tiles for current grid around center GPS |
| `_try_single_tile_match(uav_image, tile, tile_bounds)` | Wraps F09 alignment call |
| `_is_match_acceptable(result, threshold)` | Validates alignment confidence meets threshold |
## Search Grid Progression
| Level | Grid Size | Tiles | New Tiles Added |
|-------|-----------|-------|-----------------|
| 1 | 1×1 | 1 | 1 |
| 2 | 2×2 | 4 | 3 |
| 3 | 3×3 | 9 | 5 |
| 4 | 4×4 | 16 | 7 |
| 5 | 5×5 | 25 | 9 |
## Unit Tests
1. **start_search** → Creates session with grid_size=1, correct center GPS
2. **expand_search_radius from 1** → Returns 3 new tiles, grid_size becomes 4
3. **expand_search_radius from 4** → Returns 5 new tiles, grid_size becomes 9
4. **expand_search_radius at max (25)** → Returns empty list, exhausted=true
5. **try_current_grid match found** → Returns AlignmentResult, session.found=true
6. **try_current_grid no match** → Returns None
7. **mark_found** → Sets session.found=true
8. **get_search_status** → Returns correct current_grid_size, found, exhausted flags
## Integration Tests
1. **Match on first grid** → start_search → try_current_grid → match found immediately
2. **Match on 3rd expansion** → Search through 1→4→9 tiles, match found at grid_size=9
3. **Full exhaustion** → All 25 tiles searched, no match, exhausted=true
4. **Concurrent searches** → Multiple flight search sessions operate independently
_docs/02_components/11_failure_recovery_coordinator/11.03_feature_user_input_handling.md
+47
View File
@@ -0,0 +1,47 @@
# Feature: User Input Handling
## Description
Handles human-in-the-loop recovery when all automated strategies are exhausted. Creates user input requests with candidate tiles and applies user-provided GPS anchors to the factor graph.
## Component APIs Implemented
- `create_user_input_request(flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest`
- `apply_user_anchor(flight_id: str, frame_id: int, anchor: UserAnchor) -> bool`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F10 Factor Graph Optimizer | `add_absolute_factor()`, `optimize()` for anchor application |
| F08 Global Place Recognition | Candidate tiles (passed in, not called directly) |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_generate_request_id()` | Creates unique request ID |
| `_get_uav_image_for_frame(flight_id, frame_id)` | Retrieves UAV image for user display |
| `_validate_anchor(anchor)` | Validates anchor data (GPS bounds, pixel within image) |
| `_convert_anchor_to_factor(anchor, frame_id)` | Transforms UserAnchor to factor graph format |
## Request/Response Flow
1. **Request Creation**: F11 creates UserInputRequest → F02.2 sends via F15 → Client receives
2. **Anchor Application**: Client sends anchor → F01 API → F02.2 calls F11.apply_user_anchor()
## Unit Tests
1. **create_user_input_request** → Returns UserInputRequest with correct flight_id, frame_id, candidates
2. **create_user_input_request generates unique ID** → Multiple calls produce unique request_ids
3. **apply_user_anchor valid** → Returns true, calls F10.add_absolute_factor with is_user_anchor=true
4. **apply_user_anchor invalid GPS** → Returns false, no factor added
5. **apply_user_anchor invalid pixel coords** → Returns false, rejected
6. **apply_user_anchor triggers optimization** → F10.optimize() called after factor addition
## Integration Tests
1. **Full user input flow** → Search exhausted → create_user_input_request → user provides anchor → apply_user_anchor → processing resumes
2. **User anchor improves trajectory** → Before/after anchor application, trajectory accuracy improves
3. **Multiple user anchors** → Can apply anchors to different frames in sequence
_docs/02_components/11_failure_recovery_coordinator/11.04_feature_chunk_recovery_coordination.md
+72
View File
@@ -0,0 +1,72 @@
# Feature: Chunk Recovery Coordination
## Description
Coordinates chunk-based recovery when tracking is lost. Creates chunks proactively, orchestrates semantic matching via F08, LiteSAM matching with rotation sweeps via F06/F09, and merges recovered chunks into the main trajectory. Includes background processing of unanchored chunks.
## Component APIs Implemented
- `create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle`
- `try_chunk_semantic_matching(chunk_id: str) -> Optional[List[TileCandidate]]`
- `try_chunk_litesam_matching(chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]`
- `merge_chunk_to_trajectory(flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool`
- `process_unanchored_chunks(flight_id: str) -> None`
## External Tools and Services
| Component | Usage |
|-----------|-------|
| F12 Route Chunk Manager | `create_chunk()`, `get_chunk_images()`, `get_chunk_frames()`, `mark_chunk_anchored()`, `merge_chunks()`, `get_chunks_for_matching()`, `is_chunk_ready_for_matching()`, `mark_chunk_matching()` |
| F10 Factor Graph Optimizer | Chunk creation, anchor application (via F12) |
| F04 Satellite Data Manager | Tile retrieval for matching |
| F08 Global Place Recognition | `retrieve_candidate_tiles_for_chunk()` |
| F06 Image Rotation Manager | `try_chunk_rotation_steps()` for rotation sweeps |
| F09 Metric Refinement | `align_chunk_to_satellite()` (via F06) |
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_determine_merge_target(chunk_id)` | Finds temporal predecessor chunk for merging |
| `_compute_sim3_transform(alignment_result)` | Builds Sim3Transform from alignment |
| `_select_anchor_frame(chunk_id)` | Selects best frame in chunk for anchoring (middle or highest confidence) |
| `_handle_chunk_matching_failure(chunk_id)` | Decides retry vs user input request on failure |
## Rotation Sweep Strategy
- 12 rotation angles: 0°, 30°, 60°, ..., 330°
- Critical for chunks from sharp turns (unknown orientation)
- Returns best matching rotation angle in ChunkAlignmentResult
## Chunk Lifecycle States
| Status | Description |
|--------|-------------|
| `unanchored` | Chunk created, no GPS anchor |
| `matching` | Matching in progress |
| `anchored` | GPS anchor established |
| `merged` | Merged into main trajectory |
## Unit Tests
1. **create_chunk_on_tracking_loss** → Returns ChunkHandle with correct flight_id, start_frame_id, is_active=true
2. **try_chunk_semantic_matching success** → Returns List[TileCandidate] from F08
3. **try_chunk_semantic_matching no match** → Returns None
4. **try_chunk_litesam_matching match at 0°** → Returns ChunkAlignmentResult with rotation_angle=0
5. **try_chunk_litesam_matching match at 120°** → Returns ChunkAlignmentResult with rotation_angle=120
6. **try_chunk_litesam_matching no match** → Returns None after trying all rotations and tiles
7. **merge_chunk_to_trajectory** → Returns true, F12.merge_chunks called with correct parameters
8. **merge_chunk_to_trajectory determines correct merge target** → Merges to temporal predecessor
9. **_determine_merge_target no predecessor** → Returns "main" for first chunk
## Integration Tests
1. **Proactive chunk creation** → Tracking lost → chunk created immediately → processing continues in new chunk
2. **Chunk semantic matching** → Chunk with 10 frames → semantic matching finds candidates where single-frame fails
3. **Chunk LiteSAM with rotation** → Unknown orientation chunk → rotation sweep finds match at correct angle
4. **Full chunk recovery flow** → create_chunk → semantic matching → LiteSAM matching → merge
5. **Multiple chunk merging** → chunk_1 (frames 1-10), chunk_2 (frames 20-30) → both recovered and merged
6. **Background processing** → 3 unanchored chunks → process_unanchored_chunks matches and merges asynchronously
7. **Non-blocking processing** → Frame processing continues while chunk matching runs in background
8. **Chunk matching failure** → All candidates fail → user input requested for chunk
_docs/02_components/11_failure_recovery_coordinator/11._component_failure_recovery_coordinator.md
+784
View File
@@ -0,0 +1,784 @@
# Failure Recovery Coordinator
## Interface Definition
**Interface Name**: `IFailureRecoveryCoordinator`
### Interface Methods
```python
class IFailureRecoveryCoordinator(ABC):
# Status Checks
@abstractmethod
def check_confidence(self, vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment:
pass
@abstractmethod
def detect_tracking_loss(self, confidence: ConfidenceAssessment) -> bool:
pass
# Search & Recovery (Synchronous returns, NO EVENTS)
@abstractmethod
def start_search(self, flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession:
pass
@abstractmethod
def expand_search_radius(self, session: SearchSession) -> List[TileCoords]:
pass
@abstractmethod
def try_current_grid(self, session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]:
pass
@abstractmethod
def mark_found(self, session: SearchSession, result: AlignmentResult) -> bool:
pass
@abstractmethod
def get_search_status(self, session: SearchSession) -> SearchStatus:
pass
@abstractmethod
def create_user_input_request(self, flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest:
"""Returns request object. Caller (F02.2) is responsible for sending to F15."""
pass
@abstractmethod
def apply_user_anchor(self, flight_id: str, frame_id: int, anchor: UserAnchor) -> bool:
pass
# Chunk Recovery
@abstractmethod
def create_chunk_on_tracking_loss(self, flight_id: str, frame_id: int) -> ChunkHandle:
pass
@abstractmethod
def try_chunk_semantic_matching(self, chunk_id: str) -> Optional[List[TileCandidate]]:
pass
@abstractmethod
def try_chunk_litesam_matching(self, chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]:
pass
@abstractmethod
def merge_chunk_to_trajectory(self, flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool:
pass
@abstractmethod
def process_unanchored_chunks(self, flight_id: str) -> None:
"""Background task logic. Should be invoked/managed by F02.2."""
pass
```
## Component Description
### Responsibilities
- **Pure Logic Component**: Coordinates recovery strategies but delegates execution and communication to F02.2.
- **No Events**: Returns status objects or booleans. F02.2 decides state transitions based on these returns.
- **Chunk Orchestration**: Coordinates F12 and F10 operations during recovery but does not own the thread/task.
- Monitor confidence metrics (inlier count, MRE, covariance)
- Detect tracking loss and trigger recovery
- Coordinate progressive tile search (1→4→9→16→25)
- Handle human-in-the-loop when all strategies exhausted
- Apply user-provided anchors to Factor Graph
- **Proactive chunk creation on tracking loss** (via F12)
- **Chunk LiteSAM matching with rotation sweeps**
- **Chunk merging orchestration**
- **Background chunk matching processing**
### Interaction Pattern (Direct Calls)
**Caller**: F02.2 Flight Processing Engine
1. **Tracking Loss**:
* F02.2 calls `start_search()`.
* F02.2 calls `try_current_grid()` iteratively.
* If found, F02.2 calls `mark_found()`.
* If exhausted, F02.2 calls `create_user_input_request()` -> gets `Request` object -> F02.2 calls F15.
2. **Chunk Matching**:
* F02.2 calls `create_chunk_on_tracking_loss()`.
* F02.2 (background task) calls `try_chunk_semantic_matching()`.
* F02.2 calls `try_chunk_litesam_matching()`.
* F02.2 calls `merge_chunk_to_trajectory()`.
### External SSE Events (to Clients)
F11 does NOT directly send events to clients. F02.2 orchestrates client communication based on F11's return values:
- When F11.create_user_input_request() returns `UserInputRequest` → F02.2 calls F15.send_user_input_request()
- When F11.merge_chunk_to_trajectory() returns True → F02.2 calls F14.update_results_after_chunk_merge()
- When recovery fails (all F11 methods return None/False) → F02.2 calls F15.send_processing_blocked()
This separation ensures:
1. F11 is a pure business logic component (no I/O dependencies)
2. F02.2 controls all coordination and state management
3. F14/F15 handle all client-facing communication
### Scope
- Confidence monitoring
- Progressive search coordination
- User input request/response handling
- Recovery strategy orchestration
- Integration point for F04, F06, F08, F09, F10
- **Chunk lifecycle and matching coordination**
- **Multi-chunk simultaneous processing**
### Architecture Note: Single Responsibility Consideration
F11 has extensive responsibilities including progressive search, chunk creation coordination, chunk matching coordination (semantic + LiteSAM), chunk merging coordination, and user input handling. While this represents significant scope, these remain together in a single component because:
1. **Tight Coupling**: All recovery strategies share state (SearchSession, chunk status) and need coordinated fallback logic (progressive search → chunk matching → user input).
2. **Sequential Dependency**: Chunk semantic matching feeds into chunk LiteSAM matching, which feeds into chunk merging. Splitting these would create circular dependencies or complex event choreography.
3. **Pure Logic Pattern**: F11 remains a pure logic component (no I/O, no threads). All execution is delegated to F02.2, keeping F11 focused on decision-making rather than coordination.
4. **Single Point of Recovery Policy**: Having all recovery strategies in one component makes the fallback logic explicit and maintainable. The alternative—multiple coordinators—would require a meta-coordinator.
**Future Consideration**: If F11 grows beyond ~1000 lines or if specific recovery strategies need independent evolution, consider splitting into:
- `F11a_search_coordinator` - Progressive search, user input handling
- `F11b_chunk_recovery_coordinator` - Chunk semantic matching, LiteSAM matching, merging
For now, the component remains unified with clear internal organization (search methods, chunk methods).
## API Methods
### `check_confidence(vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment`
**Description**: Assesses tracking confidence from VO and LiteSAM results.
**Called By**: Main processing loop (per frame)
**Input**:
```python
vo_result: RelativePose
litesam_result: Optional[AlignmentResult]
```
**Output**:
```python
ConfidenceAssessment:
overall_confidence: float # 0-1
vo_confidence: float
litesam_confidence: float
inlier_count: int
tracking_status: str # "good", "degraded", "lost"
```
**Confidence Metrics**:
- VO inlier count and ratio
- LiteSAM match confidence
- Factor graph marginal covariance
- Reprojection error
**Thresholds**:
- **Good**: VO inliers > 50, LiteSAM confidence > 0.7
- **Degraded**: VO inliers 20-50
- **Lost**: VO inliers < 20
**Test Cases**:
1. Good tracking → "good" status
2. Low overlap → "degraded"
3. Sharp turn → "lost"
---
### `detect_tracking_loss(confidence: ConfidenceAssessment) -> bool`
**Description**: Determines if tracking is lost.
**Called By**: Main processing loop
**Input**: `ConfidenceAssessment`
**Output**: `bool` - True if tracking lost
**Test Cases**:
1. Confidence good → False
2. Confidence lost → True
---
### `start_search(flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession`
**Description**: Initiates progressive search session.
**Called By**: Main processing loop (when tracking lost)
**Input**:
```python
flight_id: str
frame_id: int
estimated_gps: GPSPoint # Dead-reckoning estimate
```
**Output**:
```python
SearchSession:
session_id: str
flight_id: str
frame_id: int
center_gps: GPSPoint
current_grid_size: int # Starts at 1
max_grid_size: int # 25
found: bool
```
**Processing Flow**:
1. Create search session
2. Set center from estimated_gps
3. Set current_grid_size = 1
4. Return session
**Test Cases**:
1. Start search → session created with grid_size=1
---
### `expand_search_radius(session: SearchSession) -> List[TileCoords]`
**Description**: Expands search grid to next size (1→4→9→16→25).
**Called By**: Internal (after try_current_grid fails)
**Input**: `SearchSession`
**Output**: `List[TileCoords]` - Tiles for next grid size
**Processing Flow**:
1. Increment current_grid_size (1→4→9→16→25)
2. Call F04.expand_search_grid() to get new tiles only
3. Return new tile coordinates
**Test Cases**:
1. Expand 1→4 → returns 3 new tiles
2. Expand 4→9 → returns 5 new tiles
3. At grid_size=25 → no more expansion
---
### `try_current_grid(session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]`
**Description**: Tries LiteSAM matching on current tile grid.
**Called By**: Internal (progressive search loop)
**Input**:
```python
session: SearchSession
tiles: Dict[str, np.ndarray] # From F04
```
**Output**: `Optional[AlignmentResult]` - Match result or None
**Processing Flow**:
1. Get UAV image for frame_id
2. For each tile in grid:
- Get tile_bounds via F04.compute_tile_bounds(tile_coords)
- Call F09.align_to_satellite(uav_image, tile, tile_bounds)
- If match found with confidence > threshold:
- mark_found(session, result)
- Return result
3. Return None if no match
**Test Cases**:
1. Match on 3rd tile → returns result
2. No match in grid → returns None
---
### `mark_found(session: SearchSession, result: AlignmentResult) -> bool`
**Description**: Marks search session as successful.
**Called By**: Internal
**Input**:
```python
session: SearchSession
result: AlignmentResult
```
**Output**: `bool` - True
**Processing Flow**:
1. Set session.found = True
2. Log success (grid_size where found)
3. Resume processing
---
### `get_search_status(session: SearchSession) -> SearchStatus`
**Description**: Gets current search status.
**Called By**: F01 REST API (for status endpoint)
**Output**:
```python
SearchStatus:
current_grid_size: int
found: bool
exhausted: bool # Reached grid_size=25 without match
```
---
### `create_user_input_request(flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest`
**Description**: Creates user input request when all search strategies exhausted.
**Called By**: Internal (when grid_size=25 and no match)
**Input**:
```python
flight_id: str
frame_id: int
candidate_tiles: List[TileCandidate] # Top-5 from G08
```
**Output**:
```python
UserInputRequest:
request_id: str
flight_id: str
frame_id: int
uav_image: np.ndarray
candidate_tiles: List[TileCandidate]
message: str
```
**Processing Flow**:
1. Get UAV image for frame_id
2. Get top-5 candidates from F08
3. Create request
4. Return Request object (Caller F02.2 handles sending)
**Test Cases**:
1. All search failed → creates request
2. Request returned to caller
---
### `apply_user_anchor(flight_id: str, frame_id: int, anchor: UserAnchor) -> bool`
**Description**: Applies user-provided GPS anchor.
**Called By**: F01 REST API (user-fix endpoint)
**Input**:
```python
flight_id: str
frame_id: int
anchor: UserAnchor:
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**: `bool` - True if applied
**Processing Flow**:
1. Validate anchor data
2. Call F10.add_absolute_factor(frame_id, gps, is_user_anchor=True)
3. F10.optimize() → refines trajectory
4. Return True (Caller F02.2 updates status)
**Test Cases**:
1. Valid anchor → applied, processing resumes
2. Invalid anchor → rejected
---
### `create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle`
**Description**: Creates a new chunk proactively when tracking is lost.
**Called By**:
- F02.2 Flight Processing Engine (when tracking lost detected)
**Input**:
```python
flight_id: str
frame_id: int # First frame in new chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str
```
**Processing Flow**:
1. Call F12 Route Chunk Manager.create_chunk()
2. F12 creates chunk in F10 Factor Graph Optimizer
3. Mark chunk as active
4. Return ChunkHandle
**Proactive Behavior**:
- Chunk created immediately (not waiting for matching to fail)
- Processing continues in new chunk
- Matching attempted asynchronously
**Test Cases**:
1. **Create chunk**: Chunk created successfully
2. **Proactive creation**: Chunk created before matching attempts
3. **Continue processing**: Processing continues in new chunk
---
### `try_chunk_semantic_matching(chunk_id: str) -> Optional[List[TileCandidate]]`
**Description**: Attempts semantic matching for a chunk using aggregate DINOv2 descriptor.
**Called By**:
- Internal (when chunk ready for matching)
- process_unanchored_chunks() (background task)
**Input**:
```python
chunk_id: str
```
**Output**:
```python
Optional[List[TileCandidate]]: Top-k candidate tiles or None if matching failed
```
**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. Call F08 Global Place Recognition.retrieve_candidate_tiles_for_chunk()
3. F08 computes chunk descriptor and queries Faiss
4. Return candidate tiles if found, None otherwise
**Test Cases**:
1. **Chunk matching**: Returns candidate tiles
2. **Featureless terrain**: Succeeds where single-image fails
3. **No match**: Returns None
---
### `try_chunk_litesam_matching(chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]`
**Description**: Attempts LiteSAM matching for chunk with rotation sweeps.
**Called By**:
- Internal (after chunk semantic matching succeeds)
- process_unanchored_chunks() (background task)
**Input**:
```python
chunk_id: str
candidate_tiles: List[TileCandidate] # From chunk semantic matching
```
**Output**:
```python
Optional[ChunkAlignmentResult]: Match result or None
```
**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. For each candidate tile:
- Get tile from F04 Satellite Data Manager
- Call F06.try_chunk_rotation_steps() (12 rotations: 0°, 30°, ..., 330°)
- F06 calls F09.align_chunk_to_satellite() for each rotation
- If match found with confidence > threshold:
- Return ChunkAlignmentResult
3. Return None if no match found
**Rotation Sweeps**:
- Critical for chunks from sharp turns (unknown orientation)
- Tries all 12 rotation angles
- Returns best matching rotation
**Test Cases**:
1. **Match on first tile**: Returns alignment result
2. **Match on 3rd tile**: Returns alignment result
3. **Match at 120° rotation**: Returns result with correct rotation angle
4. **No match**: Returns None
---
### `merge_chunk_to_trajectory(flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool`
**Description**: Merges chunk into main trajectory after successful matching.
**Called By**:
- Internal (after chunk LiteSAM matching succeeds)
- process_unanchored_chunks() (which has flight_id from chunk handle)
**Input**:
```python
flight_id: str # Flight identifier (available from chunk handle)
chunk_id: str
alignment_result: ChunkAlignmentResult:
chunk_center_gps: GPSPoint
transform: Sim3Transform
confidence: float
```
**Output**:
```python
bool: True if merge successful
```
**Processing Flow**:
1. Get chunk frames via F12.get_chunk_frames(chunk_id) → merged_frames
2. Get chunk anchor frame (middle frame or best frame)
3. Call F12.mark_chunk_anchored() with GPS (F12 coordinates with F10)
4. **Determine merge target**:
- Main chunk is typically the temporal predecessor (previous chunk by frame_id order)
- If no predecessor: merge to main trajectory (main_chunk_id="main")
- F11 determines main_chunk based on chunk frame_id ordering
5. Call F12.merge_chunks(main_chunk_id, chunk_id, transform)
- Note: `merge_chunks(main_chunk, new_chunk)` merges new_chunk INTO main_chunk
- chunk_id (source) is merged into target_chunk_id
6. F12 handles chunk state updates (deactivation, status updates)
7. F10 optimizes merged graph globally (via F12.merge_chunks())
8. Return True (Caller F02.2 coordinates result updates)
**Sim(3) Transform**:
- Translation: GPS offset
- Rotation: From alignment result
- Scale: Resolved from altitude and GSD
**Test Cases**:
1. **Merge successful**: Chunks merged successfully
2. **Global consistency**: Merged trajectory globally consistent
3. **Multiple chunks**: Can merge multiple chunks sequentially
---
### `process_unanchored_chunks(flight_id: str) -> None`
**Description**: Background task that periodically attempts matching for unanchored chunks.
**Called By**:
- Background thread (periodic, e.g., every 5 seconds)
- F02.2 Flight Processing Engine (after frame processing)
**Input**:
```python
flight_id: str
```
**Output**: None (runs asynchronously)
**Processing Flow**:
```
while flight_active:
unanchored_chunks = F12.get_chunks_for_matching(flight_id)
for chunk in unanchored_chunks:
if F12.is_chunk_ready_for_matching(chunk.chunk_id):
F12.mark_chunk_matching(chunk.chunk_id)
candidates = try_chunk_semantic_matching(chunk.chunk_id)
if candidates:
alignment = try_chunk_litesam_matching(chunk.chunk_id, candidates)
if alignment:
merge_chunk_to_trajectory(chunk.flight_id, chunk.chunk_id, alignment)
sleep(5 seconds)
```
**Background Processing**:
- **Trigger**: Started by F02.2 Flight Processing Engine after flight creation
- Runs asynchronously, doesn't block frame processing
- Periodically checks for ready chunks (every 5 seconds)
- Attempts matching and merging
- Reduces user input requests
- **Lifecycle**: Starts when flight becomes active, stops when flight completed
**Chunk Failure Flow**:
When chunk matching fails after trying all candidates:
1. Emit `ChunkMatchingFailed` event with chunk_id and flight_id (to F02.2 internal queue)
2. F02.2 receives event and decides next action:
- **Option A**: Wait for more frames and retry later (chunk may gain more distinctive features)
- **Option B**: Create user input request for the chunk
3. If user input requested:
- F02.2 calls F15.send_user_input_request() with chunk context
- Client receives via SSE
- User provides GPS anchor for one frame in chunk
- F02.2 receives user fix via handle_user_fix()
- F11.apply_user_anchor() anchors the chunk
- Processing resumes
**Test Cases**:
1. **Background matching**: Unanchored chunks matched asynchronously
2. **Chunk merging**: Chunks merged when matches found
3. **Non-blocking**: Frame processing continues during matching
4. **Chunk matching fails**: Failure handled, user input requested if needed
## Integration Tests
### Test 1: Progressive Search Flow
1. Tracking lost detected
2. start_search() → grid_size=1
3. try_current_grid(1 tile) → no match
4. expand_search_radius() → grid_size=4
5. try_current_grid(4 tiles) → match found
6. mark_found() → success
### Test 2: Full Search Exhaustion
1. start_search()
2. try grids: 1→4→9→16→25, all fail
3. create_user_input_request()
4. User provides anchor
5. apply_user_anchor() → processing resumes
### Test 3: Confidence Monitoring
1. Normal frames → confidence good
2. Low overlap frame → confidence degraded
3. Sharp turn → tracking lost, trigger search
### Test 4: Proactive Chunk Creation
1. Tracking lost detected
2. create_chunk_on_tracking_loss() → chunk created immediately
3. Processing continues in new chunk
4. Verify chunk matching attempted asynchronously
### Test 5: Chunk Semantic Matching
1. Build chunk with 10 images (plain field)
2. try_chunk_semantic_matching() → returns candidate tiles
3. Verify chunk matching succeeds where single-image fails
### Test 6: Chunk LiteSAM Matching with Rotation
1. Build chunk with unknown orientation
2. try_chunk_litesam_matching() with candidate tiles
3. Rotation sweep finds match at 120°
4. Returns ChunkAlignmentResult with correct GPS
### Test 7: Chunk Merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Anchor chunk_2 via chunk matching
3. merge_chunk_to_trajectory() → chunks merged
4. Verify global trajectory consistent
### Test 8: Background Chunk Processing
1. Create 3 unanchored chunks
2. process_unanchored_chunks() runs in background
3. Chunks matched and merged asynchronously
4. Frame processing continues uninterrupted
## Non-Functional Requirements
### Performance
- **check_confidence**: < 10ms
- **Progressive search (25 tiles)**: < 1.5s total
- **User input latency**: < 500ms from creation to SSE event
### Reliability
- Always exhausts all search strategies before requesting user input
- Guarantees processing block when awaiting user input
- Graceful recovery from all failure modes
## Dependencies
### Internal Components
- **F12 Route Chunk Manager**: Chunk operations.
- **F10 Factor Graph Optimizer**: Anchor application.
- **F04 Satellite Data Manager**: Search grids.
- **F08 Global Place Recognition**: Candidate retrieval.
- **F09 Metric Refinement**: Alignment.
- **F06 Image Rotation Manager**: Rotation sweeps.
### External Dependencies
- None
## Data Models
### ConfidenceAssessment
```python
class ConfidenceAssessment(BaseModel):
overall_confidence: float
vo_confidence: float
litesam_confidence: float
inlier_count: int
tracking_status: str
```
### SearchSession
```python
class SearchSession(BaseModel):
session_id: str
flight_id: str
frame_id: int
center_gps: GPSPoint
current_grid_size: int
max_grid_size: int
found: bool
exhausted: bool
```
### UserInputRequest
```python
class UserInputRequest(BaseModel):
request_id: str
flight_id: str
frame_id: int
uav_image: np.ndarray
candidate_tiles: List[TileCandidate]
message: str
created_at: datetime
```
### UserAnchor
```python
class UserAnchor(BaseModel):
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
confidence: float = 1.0
```
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### ChunkAlignmentResult
```python
class ChunkAlignmentResult(BaseModel):
matched: bool
chunk_id: str
chunk_center_gps: GPSPoint
rotation_angle: float
confidence: float
inlier_count: int
transform: Sim3Transform
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or (4,) quaternion
scale: float
```
### RecoveryStatus (Returned by recovery methods)
```python
class RecoveryStatus(BaseModel):
"""Status returned by F11 methods, used by F02 to determine next action."""
success: bool
method: str # "rotation_sweep", "progressive_search", "chunk_matching", "user_input"
gps: Optional[GPSPoint]
chunk_id: Optional[str]
message: Optional[str]
```
_docs/02_components/12_route_chunk_manager/12.01_feature_chunk_lifecycle_management.md
+123
View File
@@ -0,0 +1,123 @@
# Feature: Chunk Lifecycle Management
## Name
Chunk Lifecycle Management
## Description
Core operations for creating, growing, and managing the lifecycle of route chunks. Chunks are first-class entities in the Atlas multi-map architecture, created proactively on tracking loss to continue processing without failure. This feature handles the fundamental CRUD-like operations for chunk management and maintains internal state tracking.
## Component APIs Implemented
### `create_chunk(flight_id: str, start_frame_id: int) -> ChunkHandle`
Creates a new route chunk when tracking is lost or proactively by F11.
- Generates unique chunk_id
- Calls F10.create_new_chunk() to initialize subgraph
- Initializes chunk state (unanchored, active)
- Stores ChunkHandle in internal dictionary
### `add_frame_to_chunk(chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool`
Adds a frame to an active chunk during frame processing.
- Verifies chunk exists and is active
- Appends frame_id to chunk's frames list
- Calls F10.add_relative_factor_to_chunk()
- Updates end_frame_id
### `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]`
Retrieves the currently active chunk for a flight.
- Queries internal state dictionary
- Returns ChunkHandle with is_active=True or None
### `deactivate_chunk(chunk_id: str) -> bool`
Deactivates a chunk after merging or completion.
- Sets is_active=False on ChunkHandle
- Does not delete chunk data
## External Tools and Services
- **F10 Factor Graph Optimizer**: `create_new_chunk()`, `add_relative_factor_to_chunk()`
## Internal Methods
### `_generate_chunk_id() -> str`
Generates unique chunk identifier (UUID or sequential).
### `_validate_chunk_active(chunk_id: str) -> bool`
Checks if chunk exists and is_active=True.
### `_get_chunk_by_id(chunk_id: str) -> Optional[ChunkHandle]`
Internal lookup in _chunks dictionary.
### `_update_chunk_end_frame(chunk_id: str, frame_id: int)`
Updates end_frame_id after adding frame.
## Unit Tests
### Test: create_chunk_returns_active_handle
- Call create_chunk(flight_id, start_frame_id)
- Assert returned ChunkHandle has is_active=True
- Assert chunk_id is non-empty string
- Assert start_frame_id matches input
### Test: create_chunk_calls_f10
- Call create_chunk()
- Assert F10.create_new_chunk() was called with correct parameters
### Test: create_multiple_chunks_same_flight
- Create chunk_1 and chunk_2 for same flight
- Assert both chunks exist with different chunk_ids
### Test: add_frame_to_active_chunk
- Create chunk
- Call add_frame_to_chunk() with valid frame
- Assert returns True
- Assert frame_id in chunk.frames
### Test: add_frame_updates_end_frame_id
- Create chunk with start_frame_id=10
- Add frames 11, 12, 13
- Assert end_frame_id=13
### Test: add_frame_to_inactive_chunk_fails
- Create chunk, then deactivate_chunk()
- Call add_frame_to_chunk()
- Assert returns False
### Test: add_frame_to_nonexistent_chunk_fails
- Call add_frame_to_chunk("invalid_id", ...)
- Assert returns False
### Test: get_active_chunk_returns_correct_chunk
- Create chunk for flight_1
- Assert get_active_chunk(flight_1) returns chunk
- Assert get_active_chunk(flight_2) returns None
### Test: get_active_chunk_none_when_deactivated
- Create chunk, then deactivate
- Assert get_active_chunk() returns None
### Test: deactivate_chunk_success
- Create chunk
- Call deactivate_chunk()
- Assert returns True
- Assert chunk.is_active=False
### Test: deactivate_nonexistent_chunk_fails
- Call deactivate_chunk("invalid_id")
- Assert returns False
## Integration Tests
### Test: chunk_lifecycle_flow
1. create_chunk() → verify ChunkHandle
2. add_frame_to_chunk() × 10 → verify frames list grows
3. get_active_chunk() → returns the chunk
4. deactivate_chunk() → chunk deactivated
5. get_active_chunk() → returns None
### Test: multiple_chunks_isolation
1. Create chunk_1 (flight_A)
2. Create chunk_2 (flight_A)
3. Add frames to chunk_1
4. Add frames to chunk_2
5. Verify frames lists are independent
6. Verify only one can be active at a time per flight
_docs/02_components/12_route_chunk_manager/12.02_feature_chunk_data_retrieval.md
+103
View File
@@ -0,0 +1,103 @@
# Feature: Chunk Data Retrieval
## Name
Chunk Data Retrieval
## Description
Query operations for retrieving chunk data including frame lists, images, estimated bounds, and composite descriptors. These methods provide data to other components (F08, F09, F11) for matching and recovery operations. The feature handles data aggregation and transformation without modifying chunk state.
## Component APIs Implemented
### `get_chunk_frames(chunk_id: str) -> List[int]`
Retrieves ordered list of frame IDs in a chunk.
- Returns frames list from ChunkHandle
- Ordered by sequence (insertion order)
### `get_chunk_images(chunk_id: str) -> List[np.ndarray]`
Retrieves images for all frames in a chunk.
- Calls F05.get_image_by_sequence() for each frame
- Returns images in frame order
### `get_chunk_bounds(chunk_id: str) -> ChunkBounds`
Estimates GPS bounds of a chunk based on VO trajectory.
- Computes estimated center from relative poses
- Calculates radius from trajectory extent
- Returns confidence based on anchor status
### `get_chunk_composite_descriptor(chunk_id: str) -> np.ndarray`
Computes aggregate DINOv2 descriptor for semantic matching.
- Retrieves chunk images
- Calls F08.compute_chunk_descriptor() with aggregation strategy
- Returns 4096-dim or 8192-dim vector
## External Tools and Services
- **F05 Image Input Pipeline**: `get_image_by_sequence()`
- **F08 Global Place Recognition**: `compute_chunk_descriptor()`
## Internal Methods
### `_compute_trajectory_extent(chunk_id: str) -> Tuple[float, float]`
Calculates trajectory spread from VO poses for bounds estimation.
### `_estimate_center_from_poses(chunk_id: str) -> GPSPoint`
Estimates center GPS from accumulated relative poses and last known anchor.
### `_calculate_bounds_confidence(chunk_id: str) -> float`
Returns confidence (0.0-1.0) based on anchor status and trajectory length.
## Unit Tests
### Test: get_chunk_frames_returns_ordered_list
- Create chunk, add frames 10, 11, 12
- Assert get_chunk_frames() returns [10, 11, 12]
### Test: get_chunk_frames_empty_chunk
- Create chunk without adding frames
- Assert get_chunk_frames() returns [start_frame_id] only
### Test: get_chunk_frames_nonexistent_chunk
- Call get_chunk_frames("invalid_id")
- Assert returns empty list or raises exception
### Test: get_chunk_images_returns_correct_count
- Create chunk with 5 frames
- Mock F05.get_image_by_sequence()
- Assert get_chunk_images() returns 5 images
### Test: get_chunk_images_preserves_order
- Create chunk with frames [10, 11, 12]
- Assert images returned in same order as frames
### Test: get_chunk_bounds_unanchored
- Create unanchored chunk with 10 frames
- Assert get_chunk_bounds().confidence < 0.5
- Assert estimated_radius > 0
### Test: get_chunk_bounds_anchored
- Create chunk, mark as anchored
- Assert get_chunk_bounds().confidence > 0.7
### Test: get_chunk_composite_descriptor_shape
- Create chunk with 10 frames
- Mock F08.compute_chunk_descriptor()
- Assert descriptor has expected dimensions (4096 or 8192)
### Test: get_chunk_composite_descriptor_calls_f08
- Create chunk
- Call get_chunk_composite_descriptor()
- Assert F08.compute_chunk_descriptor() called with chunk images
## Integration Tests
### Test: chunk_descriptor_computation
1. Create chunk with 10 frames
2. get_chunk_images() → 10 images
3. get_chunk_composite_descriptor() → aggregated descriptor
4. Verify descriptor shape and non-zero values
### Test: chunk_bounds_accuracy
1. Create chunk with known anchor
2. Add 10 frames with VO results
3. get_chunk_bounds() → verify center near anchor
4. Verify radius reasonable for trajectory length
_docs/02_components/12_route_chunk_manager/12.03_feature_chunk_matching_coordination.md
+153
View File
@@ -0,0 +1,153 @@
# Feature: Chunk Matching Coordination
## Name
Chunk Matching Coordination
## Description
Operations related to the chunk matching workflow including readiness checks, matching status tracking, anchoring, and merging. This feature implements transactional integrity with F10 Factor Graph Optimizer using "Check-Act" pattern: F10 is called first, and only on success is internal state updated. Critical for the Atlas multi-map architecture where chunks are matched and merged into the global trajectory.
## Component APIs Implemented
### `is_chunk_ready_for_matching(chunk_id: str) -> bool`
Checks if chunk meets criteria for satellite matching attempt.
- Min frames: >= 5 (configurable)
- Max frames: <= 20 (configurable)
- Not already matched (status != "anchored" or "merged")
### `get_chunks_for_matching(flight_id: str) -> List[ChunkHandle]`
Retrieves all unanchored chunks ready for matching.
- Filters by flight_id
- Returns chunks with matching_status="unanchored" and is_ready_for_matching=True
### `mark_chunk_matching(chunk_id: str) -> bool`
Marks chunk as currently being matched.
- Updates matching_status to "matching"
- Prevents duplicate matching attempts
### `mark_chunk_anchored(chunk_id: str, frame_id: int, gps: GPSPoint) -> bool`
Anchors chunk to GPS coordinate after successful satellite matching.
- **Transactional**: Calls F10.add_chunk_anchor() first
- On success: Updates has_anchor, anchor_frame_id, anchor_gps, matching_status="anchored"
- On failure: Returns False, no state change
### `merge_chunks(main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool`
Merges new_chunk INTO main_chunk using Sim(3) transformation.
- Resolves flight_id from internal ChunkHandle for main_chunk_id
- **Transactional**: Calls F10.merge_chunk_subgraphs() first
- On success: new_chunk marked "merged" and deactivated, main_chunk extended
- On failure: Returns False, no state change
## External Tools and Services
- **F10 Factor Graph Optimizer**: `add_chunk_anchor()`, `merge_chunk_subgraphs()`
- **F03 Flight Database**: `save_chunk_state()` (after merge)
## Internal Methods
### `_check_matching_criteria(chunk_id: str) -> bool`
Validates chunk meets all matching readiness criteria.
### `_get_matching_status(chunk_id: str) -> str`
Returns current matching_status from ChunkHandle.
### `_update_anchor_state(chunk_id: str, frame_id: int, gps: GPSPoint)`
Updates ChunkHandle with anchor information after successful F10 call.
### `_mark_chunk_merged(chunk_id: str)`
Updates new_chunk state after successful merge operation.
## Unit Tests
### Test: is_chunk_ready_min_frames
- Create chunk with 3 frames
- Assert is_chunk_ready_for_matching() returns False
- Add 2 more frames (total 5)
- Assert is_chunk_ready_for_matching() returns True
### Test: is_chunk_ready_max_frames
- Create chunk with 21 frames
- Assert is_chunk_ready_for_matching() returns False
### Test: is_chunk_ready_already_anchored
- Create chunk, mark as anchored
- Assert is_chunk_ready_for_matching() returns False
### Test: get_chunks_for_matching_filters_correctly
- Create 3 chunks: unanchored+ready, anchored, unanchored+not_ready
- Assert get_chunks_for_matching() returns only first chunk
### Test: get_chunks_for_matching_filters_by_flight
- Create chunks for flight_A and flight_B
- Assert get_chunks_for_matching(flight_A) returns only flight_A chunks
### Test: mark_chunk_matching_updates_status
- Create chunk
- Call mark_chunk_matching()
- Assert matching_status="matching"
### Test: mark_chunk_anchored_transactional_success
- Create chunk
- Mock F10.add_chunk_anchor() returns success
- Call mark_chunk_anchored()
- Assert returns True
- Assert has_anchor=True, anchor_gps set, matching_status="anchored"
### Test: mark_chunk_anchored_transactional_failure
- Create chunk
- Mock F10.add_chunk_anchor() returns failure
- Call mark_chunk_anchored()
- Assert returns False
- Assert has_anchor=False (no state change)
### Test: merge_chunks_transactional_success
- Create main_chunk and new_chunk
- Mock F10.merge_chunk_subgraphs() returns success
- Call merge_chunks()
- Assert returns True
- Assert new_chunk.is_active=False
- Assert new_chunk.matching_status="merged"
### Test: merge_chunks_transactional_failure
- Create main_chunk and new_chunk
- Mock F10.merge_chunk_subgraphs() returns failure
- Call merge_chunks()
- Assert returns False
- Assert new_chunk state unchanged
### Test: merge_chunks_resolves_flight_id
- Create main_chunk for flight_A
- Verify merge_chunks() calls F10 with correct flight_id from ChunkHandle
## Integration Tests
### Test: chunk_matching_workflow
1. create_chunk() with 10 frames
2. is_chunk_ready_for_matching() → True
3. mark_chunk_matching() → status="matching"
4. mark_chunk_anchored() → status="anchored"
5. is_chunk_ready_for_matching() → False
### Test: chunk_merge_workflow
1. Create main_chunk (frames 1-10), new_chunk (frames 20-30)
2. Anchor new_chunk via mark_chunk_anchored()
3. merge_chunks(main_chunk, new_chunk, transform)
4. Verify new_chunk deactivated and marked "merged"
5. Verify main_chunk still active
### Test: transactional_integrity_anchor
1. Create chunk
2. Configure F10.add_chunk_anchor() to fail
3. Call mark_chunk_anchored()
4. Verify chunk state unchanged
5. Configure F10.add_chunk_anchor() to succeed
6. Call mark_chunk_anchored()
7. Verify chunk state updated
### Test: transactional_integrity_merge
1. Create main_chunk and new_chunk
2. Configure F10.merge_chunk_subgraphs() to fail
3. Call merge_chunks()
4. Verify both chunks' state unchanged
5. Configure F10.merge_chunk_subgraphs() to succeed
6. Call merge_chunks()
7. Verify new_chunk state updated
_docs/02_components/12_route_chunk_manager/12.04_feature_chunk_state_persistence.md
+114
View File
@@ -0,0 +1,114 @@
# Feature: Chunk State Persistence
## Name
Chunk State Persistence
## Description
Persistence operations for saving and loading chunk state via F03 Flight Database. Enables system recovery after restarts and maintains chunk state across processing sessions. Serializes internal _chunks dictionary to persistent storage.
## Component APIs Implemented
### `save_chunk_state(flight_id: str) -> bool`
Persists all chunk state for a flight.
- Serializes ChunkHandles for specified flight
- Calls F03.save_chunk_state()
- Returns success status
### `load_chunk_state(flight_id: str) -> bool`
Loads chunk state for a flight from persistent storage.
- Calls F03.load_chunk_states()
- Deserializes and populates internal _chunks dictionary
- Returns success status
## External Tools and Services
- **F03 Flight Database**: `save_chunk_state()`, `load_chunk_states()`
## Internal Methods
### `_serialize_chunks(flight_id: str) -> Dict`
Converts ChunkHandles to serializable dictionary format.
### `_deserialize_chunks(data: Dict) -> Dict[str, ChunkHandle]`
Reconstructs ChunkHandles from serialized data.
### `_filter_chunks_by_flight(flight_id: str) -> List[ChunkHandle]`
Filters internal _chunks dictionary by flight_id.
### `_merge_loaded_chunks(chunks: Dict[str, ChunkHandle])`
Merges loaded chunks into internal state without overwriting existing.
## Unit Tests
### Test: save_chunk_state_calls_f03
- Create chunks for flight_A
- Call save_chunk_state(flight_A)
- Assert F03.save_chunk_state() called with serialized data
### Test: save_chunk_state_filters_by_flight
- Create chunks for flight_A and flight_B
- Call save_chunk_state(flight_A)
- Assert only flight_A chunks serialized
### Test: save_chunk_state_success
- Create chunks
- Mock F03.save_chunk_state() returns success
- Assert save_chunk_state() returns True
### Test: save_chunk_state_failure
- Create chunks
- Mock F03.save_chunk_state() returns failure
- Assert save_chunk_state() returns False
### Test: load_chunk_state_populates_internal_dict
- Mock F03.load_chunk_states() returns chunk data
- Call load_chunk_state(flight_A)
- Assert internal _chunks dictionary populated
### Test: load_chunk_state_success
- Mock F03.load_chunk_states() returns success
- Assert load_chunk_state() returns True
### Test: load_chunk_state_failure
- Mock F03.load_chunk_states() returns failure
- Assert load_chunk_state() returns False
### Test: load_chunk_state_empty_flight
- Mock F03.load_chunk_states() returns empty
- Call load_chunk_state(flight_A)
- Assert returns True (no error)
- Assert no chunks loaded
### Test: serialize_preserves_all_fields
- Create chunk with all fields populated (anchor, frames, status)
- Serialize and deserialize
- Assert all fields match original
### Test: deserialize_handles_missing_optional_fields
- Create serialized data with optional fields missing
- Deserialize
- Assert ChunkHandle created with default values
## Integration Tests
### Test: save_load_roundtrip
1. Create chunks with various states (active, anchored, merged)
2. save_chunk_state()
3. Clear internal state
4. load_chunk_state()
5. Verify all chunks restored with correct state
### Test: persistence_across_restart
1. Create flight with chunks
2. Process frames, anchor some chunks
3. save_chunk_state()
4. Simulate restart (new instance)
5. load_chunk_state()
6. Verify can continue processing
### Test: partial_state_recovery
1. Create chunks for multiple flights
2. save_chunk_state(flight_A)
3. Clear internal state
4. load_chunk_state(flight_A)
5. Verify only flight_A chunks loaded
_docs/02_components/12_route_chunk_manager/12._component_route_chunk_manager.md
+471
View File
@@ -0,0 +1,471 @@
# Route Chunk Manager
## Interface Definition
**Interface Name**: `IRouteChunkManager`
### Interface Methods
```python
class IRouteChunkManager(ABC):
@abstractmethod
def create_chunk(self, flight_id: str, start_frame_id: int) -> ChunkHandle:
pass
@abstractmethod
def add_frame_to_chunk(self, chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool:
pass
@abstractmethod
def get_chunk_frames(self, chunk_id: str) -> List[int]:
pass
@abstractmethod
def get_chunk_images(self, chunk_id: str) -> List[np.ndarray]:
pass
@abstractmethod
def get_chunk_composite_descriptor(self, chunk_id: str) -> np.ndarray:
pass
@abstractmethod
def get_chunk_bounds(self, chunk_id: str) -> ChunkBounds:
pass
@abstractmethod
def is_chunk_ready_for_matching(self, chunk_id: str) -> bool:
pass
@abstractmethod
def mark_chunk_anchored(self, chunk_id: str, frame_id: int, gps: GPSPoint) -> bool:
"""
Transactional update:
1. Calls F10.add_chunk_anchor().
2. IF success: Updates internal state to 'anchored'.
3. Returns success status.
"""
pass
@abstractmethod
def get_chunks_for_matching(self, flight_id: str) -> List[ChunkHandle]:
pass
@abstractmethod
def get_active_chunk(self, flight_id: str) -> Optional[ChunkHandle]:
pass
@abstractmethod
def deactivate_chunk(self, chunk_id: str) -> bool:
pass
@abstractmethod
def merge_chunks(self, main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool:
"""
Merges new_chunk INTO main_chunk. Extends main_chunk with new_chunk's frames.
Transactional update:
1. Calls F10.merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform).
2. IF success: Updates internal state (new_chunk merged/deactivated).
3. Returns success status.
Note: flight_id is obtained from the ChunkHandle stored internally for main_chunk_id.
"""
pass
@abstractmethod
def mark_chunk_matching(self, chunk_id: str) -> bool:
pass
@abstractmethod
def save_chunk_state(self, flight_id: str) -> bool:
pass
@abstractmethod
def load_chunk_state(self, flight_id: str) -> bool:
pass
```
## Component Description
### Responsibilities
- **Source of Truth**: Manages chunk states (Active, Matching, Anchored, Merged).
- **Transactional Integrity**: Ensures internal state updates are atomic with respect to Factor Graph (F10) operations.
- **Implementation**: Uses "Check-Act" pattern. Calls F10 first; if F10 fails, F12 does not update state and returns error.
### Internal State Management
F12 maintains an internal dictionary of `ChunkHandle` objects keyed by `chunk_id`. Each `ChunkHandle` contains `flight_id`, allowing F12 to resolve flight context for F10 calls without requiring `flight_id` as a parameter on every method. This simplifies the API for callers while maintaining flight isolation internally.
```python
# Internal state
_chunks: Dict[str, ChunkHandle] # chunk_id -> ChunkHandle (contains flight_id)
```
### Interaction
- Called by **F02.2 Flight Processing Engine** and **F11 Failure Recovery Coordinator** (via F02.2 or direct delegation).
- Calls **F10 Factor Graph Optimizer**.
### Scope
- Chunk lifecycle management
- Chunk state tracking
- Chunk representation generation (descriptors, bounds)
- Integration point for chunk matching coordination
## API Methods
### `create_chunk(flight_id: str, start_frame_id: int) -> ChunkHandle`
**Description**: Initializes a new route chunk.
**Called By**:
- F02.2 Flight Processing Engine (when tracking lost)
- F11 Failure Recovery Coordinator (proactive chunk creation)
**Input**:
```python
flight_id: str
start_frame_id: int # First frame in chunk
```
**Output**:
```python
ChunkHandle:
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str
```
**Processing Flow**:
1. Generate unique chunk_id
2. Call F10 Factor Graph Optimizer.create_new_chunk()
3. Initialize chunk state (unanchored, active)
4. Store chunk metadata
5. Return ChunkHandle
**Test Cases**:
1. **Create chunk**: Returns ChunkHandle with is_active=True
2. **Multiple chunks**: Can create multiple chunks for same flight
3. **Chunk initialization**: Chunk initialized in factor graph
---
### `add_frame_to_chunk(chunk_id: str, frame_id: int, vo_result: RelativePose) -> bool`
**Description**: Adds a frame to an active chunk.
**Called By**:
- F02.2 Flight Processing Engine (during frame processing)
**Input**:
```python
chunk_id: str
frame_id: int
vo_result: RelativePose # From F07 Sequential VO
```
**Output**:
```python
bool: True if frame added successfully
```
**Processing Flow**:
1. Verify chunk exists and is active
2. Add frame_id to chunk's frames list
3. Store vo_result for chunk
4. Call F10.add_relative_factor_to_chunk()
5. Update chunk's end_frame_id
6. Check if chunk ready for matching
**Test Cases**:
1. **Add frame to active chunk**: Frame added successfully
2. **Add frame to inactive chunk**: Returns False
3. **Chunk growth**: Chunk frames list updated
---
### `get_chunk_frames(chunk_id: str) -> List[int]`
**Description**: Retrieves list of frame IDs in a chunk.
**Called By**:
- F08 Global Place Recognition (for chunk descriptor computation)
- F09 Metric Refinement (for chunk LiteSAM matching)
- F11 Failure Recovery Coordinator (chunk state queries)
**Input**: `chunk_id: str`
**Output**: `List[int]` # Frame IDs in chunk, ordered by sequence
---
### `get_chunk_images(chunk_id: str) -> List[np.ndarray]`
**Description**: Retrieves images for all frames in a chunk.
**Called By**:
- F08 Global Place Recognition (chunk descriptor computation)
- F09 Metric Refinement (chunk LiteSAM matching)
- F06 Image Rotation Manager (chunk rotation)
**Output**: `List[np.ndarray]` # Images for each frame in chunk
---
### `get_chunk_composite_descriptor(chunk_id: str) -> np.ndarray`
**Description**: Computes aggregate DINOv2 descriptor for chunk (for semantic matching).
**Called By**:
- F08 Global Place Recognition (chunk semantic matching)
**Output**: `np.ndarray`: Aggregated descriptor vector (4096-dim or 8192-dim)
---
### `get_chunk_bounds(chunk_id: str) -> ChunkBounds`
**Description**: Estimates GPS bounds of a chunk based on VO trajectory.
**Called By**:
- F11 Failure Recovery Coordinator (for tile search area)
- F04 Satellite Data Manager (for tile prefetching)
**Output**:
```python
ChunkBounds:
estimated_center: GPSPoint
estimated_radius: float # meters
confidence: float
```
---
### `is_chunk_ready_for_matching(chunk_id: str) -> bool`
**Description**: Checks if a chunk has enough data (frames, spread) to attempt satellite matching.
**Criteria**:
- Min frames: >= 5 frames (configurable)
- Max frames: <= 20 frames (configurable, prevents oversized chunks)
- Internal consistency: VO factors have reasonable inlier counts
- Not already matched: matching_status != "anchored" or "merged"
---
### `mark_chunk_anchored(chunk_id: str, frame_id: int, gps: GPSPoint) -> bool`
**Description**: Anchors a chunk to a specific GPS coordinate (e.g., from successful satellite matching).
**Called By**:
- F11 Failure Recovery Coordinator (after successful chunk matching)
**Input**:
```python
chunk_id: str
frame_id: int # Frame within chunk that was anchored
gps: GPSPoint
```
**Output**: `bool` - True if marked successfully
**Processing Flow**:
1. Verify chunk exists
2. Call F10.add_chunk_anchor()
3. If successful:
- Update chunk state (has_anchor=True, anchor_frame_id, anchor_gps)
- Update matching_status to "anchored"
- Trigger chunk optimization
**Test Cases**:
1. **Mark anchored**: Chunk state updated correctly
2. **Anchor in factor graph**: F10 anchor added
3. **Chunk optimization**: Chunk optimized after anchoring
---
### `get_chunks_for_matching(flight_id: str) -> List[ChunkHandle]`
**Description**: Retrieves all unanchored chunks ready for matching.
**Called By**:
- F11 Failure Recovery Coordinator (background matching task)
**Output**: `List[ChunkHandle]` # Unanchored chunks ready for matching
---
### `get_active_chunk(flight_id: str) -> Optional[ChunkHandle]`
**Description**: Gets the currently active chunk for a flight.
**Called By**:
- F02.2 Flight Processing Engine (before processing frame)
**Output**: `Optional[ChunkHandle]` # Active chunk or None
---
### `deactivate_chunk(chunk_id: str) -> bool`
**Description**: Deactivates a chunk (typically after merging or completion).
**Called By**:
- F11 Failure Recovery Coordinator (after chunk merged)
- F02.2 Flight Processing Engine (chunk lifecycle)
**Output**: `bool` - True if deactivated successfully
---
### `merge_chunks(main_chunk_id: str, new_chunk_id: str, transform: Sim3Transform) -> bool`
**Description**: Merges new_chunk INTO main_chunk. The resulting merged chunk is main_chunk (extended with new_chunk's frames). new_chunk is deactivated after merge.
**Called By**:
- F11 Failure Recovery Coordinator (after successful chunk matching)
**Input**:
```python
main_chunk_id: str # Main chunk being extended (destination, typically older/established trajectory)
new_chunk_id: str # New chunk being merged in (source, typically newer/recently anchored)
transform: Sim3Transform:
translation: np.ndarray # (3,)
rotation: np.ndarray # (3, 3) or quaternion
scale: float
```
**Output**: `bool` - True if merge successful
**flight_id Resolution**: F12 internally stores ChunkHandle objects keyed by chunk_id. The flight_id is extracted from the ChunkHandle for main_chunk_id when calling F10 methods.
**Processing Flow**:
1. Get flight_id from internally stored ChunkHandle for main_chunk_id
2. Call F10.merge_chunk_subgraphs(flight_id, new_chunk_id, main_chunk_id, transform)
3. If successful:
- Update new_chunk_id state:
- Set is_active=False
- Set matching_status="merged"
- Call deactivate_chunk(new_chunk_id)
- main_chunk remains active (now contains merged frames)
- Persist chunk state via F03 Flight Database.save_chunk_state()
4. Return True
**Test Cases**:
1. **Merge anchored chunk**: new_chunk merged into main_chunk
2. **New chunk deactivated**: new_chunk marked as merged and deactivated
3. **Main chunk extended**: main_chunk remains active with additional frames
---
### `mark_chunk_matching(chunk_id: str) -> bool`
**Description**: Explicitly marks chunk as being matched (updates matching_status to "matching").
**Called By**:
- F11 Failure Recovery Coordinator (when chunk matching starts)
**Output**: `bool` - True if marked successfully
## Integration Tests
### Test 1: Chunk Lifecycle
1. create_chunk() → chunk created
2. add_frame_to_chunk() × 10 → 10 frames added
3. is_chunk_ready_for_matching() → True
4. mark_chunk_anchored() → chunk anchored
5. deactivate_chunk() → chunk deactivated
### Test 2: Chunk Descriptor Computation
1. Create chunk with 10 frames
2. get_chunk_images() → 10 images
3. get_chunk_composite_descriptor() → aggregated descriptor
4. Verify descriptor more robust than single-image descriptor
### Test 3: Multiple Chunks
1. Create chunk_1 (frames 1-10)
2. Create chunk_2 (frames 20-30)
3. get_chunks_for_matching() → returns both chunks
4. mark_chunk_anchored(chunk_1) → chunk_1 anchored
5. get_chunks_for_matching() → returns only chunk_2
### Test 4: Chunk Merging
1. Create main_chunk (frames 1-10), new_chunk (frames 20-30)
2. Anchor new_chunk via mark_chunk_anchored()
3. merge_chunks(main_chunk, new_chunk, transform) → new_chunk merged into main_chunk
4. Verify new_chunk marked as merged and deactivated
5. Verify main_chunk extended with new frames
6. Verify F10.merge_chunk_subgraphs() called with correct parameters
### Test 5: Chunk Matching Status
1. Create chunk
2. mark_chunk_matching() → status updated to "matching"
3. mark_chunk_anchored() → status updated to "anchored"
4. Verify explicit state transitions
## Non-Functional Requirements
### Performance
- **create_chunk**: < 10ms
- **add_frame_to_chunk**: < 5ms
- **get_chunk_composite_descriptor**: < 3s for 20 images (async)
- **get_chunk_bounds**: < 10ms
### Reliability
- Chunk state persisted across restarts
- Graceful handling of missing frames
- Thread-safe chunk operations
## Dependencies
### Internal Components
- **F10 Factor Graph Optimizer**: Critical dependency for subgraph operations (`create_chunk_subgraph`, `add_relative_factor_to_chunk`, `merge_chunk_subgraphs`).
- **F03 Flight Database**: Persistence via `save_chunk_state()`, `load_chunk_states()`.
- **F05 Image Input Pipeline**: Image retrieval via `get_image_by_sequence()` for `get_chunk_images()`.
- **F08 Global Place Recognition**: Descriptor computation via `compute_chunk_descriptor()` for `get_chunk_composite_descriptor()`.
## Data Models
### ChunkHandle
```python
class ChunkHandle(BaseModel):
chunk_id: str
flight_id: str
start_frame_id: int
end_frame_id: Optional[int]
frames: List[int]
is_active: bool
has_anchor: bool
anchor_frame_id: Optional[int]
anchor_gps: Optional[GPSPoint]
matching_status: str # "unanchored", "matching", "anchored", "merged"
```
### ChunkBounds
```python
class ChunkBounds(BaseModel):
estimated_center: GPSPoint
estimated_radius: float # meters
confidence: float # 0.0 to 1.0
```
### ChunkConfig
```python
class ChunkConfig(BaseModel):
min_frames_for_matching: int = 5
max_frames_per_chunk: int = 20
descriptor_aggregation: str = "mean" # "mean", "vlad", "max"
```
### Sim3Transform
```python
class Sim3Transform(BaseModel):
translation: np.ndarray # (3,) - translation vector
rotation: np.ndarray # (3, 3) rotation matrix or (4,) quaternion
scale: float # Scale factor
```
_docs/02_components/13_coordinate_transformer/13.01_feature_enu_coordinate_management.md
+115
View File
@@ -0,0 +1,115 @@
# Feature: ENU Coordinate Management
## Description
Manages East-North-Up (ENU) coordinate system origins per flight and provides bidirectional conversions between GPS (WGS84) and ENU coordinates. ENU is a local Cartesian coordinate system used by the Factor Graph Optimizer for better numerical stability in optimization operations.
Each flight has its own ENU origin set from the flight's start_gps. All frames in a flight share the same origin for consistent coordinate conversions.
## Component APIs Implemented
### `set_enu_origin(flight_id: str, origin_gps: GPSPoint) -> None`
Sets the ENU origin for a specific flight during flight creation. Precomputes conversion factors for lat/lon to meters at origin latitude.
### `get_enu_origin(flight_id: str) -> GPSPoint`
Returns the stored ENU origin for a flight. Raises `OriginNotSetError` if called before set_enu_origin.
### `gps_to_enu(flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]`
Converts GPS coordinates to ENU (east, north, up) in meters relative to flight's origin.
Algorithm:
- delta_lat = gps.lat - origin.lat
- delta_lon = gps.lon - origin.lon
- east = delta_lon × cos(origin.lat) × 111319.5
- north = delta_lat × 111319.5
- up = 0
### `enu_to_gps(flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint`
Converts ENU coordinates back to GPS using the flight's ENU origin.
Algorithm:
- delta_lon = east / (cos(origin.lat) × 111319.5)
- delta_lat = north / 111319.5
- lat = origin.lat + delta_lat
- lon = origin.lon + delta_lon
## External Tools and Services
None - pure mathematical conversions.
## Internal Methods
### `_compute_meters_per_degree(latitude: float) -> Tuple[float, float]`
Precomputes conversion factors for a given latitude. Returns (meters_per_degree_lon, meters_per_degree_lat). Used during set_enu_origin for efficiency.
### `_get_origin_or_raise(flight_id: str) -> GPSPoint`
Helper to retrieve origin with error handling. Raises OriginNotSetError if not set.
## Unit Tests
### Test: set_enu_origin stores origin
- Call set_enu_origin with flight_id and origin GPS
- Verify origin is stored
- Verify conversion factors are precomputed
### Test: get_enu_origin returns stored origin
- Set origin via set_enu_origin
- Call get_enu_origin
- Verify returned GPS matches stored origin
### Test: get_enu_origin raises error if not set
- Call get_enu_origin for unknown flight_id
- Verify OriginNotSetError is raised
### Test: multiple flights have independent origins
- Set different origins for flight_1 and flight_2
- Verify each get_enu_origin returns correct independent origin
### Test: gps_to_enu at origin returns zero
- Set origin
- Convert origin GPS to ENU
- Verify result is (0, 0, 0)
### Test: gps_to_enu 1km east
- Set origin at known location
- Convert GPS point ~1km east of origin
- Verify east component ~1000m, north ~0
### Test: gps_to_enu 1km north
- Set origin at known location
- Convert GPS point ~1km north of origin
- Verify north component ~1000m, east ~0
### Test: gps_to_enu diagonal offset
- Convert GPS point with both lat/lon offset
- Verify correct east/north components
### Test: enu_to_gps at zero returns origin
- Set origin
- Convert (0, 0, 0) ENU to GPS
- Verify result matches origin GPS
### Test: round-trip conversion preserves coordinates
- Set origin
- Start with GPS point
- Convert to ENU via gps_to_enu
- Convert back via enu_to_gps
- Verify GPS matches original (within floating-point precision)
### Test: latitude affects east conversion factor
- Test gps_to_enu at different latitudes
- Verify east component accounts for cos(latitude) factor
## Integration Tests
### Test: ENU origin used by Factor Graph
- Create flight with start_gps via F02.1
- Verify F02.1 calls set_enu_origin with start_gps
- Verify Factor Graph can retrieve origin via get_enu_origin
### Test: ENU conversions consistent across components
- Set origin via set_enu_origin
- F10 Factor Graph uses gps_to_enu for absolute factors
- F10 uses enu_to_gps for trajectory output
- Verify GPS coordinates are consistent throughout pipeline
_docs/02_components/13_coordinate_transformer/13.02_feature_pixel_gps_projection.md
+169
View File
@@ -0,0 +1,169 @@
# Feature: Pixel-GPS Projection
## Description
Provides coordinate conversions between image pixel coordinates and GPS coordinates using camera model and pose information. This feature enables:
- Converting frame center pixels to GPS for trajectory output
- Converting detected object pixel locations to GPS coordinates (critical for external object detection integration)
- Inverse projection from GPS to pixels for visualization
- Generic point transformations via homography/affine matrices
All projections rely on the ground plane assumption and require frame pose from Factor Graph, camera parameters, and altitude.
## Component APIs Implemented
### `pixel_to_gps(flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint`
Converts pixel coordinates to GPS using camera pose and ground plane intersection.
Algorithm:
1. Unproject pixel to 3D ray using H01 Camera Model
2. Intersect ray with ground plane at altitude
3. Transform 3D point from camera frame to ENU using frame_pose
4. Convert ENU to WGS84 via enu_to_gps(flight_id, enu_point)
### `gps_to_pixel(flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]`
Inverse projection from GPS to image pixel coordinates.
Algorithm:
1. Convert GPS to ENU via gps_to_enu(flight_id, gps)
2. Transform ENU point to camera frame using frame_pose
3. Project 3D point to image plane using H01 Camera Model
4. Return pixel coordinates
### `image_object_to_gps(flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint`
**Critical method** - Converts object pixel coordinates to GPS for external object detection integration.
Processing:
1. Get frame_pose from F10.get_trajectory(flight_id)[frame_id]
2. Get camera_params from F17.get_flight_config(flight_id)
3. Get altitude from F17.get_flight_config(flight_id).altitude
4. Call pixel_to_gps(flight_id, object_pixel, frame_pose, camera_params, altitude)
5. Return GPS
### `transform_points(points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]`
Applies homography (3×3) or affine (2×3) transformation to list of points.
Algorithm:
1. Convert points to homogeneous coordinates
2. Apply transformation matrix
3. Normalize (for homography) and return
## External Tools and Services
- **H01 Camera Model**: `project()` for 3D-to-pixel, `unproject()` for pixel-to-ray
- **H02 GSD Calculator**: For GSD calculations in coordinate conversions
- **F10 Factor Graph Optimizer**: `get_trajectory(flight_id)` for frame poses
- **F17 Configuration Manager**: `get_flight_config(flight_id)` for camera params and altitude
## Internal Methods
### `_unproject_to_ray(pixel: Tuple[float, float], camera_params: CameraParameters) -> np.ndarray`
Uses H01 Camera Model to convert pixel to 3D ray direction in camera frame.
### `_intersect_ray_ground_plane(ray_origin: np.ndarray, ray_direction: np.ndarray, ground_altitude: float) -> np.ndarray`
Computes intersection point of ray with horizontal ground plane at given altitude.
### `_camera_to_enu(point_camera: np.ndarray, frame_pose: Pose) -> np.ndarray`
Transforms point from camera frame to ENU using pose rotation and translation.
### `_enu_to_camera(point_enu: np.ndarray, frame_pose: Pose) -> np.ndarray`
Inverse transformation from ENU to camera frame.
### `_to_homogeneous(points: List[Tuple[float, float]]) -> np.ndarray`
Converts 2D points to homogeneous coordinates (appends 1).
### `_from_homogeneous(points: np.ndarray) -> List[Tuple[float, float]]`
Converts from homogeneous back to 2D (divides by w).
## Unit Tests
### Test: pixel_to_gps at image center
- Create frame_pose at known ENU location
- Call pixel_to_gps with image center pixel
- Verify GPS matches expected frame center location
### Test: pixel_to_gps at image corner
- Call pixel_to_gps with corner pixel
- Verify GPS has appropriate offset from center
### Test: pixel_to_gps at different altitudes
- Same pixel, different altitudes
- Verify GPS changes appropriately with altitude (higher = wider footprint)
### Test: gps_to_pixel at frame center GPS
- Convert frame center GPS to pixel
- Verify pixel is at image center
### Test: gps_to_pixel out of view
- Convert GPS point outside camera field of view
- Verify pixel coordinates outside image bounds
### Test: pixel_to_gps/gps_to_pixel round trip
- Start with pixel
- Convert to GPS via pixel_to_gps
- Convert back via gps_to_pixel
- Verify pixel matches original (within tolerance)
### Test: image_object_to_gps at image center
- Call image_object_to_gps with center pixel
- Verify GPS matches frame center
### Test: image_object_to_gps with offset
- Call image_object_to_gps with off-center pixel
- Verify GPS has correct offset
### Test: image_object_to_gps multiple objects
- Convert multiple object pixels from same frame
- Verify each gets correct independent GPS
### Test: transform_points identity
- Apply identity matrix
- Verify points unchanged
### Test: transform_points rotation
- Apply 90° rotation matrix
- Verify points rotated correctly
### Test: transform_points translation
- Apply translation matrix
- Verify points translated correctly
### Test: transform_points homography
- Apply perspective homography
- Verify points transformed with perspective correction
### Test: transform_points affine
- Apply 2×3 affine matrix
- Verify correct transformation
## Integration Tests
### Test: Frame center GPS calculation
- Process frame through F02.2 Flight Processing Engine
- Get frame_pose from F10 Factor Graph
- Call pixel_to_gps with image center
- Verify GPS within 50m of expected
### Test: Object localization end-to-end
- External detector finds object at pixel (1500, 2000)
- Call image_object_to_gps(flight_id, frame_id, pixel)
- Verify GPS is correct
- Test with multiple objects
### Test: Object GPS updates after trajectory refinement
- Get object GPS before trajectory refinement
- Factor Graph refines trajectory with new absolute factors
- Get object GPS again
- Verify GPS updated to reflect refined trajectory
### Test: Round-trip GPS-pixel-GPS consistency
- Start with GPS point
- Convert to pixel via gps_to_pixel
- Convert back via pixel_to_gps
- Verify GPS matches original within tolerance
### Test: GSD integration via H02
- Call H02.compute_gsd() with known parameters
- Verify GSD matches expected value
- Test at different altitudes
_docs/02_components/13_coordinate_transformer/13._component_coordinate_transformer.md
+444
View File
@@ -0,0 +1,444 @@
# Coordinate Transformer
## Interface Definition
**Interface Name**: `ICoordinateTransformer`
### Interface Methods
```python
class ICoordinateTransformer(ABC):
# ENU Origin Management
@abstractmethod
def set_enu_origin(self, flight_id: str, origin_gps: GPSPoint) -> None:
pass
@abstractmethod
def get_enu_origin(self, flight_id: str) -> GPSPoint:
pass
@abstractmethod
def gps_to_enu(self, flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]:
pass
@abstractmethod
def enu_to_gps(self, flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint:
pass
# Pixel/GPS Conversions
@abstractmethod
def pixel_to_gps(self, flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint:
pass
@abstractmethod
def gps_to_pixel(self, flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]:
pass
@abstractmethod
def image_object_to_gps(self, flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint:
pass
@abstractmethod
def transform_points(self, points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]:
pass
```
## Component Description
### Responsibilities
- Pixel-to-GPS coordinate conversions
- GPS-to-pixel inverse projections
- **Critical**: Convert object pixel coordinates (from external detection system) to GPS
- Handle multiple coordinate systems: WGS84, Web Mercator, ENU, image pixels, rotated coordinates
- Camera model integration for projection operations
**Note**: GSD calculations are delegated to H02 GSD Calculator helper.
### Scope
- Coordinate system transformations
- Camera projection mathematics
- Integration with Factor Graph poses
- Object localization (pixel → GPS)
- Support for external object detection system
## API Methods
### ENU Origin Management
#### `set_enu_origin(flight_id: str, origin_gps: GPSPoint) -> None`
**Description**: Sets the ENU (East-North-Up) coordinate system origin for a specific flight. Called once during flight creation using the flight's start_gps.
**Why ENU Origin?**: Factor graph optimization works in Cartesian coordinates (meters) for better numerical stability. ENU converts GPS (degrees) to local meters relative to an origin. See `helpers/enu_origin_explanation.md` for details.
**Called By**:
- F02.1 Flight Lifecycle Manager (during create_flight)
**Input**:
```python
flight_id: str # Flight identifier
origin_gps: GPSPoint
lat: float # Origin latitude in WGS84 (flight start_gps)
lon: float # Origin longitude in WGS84 (flight start_gps)
```
**Processing Flow**:
1. Store origin_gps as ENU origin for flight_id
2. Precompute conversion factors for lat/lon to meters at origin latitude
3. All subsequent ENU operations for this flight use this origin
**Important**:
- Each flight has its own ENU origin (the flight's start_gps)
- Origin remains constant for the entire flight duration
- All frames in the flight use the same origin for ENU conversions
**Test Cases**:
1. **Set origin**: Store origin GPS
2. **Subsequent gps_to_enu calls**: Use set origin
3. **Multiple flights**: Each flight has independent origin
---
#### `get_enu_origin(flight_id: str) -> GPSPoint`
**Description**: Returns the ENU origin for a specific flight.
**Called By**:
- F10 Factor Graph Optimizer (for coordinate checks)
- F14 Result Manager (for reference)
**Input**:
```python
flight_id: str
```
**Output**:
```python
GPSPoint: The ENU origin (flight start_gps)
```
**Error Conditions**:
- Raises `OriginNotSetError` if called before set_enu_origin() for this flight_id
**Test Cases**:
1. **After set_enu_origin**: Returns stored origin
2. **Before set_enu_origin**: Raises error
---
#### `gps_to_enu(flight_id: str, gps: GPSPoint) -> Tuple[float, float, float]`
**Description**: Converts GPS coordinates to ENU (East, North, Up) relative to the flight's ENU origin.
**Called By**:
- F10 Factor Graph Optimizer (for absolute factors)
- Internal (for pixel_to_gps)
**Input**:
```python
flight_id: str # Flight identifier
gps: GPSPoint
lat: float
lon: float
```
**Output**:
```python
Tuple[float, float, float]: (east, north, up) in meters relative to flight's origin
```
**Algorithm**:
1. Compute delta_lat = gps.lat - origin.lat
2. Compute delta_lon = gps.lon - origin.lon
3. east = delta_lon × cos(origin.lat) × 111319.5 # meters/degree at equator
4. north = delta_lat × 111319.5
5. up = 0 (or computed from elevation if available)
**Test Cases**:
1. **Origin GPS**: Returns (0, 0, 0)
2. **1km East**: Returns (~1000, 0, 0)
3. **1km North**: Returns (0, ~1000, 0)
4. **Diagonal**: Returns correct east/north components
---
#### `enu_to_gps(flight_id: str, enu: Tuple[float, float, float]) -> GPSPoint`
**Description**: Converts ENU coordinates back to GPS using the flight's ENU origin.
**Called By**:
- F10 Factor Graph Optimizer (for get_trajectory)
- F14 Result Manager (for publishing GPS results)
**Input**:
```python
flight_id: str # Flight identifier
enu: Tuple[float, float, float] # (east, north, up) in meters relative to flight's origin
```
**Output**:
```python
GPSPoint: WGS84 coordinates
```
**Algorithm**:
1. delta_lon = east / (cos(origin.lat) × 111319.5)
2. delta_lat = north / 111319.5
3. lat = origin.lat + delta_lat
4. lon = origin.lon + delta_lon
**Test Cases**:
1. **Origin (0, 0, 0)**: Returns origin GPS
2. **Round-trip**: gps → enu → gps matches original (within precision)
---
### Pixel/GPS Conversions
### `pixel_to_gps(flight_id: str, pixel: Tuple[float, float], frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> GPSPoint`
**Description**: Converts pixel coordinates to GPS using camera pose and ground plane assumption.
**Called By**:
- F02.2 Flight Processing Engine (for frame center GPS)
- Internal (for image_object_to_gps)
**Input**:
```python
flight_id: str # Flight identifier (required for ENU origin lookup)
pixel: Tuple[float, float] # (x, y) in image coordinates
frame_pose: Pose # From Factor Graph (ENU coordinates)
camera_params: CameraParameters
altitude: float # Ground altitude
```
**Output**:
```python
GPSPoint:
lat: float
lon: float
```
**Algorithm**:
1. Unproject pixel to 3D ray using H01 Camera Model
2. Intersect ray with ground plane at altitude
3. Transform 3D point from camera frame to ENU using frame_pose
4. Convert ENU to WGS84 GPS using enu_to_gps(flight_id, enu_point)
**Assumptions**:
- Ground plane assumption (terrain height negligible)
- Downward-pointing camera
- Known altitude
**Error Conditions**:
- None (always returns GPS, may be inaccurate if assumptions violated)
**Test Cases**:
1. **Image center**: Returns frame center GPS
2. **Image corner**: Returns GPS at corner
3. **Object pixel**: Returns object GPS
4. **Altitude variation**: Correct GPS at different altitudes
---
### `gps_to_pixel(flight_id: str, gps: GPSPoint, frame_pose: Pose, camera_params: CameraParameters, altitude: float) -> Tuple[float, float]`
**Description**: Inverse projection from GPS to image pixel coordinates.
**Called By**:
- Visualization tools (overlay GPS annotations)
- Testing/validation
**Input**:
```python
flight_id: str # Flight identifier (required for ENU origin lookup)
gps: GPSPoint
frame_pose: Pose
camera_params: CameraParameters
altitude: float
```
**Output**:
```python
Tuple[float, float]: (x, y) pixel coordinates
```
**Algorithm**:
1. Convert GPS to ENU using gps_to_enu(flight_id, gps)
2. Transform ENU point to camera frame using frame_pose
3. Project 3D point to image plane using H01 Camera Model
4. Return pixel coordinates
**Test Cases**:
1. **Frame center GPS**: Returns image center pixel
2. **Out of view GPS**: Returns pixel outside image bounds
---
### `image_object_to_gps(flight_id: str, frame_id: int, object_pixel: Tuple[float, float]) -> GPSPoint`
**Description**: **Critical method** - Converts object pixel coordinates to GPS. Used for external object detection integration.
**Called By**:
- F02 Flight Processor (via convert_object_to_gps delegation from F01)
- F14 Result Manager (converts objects to GPS for output)
**Input**:
```python
flight_id: str # Flight identifier (needed for ENU origin and factor graph)
frame_id: int # Frame containing object
object_pixel: Tuple[float, float] # Pixel coordinates from object detector
```
**Output**:
```python
GPSPoint: GPS coordinates of object center
```
**Processing Flow**:
1. Get frame_pose from F10 Factor Graph Optimizer.get_trajectory(flight_id)[frame_id]
2. Get camera_params from F17 Configuration Manager.get_flight_config(flight_id)
3. Get altitude from F17 Configuration Manager.get_flight_config(flight_id).altitude
4. Call pixel_to_gps(object_pixel, frame_pose, camera_params, altitude)
5. Use enu_to_gps(flight_id, enu_point) for final GPS conversion
6. Return GPS
**User Story**:
- External system detects object in UAV image at pixel (1024, 768)
- Calls F02.convert_object_to_gps(flight_id="abc", frame_id=237, pixel=(1024, 768))
- F02 delegates to F13.image_object_to_gps(flight_id="abc", frame_id=237, object_pixel=(1024, 768))
- Returns GPSPoint(lat=48.123, lon=37.456)
- Object GPS can be used for navigation, targeting, etc.
**Test Cases**:
1. **Object at image center**: Returns frame center GPS
2. **Object at corner**: Returns GPS with offset
3. **Multiple objects**: Each gets correct GPS
4. **Refined trajectory**: Object GPS updates after refinement
---
### `transform_points(points: List[Tuple[float, float]], transformation: np.ndarray) -> List[Tuple[float, float]]`
**Description**: Applies homography or affine transformation to list of points.
**Called By**:
- F06 Image Rotation Manager (for rotation transforms)
- F09 Metric Refinement (homography application)
**Input**:
```python
points: List[Tuple[float, float]]
transformation: np.ndarray # 3×3 homography or 2×3 affine
```
**Output**:
```python
List[Tuple[float, float]]: Transformed points
```
**Processing Flow**:
1. Convert points to homogeneous coordinates
2. Apply transformation matrix
3. Normalize and return
**Test Cases**:
1. **Identity transform**: Points unchanged
2. **Rotation**: Points rotated correctly
3. **Homography**: Points transformed with perspective
## Integration Tests
### Test 1: Frame Center GPS Calculation
1. Get frame_pose from Factor Graph
2. pixel_to_gps(image_center) → GPS
3. Verify GPS matches expected location
4. Verify accuracy < 50m
### Test 2: Object Localization
1. External detector finds object at pixel (1500, 2000)
2. image_object_to_gps(frame_id, pixel) → GPS
3. Verify GPS correct
4. Multiple objects → all get correct GPS
### Test 3: Round-Trip Conversion
1. Start with GPS point
2. gps_to_pixel() → pixel
3. pixel_to_gps() → GPS
4. Verify GPS matches original (within tolerance)
### Test 4: GSD via H02
1. Call H02.compute_gsd() with known parameters
2. Verify matches expected value
3. Test at different altitudes
## Non-Functional Requirements
### Performance
- **pixel_to_gps**: < 5ms
- **gps_to_pixel**: < 5ms
- **image_object_to_gps**: < 10ms
### Accuracy
- **GPS accuracy**: Inherits from Factor Graph accuracy (~20m)
- **GSD calculation**: ±1% precision
- **Projection accuracy**: < 1 pixel error
### Reliability
- Handle edge cases (points outside image)
- Graceful handling of degenerate configurations
- Numerical stability
## Dependencies
### Internal Components
- **F10 Factor Graph Optimizer**: For frame poses via `get_trajectory(flight_id)` - required for `pixel_to_gps()` and `image_object_to_gps()` to get frame pose estimates.
- **F17 Configuration Manager**: For camera parameters via `get_flight_config(flight_id)`.
- **H01 Camera Model**: For projection/unprojection operations (`project()`, `unproject()`).
- **H02 GSD Calculator**: For GSD calculations in coordinate conversions.
**Note**: F13 uses internal `enu_to_gps()` and `gps_to_enu()` methods that rely on the ENU origin set via `set_enu_origin(flight_id, origin_gps)`. H06 Web Mercator Utils is NOT used for ENU conversions - ENU is a local Cartesian coordinate system.
### External Dependencies
- **numpy**: Matrix operations
- **opencv-python**: Homography operations (optional)
## Data Models
### Pose (from Factor Graph)
```python
class Pose(BaseModel):
position: np.ndarray # (3,) - [x, y, z] in ENU
orientation: np.ndarray # (4,) quaternion or (3,3) rotation
timestamp: datetime
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int # pixels
resolution_height: int # pixels
principal_point: Tuple[float, float] # (cx, cy)
distortion_coefficients: Optional[List[float]]
```
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float
lon: float
```
### CoordinateFrame
```python
class CoordinateFrame(Enum):
WGS84 = "wgs84" # GPS coordinates
ENU = "enu" # East-North-Up local frame
ECEF = "ecef" # Earth-Centered Earth-Fixed
IMAGE = "image" # Image pixel coordinates
CAMERA = "camera" # Camera frame
```
_docs/02_components/14_result_manager/14.01_feature_frame_result_persistence.md
+46
View File
@@ -0,0 +1,46 @@
# Feature: Frame Result Persistence
## Description
Handles atomic persistence and real-time publishing of individual frame processing results. Ensures consistency between `frame_results` and `waypoints` tables through transactional updates, and triggers SSE events for live client updates.
## Component APIs Implemented
### `update_frame_result(flight_id: str, frame_id: int, result: FrameResult) -> bool`
Persists frame result atomically (frame_results + waypoints tables) and publishes via SSE.
### `publish_waypoint_update(flight_id: str, frame_id: int) -> bool`
Internal method to trigger waypoint visualization update after persistence.
## External Services Used
- **F03 Flight Database**: `execute_transaction()` for atomic multi-table updates
- **F15 SSE Event Streamer**: `send_frame_result()` for real-time client notification
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_frame_transaction(flight_id, frame_id, result)` | Constructs DB transaction with frame_results INSERT/UPDATE and waypoints UPDATE |
| `_update_flight_statistics(flight_id)` | Updates flight-level statistics after frame persistence |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_update_frame_result_new_frame` | New frame result stored in frame_results table |
| `test_update_frame_result_updates_waypoint` | Waypoint table updated with latest position |
| `test_update_frame_result_transaction_atomic` | Both tables updated or neither (rollback on failure) |
| `test_update_frame_result_triggers_sse` | F15.send_frame_result() called on success |
| `test_update_frame_result_refined_flag` | Existing result updated with refined=True |
| `test_publish_waypoint_fetches_latest` | Fetches current data before publishing |
| `test_publish_waypoint_handles_transient_error` | Retries on transient F15 errors |
| `test_publish_waypoint_logs_on_db_unavailable` | Logs error and continues on DB failure |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_per_frame_processing_e2e` | Process frame → stored in F03 → SSE event sent via F15 |
| `test_statistics_updated_after_frame` | Flight statistics reflect new frame count and confidence |
_docs/02_components/14_result_manager/14.02_feature_result_retrieval.md
+44
View File
@@ -0,0 +1,44 @@
# Feature: Result Retrieval
## Description
Provides read access to frame results for REST API endpoints and SSE reconnection replay. Supports full flight result retrieval with statistics and incremental change detection for efficient reconnection.
## Component APIs Implemented
### `get_flight_results(flight_id: str) -> FlightResults`
Retrieves all frame results and computed statistics for a flight.
### `get_changed_frames(flight_id: str, since: datetime) -> List[int]`
Returns frame IDs modified since a given timestamp for incremental updates.
## External Services Used
- **F03 Flight Database**: Query frame_results and waypoints tables
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_compute_flight_statistics(frames)` | Calculates total_frames, processed_frames, refined_frames, mean_confidence |
| `_query_frames_by_timestamp(flight_id, since)` | Queries frame_results with updated_at > since filter |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_get_flight_results_returns_all_frames` | All frames for flight_id returned |
| `test_get_flight_results_includes_statistics` | FlightStatistics computed correctly |
| `test_get_flight_results_empty_flight` | Returns empty frames list with zero statistics |
| `test_get_flight_results_performance` | < 200ms for 2000 frames |
| `test_get_changed_frames_returns_modified` | Only frames with updated_at > since returned |
| `test_get_changed_frames_empty_result` | Returns empty list when no changes |
| `test_get_changed_frames_includes_refined` | Refined frames included in changed set |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_incremental_update_after_disconnect` | Client reconnects → get_changed_frames returns frames since disconnect |
| `test_get_results_matches_stored_data` | Retrieved results match what was persisted via update_frame_result |
_docs/02_components/14_result_manager/14.03_feature_batch_refinement_updates.md
+48
View File
@@ -0,0 +1,48 @@
# Feature: Batch Refinement Updates
## Description
Handles batch updates for frames that have been retrospectively improved through factor graph optimization or chunk merging. Receives GPS-converted coordinates from F02.2 (which handles ENU→GPS conversion) and updates both persistence and SSE clients.
## Component APIs Implemented
### `mark_refined(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
Updates results for frames refined by factor graph optimization (loop closure, etc.).
### `update_results_after_chunk_merge(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
Updates results for frames affected by chunk merge into main trajectory.
## External Services Used
- **F03 Flight Database**: Batch update frame_results and waypoints within transaction
- **F15 SSE Event Streamer**: `send_refinement()` for each updated frame
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_batch_refinement_transaction(flight_id, refined_results)` | Constructs batch transaction for multiple frame updates |
| `_publish_refinement_events(flight_id, frame_ids)` | Sends SSE refinement events for all updated frames |
## Unit Tests
| Test | Description |
|------|-------------|
| `test_mark_refined_updates_all_frames` | All frames in refined_results updated |
| `test_mark_refined_sets_refined_flag` | refined=True set on all updated frames |
| `test_mark_refined_updates_gps_coordinates` | GPS coordinates match RefinedFrameResult values |
| `test_mark_refined_triggers_sse_per_frame` | F15.send_refinement() called for each frame |
| `test_mark_refined_updates_waypoints` | Waypoint table updated with new positions |
| `test_chunk_merge_updates_all_frames` | All merged frames updated |
| `test_chunk_merge_sets_refined_flag` | refined=True for merged frames |
| `test_chunk_merge_triggers_sse` | SSE events sent for merged frames |
| `test_batch_transaction_atomic` | All updates succeed or all rollback |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_batch_refinement_e2e` | 100 frames processed → 40 refined → all updated in F03 → SSE events sent |
| `test_chunk_merge_e2e` | Chunk merged → affected frames updated → clients notified |
| `test_refined_frames_in_get_results` | get_flight_results returns refined=True for updated frames |
_docs/02_components/14_result_manager/14._component_result_manager.md
+342
View File
@@ -0,0 +1,342 @@
# Result Manager
## Interface Definition
**Interface Name**: `IResultManager`
### Interface Methods
```python
class IResultManager(ABC):
@abstractmethod
def update_frame_result(self, flight_id: str, frame_id: int, result: FrameResult) -> bool:
"""
Atomic update:
1. Saves result to frame_results table.
2. Updates waypoint in waypoints table.
3. All within a single transaction via F03.
"""
pass
@abstractmethod
def publish_waypoint_update(self, flight_id: str, frame_id: int) -> bool:
pass
@abstractmethod
def get_flight_results(self, flight_id: str) -> FlightResults:
pass
@abstractmethod
def mark_refined(self, flight_id: str, refined_results: List[RefinedFrameResult]) -> bool:
"""
Updates results for frames that have been retrospectively improved.
Args:
flight_id: Flight identifier
refined_results: List of RefinedFrameResult containing frame_id and GPS-converted coordinates
(caller F02.2 converts ENU to GPS before calling this method)
"""
pass
@abstractmethod
def get_changed_frames(self, flight_id: str, since: datetime) -> List[int]:
pass
@abstractmethod
def update_results_after_chunk_merge(self, flight_id: str, refined_results: List[RefinedFrameResult]) -> bool:
"""
Updates results for frames affected by chunk merge.
Args:
flight_id: Flight identifier
refined_results: List of RefinedFrameResult with GPS-converted coordinates
(caller F02.2 converts ENU to GPS before calling this method)
"""
pass
```
## Component Description
### Responsibilities
- Result consistency and publishing.
- **Atomic Updates**: Ensures consistency between normalized `waypoints` and denormalized `frame_results` via transaction requests to F03.
### Scope
- Result state management
- Flight Database integration (waypoint storage)
- SSE event triggering
- Incremental update detection
- Result persistence
## API Methods
### `update_frame_result(flight_id: str, frame_id: int, result: FrameResult) -> bool`
**Description**: Persists and publishes the result of a processed frame.
**Called By**:
- Main processing loop (after each frame)
- F10 Factor Graph (after refinement)
**Input**:
```python
flight_id: str
frame_id: int
result: FrameResult:
gps_center: GPSPoint
altitude: float
heading: float
confidence: float
timestamp: datetime
refined: bool
objects: List[ObjectLocation] # From external detector
```
**Output**: `bool` - True if updated
**Processing Flow**:
1. Construct DB transaction:
- Insert/Update `frame_results`.
- Update `waypoints` (latest position).
2. Call `F03.execute_transaction()`.
3. If success: call `F15.send_frame_result()`.
4. Update flight statistics.
**Test Cases**:
1. New frame result → stored and published
2. Refined result → updates existing, marks refined=True
---
### `publish_waypoint_update(flight_id: str, frame_id: int) -> bool`
**Description**: Specifically triggers an update for the waypoint visualization.
**Called By**:
- Internal (after update_frame_result)
**Input**:
```python
flight_id: str
frame_id: int
```
**Output**: `bool` - True if updated successfully
**Processing Flow**:
1. Fetch latest data.
2. Call `F15.send_frame_result()` (or a specific lightweight event).
3. Handle errors (retry if transient)
**Test Cases**:
1. Successful update → Waypoint stored in database
2. Database unavailable → logs error, continues
---
### `get_flight_results(flight_id: str) -> FlightResults`
**Description**: Retrieves all results for a flight.
**Called By**:
- F01 REST API (results endpoint)
- Testing/validation
**Input**: `flight_id: str`
**Output**:
```python
FlightResults:
flight_id: str
frames: List[FrameResult]
statistics: FlightStatistics
```
**Test Cases**:
1. Get all results → returns complete trajectory
---
### `mark_refined(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
**Description**: Updates results for frames that have been retrospectively improved (e.g., after loop closure or chunk merge).
**Called By**:
- F02.2 Flight Processing Engine (after factor graph optimization)
**Input**:
```python
flight_id: str
refined_results: List[RefinedFrameResult] # GPS-converted results from F02.2
```
**Output**: `bool`
**Important**: F14 does NOT call F10 or F13. The caller (F02.2) performs the following steps before calling mark_refined():
1. F02.2 gets refined poses from F10.get_trajectory(flight_id) (ENU coordinates)
2. F02.2 converts ENU to GPS via F13.enu_to_gps(flight_id, enu_tuple)
3. F02.2 constructs RefinedFrameResult objects with GPS coordinates
4. F02.2 calls F14.mark_refined() with the GPS-converted results
**Processing Flow**:
1. For each refined_result in refined_results:
- Extract frame_id, gps_center, confidence from RefinedFrameResult
- Update result with refined=True via F03 Flight Database (part of transaction)
- Update waypoint via F03 Flight Database (part of transaction)
- Call F15 SSE Event Streamer.send_refinement()
**Test Cases**:
1. Batch refinement → all frames updated and published
2. GPS coordinates match converted values
---
### `get_changed_frames(flight_id: str, since: datetime) -> List[int]`
**Description**: Gets frames changed since timestamp (for incremental updates).
**Called By**:
- F15 SSE Event Streamer (for reconnection replay)
**Input**:
```python
flight_id: str
since: datetime
```
**Output**: `List[int]` - Frame IDs changed since timestamp
**Test Cases**:
1. Get changes → returns only modified frames
2. No changes → returns empty list
---
### `update_results_after_chunk_merge(flight_id: str, refined_results: List[RefinedFrameResult]) -> bool`
**Description**: Updates results for frames affected by chunk merging into main trajectory.
**Called By**:
- F02.2 Flight Processing Engine (after chunk merge completes)
**Input**:
```python
flight_id: str
refined_results: List[RefinedFrameResult] # GPS-converted results for merged frames
```
**Output**: `bool` - True if updated successfully
**Important**: F14 does NOT call F10, F13, or F11. F02.2 coordinates the chunk merge workflow:
1. F11 returns merge result to F02.2 (direct call return, not event)
2. F02.2 gets updated poses from F10.get_trajectory(flight_id) (ENU coordinates)
3. F02.2 converts ENU to GPS via F13.enu_to_gps(flight_id, enu_tuple)
4. F02.2 constructs RefinedFrameResult objects
5. F02.2 calls F14.update_results_after_chunk_merge() with GPS-converted results
**Processing Flow**:
1. For each refined_result in refined_results:
- Extract frame_id, gps_center, confidence from RefinedFrameResult
- Update result with refined=True via F03 Flight Database (part of transaction)
- Update waypoint via F03 Flight Database (part of transaction)
- Call F15 SSE Event Streamer.send_refinement()
**Test Cases**:
1. **Chunk merge updates**: All merged frames updated and published
2. **GPS accuracy**: Updated GPS matches optimized poses
## Integration Tests
### Test 1: Per-Frame Processing
1. Process frame 237
2. update_frame_result() → stores result
3. Verify publish_waypoint_update() called (F03.update_waypoint())
4. Verify F15 SSE event sent
### Test 2: Batch Refinement
1. Process 100 frames
2. Factor graph refines frames 10-50
3. mark_refined([10-50]) → updates all
4. Verify Flight Database updated (F03.batch_update_waypoints())
5. Verify SSE refinement events sent
### Test 3: Incremental Updates
1. Process frames 1-100
2. Client disconnects at frame 50
3. Client reconnects
4. get_changed_frames(since=frame_50_time)
5. Client receives frames 51-100
## Non-Functional Requirements
### Performance
- **update_frame_result**: < 50ms
- **publish_waypoint_update**: < 100ms (non-blocking)
- **get_flight_results**: < 200ms for 2000 frames
### Reliability
- Result persistence survives crashes
- Guaranteed at-least-once delivery to Flight Database
- Idempotent updates
## Dependencies
### Internal Components
- **F03 Flight Database**: Must support transactional updates.
- **F15 SSE Event Streamer**: For real-time result streaming.
**Note**: F14 does NOT depend on F10, F13, or F11. The caller (F02.2) coordinates with those components and provides GPS-converted results to F14. This ensures unidirectional data flow and eliminates circular dependencies.
### External Dependencies
- None
## Data Models
### FrameResult
```python
class ObjectLocation(BaseModel):
object_id: str
pixel: Tuple[float, float]
gps: GPSPoint
class_name: str
confidence: float
class FrameResult(BaseModel):
frame_id: int
gps_center: GPSPoint
altitude: float
heading: float
confidence: float
timestamp: datetime
refined: bool
objects: List[ObjectLocation]
updated_at: datetime
```
### RefinedFrameResult
```python
class RefinedFrameResult(BaseModel):
"""
GPS-converted frame result provided by F02.2 after coordinate transformation.
F02.2 converts ENU poses from F10 to GPS using F13 before passing to F14.
"""
frame_id: int
gps_center: GPSPoint # GPS coordinates (converted from ENU by F02.2)
confidence: float
heading: Optional[float] = None # Updated heading if available
```
### FlightResults
```python
class FlightStatistics(BaseModel):
total_frames: int
processed_frames: int
refined_frames: int
mean_confidence: float
processing_time: float
class FlightResults(BaseModel):
flight_id: str
frames: List[FrameResult]
statistics: FlightStatistics
```
_docs/02_components/15_sse_event_streamer/15.01_feature_connection_lifecycle_management.md
+61
View File
@@ -0,0 +1,61 @@
# Feature: Connection Lifecycle Management
## Description
Manages the lifecycle of SSE (Server-Sent Events) connections including creation, tracking, health monitoring, and graceful closure. Supports multiple concurrent client connections per flight with proper resource management.
## Component APIs Implemented
### `create_stream(flight_id: str, client_id: str) -> StreamConnection`
Establishes a new SSE connection for a client subscribing to a flight's events.
### `close_stream(flight_id: str, client_id: str) -> bool`
Closes an existing SSE connection and cleans up associated resources.
### `get_active_connections(flight_id: str) -> int`
Returns the count of active SSE connections for a given flight.
## External Tools and Services
- **FastAPI/Starlette**: SSE response streaming support
- **asyncio**: Asynchronous connection handling and concurrent stream management
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_register_connection(flight_id, client_id, stream)` | Adds connection to internal registry |
| `_unregister_connection(flight_id, client_id)` | Removes connection from registry |
| `_get_connections_for_flight(flight_id)` | Returns all active streams for a flight |
| `_generate_stream_id()` | Creates unique identifier for stream |
| `_handle_client_disconnect(flight_id, client_id)` | Cleanup on unexpected disconnect |
## Data Structures
```python
# Connection registry: Dict[flight_id, Dict[client_id, StreamConnection]]
# Allows O(1) lookup by flight and client
```
## Unit Tests
| Test | Description |
|------|-------------|
| `test_create_stream_returns_valid_connection` | Verify StreamConnection has all required fields |
| `test_create_stream_unique_stream_ids` | Multiple streams get unique identifiers |
| `test_close_stream_removes_from_registry` | Connection no longer tracked after close |
| `test_close_stream_nonexistent_returns_false` | Graceful handling of invalid close |
| `test_get_active_connections_empty` | Returns 0 when no connections exist |
| `test_get_active_connections_multiple_clients` | Correctly counts concurrent connections |
| `test_multiple_flights_isolated` | Connections for different flights don't interfere |
| `test_same_client_reconnect` | Client can reconnect with same client_id |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_concurrent_client_connections` | 10 clients connect simultaneously to same flight |
| `test_connection_survives_idle_period` | Connection remains open during inactivity |
| `test_client_disconnect_detection` | System detects when client drops connection |
| `test_connection_cleanup_on_flight_completion` | All connections closed when flight ends |
_docs/02_components/15_sse_event_streamer/15.02_feature_event_broadcasting.md
+83
View File
@@ -0,0 +1,83 @@
# Feature: Event Broadcasting
## Description
Broadcasts various event types to all connected clients for a flight. Handles event formatting per SSE protocol, buffering events for temporarily disconnected clients, and replay of missed events on reconnection. Includes heartbeat mechanism for connection health.
## Component APIs Implemented
### `send_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
Broadcasts `frame_processed` event with GPS coordinates, confidence, and metadata.
### `send_search_progress(flight_id: str, search_status: SearchStatus) -> bool`
Broadcasts `search_expanded` event during failure recovery search operations.
### `send_user_input_request(flight_id: str, request: UserInputRequest) -> bool`
Broadcasts `user_input_needed` event when processing requires user intervention.
### `send_refinement(flight_id: str, frame_id: int, updated_result: FrameResult) -> bool`
Broadcasts `frame_refined` event when factor graph optimization improves a position.
### `send_heartbeat(flight_id: str) -> bool`
Sends keepalive ping to all clients to maintain connection health.
## External Tools and Services
- **FastAPI/Starlette**: SSE response streaming
- **asyncio**: Async event delivery to multiple clients
- **JSON**: Event data serialization
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_format_sse_event(event_type, event_id, data)` | Formats data per SSE protocol spec |
| `_broadcast_to_flight(flight_id, event)` | Sends event to all clients of a flight |
| `_buffer_event(flight_id, event)` | Stores event for replay to reconnecting clients |
| `_get_buffered_events(flight_id, last_event_id)` | Retrieves events after given ID for replay |
| `_prune_event_buffer(flight_id)` | Removes old events beyond retention window |
| `_generate_event_id(frame_id, event_type)` | Creates sequential event ID for ordering |
## Event Types
| Event | Format |
|-------|--------|
| `frame_processed` | `{frame_id, gps, altitude, confidence, heading, timestamp}` |
| `frame_refined` | `{frame_id, gps, refined: true}` |
| `search_expanded` | `{frame_id, grid_size, status}` |
| `user_input_needed` | `{request_id, frame_id, candidate_tiles}` |
| `:heartbeat` | SSE comment (no data payload) |
## Buffering Strategy
- Events buffered per flight with configurable retention (default: 1000 events or 5 minutes)
- Events keyed by sequential ID for replay ordering
- On reconnection with `last_event_id`, replay all events after that ID
## Unit Tests
| Test | Description |
|------|-------------|
| `test_send_frame_result_formats_correctly` | Event matches expected JSON structure |
| `test_send_refinement_includes_refined_flag` | Refined events marked appropriately |
| `test_send_search_progress_valid_status` | Status field correctly populated |
| `test_send_user_input_request_has_request_id` | Request includes unique identifier |
| `test_send_heartbeat_sse_comment_format` | Heartbeat uses `:heartbeat` comment format |
| `test_buffer_event_stores_in_order` | Events retrievable in sequence |
| `test_buffer_pruning_removes_old_events` | Buffer doesn't grow unbounded |
| `test_replay_from_last_event_id` | Correct subset returned for replay |
| `test_broadcast_returns_false_no_connections` | Graceful handling when no clients |
| `test_event_id_generation_sequential` | IDs increase monotonically |
## Integration Tests
| Test | Description |
|------|-------------|
| `test_100_frames_all_received_in_order` | Stream 100 frame results, verify completeness |
| `test_reconnection_replay` | Disconnect after 50 events, reconnect, receive 51-100 |
| `test_multiple_event_types_interleaved` | Mix of frame_processed and frame_refined events |
| `test_user_input_flow_roundtrip` | Send request, verify client receives |
| `test_heartbeat_every_30_seconds` | Verify keepalive timing |
| `test_event_latency_under_500ms` | Measure generation-to-receipt time |
| `test_high_throughput_100_events_per_second` | Sustained event rate handling |
_docs/02_components/15_sse_event_streamer/15._component_sse_event_streamer.md
+291
View File
@@ -0,0 +1,291 @@
# SSE Event Streamer
## Interface Definition
**Interface Name**: `ISSEEventStreamer`
### Interface Methods
```python
class ISSEEventStreamer(ABC):
@abstractmethod
def create_stream(self, flight_id: str, client_id: str) -> StreamConnection:
pass
@abstractmethod
def send_frame_result(self, flight_id: str, frame_result: FrameResult) -> bool:
pass
@abstractmethod
def send_search_progress(self, flight_id: str, search_status: SearchStatus) -> bool:
pass
@abstractmethod
def send_user_input_request(self, flight_id: str, request: UserInputRequest) -> bool:
pass
@abstractmethod
def send_refinement(self, flight_id: str, frame_id: int, updated_result: FrameResult) -> bool:
pass
@abstractmethod
def send_heartbeat(self, flight_id: str) -> bool:
pass
@abstractmethod
def close_stream(self, flight_id: str, client_id: str) -> bool:
pass
@abstractmethod
def get_active_connections(self, flight_id: str) -> int:
pass
```
## Component Description
### Responsibilities
- Real-time communication with clients.
- Buffering events for disconnected clients.
### Callers
- **F02.1 Flight Lifecycle Manager**: Calls `create_stream` (delegated from F01).
- **F02.2 Flight Processing Engine**: Calls `send_user_input_request`, `send_search_progress`.
- **F14 Result Manager**: Calls `send_frame_result`, `send_refinement`.
### Consistency Fix
- Previously listed F11 as caller. **Correction**: F11 returns request objects to F02.2. **F02.2 is the sole caller** for user input requests and search status updates. F11 has no dependencies on F15.
### Scope
- SSE protocol implementation
- Event formatting and sending
- Connection management
- Client reconnection handling
- Multiple concurrent streams per flight
## API Methods
### `create_stream(flight_id: str, client_id: str) -> StreamConnection`
**Description**: Establishes a server-sent events connection.
**Called By**: F01 REST API (GET /stream endpoint)
**Output**:
```python
StreamConnection:
stream_id: str
flight_id: str
client_id: str
last_event_id: Optional[str]
```
**Event Types**:
- `frame_processed`
- `frame_refined`
- `search_expanded`
- `user_input_needed`
- `processing_blocked`
- `route_api_updated`
- `route_completed`
**Test Cases**:
1. Create stream → client receives keepalive pings
2. Multiple clients → each gets own stream
---
### `send_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
**Description**: Sends frame_processed event.
**Called By**: F14 Result Manager
**Event Format**:
```json
{
"event": "frame_processed",
"id": "frame_237",
"data": {
"frame_id": 237,
"gps": {"lat": 48.123, "lon": 37.456},
"altitude": 800.0,
"confidence": 0.95,
"heading": 87.3,
"timestamp": "2025-11-24T10:30:00Z"
}
}
```
**Test Cases**:
1. Send event → all clients receive
2. Client disconnected → event buffered for replay
---
### `send_search_progress(flight_id: str, search_status: SearchStatus) -> bool`
**Description**: Sends search_expanded event.
**Called By**: F02.2 Flight Processing Engine (via F11 status return)
**Event Format**:
```json
{
"event": "search_expanded",
"data": {
"frame_id": 237,
"grid_size": 9,
"status": "searching"
}
}
```
---
### `send_user_input_request(flight_id: str, request: UserInputRequest) -> bool`
**Description**: Sends user_input_needed event.
**Called By**: F02.2 Flight Processing Engine (via F11 request object)
**Event Format**:
```json
{
"event": "user_input_needed",
"data": {
"request_id": "uuid",
"frame_id": 237,
"candidate_tiles": [...]
}
}
```
---
### `send_refinement(flight_id: str, frame_id: int, updated_result: FrameResult) -> bool`
**Description**: Sends frame_refined event.
**Called By**: F14 Result Manager
**Event Format**:
```json
{
"event": "frame_refined",
"data": {
"frame_id": 237,
"gps": {"lat": 48.1235, "lon": 37.4562},
"refined": true
}
}
```
---
### `send_heartbeat(flight_id: str) -> bool`
**Description**: Sends heartbeat/keepalive to all clients subscribed to flight.
**Called By**:
- Background heartbeat task (every 30 seconds)
- F02.2 Flight Processing Engine (periodically during processing)
**Event Format**:
```
:heartbeat
```
**Behavior**:
- Sends SSE comment (`:heartbeat`) which doesn't trigger event handlers
- Keeps TCP connection alive
- Client can use to detect connection health
**Test Cases**:
1. Send heartbeat → all clients receive ping
2. Client timeout → connection marked stale
---
### `close_stream(flight_id: str, client_id: str) -> bool`
**Description**: Closes SSE connection.
**Called By**: F01 REST API (on client disconnect)
---
### `get_active_connections(flight_id: str) -> int`
**Description**: Returns count of active SSE connections for a flight.
**Called By**:
- F02.2 Flight Processing Engine (monitoring)
- Admin tools
**Test Cases**:
1. No connections → returns 0
2. 5 clients connected → returns 5
## Integration Tests
### Test 1: Real-Time Streaming
1. Client connects
2. Process 100 frames
3. Client receives 100 frame_processed events
4. Verify order and completeness
### Test 2: Reconnection with Replay
1. Client connects
2. Process 50 frames
3. Client disconnects
4. Process 50 more frames
5. Client reconnects with last_event_id
6. Client receives frames 51-100
### Test 3: User Input Flow
1. Processing blocks
2. send_user_input_request()
3. Client receives event
4. Client responds with fix
5. Processing resumes
## Non-Functional Requirements
### Performance
- **Event latency**: < 500ms from generation to client
- **Throughput**: 100 events/second
- **Concurrent connections**: 1000+ clients
### Reliability
- Event buffering for disconnected clients
- Automatic reconnection support
- Keepalive pings every 30 seconds
## Dependencies
### Internal Components
- None (receives calls from other components)
### External Dependencies
- **FastAPI** or **Flask** SSE support
- **asyncio**: For async event streaming
## Data Models
### StreamConnection
```python
class StreamConnection(BaseModel):
stream_id: str
flight_id: str
client_id: str
created_at: datetime
last_event_id: Optional[str]
```
### SSEEvent
```python
class SSEEvent(BaseModel):
event: str
id: Optional[str]
data: Dict[str, Any]
```
_docs/02_components/16_model_manager/16.01_feature_model_lifecycle_management.md
+63
View File
@@ -0,0 +1,63 @@
# Feature: Model Lifecycle Management
## Description
Manages the complete lifecycle of ML models including loading, caching, warmup, and unloading. Handles all four models (SuperPoint, LightGlue, DINOv2, LiteSAM) with support for multiple formats (TensorRT, ONNX, PyTorch).
## Component APIs Implemented
### `load_model(model_name: str, model_format: str) -> bool`
- Loads model in specified format
- Checks if model already loaded (cache hit)
- Initializes inference engine
- Triggers warmup
- Caches for reuse
### `warmup_model(model_name: str) -> bool`
- Warms up model with dummy input
- Initializes CUDA kernels
- Pre-allocates GPU memory
- Ensures first real inference is fast
## External Tools and Services
- **TensorRT**: Loading TensorRT engine files
- **ONNX Runtime**: Loading ONNX models
- **PyTorch**: Loading PyTorch model weights (optional)
- **CUDA**: GPU memory allocation
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_check_model_cache(model_name)` | Check if model already loaded |
| `_load_tensorrt_engine(path)` | Load TensorRT engine from file |
| `_load_onnx_model(path)` | Load ONNX model from file |
| `_allocate_gpu_memory(model)` | Allocate GPU memory for model |
| `_create_dummy_input(model_name)` | Create appropriate dummy input for warmup |
| `_cache_model(model_name, engine)` | Store loaded model in cache |
## Unit Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| UT-16.01-01 | Load TensorRT model | Model loaded, returns True |
| UT-16.01-02 | Load ONNX model | Model loaded, returns True |
| UT-16.01-03 | Load already cached model | Returns True immediately (no reload) |
| UT-16.01-04 | Load invalid model name | Returns False, logs error |
| UT-16.01-05 | Load invalid model path | Returns False, logs error |
| UT-16.01-06 | Warmup SuperPoint | CUDA kernels initialized |
| UT-16.01-07 | Warmup LightGlue | CUDA kernels initialized |
| UT-16.01-08 | Warmup DINOv2 | CUDA kernels initialized |
| UT-16.01-09 | Warmup LiteSAM | CUDA kernels initialized |
| UT-16.01-10 | Warmup unloaded model | Returns False |
## Integration Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| IT-16.01-01 | Load all 4 models sequentially | All models loaded successfully |
| IT-16.01-02 | Load + warmup cycle for each model | All models ready for inference |
| IT-16.01-03 | GPU memory allocation after loading all models | ~4GB GPU memory used |
| IT-16.01-04 | First inference after warmup | Latency within target range |
_docs/02_components/16_model_manager/16.02_feature_inference_engine_provisioning.md
+68
View File
@@ -0,0 +1,68 @@
# Feature: Inference Engine Provisioning
## Description
Provides inference engines to consuming components and handles TensorRT optimization with automatic ONNX fallback. Ensures consistent inference interface regardless of underlying format and meets <5s processing requirement through acceleration.
## Component APIs Implemented
### `get_inference_engine(model_name: str) -> InferenceEngine`
- Returns inference engine for specified model
- Engine provides unified `infer(input: np.ndarray) -> np.ndarray` interface
- Consumers: F07 Sequential VO (SuperPoint, LightGlue), F08 GPR (DINOv2), F09 Metric Refinement (LiteSAM)
### `optimize_to_tensorrt(model_name: str, onnx_path: str) -> str`
- Converts ONNX model to TensorRT engine
- Applies FP16 precision (2-3x speedup)
- Performs graph fusion and kernel optimization
- One-time conversion, result cached
### `fallback_to_onnx(model_name: str) -> bool`
- Detects TensorRT failure
- Loads ONNX model as fallback
- Logs warning for monitoring
- Ensures system continues functioning
## External Tools and Services
- **TensorRT**: Model optimization and inference
- **ONNX Runtime**: Fallback inference
- **CUDA**: GPU execution
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_get_cached_engine(model_name)` | Retrieve engine from cache |
| `_build_tensorrt_engine(onnx_path)` | Build TensorRT engine from ONNX |
| `_apply_fp16_optimization(builder)` | Enable FP16 precision in TensorRT |
| `_cache_tensorrt_engine(model_name, path)` | Save TensorRT engine to disk |
| `_detect_tensorrt_failure(error)` | Determine if error requires ONNX fallback |
| `_create_inference_wrapper(engine, format)` | Create unified InferenceEngine interface |
## Unit Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| UT-16.02-01 | Get SuperPoint engine | Returns valid InferenceEngine |
| UT-16.02-02 | Get LightGlue engine | Returns valid InferenceEngine |
| UT-16.02-03 | Get DINOv2 engine | Returns valid InferenceEngine |
| UT-16.02-04 | Get LiteSAM engine | Returns valid InferenceEngine |
| UT-16.02-05 | Get unloaded model engine | Raises error or returns None |
| UT-16.02-06 | InferenceEngine.infer() with valid input | Returns features array |
| UT-16.02-07 | Optimize ONNX to TensorRT | TensorRT engine file created |
| UT-16.02-08 | TensorRT optimization with FP16 | Engine uses FP16 precision |
| UT-16.02-09 | Fallback to ONNX on TensorRT failure | ONNX model loaded, returns True |
| UT-16.02-10 | Fallback logs warning | Warning logged |
## Integration Tests
| Test | Description | Expected Result |
|------|-------------|-----------------|
| IT-16.02-01 | SuperPoint inference 100 iterations | Avg latency ~15ms (TensorRT) or ~50ms (ONNX) |
| IT-16.02-02 | LightGlue inference 100 iterations | Avg latency ~50ms (TensorRT) or ~150ms (ONNX) |
| IT-16.02-03 | DINOv2 inference 100 iterations | Avg latency ~150ms (TensorRT) or ~500ms (ONNX) |
| IT-16.02-04 | LiteSAM inference 100 iterations | Avg latency ~60ms (TensorRT) or ~200ms (ONNX) |
| IT-16.02-05 | Simulate TensorRT failure → ONNX fallback | System continues with ONNX |
| IT-16.02-06 | Full pipeline: optimize → load → infer | End-to-end works |
_docs/02_components/16_model_manager/16._component_model_manager.md
+224
View File
@@ -0,0 +1,224 @@
# Model Manager
## Interface Definition
**Interface Name**: `IModelManager`
### Interface Methods
```python
class IModelManager(ABC):
@abstractmethod
def load_model(self, model_name: str, model_format: str) -> bool:
pass
@abstractmethod
def get_inference_engine(self, model_name: str) -> InferenceEngine:
pass
@abstractmethod
def optimize_to_tensorrt(self, model_name: str, onnx_path: str) -> str:
pass
@abstractmethod
def fallback_to_onnx(self, model_name: str) -> bool:
pass
@abstractmethod
def warmup_model(self, model_name: str) -> bool:
pass
```
## Component Description
### Responsibilities
- Load ML models (TensorRT primary, ONNX fallback)
- Manage model lifecycle (loading, unloading, warmup)
- Provide inference engines for:
- SuperPoint (feature extraction)
- LightGlue (feature matching)
- DINOv2 (global descriptors)
- LiteSAM (cross-view matching)
- Handle TensorRT optimization and ONNX fallback
- Ensure <5s processing requirement through acceleration
### Scope
- Model loading and caching
- TensorRT optimization
- ONNX fallback handling
- Inference engine abstraction
- GPU memory management
## API Methods
### `load_model(model_name: str, model_format: str) -> bool`
**Description**: Loads model in specified format.
**Called By**: F02.1 Flight Lifecycle Manager (during system initialization)
**Input**:
```python
model_name: str # "SuperPoint", "LightGlue", "DINOv2", "LiteSAM"
model_format: str # "tensorrt", "onnx", "pytorch"
```
**Output**: `bool` - True if loaded
**Processing Flow**:
1. Check if model already loaded
2. Load model file
3. Initialize inference engine
4. Warm up model
5. Cache for reuse
**Test Cases**:
1. Load TensorRT model → succeeds
2. TensorRT unavailable → fallback to ONNX
3. Load all 4 models → all succeed
---
### `get_inference_engine(model_name: str) -> InferenceEngine`
**Description**: Gets inference engine for a model.
**Called By**:
- F07 Sequential VO (SuperPoint, LightGlue)
- F08 Global Place Recognition (DINOv2)
- F09 Metric Refinement (LiteSAM)
**Output**:
```python
InferenceEngine:
model_name: str
format: str
infer(input: np.ndarray) -> np.ndarray
```
**Test Cases**:
1. Get SuperPoint engine → returns engine
2. Call infer() → returns features
---
### `optimize_to_tensorrt(model_name: str, onnx_path: str) -> str`
**Description**: Converts ONNX model to TensorRT for acceleration.
**Called By**: System initialization (one-time)
**Input**:
```python
model_name: str
onnx_path: str # Path to ONNX model
```
**Output**: `str` - Path to TensorRT engine
**Processing Details**:
- FP16 precision (2-3x speedup)
- Graph fusion and kernel optimization
- One-time conversion, cached for reuse
**Test Cases**:
1. Convert ONNX to TensorRT → engine created
2. Load TensorRT engine → inference faster than ONNX
---
### `fallback_to_onnx(model_name: str) -> bool`
**Description**: Falls back to ONNX if TensorRT fails.
**Called By**: Internal (during load_model)
**Processing Flow**:
1. Detect TensorRT failure
2. Load ONNX model
3. Log warning
4. Continue with ONNX
**Test Cases**:
1. TensorRT fails → ONNX loaded automatically
2. System continues functioning
---
### `warmup_model(model_name: str) -> bool`
**Description**: Warms up model with dummy input.
**Called By**: Internal (after load_model)
**Purpose**: Initialize CUDA kernels, allocate GPU memory
**Test Cases**:
1. Warmup → first real inference fast
## Integration Tests
### Test 1: Model Loading
1. load_model("SuperPoint", "tensorrt")
2. load_model("LightGlue", "tensorrt")
3. load_model("DINOv2", "tensorrt")
4. load_model("LiteSAM", "tensorrt")
5. Verify all loaded
### Test 2: Inference Performance
1. Get inference engine
2. Run inference 100 times
3. Measure average latency
4. Verify meets performance targets
### Test 3: Fallback Scenario
1. Simulate TensorRT failure
2. Verify fallback to ONNX
3. Verify inference still works
## Non-Functional Requirements
### Performance
- **SuperPoint**: ~15ms (TensorRT), ~50ms (ONNX)
- **LightGlue**: ~50ms (TensorRT), ~150ms (ONNX)
- **DINOv2**: ~150ms (TensorRT), ~500ms (ONNX)
- **LiteSAM**: ~60ms (TensorRT), ~200ms (ONNX)
### Memory
- GPU memory: ~4GB for all 4 models
### Reliability
- Graceful fallback to ONNX
- Automatic retry on transient errors
## Dependencies
### External Dependencies
- **TensorRT**: NVIDIA inference optimization
- **ONNX Runtime**: ONNX inference
- **PyTorch**: Model weights (optional)
- **CUDA**: GPU acceleration
## Data Models
### InferenceEngine
```python
class InferenceEngine(ABC):
model_name: str
format: str
@abstractmethod
def infer(self, input: np.ndarray) -> np.ndarray:
pass
```
### ModelConfig
```python
class ModelConfig(BaseModel):
model_name: str
model_path: str
format: str
precision: str # "fp16", "fp32"
warmup_iterations: int = 3
```
_docs/02_components/17_configuration_manager/17.01_feature_system_configuration.md
+61
View File
@@ -0,0 +1,61 @@
# Feature: System Configuration
## Description
Handles loading, validation, and runtime management of system-wide configuration including camera parameters, operational area bounds, model paths, database connections, and API endpoints.
## Component APIs Implemented
- `load_config(config_path: str) -> SystemConfig`
- `validate_config(config: SystemConfig) -> ValidationResult`
- `get_camera_params(camera_id: Optional[str] = None) -> CameraParameters`
- `update_config(section: str, key: str, value: Any) -> bool`
## External Tools and Services
- **PyYAML**: Configuration file parsing (YAML format)
- **pydantic**: Data model validation and serialization
- **os/pathlib**: File system operations for config file access
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_parse_yaml_file` | Reads and parses YAML configuration file |
| `_apply_defaults` | Applies default values for missing configuration keys |
| `_validate_camera_params` | Validates camera parameters are within sensible bounds |
| `_validate_paths` | Checks that configured paths exist |
| `_validate_operational_area` | Validates operational area bounds are valid coordinates |
| `_build_camera_matrix` | Constructs camera intrinsic matrix from parameters |
| `_get_cached_config` | Returns cached configuration to avoid repeated file reads |
## Unit Tests
| Test Case | Description |
|-----------|-------------|
| `test_load_config_valid_yaml` | Load valid YAML config file, verify SystemConfig returned |
| `test_load_config_missing_file_uses_defaults` | Missing config file returns default configuration |
| `test_load_config_invalid_yaml_raises_error` | Malformed YAML raises appropriate error |
| `test_load_config_partial_uses_defaults` | Partial config merges with defaults |
| `test_validate_config_valid` | Valid configuration passes validation |
| `test_validate_config_invalid_focal_length` | Focal length <= 0 fails validation |
| `test_validate_config_invalid_sensor_size` | Sensor dimensions <= 0 fails validation |
| `test_validate_config_invalid_resolution` | Resolution <= 0 fails validation |
| `test_validate_config_invalid_operational_area` | Invalid lat/lon bounds fail validation |
| `test_validate_config_missing_paths` | Non-existent paths fail validation |
| `test_get_camera_params_default` | Get default camera returns valid parameters |
| `test_get_camera_params_specific_camera` | Get specific camera ID returns correct parameters |
| `test_get_camera_params_unknown_camera` | Unknown camera ID raises error |
| `test_update_config_valid_key` | Update existing key succeeds |
| `test_update_config_invalid_section` | Update non-existent section fails |
| `test_update_config_invalid_key` | Update non-existent key fails |
| `test_update_config_type_mismatch` | Update with wrong value type fails |
## Integration Tests
| Test Case | Description |
|-----------|-------------|
| `test_load_and_validate_full_config` | Load config file, validate, verify all sections accessible |
| `test_config_persistence_after_update` | Update config at runtime, verify changes persist in memory |
| `test_camera_params_consistency` | Camera params from get_camera_params match loaded config |
_docs/02_components/17_configuration_manager/17.02_feature_flight_configuration.md
+50
View File
@@ -0,0 +1,50 @@
# Feature: Flight Configuration
## Description
Manages flight-specific configuration including operational altitude, frame spacing, and camera parameters per flight. Provides persistence and retrieval of flight configurations created during flight initialization.
## Component APIs Implemented
- `get_flight_config(flight_id: str) -> FlightConfig`
- `save_flight_config(flight_id: str, config: FlightConfig) -> bool`
- `get_operational_altitude(flight_id: str) -> float`
- `get_frame_spacing(flight_id: str) -> float`
## External Tools and Services
- **F03 Flight Database**: Persistence layer for flight-specific configuration storage
## Internal Methods
| Method | Purpose |
|--------|---------|
| `_build_flight_config` | Constructs FlightConfig from stored data |
| `_validate_flight_config` | Validates FlightConfig before persistence |
| `_cache_flight_config` | Caches flight config in memory for fast retrieval |
| `_invalidate_cache` | Clears cached flight config when updated |
## Unit Tests
| Test Case | Description |
|-----------|-------------|
| `test_get_flight_config_existing` | Get config for existing flight returns FlightConfig |
| `test_get_flight_config_nonexistent` | Get config for non-existent flight raises error |
| `test_save_flight_config_valid` | Save valid FlightConfig returns True |
| `test_save_flight_config_invalid_flight_id` | Save with empty flight_id fails |
| `test_save_flight_config_invalid_config` | Save with invalid config data fails |
| `test_get_operational_altitude_existing` | Get altitude for existing flight returns value |
| `test_get_operational_altitude_nonexistent` | Get altitude for non-existent flight raises error |
| `test_get_operational_altitude_range` | Verify altitude within expected range (100-500m) |
| `test_get_frame_spacing_existing` | Get frame spacing returns expected distance |
| `test_get_frame_spacing_default` | Frame spacing returns reasonable default (~100m) |
## Integration Tests
| Test Case | Description |
|-----------|-------------|
| `test_save_and_retrieve_flight_config` | Save flight config via save_flight_config, retrieve via get_flight_config, verify data matches |
| `test_flight_config_persists_to_database` | Save flight config, verify stored in F03 Flight Database |
| `test_altitude_and_spacing_consistency` | Flight config altitude matches get_operational_altitude result |
| `test_multiple_flights_isolation` | Multiple flights have independent configurations |
_docs/02_components/17_configuration_manager/17._component_configuration_manager.md
+283
View File
@@ -0,0 +1,283 @@
# Configuration Manager
## Interface Definition
**Interface Name**: `IConfigurationManager`
### Interface Methods
```python
class IConfigurationManager(ABC):
@abstractmethod
def load_config(self, config_path: str) -> SystemConfig:
pass
@abstractmethod
def get_camera_params(self, camera_id: Optional[str] = None) -> CameraParameters:
pass
@abstractmethod
def validate_config(self, config: SystemConfig) -> ValidationResult:
pass
@abstractmethod
def get_flight_config(self, flight_id: str) -> FlightConfig:
pass
@abstractmethod
def update_config(self, section: str, key: str, value: Any) -> bool:
pass
@abstractmethod
def get_operational_altitude(self, flight_id: str) -> float:
"""Returns predefined operational altitude for the flight (NOT from EXIF)."""
pass
@abstractmethod
def get_frame_spacing(self, flight_id: str) -> float:
"""Returns expected distance between consecutive frames in meters."""
pass
@abstractmethod
def save_flight_config(self, flight_id: str, config: FlightConfig) -> bool:
"""Persists flight-specific configuration."""
pass
```
## Component Description
### Responsibilities
- Load system configuration from files/environment
- Provide camera parameters (focal length, sensor size, resolution)
- Manage flight-specific configurations
- Validate configuration integrity
- Support configuration updates at runtime
### Scope
- System-wide configuration
- Camera parameter management
- Operational area bounds
- Model paths and settings
- Database connections
- API endpoints
## API Methods
### `load_config(config_path: str) -> SystemConfig`
**Description**: Loads system configuration.
**Called By**: System startup
**Input**: `config_path: str` - Path to config file (YAML/JSON)
**Output**: `SystemConfig` - Complete configuration
**Test Cases**:
1. Load valid config → succeeds
2. Missing file → uses defaults
3. Invalid config → raises error
---
### `get_camera_params(camera_id: Optional[str] = None) -> CameraParameters`
**Description**: Gets camera parameters.
**Called By**: All processing components
**Output**:
```python
CameraParameters:
focal_length: float
sensor_width: float
sensor_height: float
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float]
distortion_coefficients: List[float]
```
**Test Cases**:
1. Get default camera → returns params
2. Get specific camera → returns params
---
### `validate_config(config: SystemConfig) -> ValidationResult`
**Description**: Validates configuration.
**Called By**: After load_config
**Validation Rules**:
- Camera parameters sensible
- Paths exist
- Operational area valid
- Database connection string valid
**Test Cases**:
1. Valid config → passes
2. Invalid focal length → fails
---
### `get_flight_config(flight_id: str) -> FlightConfig`
**Description**: Gets flight-specific configuration.
**Called By**: F02.1 Flight Lifecycle Manager
**Output**:
```python
FlightConfig:
camera_params: CameraParameters
altitude: float
operational_area: OperationalArea
```
**Test Cases**:
1. Get flight config → returns params
---
### `update_config(section: str, key: str, value: Any) -> bool`
**Description**: Updates config at runtime.
**Called By**: Admin tools
**Test Cases**:
1. Update value → succeeds
2. Invalid key → fails
---
### `get_operational_altitude(flight_id: str) -> float`
**Description**: Returns predefined operational altitude for the flight in meters. This is the altitude provided during flight creation, NOT extracted from EXIF metadata (images don't have GPS/altitude metadata per problem constraints).
**Called By**:
- F10 Factor Graph Optimizer (for scale resolution)
- H02 GSD Calculator (for GSD computation)
- F09 Metric Refinement (for alignment)
**Input**: `flight_id: str`
**Output**: `float` - Altitude in meters (typically 100-500m)
**Test Cases**:
1. Get existing flight altitude → returns value
2. Non-existent flight → raises error
---
### `get_frame_spacing(flight_id: str) -> float`
**Description**: Returns expected distance between consecutive frames in meters. Used for scale estimation in visual odometry.
**Called By**:
- F10 Factor Graph Optimizer (for expected displacement calculation)
**Input**: `flight_id: str`
**Output**: `float` - Expected frame spacing in meters (typically ~100m)
**Test Cases**:
1. Get frame spacing → returns expected distance
---
### `save_flight_config(flight_id: str, config: FlightConfig) -> bool`
**Description**: Persists flight-specific configuration when a flight is created.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight creation)
**Input**:
```python
flight_id: str
config: FlightConfig
```
**Output**: `bool` - True if saved successfully
**Test Cases**:
1. Save valid config → succeeds
2. Invalid flight_id → fails
## Data Models
### SystemConfig
```python
class SystemConfig(BaseModel):
camera: CameraParameters
operational_area: OperationalArea
models: ModelPaths
database: DatabaseConfig
api: APIConfig
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float]
distortion_coefficients: List[float]
```
### OperationalArea
```python
class OperationalArea(BaseModel):
name: str = "Eastern Ukraine"
min_lat: float = 45.0
max_lat: float = 52.0
min_lon: float = 22.0
max_lon: float = 40.0
```
### RecoveryConfig
```python
class RecoveryConfig(BaseModel):
"""Configuration for failure recovery and progressive search."""
search_grid_sizes: List[int] = [1, 4, 9, 16, 25] # Progressive tile search grid
min_chunk_frames_for_matching: int = 5 # Minimum frames before chunk matching
max_chunk_frames_for_matching: int = 20 # Maximum frames per chunk
user_input_threshold_tiles: int = 25 # Request user input after this many tiles
chunk_matching_interval_seconds: float = 5.0 # Background matching interval
confidence_threshold_good: float = 0.7 # Confidence for good tracking
confidence_threshold_degraded: float = 0.5 # Confidence for degraded tracking
min_inlier_count_good: int = 50 # Inliers for good tracking
min_inlier_count_tracking: int = 20 # Minimum inliers before tracking loss
```
### RotationConfig
```python
class RotationConfig(BaseModel):
"""Configuration for image rotation and heading tracking."""
rotation_step_degrees: float = 30.0 # Degrees per rotation step
litesam_max_rotation_tolerance: float = 45.0 # Max rotation LiteSAM handles
sharp_turn_threshold_degrees: float = 45.0 # Threshold for sharp turn detection
heading_history_size: int = 10 # Number of headings to track
confidence_threshold: float = 0.7 # For accepting rotation match
@property
def rotation_iterations(self) -> int:
"""Number of rotation steps (360 / step_degrees)."""
return int(360 / self.rotation_step_degrees)
```
## Dependencies
### Internal Components
- **F03 Flight Database**: For flight-specific configuration persistence (save_flight_config stores to F03)
### External Dependencies
- **pydantic**: Data model validation
- **PyYAML**: Configuration file parsing
_docs/02_components/astral_next_components_diagram.drawio
+165
View File
@@ -0,0 +1,165 @@
<mxfile host="65bd71144e">
<diagram name="ASTRAL-Next Components" id="astral-next-components">
<mxGraphModel dx="1100" dy="750" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1100" pageHeight="600" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="ASTRAL-Next Component Diagram with Jira Epic Numbers" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=16;fontStyle=1;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="300" y="10" width="500" height="30" as="geometry"/>
</mxCell>
<mxCell id="client" value="Client" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1565C0;strokeColor=#64B5F6;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="20" y="60" width="60" height="35" as="geometry"/>
</mxCell>
<mxCell id="f01" value="F01 API&#10;AZ-112" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#8B1A1A;strokeColor=#EF5350;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f02" value="F02 Flight Processor&#10;AZ-113" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#6A1B9A;strokeColor=#BA68C8;fontColor=#ffffff;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="200" y="55" width="130" height="45" as="geometry"/>
</mxCell>
<mxCell id="f03" value="F03 DB&#10;AZ-114" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="350" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f05" value="F05 Image Pipeline&#10;AZ-116" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="450" y="55" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="satellite" value="Satellite&#10;Provider" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFB300;fontColor=#ffffff;dashed=1;" vertex="1" parent="1">
<mxGeometry x="580" y="55" width="80" height="45" as="geometry"/>
</mxCell>
<mxCell id="f15" value="F15 SSE&#10;AZ-126" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFA726;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="20" y="130" width="70" height="45" as="geometry"/>
</mxCell>
<mxCell id="f08" value="F08 Place Recog&#10;AZ-119" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f11" value="F11 Failure Recovery&#10;AZ-122" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="220" y="130" width="120" height="45" as="geometry"/>
</mxCell>
<mxCell id="f12" value="F12 Chunk Mgr&#10;AZ-123" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#7B1FA2;strokeColor=#BA68C8;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="360" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f16" value="F16 Model Mgr&#10;AZ-127" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="480" y="130" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f06" value="F06 Rotation&#10;AZ-117" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="210" width="90" height="45" as="geometry"/>
</mxCell>
<mxCell id="f07" value="F07 Sequential VO&#10;AZ-118" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="210" y="210" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="f09" value="F09 Metric Refine&#10;AZ-120" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1E88E5;strokeColor=#42A5F5;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="340" y="210" width="110" height="45" as="geometry"/>
</mxCell>
<mxCell id="f17" value="F17 Config Mgr&#10;AZ-128" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#424242;strokeColor=#BDBDBD;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="470" y="210" width="100" height="45" as="geometry"/>
</mxCell>
<mxCell id="f10" value="F10 Factor Graph&#10;AZ-121" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="260" y="290" width="120" height="50" as="geometry"/>
</mxCell>
<mxCell id="f04" value="F04 Satellite Mgr&#10;AZ-115" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#CC6600;strokeColor=#FFB300;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="100" y="290" width="110" height="50" as="geometry"/>
</mxCell>
<mxCell id="f13" value="F13 Coord Transform&#10;AZ-124" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#388E3C;strokeColor=#66BB6A;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="400" y="290" width="120" height="50" as="geometry"/>
</mxCell>
<mxCell id="f14" value="F14 Result Manager&#10;AZ-125" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#E65100;strokeColor=#FFA726;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="250" y="370" width="130" height="45" as="geometry"/>
</mxCell>
<mxCell id="helpers" value="H01-H08 Helpers&#10;AZ-129" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#546E7A;strokeColor=#78909C;fontColor=#ffffff;" parent="1" vertex="1">
<mxGeometry x="540" y="290" width="110" height="50" as="geometry"/>
</mxCell>
<mxCell id="c1" value="HTTP" style="strokeColor=#FFFFFF;fontColor=#ffffff;fontSize=8;" edge="1" parent="1" source="client" target="f01">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c2" value="SSE" style="strokeColor=#FFFFFF;fontColor=#ffffff;fontSize=8;dashed=1;" edge="1" parent="1" source="f15" target="client">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c3" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f01" target="f02">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c4" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f03">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c4b" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f05">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c5" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f11">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c6" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f07">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c7" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f02" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c10" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f08">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c11" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f09">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c12" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f12">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c13" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f11" target="f06">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c14" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f07" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c15" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f09" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c16" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f12" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c17" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f10" target="f13">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c18" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f10" target="f14">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c19" style="strokeColor=#FFFFFF;dashed=1;" edge="1" parent="1" source="f14" target="f15">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c20" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f08" target="f04">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c21" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f04" target="satellite">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="155" y="370"/>
<mxPoint x="620" y="370"/>
</Array>
</mxGeometry>
</mxCell>
<mxCell id="c22" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f07" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c23" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f08" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c24" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f09" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c25" style="strokeColor=#FFFFFF;dashed=1;" edge="1" parent="1" source="f17" target="f16">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c26" style="strokeColor=#78909C;dashed=1;" edge="1" parent="1" source="helpers" target="f10">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c27" style="strokeColor=#78909C;dashed=1;" edge="1" parent="1" source="helpers" target="f13">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="c28" style="strokeColor=#FFFFFF;" edge="1" parent="1" source="f06" target="f09">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="legend" value="── sync call - - - event/config" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=10;fontColor=#888888;" vertex="1" parent="1">
<mxGeometry x="20" y="430" width="200" height="20" as="geometry"/>
</mxCell>
<mxCell id="legend2" value="Colors: Red=API | Purple=Orchestration | Blue=Visual Processing | Green=State | Orange=Data/IO | Gray=Infrastructure" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=9;fontColor=#666666;" vertex="1" parent="1">
<mxGeometry x="20" y="450" width="650" height="20" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
_docs/02_components/astral_next_components_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 333 KiB

_docs/02_components/decomposition_plan.md
+499
View File
@@ -0,0 +1,499 @@
# ASTRAL-Next System Component Decomposition Plan
## Design Principle: Interface-Based Architecture
**CRITICAL REQUIREMENT**: Each component MUST implement a well-defined interface to ensure interchangeability with different implementations.
**Benefits**:
- Swap implementations (e.g., replace LiteSAM with TransFG, GTSAM with Ceres)
- Enable unit testing with mocks
- Support multiple backends (TensorRT vs ONNX, different databases)
- Facilitate future enhancements without breaking contracts
**Interface Specification**: Each component spec must define:
- Interface name (e.g., `ISatelliteDataManager`, `IMetricRefinement`)
- All public methods with strict contracts
- Input/output data structures
- Error conditions and exceptions
- Performance guarantees
---
## System Architecture Overview
**Single unified Flight API:**
- Flight CRUD operations (create, read, update, delete)
- Waypoint management within flights
- Geofence management
- Tri-layer localization (SuperPoint+LightGlue, DINOv2, LiteSAM)
- Calls satellite provider for tiles
- Rotation preprocessing (LiteSAM 45° limit)
- Per-frame waypoint updates
- Progressive tile search (1→4→9→16→25)
- SSE streaming for real-time results
---
## FLIGHT API COMPONENTS (17 components)
### Core API Layer
**F01_flight_api**
**Interface**: `IFlightAPI`
**Endpoints**: `POST /flights`, `GET /flights/{flightId}`, `DELETE /flights/{flightId}`, `PUT /flights/{flightId}/waypoints/{waypointId}`, `POST .../images/batch`, `POST .../user-fix`, `GET .../status`, `GET .../stream`
**F02.1_flight_lifecycle_manager**
**Interface**: `IFlightLifecycleManager`
**API**: `create_flight()`, `get_flight()`, `get_flight_state()`, `delete_flight()`, `update_waypoint()`, `batch_update_waypoints()`, `validate_waypoint()`, `validate_geofence()`, `queue_images()`, `handle_user_fix()`, `create_client_stream()`, `convert_object_to_gps()`, `initialize_system()`
**F02.2_flight_processing_engine**
**Interface**: `IFlightProcessingEngine`
**API**: `start_processing()`, `stop_processing()`, `process_frame()`, `apply_user_fix()`, `handle_tracking_loss()`, `get_active_chunk()`, `create_new_chunk()`
**F03_flight_database**
**Interface**: `IFlightDatabase`
**API**: `insert_flight()`, `update_flight()`, `query_flights()`, `get_flight_by_id()`, `delete_flight()`, `get_waypoints()`, `insert_waypoint()`, `update_waypoint()`, `batch_update_waypoints()`, `save_flight_state()`, `load_flight_state()`, `save_frame_result()`, `get_frame_results()`, `save_heading()`, `get_heading_history()`, `get_latest_heading()`, `save_image_metadata()`, `get_image_path()`, `get_image_metadata()`
### Data Management
**F04_satellite_data_manager**
**Interface**: `ISatelliteDataManager`
**API**: `fetch_tile()`, `fetch_tile_grid()`, `prefetch_route_corridor()`, `progressive_fetch()`, `cache_tile()`, `get_cached_tile()`, `compute_tile_coords()`, `expand_search_grid()`, `compute_tile_bounds()`
**Features**: Progressive retrieval, tile caching, grid calculations
**F05_image_input_pipeline**
**Interface**: `IImageInputPipeline`
**API**: `queue_batch()`, `process_next_batch()`, `validate_batch()`, `store_images()`, `get_next_image()`, `get_image_by_sequence()`
**Features**: FIFO queuing, validation, storage
**F06_image_rotation_manager**
**Interface**: `IImageRotationManager`
**API**: `rotate_image_360()`, `try_rotation_steps()`, `calculate_precise_angle()`, `get_current_heading()`, `update_heading()`, `detect_sharp_turn()`, `requires_rotation_sweep()`
**Features**: 30° rotation sweeps, heading tracking
### Visual Processing
**F07_sequential_visual_odometry**
**Interface**: `ISequentialVisualOdometry`
**API**: `compute_relative_pose()`, `extract_features()`, `match_features()`, `estimate_motion()`
**F08_global_place_recognition**
**Interface**: `IGlobalPlaceRecognition`
**API**: `retrieve_candidate_tiles()`, `compute_location_descriptor()`, `query_database()`, `rank_candidates()`
**F09_metric_refinement**
**Interface**: `IMetricRefinement`
**API**: `align_to_satellite(uav_image, satellite_tile, tile_bounds)`, `compute_homography()`, `extract_gps_from_alignment()`, `compute_match_confidence()`
### State Estimation
**F10_factor_graph_optimizer**
**Interface**: `IFactorGraphOptimizer`
**API**: `add_relative_factor()`, `add_absolute_factor()`, `add_altitude_prior()`, `optimize()`, `get_trajectory()`, `get_marginal_covariance()`
**F11_failure_recovery_coordinator**
**Interface**: `IFailureRecoveryCoordinator`
**API**: `check_confidence()`, `detect_tracking_loss()`, `start_search()`, `expand_search_radius()`, `try_current_grid()`, `create_user_input_request()`, `apply_user_anchor()`
**F12_route_chunk_manager**
**Interface**: `IRouteChunkManager`
**API**: `create_chunk()`, `add_frame_to_chunk()`, `get_chunk_frames()`, `get_chunk_images()`, `get_chunk_composite_descriptor()`, `get_chunk_bounds()`, `is_chunk_ready_for_matching()`, `mark_chunk_anchored()`, `get_chunks_for_matching()`, `get_active_chunk()`, `deactivate_chunk()`
**Features**: Chunk lifecycle management, chunk state tracking, chunk matching coordination
**F13_coordinate_transformer**
**Interface**: `ICoordinateTransformer`
**API**: `set_enu_origin()`, `get_enu_origin()`, `gps_to_enu()`, `enu_to_gps()`, `pixel_to_gps()`, `gps_to_pixel()`, `image_object_to_gps()`, `transform_points()`
**Dependencies**: F10 Factor Graph Optimizer (for frame poses), H02 GSD Calculator (for GSD computation)
### Results & Communication
**F14_result_manager**
**Interface**: `IResultManager`
**API**: `update_frame_result()`, `publish_waypoint_update()`, `get_flight_results()`, `mark_refined()`
**F15_sse_event_streamer**
**Interface**: `ISSEEventStreamer`
**API**: `create_stream()`, `send_frame_result()`, `send_search_progress()`, `send_user_input_request()`, `send_refinement()`
### Infrastructure
**F16_model_manager**
**Interface**: `IModelManager`
**API**: `load_model()`, `get_inference_engine()`, `optimize_to_tensorrt()`, `fallback_to_onnx()`
**F17_configuration_manager**
**Interface**: `IConfigurationManager`
**API**: `load_config()`, `get_camera_params()`, `validate_config()`, `get_flight_config()`
---
## HELPER COMPONENTS (8 components)
**H01_camera_model** - `ICameraModel`
**H02_gsd_calculator** - `IGSDCalculator`
**H03_robust_kernels** - `IRobustKernels`
**H04_faiss_index_manager** - `IFaissIndexManager`
**H05_performance_monitor** - `IPerformanceMonitor`
**H06_web_mercator_utils** - `IWebMercatorUtils`
**H07_image_rotation_utils** - `IImageRotationUtils`
**H08_batch_validator** - `IBatchValidator`
---
## System Startup Initialization Order
**Startup sequence** (blocking, sequential):
| Order | Component | Method | Purpose | Dependencies |
|-------|-----------|--------|---------|--------------|
| 1 | F17 Configuration Manager | `load_config()` | Load system configuration | None |
| 2 | F03 Flight Database | Initialize connections | Establish DB connection pool | F17 |
| 3 | F16 Model Manager | `load_model("SuperPoint")` | Load SuperPoint feature extractor | F17 |
| 4 | F16 Model Manager | `load_model("LightGlue")` | Load LightGlue matcher | F17 |
| 5 | F16 Model Manager | `load_model("DINOv2")` | Load DINOv2 for place recognition | F17 |
| 6 | F16 Model Manager | `load_model("LiteSAM")` | Load LiteSAM for cross-view matching | F17 |
| 7 | F04 Satellite Data Manager | Initialize cache | Initialize tile cache directory | F17 |
| 8 | F08 Global Place Recognition | `load_index()` | Load pre-built Faiss index from satellite provider | F04, F16, H04 |
| 9 | F12 Route Chunk Manager | Initialize | Initialize chunk state tracking | F10 |
| 10 | F02 Flight Processor | Ready | Ready to accept flights | All above |
| 11 | F01 Flight API | Start server | Start FastAPI/Uvicorn | F02 |
**Estimated total startup time**: ~30 seconds (dominated by model loading)
**Shutdown sequence** (reverse order):
1. F01 Flight API - Stop accepting requests
2. F02 Flight Processor - Complete or cancel active flights
3. F11 Failure Recovery Coordinator - Stop background chunk matching
4. F12 Route Chunk Manager - Save chunk state
5. F16 Model Manager - Unload models
6. F03 Flight Database - Close connections
7. F04 Satellite Data Manager - Flush cache
---
## Comprehensive Component Interaction Matrix
### System Initialization
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.1 | F17 | `load_config()` | Load system configuration |
| F02.1 | F16 | `load_model()` × 4 | Load SuperPoint, LightGlue, DINOv2, LiteSAM |
| F04 | F08 | Satellite tiles + index | F08 loads pre-built Faiss index from provider |
| F08 | H04 | `load_index()` | Load satellite descriptor index |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model for descriptor computation |
### Flight Creation
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `POST /flights` | Create flight |
| F01 | F02.1 | `create_flight()` | Initialize flight state |
| F02.1 | F17 | `get_flight_config()` | Get camera params, altitude |
| F02.1 | F13 | `set_enu_origin(flight_id, start_gps)` | Set ENU coordinate origin |
| F02.1 | F04 | `prefetch_route_corridor()` | Prefetch tiles |
| F04 | Satellite Provider | `GET /api/satellite/tiles/batch` | HTTP batch download (includes tile metadata) |
| F04 | H06 | `compute_tile_bounds()` | Tile coordinate calculations |
| F02.1 | F03 | `insert_flight()` | Persist flight data |
### SSE Stream Creation
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `GET .../stream` | Open SSE connection |
| F01 | F02.1 | `create_client_stream()` | Route through lifecycle manager |
| F02.1 | F15 | `create_stream()` | Establish SSE channel |
### Image Upload
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| Client | F01 | `POST .../images/batch` | Upload 10-50 images |
| F01 | F02.1 | `queue_images()` | Route through lifecycle manager |
| F02.1 | F05 | `queue_batch()` | Queue for processing |
| F05 | H08 | `validate_batch()` | Validate sequence, format |
| F05 | F03 | `save_image_metadata()` | Persist image metadata |
### Per-Frame Processing (First Frame / Sharp Turn)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F05 | `get_next_image()` | Get image for processing |
| F02.2 | F06 | `requires_rotation_sweep()` | Check if sweep needed |
| F06 | H07 | `rotate_image()` × 12 | Rotate in 30° steps |
| F06 | F09 | `align_to_satellite(img, tile, bounds)` × 12 | Try LiteSAM each rotation |
| F02.2 | F04 | `get_cached_tile()` + `compute_tile_bounds()` | Get expected tile with bounds |
| F09 | F16 | `get_inference_engine("LiteSAM")` | Get model |
| F06 | H07 | `calculate_rotation_from_points()` | Precise angle from homography |
| F02.2 | F03 | `save_heading()` | Store UAV heading |
### Per-Frame Processing (Sequential VO)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F12 | `get_active_chunk()` | Get active chunk for frame |
| F02.2 | F07 | `compute_relative_pose()` | Provide image and chunk context |
| F07 | F16 | `get_inference_engine("SuperPoint")` | Get feature extractor |
| F07 | F16 | `get_inference_engine("LightGlue")` | Get matcher |
| F07 | H05 | `start_timer()`, `end_timer()` | Monitor timing |
| F02.2 | F10 | `add_relative_factor_to_chunk()` | Add pose measurement to chunk subgraph |
| F02.2 | F12 | `add_frame_to_chunk()` | Add frame to chunk |
### Tracking Good (Drift Correction)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F11 | `check_confidence()` | Check tracking quality |
| F02.2 | F04 | `fetch_tile()` + `compute_tile_bounds()` | Get single tile with bounds |
| F02.2 | F09 | `align_to_satellite(img, tile, bounds)` | Align to 1 tile |
| F02.2 | F10 | `add_absolute_factor()` | Add GPS measurement |
### Tracking Lost (Progressive Search + Chunk Building)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F02.2 | F11 | `check_confidence()` → FAIL | Low confidence |
| F11 | F12 | `create_chunk_on_tracking_loss()` | **Proactive chunk creation** |
| F12 | F10 | `create_chunk_subgraph()` | Create chunk in factor graph |
| F12 | F03 | `save_chunk_state()` | Persist chunk state for recovery |
| F02.2 | F12 | `get_active_chunk()` | Get new active chunk |
| F11 | F06 | `requires_rotation_sweep()` | Trigger rotation sweep (single-image) |
| F11 | F08 | `retrieve_candidate_tiles()` | Coarse localization (single-image) |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model |
| F08 | H04 | `search()` | Query Faiss index |
| F08 | F04 | `get_tile_by_gps()` × 5 | Get candidate tiles |
| F11 | F04 | `expand_search_grid(4)` | Get 2×2 grid |
| F11 | F09 | `align_to_satellite(img, tile, bounds)` | Try LiteSAM on tiles |
| F11 (fail) | F04 | `expand_search_grid(9)` | Expand to 3×3 |
| F11 (fail) | F04 | `expand_search_grid(16)` | Expand to 4×4 |
| F11 (fail) | F04 | `expand_search_grid(25)` | Expand to 5×5 |
| F11 (fail) | F12 | Continue building chunk | **Chunk building continues** |
| F11 (background) | F12 | `get_chunks_for_matching()` | Get unanchored chunks |
| F11 (background) | F08 | `retrieve_candidate_tiles_for_chunk()` | **Chunk semantic matching** |
| F11 (background) | F06 | `try_chunk_rotation_steps()` | **Chunk rotation sweeps** |
| F11 (background) | F09 | `align_chunk_to_satellite()` | **Chunk LiteSAM matching** |
| F11 (background) | F12 | `mark_chunk_anchored()` + `merge_chunks(main, new)` | **Chunk merging via F12** |
| F11 (fail) | F02.2 | Returns `UserInputRequest` | Request human help (last resort) |
| F02.2 | F15 | `send_user_input_request()` | Send SSE event to client |
### Optimization & Results
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F10 | H03 | `huber_loss()`, `cauchy_loss()` | Apply robust kernels |
| F10 | Internal | `optimize()` | Run iSAM2 optimization |
| F02.2 | F10 | `get_trajectory()` | Get optimized poses |
| F02.2 | F13 | `enu_to_gps()` | Convert ENU to GPS |
| F13 | H01 | `project()`, `unproject()` | Camera operations |
| F13 | H02 | `compute_gsd()` | GSD calculations |
| F13 | H06 | `tile_to_latlon()` | Coordinate transforms |
| F02.2 | F14 | Frame GPS + object coords | Provide results |
| F14 | F03 | `update_waypoint()` | Per-frame waypoint update |
| F14 | F15 | `send_frame_result()` | Publish to client |
| F15 | Client | SSE `frame_processed` | Real-time delivery |
| F14 | F03 | `save_frame_result()` | Persist frame result |
### User Input Recovery
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F15 | Client | SSE `user_input_needed` | Notify client |
| Client | F01 | `POST .../user-fix` | Provide anchor |
| F01 | F02.1 | `handle_user_fix()` | Route through lifecycle manager |
| F02.1 | F02.2 | `apply_user_fix()` | Delegate to processing engine |
| F02.2 | F11 | `apply_user_anchor()` | Apply fix |
| F11 | F10 | `add_absolute_factor()` (high confidence) | Hard constraint |
| F10 | Internal | `optimize()` | Re-optimize |
| F02.2 | F15 | `send_frame_result()` | Publish result via SSE |
### Asynchronous Refinement
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F10 | Internal (background) | `optimize()` | Back-propagate anchors |
| F02.2 | F10 | `get_trajectory()` | Get refined poses (ENU) |
| F02.2 | F13 | `enu_to_gps()` | Convert ENU poses to GPS |
| F02.2 | F14 | `mark_refined(List[RefinedFrameResult])` | Pass GPS-converted results |
| F14 | F03 | `batch_update_waypoints()` | Batch update waypoints |
| F14 | F15 | `send_refinement()` × N | Send updates |
| F15 | Client | SSE `frame_refined` × N | Incremental updates |
**Note**: F14 does NOT call F10 or F13. F02.2 performs coordinate conversion and provides GPS results to F14.
### Chunk Matching (Background)
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F11 (background) | F12 | `get_chunks_for_matching()` | Get unanchored chunks ready for matching |
| F11 | F12 | `get_chunk_images()` | Get chunk images |
| F12 | F05 | `get_image_by_sequence()` | **Load images for chunk** (F12 delegates to F05 for actual image retrieval) |
| F11 | F08 | `retrieve_candidate_tiles_for_chunk()` | Chunk semantic matching (aggregate DINOv2) |
| F08 | F16 | `get_inference_engine("DINOv2")` | Get model for descriptor computation |
| F08 | H04 | `search()` | Query Faiss with chunk descriptor |
| F11 | F06 | `try_chunk_rotation_steps()` | Chunk rotation sweeps (12 rotations) |
| F06 | F09 | `align_chunk_to_satellite()` × 12 | Try LiteSAM for each rotation |
| F11 | F10 | `add_chunk_anchor()` | Anchor chunk with GPS |
| F11 | F12 | `merge_chunks(main_chunk, new_chunk)` | Merge new_chunk into main_chunk (Sim3 transform) |
| F10 | Internal | `optimize_global()` | Global optimization after merging |
| F11 | F12 | `mark_chunk_anchored()` | Update chunk state |
### Cross-Cutting Concerns
| Source | Target | Method | Purpose |
|--------|--------|--------|---------|
| F17 | ALL | `get_*_config()` | Provide configuration |
| H05 | F07, F08, F09, F10, F11 | `start_timer()`, `end_timer()` | Performance monitoring |
---
## Interaction Coverage Verification
✅ **Initialization**: F02.1→F16, F17; F04→F08→H04
✅ **Flight creation**: Client→F01→F02.1→F04,F12,F16,F17,F14
✅ **Image upload**: Client→F01→F02.1→F05→H08
✅ **Rotation sweep**: F06→H07,F09 (12 iterations)
✅ **Sequential VO**: F02.2→F07→F16,F10(chunk),F12,H05
✅ **Drift correction**: F02.2→F04,F09,F10
✅ **Tracking loss**: F02.2→F11→F12(proactive chunk),F06,F08,F04(progressive),F09
✅ **Chunk building**: F02.2→F12→F10,F07
✅ **Chunk image retrieval**: F12→F05(get_image_by_sequence for chunk images)
✅ **Chunk semantic matching**: F11→F12→F08(chunk descriptor)
✅ **Chunk LiteSAM matching**: F11→F06(chunk rotation)→F09(chunk alignment)
✅ **Chunk merging**: F11→F10(Sim3 transform)
✅ **Global PR**: F08→F16,H04,F04
✅ **Optimization**: F10→H03(chunk optimization, global optimization)
✅ **Coordinate transform**: F13→F10,H01,H02,H06
✅ **Results**: F02.2→F13→F14→F15,F03
✅ **User input**: Client→F01→F02.1→F02.2→F11→F10
✅ **Refinement**: F10→F14→F03,F15
✅ **Configuration**: F17→ALL
✅ **Performance**: H05→processing components
**All major component interactions are covered.**
---
## Deliverables
**Component Count**: 25 total
- Flight API: 17 (F01-F17)
- Helpers: 8 (H01-H08)
**For each component**, create `docs/02_components/[##]_[component_name]/[component_name]_spec.md`:
1. **Interface Definition** (interface name, methods, contracts)
2. **Component Description** (responsibilities, scope)
3. **API Methods** (inputs, outputs, errors, which components call it, test cases)
4. **Integration Tests**
5. **Non-Functional Requirements** (performance, accuracy targets)
6. **Dependencies** (which components it calls)
7. **Data Models**
### Component Specifications Status
- [x] F01 Flight API (merged from R01 Route REST API)
- [x] F02.1 Flight Lifecycle Manager (split from F02 Flight Processor)
- [x] F02.2 Flight Processing Engine (split from F02 Flight Processor)
- [x] F03 Flight Database (merged from R04, G17)
- [x] F04 Satellite Data Manager
- [x] F05 Image Input Pipeline
- [x] F06 Image Rotation Manager
- [x] F07 Sequential Visual Odometry
- [x] F08 Global Place Recognition
- [x] F09 Metric Refinement
- [x] F10 Factor Graph Optimizer
- [x] F11 Failure Recovery Coordinator
- [x] F12 Route Chunk Manager (Atlas multi-map chunk lifecycle)
- [x] F13 Coordinate Transformer (with ENU origin)
- [x] F14 Result Manager
- [x] F15 SSE Event Streamer
- [x] F16 Model Manager
- [x] F17 Configuration Manager
- [x] Helper components (H01-H08)
---
## Architecture Notes
### F02 Split: Lifecycle Manager + Processing Engine
F02 has been split into two components with clear Single Responsibility:
**F02.1 Flight Lifecycle Manager**:
- Flight CRUD operations
- System initialization
- API request routing
- SSE stream creation
- User fix delegation
**F02.2 Flight Processing Engine**:
- Main processing loop
- Frame-by-frame processing orchestration
- Recovery coordination with F11
- Chunk management coordination with F12
- Runs in background thread per active flight
**Rationale**: This split aligns with Single Responsibility Principle. F02.1 handles external-facing operations (API layer), while F02.2 handles internal processing (background engine).
**Decision**: Keep as single component. Clear internal organization with separate methods for state vs processing concerns. Event-based communication with F11 keeps recovery logic separate.
### Error Recovery Strategy
**Per-Component Recovery**:
- **F02**: Persists flight state via F03 on each significant update. On restart, loads last known state.
- **F07**: Stateless - reprocesses frame if VO fails
- **F10**: Factor graph state persisted periodically. On restart, rebuilds from F03 checkpoint.
- **F11**: Chunk state persisted via F12→F03. Recovery continues from last chunk state.
- **F12**: All chunk state persisted to F03. Restores chunk handles on restart.
**System-Wide Recovery**:
1. On crash, F02 loads flight state from F03
2. F12 restores chunk state from F03
3. Processing resumes from last successfully processed frame
4. Incomplete chunks continue building/matching
**Event Recovery**:
- Events are fire-and-forget (no persistence)
- Subscribers rebuild state from F03 on restart
### Error Propagation Strategy
**Principle**: Errors propagate upward through the component hierarchy. Lower-level components throw exceptions or return error results; higher-level coordinators handle recovery.
**Error Propagation Chain**:
```
H01-H08 (Helpers) → F07/F08/F09 (Visual Processing) → F11 (Recovery) → F02 (Coordinator) → F01 (API) → Client
↓
Events → F14 → F15 → SSE → Client
```
**Error Categories**:
1. **Recoverable** (F11 handles):
- Tracking loss → Progressive search
- Low confidence → Rotation sweep
- Chunk matching fails → User input request
2. **Propagated to Client** (via SSE):
- User input needed → `user_input_needed` event
- Processing blocked → `processing_blocked` event
- Flight completed → `flight_completed` event
3. **Fatal** (F02 handles, returns to F01):
- Database connection lost → HTTP 503
- Model loading failed → HTTP 500
- Invalid configuration → HTTP 400
**User Fix Flow**:
```
Client ---> F01 (POST /user-fix) ---> F02.handle_user_fix() ---> F11.apply_user_anchor()
↓
F10.add_chunk_anchor()
↓
F02 receives UserFixApplied event
↓
F14.publish_result() ---> F15 ---> SSE ---> Client
```
_docs/02_components/helpers/h01_camera_model_spec.md
+101
View File
@@ -0,0 +1,101 @@
# Camera Model Helper
## Interface Definition
**Interface Name**: `ICameraModel`
### Interface Methods
```python
class ICameraModel(ABC):
@abstractmethod
def project(self, point_3d: np.ndarray, camera_params: CameraParameters) -> Tuple[float, float]:
pass
@abstractmethod
def unproject(self, pixel: Tuple[float, float], depth: float, camera_params: CameraParameters) -> np.ndarray:
pass
@abstractmethod
def get_focal_length(self, camera_params: CameraParameters) -> Tuple[float, float]:
pass
@abstractmethod
def apply_distortion(self, pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]:
pass
@abstractmethod
def remove_distortion(self, pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]:
pass
```
## Component Description
Pinhole camera projection model with Brown-Conrady distortion handling.
## API Methods
### `project(point_3d: np.ndarray, camera_params: CameraParameters) -> Tuple[float, float]`
**Description**: Projects 3D point to 2D image pixel.
**Formula**:
```
x = fx * X/Z + cx
y = fy * Y/Z + cy
```
---
### `unproject(pixel: Tuple[float, float], depth: float, camera_params: CameraParameters) -> np.ndarray`
**Description**: Unprojects pixel to 3D ray at given depth.
**Formula**:
```
X = (x - cx) * depth / fx
Y = (y - cy) * depth / fy
Z = depth
```
---
### `get_focal_length(camera_params: CameraParameters) -> Tuple[float, float]`
**Description**: Returns (fx, fy) in pixels.
**Formula**:
```
fx = focal_length_mm * image_width / sensor_width_mm
fy = focal_length_mm * image_height / sensor_height_mm
```
---
### `apply_distortion(pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]`
**Description**: Applies radial and tangential distortion (Brown-Conrady model).
---
### `remove_distortion(pixel: Tuple[float, float], distortion_coeffs: List[float]) -> Tuple[float, float]`
**Description**: Removes distortion from observed pixel.
## Dependencies
**External**: opencv-python, numpy
## Data Models
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int
resolution_height: int
principal_point: Tuple[float, float] # (cx, cy) pixels
distortion_coefficients: List[float] # [k1, k2, p1, p2, k3]
```
_docs/02_components/helpers/h02_gsd_calculator_spec.md
+78
View File
@@ -0,0 +1,78 @@
# GSD Calculator Helper
## Interface Definition
**Interface Name**: `IGSDCalculator`
### Interface Methods
```python
class IGSDCalculator(ABC):
@abstractmethod
def compute_gsd(self, altitude: float, camera_params: CameraParameters) -> float:
pass
@abstractmethod
def altitude_to_scale(self, altitude: float, focal_length: float) -> float:
pass
@abstractmethod
def meters_per_pixel(self, lat: float, zoom: int) -> float:
pass
@abstractmethod
def gsd_from_camera(self, altitude: float, focal_length: float, sensor_width: float, image_width: int) -> float:
pass
```
## Component Description
Ground Sampling Distance computations for altitude and coordinate systems.
## API Methods
### `compute_gsd(altitude: float, camera_params: CameraParameters) -> float`
**Description**: Computes GSD from altitude and camera parameters.
**Formula**:
```
GSD = (altitude * sensor_width) / (focal_length * image_width)
```
**Example**: altitude=800m, focal=24mm, sensor=36mm, width=6000px → GSD=0.2 m/pixel
---
### `altitude_to_scale(altitude: float, focal_length: float) -> float`
**Description**: Converts altitude to scale factor for VO.
---
### `meters_per_pixel(lat: float, zoom: int) -> float`
**Description**: Computes GSD for Web Mercator tiles at zoom level.
**Formula**:
```
meters_per_pixel = 156543.03392 * cos(lat * π/180) / 2^zoom
```
**Example**: lat=48°N, zoom=19 → ~0.3 m/pixel
---
### `gsd_from_camera(altitude: float, focal_length: float, sensor_width: float, image_width: int) -> float`
**Description**: Direct GSD calculation from parameters.
## Dependencies
**External**: numpy
## Test Cases
1. Standard camera at 800m → GSD ~0.1-0.3 m/pixel
2. Web Mercator zoom 19 at Ukraine → ~0.3 m/pixel
_docs/02_components/helpers/h03_robust_kernels_spec.md
+74
View File
@@ -0,0 +1,74 @@
# Robust Kernels Helper
## Interface Definition
**Interface Name**: `IRobustKernels`
### Interface Methods
```python
class IRobustKernels(ABC):
@abstractmethod
def huber_loss(self, error: float, threshold: float) -> float:
pass
@abstractmethod
def cauchy_loss(self, error: float, k: float) -> float:
pass
@abstractmethod
def compute_weight(self, error: float, kernel_type: str, params: Dict[str, float]) -> float:
pass
```
## Component Description
Huber/Cauchy loss functions for outlier rejection in optimization.
## API Methods
### `huber_loss(error: float, threshold: float) -> float`
**Description**: Huber robust loss function.
**Formula**:
```
if |error| <= threshold:
loss = 0.5 * error^2
else:
loss = threshold * (|error| - 0.5 * threshold)
```
**Purpose**: Quadratic for small errors, linear for large errors (outliers).
---
### `cauchy_loss(error: float, k: float) -> float`
**Description**: Cauchy robust loss function.
**Formula**:
```
loss = (k^2 / 2) * log(1 + (error/k)^2)
```
**Purpose**: More aggressive outlier rejection than Huber.
---
### `compute_weight(error: float, kernel_type: str, params: Dict[str, float]) -> float`
**Description**: Computes robust weight for error.
**Usage**: Factor Graph applies weights to downweight outliers.
## Dependencies
**External**: numpy
## Test Cases
1. Small error → weight ≈ 1.0
2. Large error (350m outlier) → weight ≈ 0.1 (downweighted)
3. Huber vs Cauchy → Cauchy more aggressive
_docs/02_components/helpers/h04_faiss_index_manager_spec.md
+131
View File
@@ -0,0 +1,131 @@
# Faiss Index Manager Helper
## Interface Definition
**Interface Name**: `IFaissIndexManager`
### Interface Methods
```python
class IFaissIndexManager(ABC):
@abstractmethod
def build_index(self, descriptors: np.ndarray, index_type: str) -> FaissIndex:
pass
@abstractmethod
def add_descriptors(self, index: FaissIndex, descriptors: np.ndarray) -> bool:
pass
@abstractmethod
def search(self, index: FaissIndex, query: np.ndarray, k: int) -> Tuple[np.ndarray, np.ndarray]:
pass
@abstractmethod
def save_index(self, index: FaissIndex, path: str) -> bool:
pass
@abstractmethod
def load_index(self, path: str) -> FaissIndex:
pass
@abstractmethod
def is_gpu_available(self) -> bool:
pass
@abstractmethod
def set_device(self, device: str) -> bool:
"""Set device: 'gpu' or 'cpu'."""
pass
```
## Component Description
Manages Faiss indices for DINOv2 descriptor similarity search. H04 provides generic Faiss index operations used by:
### Satellite Index (Primary Use Case)
- **Index format**: IVF1000 (Inverted File with 1000 clusters)
- **Index source**: Pre-built by external Satellite Provider (Maxar, Google Maps, Copernicus, etc.)
- **Index delivery**: Provider delivers index file + tile metadata when tiles are fetched on demand
- **Index updates**: Provider rebuilds index when new satellite tiles become available
- **Usage**: F08 Global Place Recognition loads this index via H04.load_index()
### UAV Index (Optional, Future Use)
For loop closure and chunk-to-chunk matching:
1. **Loop closure detection**: Find when UAV revisits previously seen areas
2. **Chunk-to-chunk matching**: Match disconnected chunks to each other
3. **Flight-to-flight matching**: Match current flight to previous flights
**Note**: H04 is a low-level utility that manages ANY Faiss index. It does NOT know whether the index contains satellite or UAV descriptors.
## API Methods
### `build_index(descriptors: np.ndarray, index_type: str) -> FaissIndex`
**Description**: Builds Faiss index from descriptors.
**Index Types**:
- **"IVF"**: Inverted File (fast for large databases)
- **"HNSW"**: Hierarchical Navigable Small World (best accuracy/speed trade-off)
- **"Flat"**: Brute force (exact, slow for large datasets)
**Input**: (N, D) descriptors array
---
### `add_descriptors(index: FaissIndex, descriptors: np.ndarray) -> bool`
**Description**: Adds more descriptors to existing index.
---
### `search(index: FaissIndex, query: np.ndarray, k: int) -> Tuple[np.ndarray, np.ndarray]`
**Description**: Searches for k nearest neighbors.
**Output**: (distances, indices) - shape (k,)
---
### `save_index(index: FaissIndex, path: str) -> bool`
**Description**: Saves index to disk for fast startup.
---
### `load_index(path: str) -> FaissIndex`
**Description**: Loads pre-built index from disk.
## Dependencies
**External**: faiss-gpu or faiss-cpu
## GPU/CPU Fallback
H04 supports automatic fallback from GPU to CPU:
- `is_gpu_available()`: Returns True if faiss-gpu is available and CUDA works
- `set_device("gpu")`: Use GPU acceleration (faster for large indexes)
- `set_device("cpu")`: Use CPU (fallback when GPU unavailable)
## Current vs Future Use Cases
### Current Use (MVP)
- **Satellite Index Loading**: F08 uses `load_index()` to load pre-built satellite descriptor index from provider.
- **Similarity Search**: F08 uses `search()` to find candidate satellite tiles.
### Future Use Cases (build_index, add_descriptors)
The `build_index()` and `add_descriptors()` methods are reserved for future features:
1. **UAV Loop Closure Detection**: Build index of UAV frame descriptors to detect when UAV revisits previously seen areas.
2. **Chunk-to-Chunk Matching**: Build index of chunk descriptors for matching disconnected trajectory segments.
3. **Flight-to-Flight Matching**: Match current flight to previous flights for multi-flight consistency.
**Note**: For MVP, F08 does NOT build satellite indexes - they are provided pre-built by the satellite data provider.
## Test Cases
1. Build index with 10,000 UAV image descriptors → succeeds
2. Search query UAV descriptor → returns top-k similar UAV frames
3. Save/load index → index restored correctly
4. GPU unavailable → automatically falls back to CPU
5. Add descriptors incrementally → index grows correctly
_docs/02_components/helpers/h05_performance_monitor_spec.md
+93
View File
@@ -0,0 +1,93 @@
# Performance Monitor Helper
## Interface Definition
**Interface Name**: `IPerformanceMonitor`
### Interface Methods
```python
class IPerformanceMonitor(ABC):
@abstractmethod
def start_timer(self, operation: str) -> str:
pass
@abstractmethod
def end_timer(self, timer_id: str) -> float:
pass
@abstractmethod
def get_statistics(self, operation: str) -> PerformanceStats:
pass
@abstractmethod
def check_sla(self, operation: str, threshold: float) -> bool:
pass
@abstractmethod
def get_bottlenecks(self) -> List[Tuple[str, float]]:
pass
```
## Component Description
Tracks processing times, ensures <5s constraint per frame.
## API Methods
### `start_timer(operation: str) -> str`
**Description**: Starts timing an operation.
**Returns**: timer_id (UUID)
---
### `end_timer(timer_id: str) -> float`
**Description**: Ends timer and records duration.
**Returns**: Duration in seconds
---
### `get_statistics(operation: str) -> PerformanceStats`
**Description**: Gets statistics for an operation.
**Output**:
```python
PerformanceStats:
operation: str
count: int
mean: float
p50: float
p95: float
p99: float
max: float
```
---
### `check_sla(operation: str, threshold: float) -> bool`
**Description**: Checks if operation meets SLA threshold.
**Example**: check_sla("frame_processing", 5.0) → True if < 5s
---
### `get_bottlenecks() -> List[Tuple[str, float]]`
**Description**: Returns slowest operations.
## Dependencies
**External**: time, statistics
## Test Cases
1. Start/end timer → records duration
2. Get statistics → returns percentiles
3. Check SLA → returns True if meeting targets
_docs/02_components/helpers/h06_web_mercator_utils_spec.md
+94
View File
@@ -0,0 +1,94 @@
# Web Mercator Utils Helper
## Interface Definition
**Interface Name**: `IWebMercatorUtils`
### Interface Methods
```python
class IWebMercatorUtils(ABC):
@abstractmethod
def latlon_to_tile(self, lat: float, lon: float, zoom: int) -> Tuple[int, int]:
pass
@abstractmethod
def tile_to_latlon(self, x: int, y: int, zoom: int) -> Tuple[float, float]:
pass
@abstractmethod
def compute_tile_bounds(self, x: int, y: int, zoom: int) -> TileBounds:
pass
@abstractmethod
def get_zoom_gsd(self, lat: float, zoom: int) -> float:
pass
```
## Component Description
Web Mercator projection (EPSG:3857) for tile coordinates. Used for Google Maps tiles.
## API Methods
### `latlon_to_tile(lat: float, lon: float, zoom: int) -> Tuple[int, int]`
**Description**: Converts GPS to tile coordinates.
**Formula**:
```
n = 2^zoom
x = floor((lon + 180) / 360 * n)
lat_rad = lat * π / 180
y = floor((1 - log(tan(lat_rad) + sec(lat_rad)) / π) / 2 * n)
```
**Returns**: (x, y) tile coordinates
---
### `tile_to_latlon(x: int, y: int, zoom: int) -> Tuple[float, float]`
**Description**: Converts tile coordinates to GPS (NW corner).
**Formula** (inverse of above)
---
### `compute_tile_bounds(x: int, y: int, zoom: int) -> TileBounds`
**Description**: Computes GPS bounding box of tile.
**Returns**:
```python
TileBounds:
nw: GPSPoint # North-West
ne: GPSPoint # North-East
sw: GPSPoint # South-West
se: GPSPoint # South-East
center: GPSPoint
gsd: float
```
---
### `get_zoom_gsd(lat: float, zoom: int) -> float`
**Description**: Gets GSD for zoom level at latitude.
**Formula**:
```
gsd = 156543.03392 * cos(lat * π/180) / 2^zoom
```
## Dependencies
**External**: numpy
## Test Cases
1. GPS to tile at zoom 19 → valid tile coords
2. Tile to GPS → inverse correct
3. Compute bounds → 4 corners valid
4. GSD at zoom 19, Ukraine → ~0.3 m/pixel
_docs/02_components/helpers/h07_image_rotation_utils_spec.md
+92
View File
@@ -0,0 +1,92 @@
# Image Rotation Utils Helper
## Interface Definition
**Interface Name**: `IImageRotationUtils`
### Interface Methods
```python
class IImageRotationUtils(ABC):
@abstractmethod
def rotate_image(self, image: np.ndarray, angle: float, center: Optional[Tuple[int, int]] = None) -> np.ndarray:
pass
@abstractmethod
def calculate_rotation_from_points(self, src_points: np.ndarray, dst_points: np.ndarray) -> float:
pass
@abstractmethod
def normalize_angle(self, angle: float) -> float:
pass
@abstractmethod
def compute_rotation_matrix(self, angle: float, center: Tuple[int, int]) -> np.ndarray:
pass
```
## Component Description
Image rotation operations, angle calculations from point shifts.
## API Methods
### `rotate_image(image: np.ndarray, angle: float, center: Optional[Tuple[int, int]] = None) -> np.ndarray`
**Description**: Rotates image around center.
**Implementation**: Uses cv2.getRotationMatrix2D + cv2.warpAffine
**Parameters**:
- **angle**: Degrees (0-360)
- **center**: Rotation center (default: image center)
**Returns**: Rotated image (same dimensions)
---
### `calculate_rotation_from_points(src_points: np.ndarray, dst_points: np.ndarray) -> float`
**Description**: Calculates rotation angle from point correspondences.
**Input**: (N, 2) arrays of matching points
**Algorithm**:
1. Compute centroids
2. Calculate angle from centroid shifts
3. Return angle in degrees
**Use Case**: Extract precise angle from LiteSAM homography
---
### `normalize_angle(angle: float) -> float`
**Description**: Normalizes angle to 0-360 range.
**Formula**:
```
angle = angle % 360
if angle < 0:
angle += 360
```
---
### `compute_rotation_matrix(angle: float, center: Tuple[int, int]) -> np.ndarray`
**Description**: Computes 2D rotation matrix.
**Returns**: 2×3 affine transformation matrix
## Dependencies
**External**: opencv-python, numpy
## Test Cases
1. Rotate 90° → image rotated correctly
2. Calculate angle from points → accurate angle
3. Normalize 370° → 10°
4. Rotation matrix → correct transformation
_docs/02_components/helpers/h08_batch_validator_spec.md
+329
View File
@@ -0,0 +1,329 @@
# Batch Validator Helper
## Interface Definition
**Interface Name**: `IBatchValidator`
### Interface Methods
```python
class IBatchValidator(ABC):
@abstractmethod
def validate_batch_size(self, batch: ImageBatch) -> ValidationResult:
pass
@abstractmethod
def check_sequence_continuity(self, batch: ImageBatch, expected_start: int) -> ValidationResult:
pass
@abstractmethod
def validate_naming_convention(self, filenames: List[str]) -> ValidationResult:
pass
@abstractmethod
def validate_format(self, image_data: bytes) -> ValidationResult:
pass
```
## Component Description
### Responsibilities
- Validate image batch integrity
- Check sequence continuity and naming conventions
- Validate image format and dimensions
- Ensure batch size constraints (10-50 images)
- Support strict sequential ordering (ADxxxxxx.jpg)
### Scope
- Batch validation for F05 Image Input Pipeline
- Image format validation
- Filename pattern matching
- Sequence gap detection
## API Methods
### `validate_batch_size(batch: ImageBatch) -> ValidationResult`
**Description**: Validates batch contains 10-50 images.
**Called By**:
- F05 Image Input Pipeline (before queuing)
**Input**:
```python
batch: ImageBatch:
images: List[bytes]
filenames: List[str]
start_sequence: int
end_sequence: int
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
- **Minimum batch size**: 10 images
- **Maximum batch size**: 50 images
- **Reason**: Balance between upload overhead and processing granularity
**Error Conditions**:
- Returns `valid=False` with error message (not an exception)
**Test Cases**:
1. **Valid batch (20 images)**: Returns `valid=True`
2. **Too few images (5)**: Returns `valid=False`, error="Batch size 5 below minimum 10"
3. **Too many images (60)**: Returns `valid=False`, error="Batch size 60 exceeds maximum 50"
4. **Empty batch**: Returns `valid=False`
---
### `check_sequence_continuity(batch: ImageBatch, expected_start: int) -> ValidationResult`
**Description**: Validates images form consecutive sequence with no gaps.
**Called By**:
- F05 Image Input Pipeline (before queuing)
**Input**:
```python
batch: ImageBatch
expected_start: int # Expected starting sequence number
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
1. **Sequence starts at expected_start**: First image sequence == expected_start
2. **Consecutive numbers**: No gaps in sequence (AD000101, AD000102, AD000103, ...)
3. **Filename extraction**: Parse sequence from ADxxxxxx.jpg pattern
4. **Strict ordering**: Images must be in sequential order
**Algorithm**:
```python
sequences = [extract_sequence(filename) for filename in batch.filenames]
if sequences[0] != expected_start:
return invalid("Expected start {expected_start}, got {sequences[0]}")
for i in range(len(sequences) - 1):
if sequences[i+1] != sequences[i] + 1:
return invalid(f"Gap detected: {sequences[i]} -> {sequences[i+1]}")
return valid()
```
**Error Conditions**:
- Returns `valid=False` with specific gap information
**Test Cases**:
1. **Valid sequence (101-150)**: expected_start=101 → valid=True
2. **Wrong start**: expected_start=101, got 102 → valid=False
3. **Gap in sequence**: AD000101, AD000103 (missing 102) → valid=False
4. **Out of order**: AD000102, AD000101 → valid=False
---
### `validate_naming_convention(filenames: List[str]) -> ValidationResult`
**Description**: Validates filenames match ADxxxxxx.jpg pattern.
**Called By**:
- Internal (during check_sequence_continuity)
- F05 Image Input Pipeline
**Input**:
```python
filenames: List[str]
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
1. **Pattern**: `AD\d{6}\.(jpg|JPG|png|PNG)`
2. **Examples**: AD000001.jpg, AD000237.JPG, AD002000.png
3. **Case insensitive**: Accepts .jpg, .JPG, .Jpg
4. **6 digits required**: Zero-padded to 6 digits
**Regex Pattern**: `^AD\d{6}\.(jpg|JPG|png|PNG)$`
**Error Conditions**:
- Returns `valid=False` listing invalid filenames
**Test Cases**:
1. **Valid names**: ["AD000001.jpg", "AD000002.jpg"] → valid=True
2. **Invalid prefix**: "IMG_0001.jpg" → valid=False
3. **Wrong digit count**: "AD001.jpg" (3 digits) → valid=False
4. **Missing extension**: "AD000001" → valid=False
5. **Invalid extension**: "AD000001.bmp" → valid=False
---
### `validate_format(image_data: bytes) -> ValidationResult`
**Description**: Validates image file format and properties.
**Called By**:
- F05 Image Input Pipeline (per-image validation)
**Input**:
```python
image_data: bytes # Raw image file bytes
```
**Output**:
```python
ValidationResult:
valid: bool
errors: List[str]
```
**Validation Rules**:
1. **Format**: Valid JPEG or PNG
2. **Dimensions**: 640×480 to 6252×4168 pixels
3. **File size**: < 10MB per image
4. **Image readable**: Not corrupted
5. **Color channels**: RGB (3 channels)
**Algorithm**:
```python
try:
image = PIL.Image.open(BytesIO(image_data))
width, height = image.size
if image.format not in ['JPEG', 'PNG']:
return invalid("Format must be JPEG or PNG")
if width < 640 or height < 480:
return invalid("Dimensions too small")
if width > 6252 or height > 4168:
return invalid("Dimensions too large")
if len(image_data) > 10 * 1024 * 1024:
return invalid("File size exceeds 10MB")
return valid()
except Exception as e:
return invalid(f"Corrupted image: {e}")
```
**Error Conditions**:
- Returns `valid=False` with specific error
**Test Cases**:
1. **Valid JPEG (2048×1536)**: valid=True
2. **Valid PNG (6252×4168)**: valid=True
3. **Too small (320×240)**: valid=False
4. **Too large (8000×6000)**: valid=False
5. **File too big (15MB)**: valid=False
6. **Corrupted file**: valid=False
7. **BMP format**: valid=False
## Integration Tests
### Test 1: Complete Batch Validation
1. Create batch with 20 images, AD000101.jpg - AD000120.jpg
2. validate_batch_size() → valid
3. validate_naming_convention() → valid
4. check_sequence_continuity(expected_start=101) → valid
5. validate_format() for each image → all valid
### Test 2: Invalid Batch Detection
1. Create batch with 60 images → validate_batch_size() fails
2. Create batch with gap (AD000101, AD000103) → check_sequence_continuity() fails
3. Create batch with IMG_0001.jpg → validate_naming_convention() fails
4. Create batch with corrupted image → validate_format() fails
### Test 3: Edge Cases
1. Batch with exactly 10 images → valid
2. Batch with exactly 50 images → valid
3. Batch with 51 images → invalid
4. Batch starting at AD999995.jpg (near max) → valid
## Non-Functional Requirements
### Performance
- **validate_batch_size**: < 1ms
- **check_sequence_continuity**: < 10ms for 50 images
- **validate_naming_convention**: < 5ms for 50 filenames
- **validate_format**: < 20ms per image (with PIL)
- **Total batch validation**: < 100ms for 50 images
### Reliability
- Never raises exceptions (returns ValidationResult with errors)
- Handles edge cases gracefully
- Clear, actionable error messages
### Maintainability
- Configurable validation rules (min/max batch size, dimensions)
- Easy to add new validation rules
- Comprehensive error reporting
## Dependencies
### Internal Components
- None (pure utility, no internal dependencies)
### External Dependencies
- **Pillow (PIL)**: Image format validation and dimension checking
- **re** (regex): Filename pattern matching
## Data Models
### ImageBatch
```python
class ImageBatch(BaseModel):
images: List[bytes] # Raw image data
filenames: List[str] # e.g., ["AD000101.jpg", ...]
start_sequence: int # 101
end_sequence: int # 150
batch_number: int # Sequential batch number
```
### ValidationResult
```python
class ValidationResult(BaseModel):
valid: bool
errors: List[str] = [] # Empty if valid
warnings: List[str] = [] # Optional warnings
```
### ValidationRules (Configuration)
```python
class ValidationRules(BaseModel):
min_batch_size: int = 10
max_batch_size: int = 50
min_width: int = 640
min_height: int = 480
max_width: int = 6252
max_height: int = 4168
max_file_size_mb: int = 10
allowed_formats: List[str] = ["JPEG", "PNG"]
filename_pattern: str = r"^AD\d{6}\.(jpg|JPG|png|PNG)$"
```
### Sequence Extraction
```python
def extract_sequence(filename: str) -> int:
"""
Extracts sequence number from filename.
Example: "AD000237.jpg" -> 237
"""
match = re.match(r"AD(\d{6})\.", filename)
if match:
return int(match.group(1))
raise ValueError(f"Invalid filename format: {filename}")
```
_docs/02_components/structure_plan.md
+284
View File
@@ -0,0 +1,284 @@
# Initial Structure for gps-denied
## Technology Stack
- **Python**: 3.10+ (GTSAM compatibility)
- **Web Framework**: FastAPI (async)
- **Database**: PostgreSQL + SQLAlchemy ORM
- **Validation**: Pydantic v2
- **ML Runtime**: TensorRT (primary), ONNX Runtime (fallback) - NO PyTorch needed for inference
- **Graph Optimization**: GTSAM
- **Similarity Search**: Faiss
## Project Structure
```
gps-denied/
├── main.py # FastAPI app entry point
├── pyproject.toml
├── .gitignore
├── .env.example
├── README.md
│
├── models/ # Pydantic DTOs (1 file per model)
│ ├── __init__.py # Re-exports all models
│ ├── core/ # Core shared models
│ │ ├── __init__.py
│ │ ├── gps_point.py
│ │ ├── camera_parameters.py
│ │ ├── pose.py
│ │ ├── polygon.py
│ │ └── validation_result.py
│ ├── flight/ # Flight domain models
│ │ ├── __init__.py
│ │ ├── flight.py
│ │ ├── flight_state.py
│ │ ├── waypoint.py
│ │ ├── geofences.py
│ │ └── heading_record.py
│ ├── processing/ # Visual processing models
│ │ ├── __init__.py
│ │ ├── relative_pose.py
│ │ ├── motion.py
│ │ ├── matches.py
│ │ ├── alignment_result.py
│ │ └── rotation_result.py
│ ├── chunks/ # Chunk-related models
│ │ ├── __init__.py
│ │ ├── chunk_handle.py
│ │ ├── chunk_bounds.py
│ │ └── sim3_transform.py
│ ├── satellite/ # Satellite tile models
│ │ ├── __init__.py
│ │ ├── tile_coords.py
│ │ ├── tile_bounds.py
│ │ └── tile_candidate.py
│ ├── recovery/ # Recovery/search models
│ │ ├── __init__.py
│ │ ├── search_session.py
│ │ ├── confidence_assessment.py
│ │ ├── user_anchor.py
│ │ └── user_input_request.py
│ ├── results/ # Result models
│ │ ├── __init__.py
│ │ ├── frame_result.py
│ │ ├── flight_results.py
│ │ ├── refined_frame_result.py
│ │ └── optimization_result.py
│ ├── images/ # Image-related models
│ │ ├── __init__.py
│ │ ├── image_data.py
│ │ ├── image_metadata.py
│ │ ├── image_batch.py
│ │ └── processing_status.py
│ ├── config/ # Configuration models
│ │ ├── __init__.py
│ │ ├── system_config.py
│ │ ├── flight_config.py
│ │ ├── database_config.py
│ │ ├── model_config.py
│ │ ├── rotation_config.py
│ │ └── recovery_config.py
│ └── api/ # API request/response models
│ ├── __init__.py
│ ├── flight_requests.py
│ ├── flight_responses.py
│ ├── batch_requests.py
│ └── user_fix_requests.py
│
├── components/ # Interface (base.py) + implementation together
│ ├── __init__.py
│ ├── flight_api/
│ │ ├── __init__.py
│ │ ├── base.py # ABC interface
│ │ └── flight_api.py # Implementation
│ ├── flight_lifecycle_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── flight_lifecycle_manager.py
│ ├── flight_processing_engine/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── flight_processing_engine.py
│ ├── flight_database/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── flight_database.py
│ ├── satellite_data_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── satellite_data_manager.py
│ ├── image_input_pipeline/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── image_input_pipeline.py
│ ├── image_rotation_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── image_rotation_manager.py
│ ├── sequential_visual_odometry/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── sequential_visual_odometry.py
│ ├── global_place_recognition/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── global_place_recognition.py
│ ├── metric_refinement/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── metric_refinement.py
│ ├── factor_graph_optimizer/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── factor_graph_optimizer.py
│ ├── failure_recovery_coordinator/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── failure_recovery_coordinator.py
│ ├── route_chunk_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── route_chunk_manager.py
│ ├── coordinate_transformer/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── coordinate_transformer.py
│ ├── result_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── result_manager.py
│ ├── sse_event_streamer/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── sse_event_streamer.py
│ ├── model_manager/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── model_manager.py
│ └── configuration_manager/
│ ├── __init__.py
│ ├── base.py
│ └── configuration_manager.py
│
├── helpers/ # Single file per helper
│ ├── __init__.py
│ ├── camera_model.py
│ ├── gsd_calculator.py
│ ├── robust_kernels.py
│ ├── faiss_index_manager.py
│ ├── performance_monitor.py
│ ├── web_mercator_utils.py
│ ├── image_rotation_utils.py
│ └── batch_validator.py
│
├── db/ # Database layer
│ ├── __init__.py
│ ├── connection.py
│ ├── models.py # SQLAlchemy ORM models
│ └── migrations/
│
├── api/ # FastAPI routes
│ ├── __init__.py
│ ├── routes/
│ │ ├── __init__.py
│ │ ├── flights.py
│ │ ├── images.py
│ │ └── stream.py
│ └── dependencies.py
│
└── _docs/ # Documentation
```
## Dependencies (pyproject.toml)
```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "azaion-gps-denied-desktop"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
"fastapi>=0.109.0",
"uvicorn[standard]>=0.27.0",
"pydantic>=2.5.0",
"sqlalchemy[asyncio]>=2.0.0",
"asyncpg>=0.29.0",
"alembic>=1.13.0",
"numpy>=1.26.0",
"opencv-python>=4.9.0",
"sse-starlette>=2.0.0",
"python-multipart>=0.0.6",
"httpx>=0.26.0",
"pyyaml>=6.0",
"gtsam>=4.2",
]
[project.optional-dependencies]
ml = [
"tensorrt>=10.0.0",
"onnxruntime-gpu>=1.17.0",
"faiss-gpu>=1.7.4",
]
dev = [
"pytest>=7.4.0",
"pytest-asyncio>=0.21.0",
"pytest-cov>=4.1.0",
]
[tool.hatch.build.targets.wheel]
packages = ["models", "components", "helpers", "db", "api"]
```
Note: PyTorch removed - TensorRT and ONNX Runtime handle inference without it.
## Implementation Phases
### Phase 1: Project Setup ✓
- Create package structure with `__init__.py` files
- Create `pyproject.toml`, `.gitignore`, `README.md`, `.env.example`
- Setup uv for dependency management
### Phase 2: Core DTOs ✓
- `models/core/` - GPSPoint, CameraParameters, Pose, Polygon, ValidationResult
- `models/flight/` - Flight, FlightState, Waypoint, Geofences, HeadingRecord
- `models/processing/` - RelativePose, AlignmentResult, Matches, Motion, RotationResult
- `models/chunks/` - ChunkHandle, ChunkBounds, Sim3Transform
- All remaining model subdirectories (satellite, recovery, results, images, config, api)
### Phase 3: Components with Interfaces ✓
- Each component folder contains:
- `base.py` - ABC interface
- `{component_name}.py` - Implementation stub with `NotImplementedError`
- All 18 components implemented
### Phase 4: Helpers ✓
- Single file helpers with class stubs
- All 8 helpers implemented
### Phase 5: Database Schema ✓
- SQLAlchemy ORM models (Flight, Waypoint, FrameResult, Chunk)
- Async connection management with asyncpg
- Alembic migrations folder created
### Phase 6: API Routes ✓
- FastAPI routes for flights, images, SSE streaming
- Dependency injection setup
## Completed Tasks
- [x] Create package structure, pyproject.toml, .gitignore
- [x] Create all Pydantic DTOs (models/ directory)
- [x] Create components with base.py (ABC) + stub implementations
- [x] Create helper files with empty implementations
- [x] Create SQLAlchemy models and Alembic setup
- [x] Create API routes and dependencies
_docs/02_components/system_flows.md
+901
View File
@@ -0,0 +1,901 @@
# ASTRAL-Next System Flows
> **See also**: [System Flow Diagrams (Mermaid)](./system_flows_diagrams.md) for interactive visual diagrams.
## System Overview
ASTRAL-Next is a GPS-denied UAV visual localization system using tri-layer matching (SuperPoint+LightGlue for VO, DINOv2 for place recognition, LiteSAM for cross-view matching) with an Atlas-style multi-map chunk architecture for robust tracking and recovery.
## Components Summary
| ID | Component | Interface | Purpose |
|----|-----------|-----------|---------|
| F01 | Flight API | `IFlightAPI` | REST endpoints, SSE streaming |
| F02.1 | Flight Lifecycle Manager | `IFlightLifecycleManager` | Flight CRUD, init, API delegation |
| F02.2 | Flight Processing Engine | `IFlightProcessingEngine` | Processing loop, recovery orchestration |
| F03 | Flight Database | `IFlightDatabase` | Persistence layer |
| F04 | Satellite Data Manager | `ISatelliteDataManager` | Tile fetching, caching, progressive search |
| F05 | Image Input Pipeline | `IImageInputPipeline` | Image ingestion, validation, storage |
| F06 | Image Rotation Manager | `IImageRotationManager` | Rotation sweeps, heading tracking |
| F07 | Sequential Visual Odometry | `ISequentialVisualOdometry` | Frame-to-frame VO (SuperPoint+LightGlue) |
| F08 | Global Place Recognition | `IGlobalPlaceRecognition` | Coarse localization (DINOv2+VLAD) |
| F09 | Metric Refinement | `IMetricRefinement` | Precise alignment (LiteSAM) |
| F10 | Factor Graph Optimizer | `IFactorGraphOptimizer` | GTSAM-based trajectory optimization |
| F11 | Failure Recovery Coordinator | `IFailureRecoveryCoordinator` | Recovery orchestration, chunk matching |
| F12 | Route Chunk Manager | `IRouteChunkManager` | Chunk lifecycle management |
| F13 | Coordinate Transformer | `ICoordinateTransformer` | Coordinate conversions (GPS↔ENU↔Pixel) |
| F14 | Result Manager | `IResultManager` | Result tracking and publishing |
| F15 | SSE Event Streamer | `ISSEEventStreamer` | Real-time event streaming |
| F16 | Model Manager | `IModelManager` | ML model loading (TensorRT/ONNX) |
| F17 | Configuration Manager | `IConfigurationManager` | System configuration |
### Helper Components
| ID | Helper | Interface | Purpose |
|----|--------|-----------|---------|
| H01 | Camera Model | `ICameraModel` | Projection/unprojection |
| H02 | GSD Calculator | `IGSDCalculator` | Ground sampling distance |
| H03 | Robust Kernels | `IRobustKernels` | Huber/Cauchy loss functions |
| H04 | Faiss Index Manager | `IFaissIndexManager` | Similarity search |
| H05 | Performance Monitor | `IPerformanceMonitor` | Timing measurements |
| H06 | Web Mercator Utils | `IWebMercatorUtils` | Tile coordinate calculations |
| H07 | Image Rotation Utils | `IImageRotationUtils` | Image rotation operations |
| H08 | Batch Validator | `IBatchValidator` | Image batch validation |
---
## Flow 1: System Initialization
**Purpose**: Initialize all system components on startup.
**Sequence**:
```
┌─────────────────┐
│ System Start │
└────────┬────────┘
▼
┌─────────────────┐
│ F17 load_config │ ← Load system configuration
└────────┬────────┘
▼
┌─────────────────┐
│ F03 Initialize │ ← Establish DB connection pool
│ connections │
└────────┬────────┘
▼
┌─────────────────────────────────────────┐
│ F16 load_model() × 4 │
│ ├─ SuperPoint (feature extraction) │
│ ├─ LightGlue (feature matching) │
│ ├─ DINOv2 (place recognition) │
│ └─ LiteSAM (cross-view matching) │
└────────┬────────────────────────────────┘
▼
┌─────────────────┐
│F04 Initialize │ ← Initialize tile cache
│ cache │
└────────┬────────┘
▼
┌─────────────────────────────────────────┐
│ F08 load_index() │
│ ← Load pre-built Faiss index from │
│ satellite provider │
└────────┬────────────────────────────────┘
▼
┌─────────────────┐
│ F12 Initialize │ ← Initialize chunk state tracking
└────────┬────────┘
▼
┌─────────────────┐
│ F02.1 Ready │ ← Ready to accept flights
└────────┬────────┘
▼
┌─────────────────┐
│ F01 Start │ ← Start FastAPI/Uvicorn
│ server │
└─────────────────┘
```
**Duration**: ~30 seconds (dominated by model loading)
---
## Flow 2: Flight Creation
**Purpose**: Create a new flight with initial configuration and prefetch satellite data.
**Trigger**: `POST /flights`
**Sequence**:
```
┌──────────┐ POST /flights ┌─────┐
│ Client │ ───────────────────► │ F01 │
└──────────┘ └──┬──┘
│ create_flight()
▼
┌─────────┐
│ F02.1 │ Flight Lifecycle Manager
└────┬────┘
┌────────────────────────┬─┴─┬────────────────────────┐
│ │ │ │
▼ │ ▼ ▼
┌───────────────┐ │ ┌───────────────┐ ┌───────────────┐
│F17 get_flight │ │ │F13 set_enu │ │F04 prefetch │
│ _config() │ │ │ _origin() │ │ _route │
└───────────────┘ │ └───────────────┘ │ _corridor() │
│ └───────┬───────┘
│ │
▼ ▼
┌─────────────┐ ┌─────────────────┐
│F03 insert │ │ Satellite │
│ _flight() │ │ Provider API │
└─────────────┘ │ GET tiles/batch │
└─────────────────┘
│
┌──────┴──────┐
│ F14 │
│create_stream│
└─────────────┘
│
┌──────┴──────┐
SSE Connection │ Client │
◄─────────────────────────────│ GET stream │
└─────────────┘
```
**Output**: `flight_id`, SSE stream established
---
## Flow 3: Image Upload
**Purpose**: Upload batch of UAV images for processing.
**Trigger**: `POST /flights/{flightId}/images/batch`
**Sequence**:
```
┌──────────┐ POST images/batch ┌─────┐
│ Client │ ──────────────────► │ F01 │
│ │ (10-50 images) └──┬──┘
└──────────┘ │ queue_batch()
▼
┌─────────────┐
│ F05 │ Image Input Pipeline
└──────┬──────┘
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│H08 validate │ │ Store to │ │F03 save │
│ _batch() │ │ disk │ │ _image │
└─────────────┘ └─────────────┘ │ _metadata()│
└─────────────┘
```
**Validation Rules**:
- Batch size: 10-50 images
- Naming convention: ADxxxxxx.jpg
- Sequential numbering
- Image dimensions: 640×480 to 6252×4168
---
## Flow 4: Normal Frame Processing (Tracking Good)
**Purpose**: Process a single frame when tracking quality is good.
**Sequence**:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ F02.2 Flight Processing Engine │
│ │
│ ┌─────────────┐ │
│ │ get_next │◄───────────────────────────────────────────────────────┐│
│ │ _image() │ ││
│ └──────┬──────┘ ││
│ │ F05 ││
│ ▼ ││
│ ┌─────────────────────────────────────────────────────────────┐ ││
│ │ get_active_chunk() via F12 │ ││
│ └──────────────────────────────┬──────────────────────────────┘ ││
│ ▼ ││
│ ┌─────────────────────────────────────────────────────────────┐ ││
│ │ requires_rotation_sweep()? via F06 │ ││
│ │ ├─ YES → Flow 5 (First Frame/Sharp Turn) │ ││
│ │ └─ NO → Pre-rotate to current heading │ ││
│ └──────────────────────────────┬──────────────────────────────┘ ││
│ ▼ ││
│ ┌─────────────────────────────────────────────────────────────┐ ││
│ │ F07 compute_relative_pose() │ ││
│ │ ├─ SuperPoint extract (prev + curr) via F16 │ ││
│ │ └─ LightGlue match via F16 │ ││
│ └──────────────────────────────┬──────────────────────────────┘ ││
│ │ ││
│ ┌─────────────────┼─────────────────┐ ││
│ │ Tracking Good? │ Tracking Lost? │ ││
│ ▼ ▼ ▼ ││
│ ┌─────────────────┐ ┌─────────────────────────────────┐ ││
│ │F12 add_frame │ │ →Flow 6 (Tracking Loss/Recovery)│ ││
│ │ _to_chunk() │ └─────────────────────────────────┘ ││
│ └────────┬────────┘ ││
│ ▼ ││
│ ┌─────────────────────────────────────────────────────────────┐ ││
│ │ F04 fetch_tile() + compute_tile_bounds() │ ││
│ └──────────────────────────────┬──────────────────────────────┘ ││
│ ▼ ││
│ ┌─────────────────────────────────────────────────────────────┐ ││
│ │ F09 align_to_satellite(image, tile, tile_bounds) │ ││
│ │ └─ LiteSAM matching via F16 │ ││
│ └──────────────────────────────┬──────────────────────────────┘ ││
│ │ ││
│ ┌─────────────────┴─────────────────┐ ││
│ │ Match Found? │ ││
│ ▼ ▼ ││
│ ┌──────────────────────────────────┐ ┌─────────────────────────┐ ││
│ │F10 add_absolute_factor(flight_id,│ │ Skip absolute anchor │ ││
│ │ frame_id, gps, covariance, │ │ (VO-only frame) │ ││
│ │ is_user_anchor=False) │ └─────────────────────────┘ ││
│ │F06 update_heading() │ ││
│ └────────────┬─────────────────────┘ ││
│ │ ││
│ └─────────────────────┬───────────────────────────────────┘│
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ F10 optimize_chunk(flight_id, chunk_id, iterations) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ F13 enu_to_gps() → Convert ENU pose to GPS │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ F14 update_frame_result() + publish_waypoint_update() │ │
│ │ └─ F15 send_frame_result() → SSE "frame_processed" │ │
│ │ └─ F03 save_frame_result() │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
**Performance**: < 5 seconds per frame
---
## Flow 5: First Frame / Sharp Turn (Rotation Sweep)
**Purpose**: Establish UAV heading when unknown or after sharp turn.
**Trigger**: `requires_rotation_sweep()` returns True
**Sequence**:
```
┌───────────────────────────────────────────────────────────────────────────┐
│ F06 Image Rotation Manager │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ try_rotation_steps(image, satellite_tile, tile_bounds) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┴────────────────────────────┐ │
│ │ For angle in [0°, 30°, 60°, ... 330°]: │ │
│ │ │ │
│ │ ┌─────────────────────────────────────┐ │ │
│ │ │ H07 rotate_image(image, angle) │ │ │
│ │ └───────────────────┬─────────────────┘ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F09 align_to_satellite(rotated, │ │ │
│ │ │ satellite_tile, tile_bounds) │ │ │
│ │ └───────────────────┬─────────────────┘ │ │
│ │ │ │ │
│ │ ┌──────────────┴──────────────┐ │ │
│ │ │ Match Found? │ │ │
│ │ ▼ ▼ │ │
│ │ ┌───────────────┐ ┌───────────────┐ │ │
│ │ │ Calculate │ │ Try next │ │ │
│ │ │ precise angle │ │ rotation │ │ │
│ │ └───────┬───────┘ └───────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F06 update_heading() → returns │ │ │
│ │ │ F02.2 calls F03 save_heading() │ │ │
│ │ └─────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────┴────────────┐ │
│ │ Return RotationResult │ │
│ │ or None │ │
│ └─────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────┘
```
**Output**: RotationResult with precise heading angle
---
## Flow 6: Tracking Loss / Recovery (Progressive Search)
**Purpose**: Recover localization after tracking loss.
**Trigger**: VO inlier count < 20 or LiteSAM match fails
**Sequence**:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ F02.2 calls F11 methods (direct returns, NO EVENTS) │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. F02.2 calls F11.start_search() → returns SearchSession │ │
│ │ F02.2 updates status to "recovering" │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 2. F02.2 calls F11.create_chunk_on_tracking_loss() via F12 │ │
│ │ └─ Proactive chunk creation (processing continues) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 3. Single-image recovery attempt: │ │
│ │ ├─ F06 requires_rotation_sweep() → trigger sweep │ │
│ │ ├─ F08 retrieve_candidate_tiles() (DINOv2) │ │
│ │ └─ Progressive tile search (1→4→9→16→25): │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────┐ │ │
│ │ │ For grid_size in [1, 4, 9, 16, 25]: │ │ │
│ │ │ ├─ F11.expand_search_radius() → tiles │ │ │
│ │ │ ├─ F02.2 fetches tiles via F04 │ │ │
│ │ │ ├─ F11.try_current_grid() → AlignmentResult │ │ │
│ │ │ │ │ │ │
│ │ │ └─ If match found: BREAK │ │ │
│ │ └───────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌────────────────────┴────────────────────┐ │
│ │ Single-image match found? │ │
│ ▼ ▼ │
│ ┌─────────────────────┐ ┌────────────────────────────┐ │
│ │ F11.mark_found() │ │ Continue chunk building │ │
│ │ Resume normal flow │ │ → Flow 7 (Chunk Building) │ │
│ └─────────────────────┘ │ → Flow 8 (Chunk Matching) │ │
│ │ (Background) │ │
│ └────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ If all strategies exhausted: │ │
│ │ → Flow 10 (User Input Recovery) │ │
│ └─────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 7: Chunk Building
**Purpose**: Build route chunks when tracking is lost, continuing VO within chunk.
**Sequence**:
```
┌───────────────────────────────────────────────────────────────────────────┐
│ F12 Route Chunk Manager │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ create_chunk(flight_id, start_frame_id) │ │
│ │ ├─ F10 create_chunk_subgraph(flight_id, chunk_id, │ │
│ │ │ start_frame_id) ← Factor graph subgraph │ │
│ │ ├─ Initialize chunk state (unanchored, active) │ │
│ │ └─ F03 save_chunk_state() │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┴────────────────────────────┐ │
│ │ For each frame in chunk: │ │
│ │ │ │
│ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F07 compute_relative_pose() │ │ │
│ │ └───────────────────┬─────────────────┘ │ │
│ │ ▼ │ │
│ │ ┌──────────────────────────────────────┐ │ │
│ │ │ F12 add_frame_to_chunk() │ │ │
│ │ │ └─ F10 add_relative_factor_to_chunk│ │ │
│ │ │ (flight_id, chunk_id, frame_i, │ │ │
│ │ │ frame_j, relative_pose, cov) │ │ │
│ │ └───────────────────┬──────────────────┘ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────┐ │ │
│ │ │ F10 optimize_chunk(flight_id, │ │ │
│ │ │ chunk_id, iterations) (local) │ │ │
│ │ └─────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ is_chunk_ready_for_matching()? │ │
│ │ ├─ Min 5 frames │ │
│ │ ├─ Max 20 frames │ │
│ │ └─ Internal consistency (good VO inlier counts) │ │
│ └─────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 8: Background Chunk Matching
**Purpose**: Asynchronously match unanchored chunks to satellite data.
**Trigger**: Background task (every 5 seconds)
**Sequence**:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ F11 process_unanchored_chunks() (Background Task) │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ while flight_active: │ │
│ │ ├─ unanchored = F12 get_chunks_for_matching() │ │
│ │ │ │ │
│ │ └─ for chunk in unanchored: │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │ if F12 is_chunk_ready_for_matching(chunk_id): │ │ │
│ │ │ ├─ F12 mark_chunk_matching(chunk_id) │ │ │
│ │ │ │ │ │ │
│ │ │ ├─ STEP 1: Chunk Semantic Matching │ │ │
│ │ │ │ ├─ F12 get_chunk_images() │ │ │
│ │ │ │ └─ F08 retrieve_candidate_tiles_for_chunk() │ │ │
│ │ │ │ └─ F08 compute_chunk_descriptor() │ │ │
│ │ │ │ └─ H04 Faiss search() │ │ │
│ │ │ │ │ │ │
│ │ │ ├─ STEP 2: Chunk LiteSAM Matching (with rotation) │ │ │
│ │ │ │ ├─ For each candidate tile: │ │ │
│ │ │ │ │ ├─ F04 get_tile + compute_tile_bounds() │ │ │
│ │ │ │ │ ├─ F06 try_chunk_rotation_steps() │ │ │
│ │ │ │ │ │ └─ F09 align_chunk_to_satellite() │ │ │
│ │ │ │ │ └─ If match: BREAK │ │ │
│ │ │ │ │ │ │
│ │ │ └─ If match found: │ │ │
│ │ │ └─ → Flow 9 (Chunk Merging) │ │ │
│ │ │ │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ sleep(5 seconds) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 9: Chunk Merging
**Purpose**: Merge anchored chunk into main trajectory.
**Trigger**: Successful chunk matching
**Sequence**:
```
┌───────────────────────────────────────────────────────────────────────-──┐
│ F02.2 orchestrates via F11.merge_chunk_to_trajectory() │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. Get chunk frames: F12 get_chunk_frames(new_chunk_id) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 2. Anchor chunk: F12 mark_chunk_anchored(new_chunk_id, gps) │ │
│ │ └─ F10 add_chunk_anchor(flight_id, new_chunk_id, frame_id│ │
│ │ gps, covariance) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 3. Determine main chunk (predecessor or "main") │ │
│ │ └─ Returns main_chunk_id (predecessor or "main") │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 4. Merge: F12 merge_chunks(main_chunk_id,new_chunk_id,Sim3) │ │
│ │ ├─ F10 merge_chunk_subgraphs(flight_id, new_chunk_id, │ │
│ │ │ main_chunk_id, transform) ← Apply Sim(3) │ │
│ │ ├─ F12 deactivate_chunk(new_chunk_id) │ │
│ │ └─ F03 save_chunk_state() │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 5. Optimize global: F10 optimize_global(flight_id, │ │
│ │ iterations) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 6. F11 returns True → F02.2 coordinates result updates: │ │
│ │ ├─ F02.2 calls F10 get_trajectory(flight_id) → ENU poses │ │
│ │ ├─ F02.2 calls F13 enu_to_gps() for each frame │ │
│ │ ├─ F02.2 constructs List[RefinedFrameResult] │ │
│ │ └─ F02.2 calls F14.update_results_after_chunk_merge() │ │
│ │ ├─ F03 save_frame_result() + update_waypoint() │ │
│ │ └─ F15 send_refinement() → SSE "frame_refined" × N │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
**Sim(3) Transform**: Translation + Rotation + Scale alignment between chunks
**Note**: F14 does NOT call F10/F13. F02.2 performs coordinate conversion and passes GPS results.
---
## Flow 10: User Input Recovery
**Purpose**: Request human assistance when all automatic recovery fails.
**Trigger**: Progressive search exhausted (25 tiles), chunk matching failed
**Sequence**:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ F02.2 orchestrates via F11.create_user_input_request() │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. F11.create_user_input_request() → returns UserInputRequest│ │
│ │ └─ Gets UAV image, top-5 candidates from F08 │ │
│ │ └─ Returns request object (does NOT call F15) │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 2. F02.2 receives UserInputRequest │ │
│ │ ├─ F02.2 calls F15.send_user_input_request() │ │
│ │ │ → SSE "user_input_needed" │ │
│ │ └─ F02.2 updates status to "BLOCKED" │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
│
▼ Client receives SSE event
┌───────────────────────────────────────────────────────────────────────────┐
│ Client UI │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Display UAV image + candidate satellite tiles │ │
│ │ User clicks corresponding point on satellite tile │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ POST /flights/{flightId}/user-fix │ │
│ │ body: { frame_id, uav_pixel, satellite_gps } │ │
│ └─────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ F11 apply_user_anchor() │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. Validate anchor data │ │
│ │ 2. F10 add_absolute_factor(is_user_anchor=True) ← σ=5m │ │
│ │ 3. F10 optimize() │ │
│ │ 4. EMIT UserFixApplied event │ │
│ │ └─ F02.2 updates status to "PROCESSING" │ │
│ │ 5. Resume processing loop │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 11: Asynchronous Refinement
**Purpose**: Back-propagate optimization improvements to previous frames.
**Trigger**: New absolute GPS factor added, chunk merged
**Sequence**:
```
┌─────────────────────────────────────────────────────────────-────────────┐
│ F10 Background Optimization │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. New absolute factor triggers batch optimization │ │
│ │ 2. F10 optimize(iterations=50-100) │ │
│ │ └─ Levenberg-Marquardt with robust kernels │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 3. Identify refined frames │ │
│ │ 4. F14 mark_refined(frame_ids) │ │
│ │ ├─ For each frame: │ │
│ │ │ ├─ F10 get_trajectory(flight_id) → ENU pose │ │
│ │ │ ├─ F13 enu_to_gps(flight_id, enu_pose) │ │
│ │ │ ├─ F03 save_frame_result(refined=True) │ │
│ │ │ └─ F03 update_waypoint() │ │
│ │ │ │ │
│ │ └─ F15 send_refinement() × N → SSE "frame_refined" × N │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 12: Object to GPS Conversion
**Purpose**: Convert detected object pixel coordinates to GPS (external integration).
**Trigger**: `POST /flights/{flightId}/frames/{frameId}/object-to-gps`
**Sequence**:
```
┌──────────────────┐ POST object-to-gps ┌─────┐
│ External System │ ────────────────────►│ F01 │
│ (Azaion.Inference│ {pixel_x, pixel_y} └──┬──┘
└──────────────────┘ │
▼
┌─────────────┐
│ F13 │ Coordinate Transformer
└──────┬──────┘
┌───────────────────┼───────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│F10 get_pose │ │F17 camera │ │ H01 Camera │
│ (frame_id) │ │ _params │ │ Model │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───────────────────┼───────────────────┘
▼
┌─────────────┐
│ pixel_to_gps│
│ calculation │
└──────┬──────┘
│
▼
┌───────────────────────────┐
│ Return ObjectGPSResponse │
│ { gps, accuracy_meters }│
└───────────────────────────┘
```
---
## Flow 13: Flight Completion
**Purpose**: Complete flight processing and clean up resources.
**Trigger**: All images processed successfully
**Sequence**:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ F02.1 Flight Lifecycle Manager │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 1. All frames processed (Signal from F02.2) │ │
│ │ 2. Wait for pending chunk merges (if any) │ │
│ │ 3. F10 optimize_global(flight_id, iterations=100) │ │
│ │ ← Final optimization │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 4. F14 mark_refined(all_frames) │ │
│ │ └─ Final refinement updates │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 5. F03 save_flight_state(status="completed") │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 6. F15 send event → SSE "flight_completed" │ │
│ └──────────────────────────────┬──────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ 7. Cleanup: │ │
│ │ ├─ Stop background chunk matching task │ │
│ │ ├─ F04 clear_flight_cache(flight_id) (optional) │ │
│ │ └─ Release flight resources │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Flow 14: System Shutdown
**Purpose**: Gracefully shutdown all components.
**Sequence**:
```
┌─────────────────┐
│ Shutdown Signal │
└────────┬────────┘
▼
┌─────────────────┐
│ F01 Stop │ ← Stop accepting requests
│ accepting │
└────────┬────────┘
▼
┌─────────────────┐
│ F02.1 Complete/ │ ← Complete or cancel active flights
│ Cancel │
└────────┬────────┘
▼
┌─────────────────┐
│ F11 Stop │ ← Stop background chunk matching
│ background │
└────────┬────────┘
▼
┌─────────────────┐
│ F12 Save chunk │ ← Save chunk state for recovery
│ state │
└────────┬────────┘
▼
┌─────────────────┐
│ F16 Unload │ ← Unload ML models, free GPU memory
│ models │
└────────┬────────┘
▼
┌─────────────────┐
│ F03 Close │ ← Close database connections
│ connections │
└────────┬────────┘
▼
┌─────────────────┐
│ F04 Flush cache │ ← Flush satellite tile cache
└────────┬────────┘
▼
┌─────────────────┐
│ Shutdown │
│ Complete │
└─────────────────┘
```
---
## Component Interaction Summary
### Data Flow Direction
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ EXTERNAL │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Client │ │ Satellite │ │ External │ │
│ │ (UI) │ │ Provider │ │ Detector │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
└─────────┼───────────────────┼───────────────────┼───────────────────────────┘
│ REST/SSE │ HTTP │ REST
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API LAYER │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ F01 Flight API │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ ORCHESTRATION LAYER │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ F02.1 Flight Lifecycle Manager │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ (Spawns/Manages) │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ F02.2 Flight Processing Engine │ │
│ │ (Processing loop, State machine, Recovery orchestration) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
├─────────────────────────────────────────────────────────────┐
▼ ▼
┌─────────────────────────────────────────────┐ ┌────────────────────────────┐
│ DATA MANAGEMENT LAYER │ │ RECOVERY LAYER │
│ ┌─────────────┐ ┌─────────────┐ │ │ ┌───────────────────────┐ │
│ │ F04 │ │ F05 │ │ │ │ F11 │ │
│ │ Satellite │ │ Image │ │ │ │ Failure Recovery │ │
│ │ Data │ │ Input │ │ │ │ (Logic & Strategies) │ │
│ └─────────────┘ └─────────────┘ │ │ └───────────────────────┘ │
│ │ │ │ │
│ ┌─────────────┐ ┌─────────────┐ │ │ ▼ │
│ │ F12 │ │ F03 │ │ │ ┌───────────────────────┐ │
│ │Route Chunk │ │ Flight │ │ │ │ F12 │ │
│ │ Manager │ │ Database │ │ │ │ (Chunk state source) │ │
│ └─────────────┘ └─────────────┘ │ │ └───────────────────────┘ │
└─────────────────────────────────────────────┘ └────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ VISUAL PROCESSING LAYER │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌──────────────┐ │
│ │ F06 │ │ F07 │ │ F08 │ │ F09 │ │
│ │ Rotation │ │ Sequential VO │ │ Global │ │ Metric │ │
│ │ Manager │ │(SuperPoint+ │ │ Place │ │ Refinement │ │
│ │ (30° sweeps) │ │ LightGlue) │ │ Recognition │ │ (LiteSAM) │ │
│ │ │ │ │ │ (DINOv2) │ │ │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ STATE ESTIMATION LAYER │
│ ┌──────────────────────────────────────────────────────────────────k──┐ │
│ │ F10 Factor Graph Optimizer │ │
│ │ (GTSAM, iSAM2, Robust kernels, Chunk subgraphs, Sim(3) merging) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ OUTPUT LAYER │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ F13 │ │ F14 │ │ F15 │ │
│ │ Coordinate │ │ Result │ │ SSE │ │
│ │ Transformer │ │ Manager │ │ Streamer │ │
│ │ (ENU↔GPS↔Px) │ │ │ │ │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────-┐
│ INFRASTRUCTURE LAYER │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────────────────-┐│
│ │ F16 │ │ F17 │ │ HELPERS ││
│ │ Model │ │Configuration │ │ H01-H08 (Camera, GSD, Kernels, ││
│ │ Manager │ │ Manager │ │ Faiss, Monitor, Mercator, etc) ││
│ └───────────────┘ └───────────────┘ └────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────────────────────┘
```
---
## Status-Based Recovery Communication
F11 Failure Recovery Coordinator uses **Direct Call Returns** to communicate with F02.2 Flight Processing Engine. Events have been removed to eliminate circular dependencies and clarify control flow.
| Action | Caller | Called | Return/Outcome |
|--------|--------|--------|----------------|
| `start_search` | F02.2 | F11 | `SearchSession` object |
| `try_current_grid` | F02.2 | F11 | `AlignmentResult` (or None) |
| `create_user_input_request` | F02.2 | F11 | `UserInputRequest` object |
| `apply_user_anchor` | F02.2 | F11 | Success boolean |
F02.2 is responsible for updating the flight status (e.g., "recovering", "blocked", "processing") based on these return values.
---
## SSE Events to Client
| Event | Trigger | Data |
|-------|---------|------|
| `frame_processed` | Frame completed | frame_id, gps, altitude, heading, confidence |
| `frame_refined` | Trajectory refined | frame_id, updated gps, refined=true |
| `search_expanded` | Progressive search | frame_id, grid_size, status |
| `user_input_needed` | Recovery exhausted | request_id, frame_id, candidate_tiles |
| `processing_blocked` | Status change | reason, frame_id |
| `flight_completed` | Flight done | statistics |
---
## Performance Targets
| Flow | Target |
|------|--------|
| System Initialization | < 30 seconds |
| Flight Creation | < 500ms response |
| Image Batch Upload | < 2 seconds (50 images) |
| Per-Frame Processing | < 5 seconds |
| Rotation Sweep (12 rotations) | < 1.2 seconds |
| Progressive Search (25 tiles) | < 1.5 seconds |
| Chunk Matching | < 3 seconds (background) |
| SSE Event Latency | < 500ms |
---
## Accuracy Targets
| Metric | Target |
|--------|--------|
| GPS Accuracy | 60% < 20m, 80% < 50m |
| Mean Reprojection Error | < 1.0 pixels |
| Place Recognition Recall@5 | > 85% |
| LiteSAM Success Rate | > 95% (when rotation correct) |
_docs/02_components/system_flows_diagrams.md
+603
View File
@@ -0,0 +1,603 @@
# ASTRAL-Next System Flow Diagrams
## Component Architecture Diagram
```mermaid
graph TB
subgraph External["External Systems"]
Client[Client UI]
SatProv[Satellite Provider]
ExtDet[External Detector]
end
subgraph API["API Layer"]
F01[F01 Flight API]
end
subgraph Orchestration["Orchestration Layer"]
F02[F02 Flight Processor]
end
subgraph Data["Data Management"]
F03[F03 Flight Database]
F04[F04 Satellite Data Manager]
F05[F05 Image Input Pipeline]
F12[F12 Route Chunk Manager]
end
subgraph Recovery["Recovery Layer"]
F11[F11 Failure Recovery Coordinator]
end
subgraph Visual["Visual Processing"]
F06[F06 Image Rotation Manager]
F07[F07 Sequential Visual Odometry]
F08[F08 Global Place Recognition]
F09[F09 Metric Refinement]
end
subgraph State["State Estimation"]
F10[F10 Factor Graph Optimizer]
end
subgraph Output["Output Layer"]
F13[F13 Coordinate Transformer]
F14[F14 Result Manager]
F15[F15 SSE Event Streamer]
end
subgraph Infra["Infrastructure"]
F16[F16 Model Manager]
F17[F17 Configuration Manager]
end
Client -->|REST| F01
F15 -->|SSE| Client
ExtDet -->|REST| F01
F04 -->|HTTP| SatProv
F01 --> F02
F02 --> F03
F02 --> F05
F02 --> F11
F02 --> F07
F02 --> F10
F02 --> F12
F02 --> F04
F11 --> F08
F11 --> F09
F11 --> F12
F11 -.->|events| F02
F11 -.->|events| F14
F06 --> F09
F07 --> F10
F09 --> F10
F12 --> F10
F10 --> F13
F14 --> F13
F14 --> F15
F14 --> F03
F07 --> F16
F08 --> F16
F09 --> F16
F02 --> F17
```
## Flow 1: System Initialization
```mermaid
sequenceDiagram
participant Main as System
participant F17 as Config Manager
participant F03 as Flight Database
participant F16 as Model Manager
participant F04 as Satellite Manager
participant F08 as Place Recognition
participant F12 as Chunk Manager
participant F02 as Flight Processor
participant F01 as Flight API
Main->>F17: load_config()
F17-->>Main: SystemConfig
Main->>F03: Initialize connections
F03-->>Main: Connection pool ready
Main->>F16: load_model("SuperPoint")
Main->>F16: load_model("LightGlue")
Main->>F16: load_model("DINOv2")
Main->>F16: load_model("LiteSAM")
F16-->>Main: Models loaded (~25s)
Main->>F04: Initialize cache
F04-->>Main: Cache ready
Main->>F08: load_index()
F08-->>Main: Faiss index loaded
Main->>F12: Initialize
F12-->>Main: Chunk tracking ready
Main->>F02: Ready
F02-->>Main: Ready to accept flights
Main->>F01: Start server
F01-->>Main: FastAPI running
```
## Flow 2: Flight Creation
```mermaid
sequenceDiagram
participant C as Client
participant F01 as Flight API
participant F02 as Flight Processor
participant F17 as Config Manager
participant F13 as Coord Transformer
participant F04 as Satellite Manager
participant F03 as Flight Database
participant F15 as SSE Streamer
C->>F01: POST /flights
F01->>F02: create_flight()
F02->>F17: get_flight_config()
F17-->>F02: CameraParams, Altitude
F02->>F13: set_enu_origin(start_gps)
F02->>F04: prefetch_route_corridor()
F04-->>F02: Prefetching async
F02->>F03: insert_flight()
F03-->>F02: flight_id
F01-->>C: 201 Created {flight_id}
C->>F01: GET /flights/{id}/stream
F01->>F15: create_stream()
F15-->>C: SSE Connection
```
## Flow 3: Normal Frame Processing
```mermaid
sequenceDiagram
participant F02 as Flight Processor
participant F05 as Image Pipeline
participant F12 as Chunk Manager
participant F06 as Rotation Manager
participant F07 as Sequential VO
participant F04 as Satellite Manager
participant F09 as Metric Refinement
participant F10 as Factor Graph
participant F13 as Coord Transformer
participant F14 as Result Manager
participant F15 as SSE Streamer
F02->>F05: get_next_image()
F05-->>F02: ImageData
F02->>F12: get_active_chunk()
F12-->>F02: ChunkHandle
F02->>F06: requires_rotation_sweep()
F06-->>F02: false (heading known)
F02->>F07: compute_relative_pose_in_chunk()
F07-->>F02: RelativePose
F02->>F12: add_frame_to_chunk()
F12->>F10: add_relative_factor_to_chunk()
F02->>F04: fetch_tile() + compute_tile_bounds()
F04-->>F02: tile, tile_bounds
F02->>F09: align_to_satellite(img, tile, bounds)
F09-->>F02: AlignmentResult (GPS)
F02->>F10: add_absolute_factor()
F02->>F10: optimize_chunk()
F10-->>F02: OptimizationResult
F02->>F13: enu_to_gps()
F13-->>F02: GPSPoint
F02->>F14: update_frame_result()
F14->>F15: send_frame_result()
F15-->>Client: SSE "frame_processed"
```
## Flow 4: Rotation Sweep (First Frame / Sharp Turn)
```mermaid
sequenceDiagram
participant F02 as Flight Processor
participant F06 as Rotation Manager
participant H07 as Rotation Utils
participant F09 as Metric Refinement
participant F03 as Flight Database
F02->>F06: try_rotation_steps(img, tile, bounds)
loop For angle in [0°, 30°, ... 330°]
F06->>H07: rotate_image(img, angle)
H07-->>F06: rotated_img
F06->>F09: align_to_satellite(rotated_img, tile, bounds)
F09-->>F06: AlignmentResult
alt Match Found (confidence > 0.7)
F06->>F06: calculate_precise_angle()
F06->>F03: save_heading()
F06-->>F02: RotationResult
end
end
alt No Match Found
F06-->>F02: None (trigger recovery)
end
```
## Flow 5: Tracking Loss Recovery
```mermaid
sequenceDiagram
participant F02 as Flight Processor
participant F11 as Failure Recovery
participant F12 as Chunk Manager
participant F06 as Rotation Manager
participant F08 as Place Recognition
participant F04 as Satellite Manager
participant F09 as Metric Refinement
F02->>F11: start_search(frame_id, estimated_gps)
F11-->>F02: SearchSession
Note over F11: Emit RecoveryStarted event
F11->>F12: create_chunk_on_tracking_loss()
F12-->>F11: ChunkHandle (processing continues)
F11->>F06: requires_rotation_sweep()
F11->>F08: retrieve_candidate_tiles()
loop Progressive Search [1, 4, 9, 16, 25]
F11->>F04: expand_search_grid(grid_size)
F04-->>F11: tiles
loop For each tile
F11->>F04: compute_tile_bounds()
F11->>F09: align_to_satellite(img, tile, bounds)
alt Match Found
Note over F11: Emit RecoverySucceeded
F11-->>F02: RecoveryResult(success=true)
end
end
end
alt All Failed
Note over F11: Emit UserInputNeeded
F11-->>F02: RecoveryResult(success=false)
end
```
## Flow 6: Chunk Matching (Background)
```mermaid
sequenceDiagram
participant F11 as Failure Recovery
participant F12 as Chunk Manager
participant F08 as Place Recognition
participant F06 as Rotation Manager
participant F04 as Satellite Manager
participant F09 as Metric Refinement
participant F10 as Factor Graph
loop Every 5 seconds
F11->>F12: get_chunks_for_matching()
F12-->>F11: List[ChunkHandle]
loop For each unanchored chunk
F11->>F12: is_chunk_ready_for_matching()
alt Chunk Ready
F11->>F12: mark_chunk_matching()
Note over F11: Step 1: Semantic Matching
F11->>F12: get_chunk_images()
F11->>F08: retrieve_candidate_tiles_for_chunk()
F08-->>F11: List[TileCandidate]
Note over F11: Step 2: LiteSAM with Rotation
loop For each candidate tile
F11->>F04: get_tile + compute_tile_bounds()
F11->>F06: try_chunk_rotation_steps()
F06->>F09: align_chunk_to_satellite()
alt Match Found
F09-->>F11: ChunkAlignmentResult
end
end
alt Match Found
F11->>F12: mark_chunk_anchored()
F12->>F10: add_chunk_anchor()
F11->>F12: merge_chunks()
F12->>F10: merge_chunks(Sim3)
F11->>F10: optimize_global()
Note over F11: Emit ChunkMerged event
end
end
end
end
```
## Flow 7: User Input Recovery
```mermaid
sequenceDiagram
participant F11 as Failure Recovery
participant F08 as Place Recognition
participant F15 as SSE Streamer
participant C as Client
participant F01 as Flight API
participant F10 as Factor Graph
participant F02 as Flight Processor
F11->>F08: retrieve_candidate_tiles()
F08-->>F11: Top-5 candidates
F11->>F15: send_user_input_request()
F15-->>C: SSE "user_input_needed"
Note over F11: Emit UserInputNeeded event
Note over F02: Status = BLOCKED
C->>F01: POST /user-fix {pixel, gps}
F01->>F11: apply_user_anchor()
F11->>F10: add_absolute_factor(is_user_anchor=true)
F11->>F10: optimize()
Note over F11: Emit UserFixApplied event
Note over F02: Status = PROCESSING
F11-->>F01: Success
F01-->>C: 200 OK
```
## Flow 8: Result Publishing & Refinement
```mermaid
sequenceDiagram
participant F10 as Factor Graph
participant F14 as Result Manager
participant F13 as Coord Transformer
participant F03 as Flight Database
participant F15 as SSE Streamer
participant C as Client
Note over F10: New absolute factor added
F10->>F10: optimize(batch)
F10->>F14: mark_refined(frame_ids)
loop For each refined frame
F14->>F10: get_trajectory()
F10-->>F14: Pose (ENU)
F14->>F13: enu_to_gps(flight_id, enu)
F13-->>F14: GPSPoint
F14->>F03: save_frame_result(refined=true)
F14->>F03: update_waypoint()
F14->>F15: send_refinement()
F15-->>C: SSE "frame_refined"
end
```
## Flow 9: Object to GPS Conversion
```mermaid
sequenceDiagram
participant Ext as External Detector
participant F01 as Flight API
participant F13 as Coord Transformer
participant F10 as Factor Graph
participant F17 as Config Manager
participant H01 as Camera Model
Ext->>F01: POST /object-to-gps {pixel_x, pixel_y}
F01->>F13: image_object_to_gps(pixel, frame_id)
F13->>F10: get_pose(frame_id)
F10-->>F13: Pose (ENU)
F13->>F17: get_camera_params()
F17-->>F13: CameraParameters
F13->>H01: unproject(pixel)
F13->>F13: intersect_ground_plane()
F13->>F13: enu_to_gps()
F13-->>F01: GPSPoint
F01-->>Ext: {gps, accuracy_meters}
```
## Complete System Flow Overview
```mermaid
flowchart TB
subgraph Init["System Initialization"]
direction TB
I1[Load Config F17]
I2[Init DB F03]
I3[Load Models F16]
I4[Init Cache F04]
I5[Load Faiss F08]
I6[Start API F01]
I1 --> I2 --> I3 --> I4 --> I5 --> I6
end
subgraph Flight["Flight Lifecycle"]
direction TB
FL1[Create Flight]
FL2[Upload Images]
FL3[Process Frames]
FL4[Complete Flight]
FL1 --> FL2 --> FL3 --> FL4
end
subgraph Process["Frame Processing"]
direction TB
P1{First Frame?}
P2[Rotation Sweep]
P3[Sequential VO]
P4{Tracking OK?}
P5[Single Tile Match]
P6[Optimize]
P7[Publish Result]
P1 -->|Yes| P2
P1 -->|No| P3
P2 --> P3
P3 --> P4
P4 -->|Yes| P5
P5 --> P6
P6 --> P7
end
subgraph Recovery["Recovery Flow"]
direction TB
R1[Create Chunk]
R2[Progressive Search]
R3{Match Found?}
R4[Build Chunk]
R5[Chunk Matching]
R6{Chunk Match?}
R7[Merge Chunk]
R8[Request User Input]
R9[Apply User Anchor]
P4 -->|No| R1
R1 --> R2
R2 --> R3
R3 -->|Yes| P6
R3 -->|No| R4
R4 --> R5
R5 --> R6
R6 -->|Yes| R7
R7 --> P6
R6 -->|No| R8
R8 --> R9
R9 --> P6
end
Init --> Flight
FL3 --> Process
```
## Event Flow Diagram
```mermaid
flowchart LR
subgraph Events["F11 Emits Events"]
E1[RecoveryStarted]
E2[RecoverySucceeded]
E3[RecoveryFailed]
E4[UserInputNeeded]
E5[UserFixApplied]
E6[ChunkCreated]
E7[ChunkAnchored]
E8[ChunkMerged]
end
subgraph F02Sub["F02 Subscribes"]
S1[Update status: recovering]
S2[Update status: processing]
S3[Update status: blocked]
S4[Update status: blocked]
S5[Resume processing]
S6[Log chunk creation]
S7[Log anchor]
S8[Trigger result update]
end
subgraph F14Sub["F14 Subscribes"]
R1[Update merged frame results]
end
E1 --> S1
E2 --> S2
E3 --> S3
E4 --> S4
E5 --> S5
E6 --> S6
E7 --> S7
E8 --> S8
E8 --> R1
```
## Data Flow Through Layers
```mermaid
flowchart TB
subgraph Input["Input"]
IMG[UAV Images]
SAT[Satellite Tiles]
USR[User Anchors]
end
subgraph Processing["Processing"]
SP[SuperPoint Features]
LG[LightGlue Matches]
DINO[DINOv2 Descriptors]
LITE[LiteSAM Homography]
end
subgraph State["State Estimation"]
REL[Relative Factors]
ABS[Absolute Factors]
CHUNK[Chunk Subgraphs]
OPT[iSAM2 Optimization]
end
subgraph Output["Output"]
ENU[ENU Trajectory]
GPS[GPS Coordinates]
SSE[SSE Events]
DB[Database]
end
IMG --> SP
IMG --> DINO
IMG --> LITE
SAT --> LITE
SAT --> DINO
USR --> ABS
SP --> LG
LG --> REL
DINO --> ABS
LITE --> ABS
REL --> CHUNK
ABS --> CHUNK
CHUNK --> OPT
OPT --> ENU
ENU --> GPS
GPS --> SSE
GPS --> DB
```