add features

docs/02_components/03_flight_database/03.01_feature_flight_crud_operations.md

@@ -0,0 +1,114 @@
+# Feature: Flight & Waypoint CRUD Operations
+
+## Description
+Core database infrastructure and CRUD operations for flights, waypoints, and geofences. Provides connection pooling, transaction support, and all primary data operations. This feature handles the main entities that define a flight and its route.
+
+## Component APIs Implemented
+- `execute_transaction(operations: List[Callable]) -> bool`
+- `insert_flight(flight: Flight) -> str`
+- `update_flight(flight: Flight) -> bool`
+- `query_flights(filters: Dict[str, Any], limit: int, offset: int) -> List[Flight]`
+- `get_flight_by_id(flight_id: str) -> Optional[Flight]`
+- `delete_flight(flight_id: str) -> bool`
+- `get_waypoints(flight_id: str, limit: Optional[int] = None) -> List[Waypoint]`
+- `insert_waypoint(flight_id: str, waypoint: Waypoint) -> str`
+- `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> bool`
+- `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchResult`
+
+## External Tools and Services
+- **PostgreSQL**: Primary database
+- **SQLAlchemy**: ORM and connection pooling
+- **Alembic**: Schema migrations
+
+## Internal Methods
+| Method | Purpose |
+|--------|---------|
+| `_get_connection()` | Acquire connection from pool |
+| `_release_connection(conn)` | Return connection to pool |
+| `_execute_with_retry(query, params, retries=3)` | Execute query with automatic retry on transient errors |
+| `_build_flight_from_row(row)` | Map database row to Flight object |
+| `_build_waypoint_from_row(row)` | Map database row to Waypoint object |
+| `_serialize_camera_params(params)` | Serialize CameraParameters to JSONB |
+| `_deserialize_camera_params(jsonb)` | Deserialize JSONB to CameraParameters |
+| `_build_filter_query(filters)` | Build WHERE clause from filter dict |
+
+## Unit Tests
+1. **Connection management**
+   - Pool returns connections correctly
+   - Connection recycling after timeout
+   - Health check on stale connections
+
+2. **execute_transaction**
+   - All operations succeed → commit
+   - One operation fails → rollback all
+   - Connection error → retry and succeed
+   - Nested transactions handled correctly
+
+3. **insert_flight**
+   - Valid flight with 100 waypoints → all persisted
+   - Duplicate flight_id → raises IntegrityError
+   - Partial failure mid-insert → complete rollback
+   - Empty waypoints list → flight created with no waypoints
+
+4. **update_flight**
+   - Existing flight → returns True, fields updated
+   - Non-existent flight → returns False
+   - Only specified fields updated, others unchanged
+
+5. **query_flights**
+   - Filter by name → returns matching flights
+   - Filter by status → returns matching flights
+   - Pagination (offset=100, limit=50) → returns correct slice
+   - No matches → returns empty list
+
+6. **get_flight_by_id**
+   - Existing flight → returns complete Flight with waypoints
+   - Non-existent flight → returns None
+   - Large flight (3000 waypoints) → returns within 150ms
+
+7. **delete_flight**
+   - Existing flight → returns True, cascade to all tables
+   - Non-existent flight → returns False
+   - Verify cascade deletes waypoints, geofences, etc.
+
+8. **get_waypoints**
+   - All waypoints (limit=None) → returns complete list
+   - Limited (limit=100) → returns first 100
+   - Non-existent flight → returns empty list
+
+9. **insert_waypoint**
+   - Valid insertion → returns waypoint_id
+   - Non-existent flight → raises ForeignKeyError
+
+10. **update_waypoint**
+    - Existing waypoint → returns True
+    - Non-existent waypoint → returns False
+    - Concurrent updates → no data corruption
+
+11. **batch_update_waypoints**
+    - Batch of 100 → all succeed
+    - Partial failure → returns failed_ids
+    - Empty batch → returns success with updated_count=0
+
+## Integration Tests
+1. **Complete flight lifecycle**
+   - insert_flight() → update_waypoint() × 100 → get_flight_by_id() → delete_flight()
+   - Verify all data persisted and cascade delete works
+
+2. **High-frequency waypoint updates**
+   - Insert flight with 2000 waypoints
+   - Concurrent update_waypoint() calls (100/sec)
+   - Verify throughput > 200 ops/sec
+   - Verify no data corruption
+
+3. **Transaction atomicity**
+   - Begin transaction with 5 operations
+   - Fail on operation 3
+   - Verify operations 1-2 rolled back
+
+4. **Connection pool behavior**
+   - Exhaust pool → new requests wait
+   - Pool recovery after connection failures
+   - Connection reuse efficiency
+
+

docs/02_components/03_flight_database/03.02_feature_processing_state_persistence.md

@@ -0,0 +1,103 @@
+# Feature: Processing State Persistence
+
+## Description
+Persistence layer for flight processing state, frame results, and heading history. Supports crash recovery by persisting processing progress and enables temporal smoothing through heading history tracking. Critical for maintaining processing continuity and data integrity during frame refinement operations.
+
+## Component APIs Implemented
+- `save_flight_state(flight_state: FlightState) -> bool`
+- `load_flight_state(flight_id: str) -> Optional[FlightState]`
+- `query_processing_history(filters: Dict[str, Any]) -> List[FlightState]`
+- `save_frame_result(flight_id: str, frame_result: FrameResult) -> bool`
+- `get_frame_results(flight_id: str) -> List[FrameResult]`
+- `save_heading(flight_id: str, frame_id: int, heading: float, timestamp: datetime) -> bool`
+- `get_heading_history(flight_id: str, last_n: Optional[int] = None) -> List[HeadingRecord]`
+- `get_latest_heading(flight_id: str) -> Optional[float]`
+
+## External Tools and Services
+- **PostgreSQL**: Primary database
+- **SQLAlchemy**: ORM and connection pooling
+
+## Internal Methods
+| Method | Purpose |
+|--------|---------|
+| `_build_flight_state_from_row(row)` | Map database row to FlightState object |
+| `_build_frame_result_from_row(row)` | Map database row to FrameResult object |
+| `_build_heading_record_from_row(row)` | Map database row to HeadingRecord object |
+| `_upsert_flight_state(state)` | Insert or update flight state (single source of truth) |
+| `_upsert_frame_result(flight_id, result)` | Insert or update frame result (handles refinement updates) |
+
+## Unit Tests
+1. **save_flight_state**
+   - New state → created successfully
+   - Update existing state → overwrites previous
+   - Verify all fields persisted correctly
+   - Verify heading NOT stored in flight_state (use heading_history)
+
+2. **load_flight_state**
+   - Existing state → returns FlightState object
+   - Non-existent flight → returns None
+   - Verify all fields deserialized correctly
+
+3. **query_processing_history**
+   - Filter by date range → returns flights in range
+   - Filter by status → returns matching flights
+   - Combined filters → returns intersection
+   - No matches → returns empty list
+
+4. **save_frame_result**
+   - New frame → persisted successfully
+   - Update on refinement → overwrites with refined=True
+   - Verify GPS, altitude, heading, confidence persisted
+   - Verify timestamp and updated_at set correctly
+
+5. **get_frame_results**
+   - Flight with 500 frames → returns all results ordered by frame_id
+   - No results → returns empty list
+   - Performance: 2000 frames returned within 100ms
+
+6. **save_heading**
+   - New heading → persisted correctly
+   - Overwrite same frame_id → updates value
+   - Verify timestamp persisted
+   - Heading range validation (0-360)
+
+7. **get_heading_history**
+   - All headings (last_n=None) → returns complete history
+   - Last 10 headings → returns 10 most recent by timestamp
+   - Non-existent flight → returns empty list
+   - Ordered by frame_id descending
+
+8. **get_latest_heading**
+   - Has history → returns latest heading value
+   - No history → returns None
+   - After multiple saves → returns most recent
+
+## Integration Tests
+1. **Processing state recovery**
+   - Save state with frames_processed=250
+   - Simulate crash (close connection)
+   - Reconnect and load_flight_state()
+   - Verify state intact, processing can resume
+
+2. **Frame result refinement flow**
+   - save_frame_result() with refined=False
+   - Later, save_frame_result() with refined=True
+   - get_frame_results() → shows refined=True
+   - Verify GPS coordinates updated
+
+3. **Heading history for smoothing**
+   - save_heading() × 100 frames
+   - get_heading_history(last_n=10)
+   - Verify correct 10 headings returned for smoothing calculation
+   - Verify ordering correct for temporal analysis
+
+4. **High-frequency persistence**
+   - Concurrent save_frame_result() and save_heading() calls
+   - Measure throughput > 200 ops/sec
+   - Verify no data loss or corruption
+
+5. **Transactional consistency**
+   - Transaction: update frame_result + update waypoint
+   - Verify atomic update or complete rollback
+
+

docs/02_components/03_flight_database/03.03_feature_auxiliary_data_persistence.md

@@ -0,0 +1,99 @@
+# Feature: Auxiliary Data Persistence
+
+## Description
+Persistence layer for supporting data including image metadata and chunk state. Image metadata enables image pipeline to track stored files, while chunk state persistence is critical for crash recovery of route chunk processing. These operations support other components' specific data requirements.
+
+## Component APIs Implemented
+- `save_image_metadata(flight_id: str, frame_id: int, file_path: str, metadata: Dict) -> bool`
+- `get_image_path(flight_id: str, frame_id: int) -> Optional[str]`
+- `get_image_metadata(flight_id: str, frame_id: int) -> Optional[Dict]`
+- `save_chunk_state(flight_id: str, chunk: ChunkHandle) -> bool`
+- `load_chunk_states(flight_id: str) -> List[ChunkHandle]`
+- `delete_chunk_state(flight_id: str, chunk_id: str) -> bool`
+
+## External Tools and Services
+- **PostgreSQL**: Primary database with JSONB support
+- **SQLAlchemy**: ORM and connection pooling
+
+## Internal Methods
+| Method | Purpose |
+|--------|---------|
+| `_build_chunk_handle_from_row(row)` | Map database row to ChunkHandle object |
+| `_serialize_chunk_frames(frames)` | Serialize frame list to JSONB array |
+| `_deserialize_chunk_frames(jsonb)` | Deserialize JSONB array to frame list |
+| `_serialize_metadata(metadata)` | Serialize metadata dict to JSONB |
+| `_deserialize_metadata(jsonb)` | Deserialize JSONB to metadata dict |
+| `_upsert_chunk_state(flight_id, chunk)` | Insert or update chunk state |
+
+## Unit Tests
+1. **save_image_metadata**
+   - New image → metadata persisted with file_path
+   - Overwrite same frame_id → updates metadata
+   - Verify JSONB serialization of metadata dict
+   - Verify uploaded_at timestamp set
+
+2. **get_image_path**
+   - Existing image → returns file path string
+   - Non-existent frame → returns None
+   - Non-existent flight → returns None
+
+3. **get_image_metadata**
+   - Existing image → returns metadata dict
+   - Verify deserialization of original_name, width, height, file_size
+   - Non-existent → returns None
+
+4. **save_chunk_state**
+   - New chunk → persisted successfully
+   - Update existing chunk → state updated
+   - Verify all fields: chunk_id, start_frame_id, end_frame_id, frames
+   - Verify anchor fields: has_anchor, anchor_frame_id, anchor_gps
+   - Verify matching_status persisted
+   - Verify frames JSONB array serialization
+
+5. **load_chunk_states**
+   - Flight with 3 chunks → returns all chunk handles
+   - No chunks → returns empty list
+   - Verify correct deserialization of ChunkHandle fields
+   - Verify frames list deserialized from JSONB
+
+6. **delete_chunk_state**
+   - Existing chunk → deleted, returns True
+   - Non-existent chunk → returns False
+   - Verify other chunks unaffected
+
+## Integration Tests
+1. **Image metadata lifecycle**
+   - save_image_metadata() for 100 frames
+   - get_image_path() for each → all paths returned
+   - Delete flight → cascade deletes image metadata
+   - Verify no orphan records
+
+2. **Chunk state crash recovery**
+   - Create flight, save 5 chunk states
+   - Simulate crash (close connection)
+   - Reconnect, load_chunk_states()
+   - Verify all 5 chunks restored with correct state
+   - Verify frames lists intact
+
+3. **Chunk lifecycle operations**
+   - save_chunk_state() → active chunk
+   - Update chunk: add frames, set has_anchor=True
+   - save_chunk_state() → verify update
+   - delete_chunk_state() after merge → verify removed
+
+4. **Concurrent chunk operations**
+   - Multiple chunks saved concurrently
+   - Verify no data corruption
+   - Verify unique chunk_ids enforced
+
+5. **JSONB query performance**
+   - Save chunks with large frames arrays (500+ frames)
+   - load_chunk_states() performance within 50ms
+   - Verify JSONB indexing effectiveness
+
+6. **Foreign key constraints**
+   - save_chunk_state with invalid anchor_frame_id → proper error handling
+   - Verify FK constraint fk_anchor_frame enforced
+   - Delete referenced image → anchor_frame_id set to NULL
+
+