add tests

gen_tests updated solution.md updated
2026-04-23 01:06:36 +00:00 · 2025-11-24 22:57:46 +02:00
parent f50006d100
commit 4f8c18a066
49 changed files with 7209 additions and 3 deletions
@@ -0,0 +1,395 @@
+# Integration Test: Flight Manager
+
+## Summary
+Validate the Flight Manager component responsible for managing flight sessions, coordinating image processing, and tracking flight state throughout the ASTRAL-Next pipeline.
+
+## Component Under Test
+**Component**: Flight Manager
+**Location**: `gps_denied_02_flight_manager`
+**Dependencies**:
+- Database Layer (flight persistence)
+- Image Input Pipeline
+- Sequential Visual Odometry (L1)
+- Global Place Recognition (L2)
+- Metric Refinement (L3)
+- Factor Graph Optimizer
+- Failure Recovery Coordinator
+- Result Manager
+
+## Detailed Description
+This test validates that the Flight Manager can:
+1. Create and initialize new flight sessions
+2. Manage flight lifecycle (created → processing → completed → archived)
+3. Queue and dispatch images for processing
+4. Coordinate between all processing layers (L1, L2, L3, Factor Graph)
+5. Track processing progress and statistics
+6. Handle processing failures and recovery
+7. Manage concurrent flights
+8. Persist flight state across system restarts
+9. Enforce constraints (image ordering, missing frames)
+10. Trigger user input requests when automated processing fails
+
+The Flight Manager is the central orchestrator coordinating all components in the ASTRAL-Next system.
+
+## Input Data
+
+### Test Case 1: Create New Flight
+- **Input**:
+  - Flight name: "Test_Baseline_Flight"
+  - Start GPS: 48.275292, 37.385220
+  - Altitude: 400m
+  - Camera params: focal_length=25mm, sensor_width=23.5mm, resolution=6252x4168
+- **Expected**: Flight created with unique ID, state = "created"
+
+### Test Case 2: Add Images to Flight
+- **Flight**: Test_Baseline_Flight
+- **Images**: AD000001-AD000010 (10 images in order)
+- **Expected**: All images queued, sequence maintained
+
+### Test Case 3: Process Flight (Normal)
+- **Flight**: Test_Baseline_Flight with AD000001-AD000010
+- **Expected**:
+  - State transitions: created → processing → completed
+  - All 10 images processed successfully
+  - Results available
+
+### Test Case 4: Process with Sharp Turn
+- **Flight**: Test_Sharp_Turn
+- **Images**: AD000042, AD000044, AD000045, AD000046 (skip AD000043)
+- **Expected**:
+  - Detect missing frame AD000043
+  - L1 tracking fails, L2 recovers location
+  - Flight completes successfully
+
+### Test Case 5: Process with Outlier
+- **Flight**: Test_Outlier
+- **Images**: AD000045-AD000050 (includes 268m outlier)
+- **Expected**:
+  - Outlier detected by Factor Graph
+  - Robust cost function handles outlier
+  - Other images processed correctly
+
+### Test Case 6: Process Long Flight
+- **Flight**: Test_Long_Flight
+- **Images**: All 60 images (AD000001-AD000060)
+- **Expected**:
+  - Processing completes without failure
+  - Registration rate > 95% (AC-9)
+  - Accuracy targets met (AC-1, AC-2)
+
+### Test Case 7: Handle Processing Failure
+- **Flight**: Test_Failure
+- **Images**: AD000001-AD000005
+- **Scenario**: Simulate L1, L2, L3 all failing for AD000003
+- **Expected**:
+  - Failure detected
+  - Failure Recovery Coordinator invoked
+  - User input requested (AC-6)
+  - Flight state = "awaiting_user_input"
+
+### Test Case 8: Apply User Fix
+- **Flight**: Test_Failure (from Test Case 7)
+- **User Input**: GPS for AD000003 = 48.274520, 37.381657
+- **Expected**:
+  - User fix accepted
+  - Processing resumes
+  - Factor Graph incorporates fix
+  - Flight completes
+
+### Test Case 9: Concurrent Flights
+- **Flights**: 3 flights processing simultaneously
+  - Flight A: AD000001-AD000020
+  - Flight B: AD000021-AD000040
+  - Flight C: AD000041-AD000060
+- **Expected**:
+  - All 3 flights process without interference
+  - No resource contention issues
+  - All complete successfully
+
+### Test Case 10: Flight State Persistence
+- **Scenario**:
+  - Start flight with AD000001-AD000030
+  - Process 15 images
+  - Simulate system restart
+  - Reload flight state
+  - Continue processing remaining images
+- **Expected**: Flight resumes from last checkpoint
+
+### Test Case 11: Get Flight Statistics
+- **Flight**: Completed flight with 60 images
+- **Expected Statistics**:
+  - total_images: 60
+  - processed_images: 60
+  - failed_images: 0-2
+  - success_rate: > 0.95
+  - mean_error_m: < 30
+  - processing_time_s: < 300
+  - registration_rate: > 0.95
+
+### Test Case 12: Archive Flight
+- **Flight**: Completed flight
+- **Expected**:
+  - Flight state = "archived"
+  - Results still accessible
+  - No longer in active processing queue
+
+## Expected Output
+
+For each test case:
+```json
+{
+  "flight_id": "unique_flight_identifier",
+  "flight_name": "string",
+  "state": "created|processing|completed|failed|awaiting_user_input|archived",
+  "created_at": "timestamp",
+  "updated_at": "timestamp",
+  "statistics": {
+    "total_images": <integer>,
+    "processed_images": <integer>,
+    "failed_images": <integer>,
+    "awaiting_user_input": <integer>,
+    "success_rate": <float>,
+    "mean_error_m": <float>,
+    "registration_rate": <float>,
+    "processing_time_s": <float>
+  },
+  "current_image": "string|null",
+  "next_action": "process_next|await_user_input|complete|none"
+}
+```
+
+## Success Criteria
+
+**Test Case 1 (Create)**:
+- Flight created successfully
+- Unique flight_id assigned
+- State = "created"
+- All parameters stored correctly
+
+**Test Case 2 (Add Images)**:
+- All 10 images queued
+- Sequence numbers assigned (1-10)
+- No duplicates
+
+**Test Case 3 (Process Normal)**:
+- All images processed
+- State = "completed"
+- success_rate = 1.0
+- Processing time < 60 seconds (10 images)
+
+**Test Case 4 (Sharp Turn)**:
+- Missing frame detected
+- L2 successfully recovers location for AD000044
+- success_rate ≥ 0.75 (3/4 or better)
+
+**Test Case 5 (Outlier)**:
+- Outlier detected and handled
+- success_rate ≥ 0.83 (5/6 or better)
+- Non-outlier images have low error
+
+**Test Case 6 (Long Flight)**:
+- All 60 images processed
+- registration_rate > 0.95 (AC-9)
+- success_rate > 0.80
+- Processing time < 300 seconds (< 5s per image, AC-7)
+
+**Test Case 7 (Failure)**:
+- Failure detected for AD000003
+- State transitions to "awaiting_user_input"
+- User notification sent via SSE
+- Processing paused appropriately
+
+**Test Case 8 (User Fix)**:
+- User fix accepted
+- Processing resumes automatically
+- State transitions back to "processing"
+- Flight completes successfully
+
+**Test Case 9 (Concurrent)**:
+- All 3 flights complete
+- No race conditions
+- No resource exhaustion
+- Each flight independent
+
+**Test Case 10 (Persistence)**:
+- State saved correctly
+- Reloaded state matches pre-restart
+- Processing continues from checkpoint
+- No image reprocessing
+
+**Test Case 11 (Statistics)**:
+- All statistics calculated correctly
+- Statistics updated in real-time
+- Match actual processing results
+
+**Test Case 12 (Archive)**:
+- State = "archived"
+- Flight no longer active
+- Results preserved
+
+## Maximum Expected Time
+- **Create flight**: < 500ms
+- **Add 10 images**: < 2 seconds
+- **Process 10 images**: < 60 seconds
+- **Process 60 images**: < 300 seconds (5s per image, AC-7)
+- **Apply user fix**: < 1 second
+- **Get statistics**: < 200ms
+- **Total test suite**: < 600 seconds (10 minutes)
+
+## Test Execution Steps
+
+1. **Setup Phase**:
+   a. Initialize Flight Manager
+   b. Clear any existing test flights from database
+   c. Prepare test images
+   d. Configure processing parameters
+
+2. **Test Case 1 - Create**:
+   a. Call create_flight() with parameters
+   b. Verify flight_id returned
+   c. Check database for flight record
+   d. Validate initial state
+
+3. **Test Case 2 - Add Images**:
+   a. Call add_images() with 10 images
+   b. Verify all queued
+   c. Check sequence assignment
+   d. Validate database state
+
+4. **Test Case 3 - Process Normal**:
+   a. Call start_processing()
+   b. Monitor state transitions
+   c. Wait for completion
+   d. Verify results
+
+5. **Test Case 4 - Sharp Turn**:
+   a. Create flight with gap in sequence
+   b. Start processing
+   c. Monitor L1 failure, L2 recovery
+   d. Verify completion
+
+6. **Test Case 5 - Outlier**:
+   a. Process flight with outlier
+   b. Monitor Factor Graph handling
+   c. Verify outlier detection
+   d. Check final results
+
+7. **Test Case 6 - Long Flight**:
+   a. Process all 60 images
+   b. Monitor progress continuously
+   c. Measure processing time
+   d. Validate against AC-1, AC-2, AC-7, AC-9
+
+8. **Test Case 7 - Failure**:
+   a. Simulate triple failure for one image
+   b. Verify failure detection
+   c. Check state transition
+   d. Confirm user input request
+
+9. **Test Case 8 - User Fix**:
+   a. Submit user fix
+   b. Verify acceptance
+   c. Monitor processing resume
+   d. Check incorporation into results
+
+10. **Test Case 9 - Concurrent**:
+    a. Start 3 flights simultaneously
+    b. Monitor all 3 in parallel
+    c. Verify isolation
+    d. Wait for all completions
+
+11. **Test Case 10 - Persistence**:
+    a. Start long flight
+    b. Save state mid-processing
+    c. Simulate restart (reload from DB)
+    d. Continue and complete
+
+12. **Test Case 11 - Statistics**:
+    a. Query statistics after completion
+    b. Validate calculations
+    c. Compare with ground truth
+    d. Check all metrics present
+
+13. **Test Case 12 - Archive**:
+    a. Archive completed flight
+    b. Verify state change
+    c. Check accessibility
+    d. Ensure not in active queue
+
+## Pass/Fail Criteria
+
+**Overall Test Passes If**:
+- All 12 test cases meet their success criteria
+- Flight state machine transitions correctly
+- All images processed (or user input requested)
+- No data loss or corruption
+- Concurrent flights isolated
+- State persistence works correctly
+
+**Test Fails If**:
+- Any flight enters invalid state
+- Images processed out of order
+- Duplicate processing occurs
+- Statistics incorrect
+- Race conditions in concurrent processing
+- State persistence fails
+- Memory or resource leaks
+
+## Additional Validation
+
+**State Machine Validation**:
+Valid state transitions:
+- created → processing
+- processing → completed
+- processing → awaiting_user_input
+- awaiting_user_input → processing
+- processing → failed
+- completed → archived
+- failed → archived
+
+Invalid transitions should be rejected.
+
+**Image Queue Management**:
+- FIFO processing order
+- No image skipped (unless failure)
+- Sequence numbers maintained
+- Duplicate detection
+
+**Resource Management**:
+- Memory usage bounded per flight
+- No orphaned resources after completion
+- Cleanup on flight deletion
+- Limits on concurrent flights (if configured)
+
+**Error Recovery**:
+- Graceful handling of component failures
+- Retry logic for transient errors
+- User intervention for persistent failures
+- Clear error messages
+
+**Integration with Acceptance Criteria**:
+- **AC-6**: User input mechanism tested (Test Cases 7, 8)
+- **AC-7**: Processing time < 5s per image (Test Case 6)
+- **AC-8**: Real-time results via SSE (monitored during processing)
+- **AC-9**: Registration rate > 95% (Test Case 6)
+
+**Performance Metrics**:
+- Throughput: images processed per second
+- Latency: time from image queued to result available
+- Overhead: Flight Manager processing vs actual vision processing
+- Scalability: performance with 1, 10, 100 flights
+
+##Database Operations**:
+- Atomic transactions for state changes
+- Proper indexing for queries
+- No deadlocks in concurrent operations
+- Backup and recovery procedures
+
+**Configuration Options**:
+Test various configuration:
+- Max concurrent images processing
+- Retry attempts for failed processing
+- Timeout values
+- Buffer sizes
+- Checkpoint frequency for persistence
+