gps-denied-onboard/docs/03_tests/11_flight_manager_integration_spec.md

Integration Test: Flight Manager

Summary

Validate the Flight Manager component responsible for managing flight sessions, coordinating image processing, and tracking flight state throughout the ASTRAL-Next pipeline.

Component Under Test

Component: Flight Manager Location: gps_denied_02_flight_manager Dependencies:

• Database Layer (flight persistence)
• Image Input Pipeline
• Sequential Visual Odometry (L1)
• Global Place Recognition (L2)
• Metric Refinement (L3)
• Factor Graph Optimizer
• Failure Recovery Coordinator
• Result Manager

Detailed Description

This test validates that the Flight Manager can:

1. Create and initialize new flight sessions
2. Manage flight lifecycle (created → processing → completed → archived)
3. Queue and dispatch images for processing
4. Coordinate between all processing layers (L1, L2, L3, Factor Graph)
5. Track processing progress and statistics
6. Handle processing failures and recovery
7. Manage concurrent flights
8. Persist flight state across system restarts
9. Enforce constraints (image ordering, missing frames)
10. Trigger user input requests when automated processing fails

The Flight Manager is the central orchestrator coordinating all components in the ASTRAL-Next system.

Input Data

Test Case 1: Create New Flight

• Input:
  • Flight name: "Test_Baseline_Flight"
  • Start GPS: 48.275292, 37.385220
  • Altitude: 400m
  • Camera params: focal_length=25mm, sensor_width=23.5mm, resolution=6252x4168
• Expected: Flight created with unique ID, state = "created"

Test Case 2: Add Images to Flight

• Flight: Test_Baseline_Flight
• Images: AD000001-AD000010 (10 images in order)
• Expected: All images queued, sequence maintained

Test Case 3: Process Flight (Normal)

• Flight: Test_Baseline_Flight with AD000001-AD000010
• Expected:
  • State transitions: created → processing → completed
  • All 10 images processed successfully
  • Results available

Test Case 4: Process with Sharp Turn

• Flight: Test_Sharp_Turn
• Images: AD000042, AD000044, AD000045, AD000046 (skip AD000043)
• Expected:
  • Detect missing frame AD000043
  • L1 tracking fails, L2 recovers location
  • Flight completes successfully

Test Case 5: Process with Outlier

• Flight: Test_Outlier
• Images: AD000045-AD000050 (includes 268m outlier)
• Expected:
  • Outlier detected by Factor Graph
  • Robust cost function handles outlier
  • Other images processed correctly

Test Case 6: Process Long Flight

• Flight: Test_Long_Flight
• Images: All 60 images (AD000001-AD000060)
• Expected:
  • Processing completes without failure
  • Registration rate > 95% (AC-9)
  • Accuracy targets met (AC-1, AC-2)

Test Case 7: Handle Processing Failure

• Flight: Test_Failure
• Images: AD000001-AD000005
• Scenario: Simulate L1, L2, L3 all failing for AD000003
• Expected:
  • Failure detected
  • Failure Recovery Coordinator invoked
  • User input requested (AC-6)
  • Flight state = "awaiting_user_input"

Test Case 8: Apply User Fix

• Flight: Test_Failure (from Test Case 7)
• User Input: GPS for AD000003 = 48.274520, 37.381657
• Expected:
  • User fix accepted
  • Processing resumes
  • Factor Graph incorporates fix
  • Flight completes

Test Case 9: Concurrent Flights

• Flights: 3 flights processing simultaneously
  • Flight A: AD000001-AD000020
  • Flight B: AD000021-AD000040
  • Flight C: AD000041-AD000060
• Expected:
  • All 3 flights process without interference
  • No resource contention issues
  • All complete successfully

Test Case 10: Flight State Persistence

• Scenario:
  • Start flight with AD000001-AD000030
  • Process 15 images
  • Simulate system restart
  • Reload flight state
  • Continue processing remaining images
• Expected: Flight resumes from last checkpoint

Test Case 11: Get Flight Statistics

• Flight: Completed flight with 60 images
• Expected Statistics:
  • total_images: 60
  • processed_images: 60
  • failed_images: 0-2
  • success_rate: > 0.95
  • mean_error_m: < 30
  • processing_time_s: < 300
  • registration_rate: > 0.95

Test Case 12: Archive Flight

• Flight: Completed flight
• Expected:
  • Flight state = "archived"
  • Results still accessible
  • No longer in active processing queue

Expected Output

For each test case:

{
  "flight_id": "unique_flight_identifier",
  "flight_name": "string",
  "state": "created|processing|completed|failed|awaiting_user_input|archived",
  "created_at": "timestamp",
  "updated_at": "timestamp",
  "statistics": {
    "total_images": <integer>,
    "processed_images": <integer>,
    "failed_images": <integer>,
    "awaiting_user_input": <integer>,
    "success_rate": <float>,
    "mean_error_m": <float>,
    "registration_rate": <float>,
    "processing_time_s": <float>
  },
  "current_image": "string|null",
  "next_action": "process_next|await_user_input|complete|none"
}

Success Criteria

Test Case 1 (Create):

• Flight created successfully
• Unique flight_id assigned
• State = "created"
• All parameters stored correctly

Test Case 2 (Add Images):

• All 10 images queued
• Sequence numbers assigned (1-10)
• No duplicates

Test Case 3 (Process Normal):

• All images processed
• State = "completed"
• success_rate = 1.0
• Processing time < 60 seconds (10 images)

Test Case 4 (Sharp Turn):

• Missing frame detected
• L2 successfully recovers location for AD000044
• success_rate ≥ 0.75 (3/4 or better)

Test Case 5 (Outlier):

• Outlier detected and handled
• success_rate ≥ 0.83 (5/6 or better)
• Non-outlier images have low error

Test Case 6 (Long Flight):

• All 60 images processed
• registration_rate > 0.95 (AC-9)
• success_rate > 0.80
• Processing time < 300 seconds (< 5s per image, AC-7)

Test Case 7 (Failure):

• Failure detected for AD000003
• State transitions to "awaiting_user_input"
• User notification sent via SSE
• Processing paused appropriately

Test Case 8 (User Fix):

• User fix accepted
• Processing resumes automatically
• State transitions back to "processing"
• Flight completes successfully

Test Case 9 (Concurrent):

• All 3 flights complete
• No race conditions
• No resource exhaustion
• Each flight independent

Test Case 10 (Persistence):

• State saved correctly
• Reloaded state matches pre-restart
• Processing continues from checkpoint
• No image reprocessing

Test Case 11 (Statistics):

• All statistics calculated correctly
• Statistics updated in real-time
• Match actual processing results

Test Case 12 (Archive):

• State = "archived"
• Flight no longer active
• Results preserved

Maximum Expected Time

• Create flight: < 500ms
• Add 10 images: < 2 seconds
• Process 10 images: < 60 seconds
• Process 60 images: < 300 seconds (5s per image, AC-7)
• Apply user fix: < 1 second
• Get statistics: < 200ms
• Total test suite: < 600 seconds (10 minutes)

Test Execution Steps

1. Setup Phase: a. Initialize Flight Manager b. Clear any existing test flights from database c. Prepare test images d. Configure processing parameters

2. Test Case 1 - Create: a. Call create_flight() with parameters b. Verify flight_id returned c. Check database for flight record d. Validate initial state

3. Test Case 2 - Add Images: a. Call add_images() with 10 images b. Verify all queued c. Check sequence assignment d. Validate database state

4. Test Case 3 - Process Normal: a. Call start_processing() b. Monitor state transitions c. Wait for completion d. Verify results

5. Test Case 4 - Sharp Turn: a. Create flight with gap in sequence b. Start processing c. Monitor L1 failure, L2 recovery d. Verify completion

6. Test Case 5 - Outlier: a. Process flight with outlier b. Monitor Factor Graph handling c. Verify outlier detection d. Check final results

7. Test Case 6 - Long Flight: a. Process all 60 images b. Monitor progress continuously c. Measure processing time d. Validate against AC-1, AC-2, AC-7, AC-9

8. Test Case 7 - Failure: a. Simulate triple failure for one image b. Verify failure detection c. Check state transition d. Confirm user input request

9. Test Case 8 - User Fix: a. Submit user fix b. Verify acceptance c. Monitor processing resume d. Check incorporation into results

10. Test Case 9 - Concurrent: a. Start 3 flights simultaneously b. Monitor all 3 in parallel c. Verify isolation d. Wait for all completions

11. Test Case 10 - Persistence: a. Start long flight b. Save state mid-processing c. Simulate restart (reload from DB) d. Continue and complete

12. Test Case 11 - Statistics: a. Query statistics after completion b. Validate calculations c. Compare with ground truth d. Check all metrics present

13. Test Case 12 - Archive: a. Archive completed flight b. Verify state change c. Check accessibility d. Ensure not in active queue

Pass/Fail Criteria

Overall Test Passes If:

• All 12 test cases meet their success criteria
• Flight state machine transitions correctly
• All images processed (or user input requested)
• No data loss or corruption
• Concurrent flights isolated
• State persistence works correctly

Test Fails If:

• Any flight enters invalid state
• Images processed out of order
• Duplicate processing occurs
• Statistics incorrect
• Race conditions in concurrent processing
• State persistence fails
• Memory or resource leaks

Additional Validation

State Machine Validation: Valid state transitions:

• created → processing
• processing → completed
• processing → awaiting_user_input
• awaiting_user_input → processing
• processing → failed
• completed → archived
• failed → archived

Invalid transitions should be rejected.

Image Queue Management:

• FIFO processing order
• No image skipped (unless failure)
• Sequence numbers maintained
• Duplicate detection

Resource Management:

• Memory usage bounded per flight
• No orphaned resources after completion
• Cleanup on flight deletion
• Limits on concurrent flights (if configured)

Error Recovery:

• Graceful handling of component failures
• Retry logic for transient errors
• User intervention for persistent failures
• Clear error messages

Integration with Acceptance Criteria:

• AC-6: User input mechanism tested (Test Cases 7, 8)
• AC-7: Processing time < 5s per image (Test Case 6)
• AC-8: Real-time results via SSE (monitored during processing)
• AC-9: Registration rate > 95% (Test Case 6)

Performance Metrics:

• Throughput: images processed per second
• Latency: time from image queued to result available
• Overhead: Flight Manager processing vs actual vision processing
• Scalability: performance with 1, 10, 100 flights

##Database Operations**:

• Atomic transactions for state changes
• Proper indexing for queries
• No deadlocks in concurrent operations
• Backup and recovery procedures

Configuration Options: Test various configuration:

• Max concurrent images processing
• Retry attempts for failed processing
• Timeout values
• Buffer sizes
• Checkpoint frequency for persistence