gps-denied-onboard/docs/03_tests/00_test_summary.md

ASTRAL-Next Test Specifications Summary

Overview

Comprehensive test specifications for the GPS-denied navigation system following the QA testing pyramid approach.

Total Test Specifications: 56

Test Organization

Integration Tests (01-16): Component Level

Tests individual system components in isolation with their dependencies.

Vision Pipeline (01-04):

• 01: Sequential Visual Odometry (SuperPoint + LightGlue)
• 02: Global Place Recognition (AnyLoc)
• 03: Metric Refinement (LiteSAM)
• 04: Factor Graph Optimizer (GTSAM)

Data Management (05-08):

• 05: Satellite Data Manager
• 06: Coordinate Transformer
• 07: Image Input Pipeline
• 08: Image Rotation Manager

Service Infrastructure (09-12):

• 09: REST API (FastAPI endpoints for flight management, image uploads, user fixes)
• 10: SSE Event Streamer (Server-Sent Events for real-time result streaming)
• 11: Flight Manager
• 12: Result Manager

Support Components (13-16):

• 13: Model Manager (TensorRT)
• 14: Failure Recovery Coordinator
• 15: Configuration Manager
• 16: Database Layer

System Integration Tests (21-25): Multi-Component Flows

Tests integration between multiple components.

• 21: End-to-End Normal Flight
• 22: Satellite to Vision Pipeline
• 23: Vision to Optimization Pipeline
• 24: Multi-Component Error Propagation
• 25: Real-Time Streaming Pipeline

Acceptance Tests (31-43): Requirements Validation

Tests mapped to 10 acceptance criteria.

Accuracy (31-32):

• 31: AC-1 - 80% < 50m error
• 32: AC-2 - 60% < 20m error

Robustness (33-35):

• 33: AC-3 - 350m outlier handling
• 34: AC-4 - Sharp turn recovery
• 35: AC-5 - Multi-fragment route connection

User Interaction (36):

• 36: AC-6 - User input after 3 failures

Performance (37-38):

• 37: AC-7 - <5s per image
• 38: AC-8 - Real-time streaming + refinement

Quality Metrics (39-40):

• 39: AC-9 - Registration rate >95%
• 40: AC-10 - MRE <1.0 pixels

Cross-Cutting (41-43):

• 41: Long flight (3000 images)
• 42: Degraded satellite data
• 43: Complete system validation

Chunk-Based Recovery (55-56):

• 55: Chunk rotation recovery (rotation sweeps for chunks)
• 56: Multi-chunk simultaneous processing (Atlas architecture)

GPS-Analyzed Scenario Tests (51-54): Real Data

Tests using GPS-analyzed test datasets.

• 51: Test_Baseline (AD000001-030) - Standard flight
• 52: Test_Outlier_350m (AD000045-050) - Outlier scenario
• 53: Test_Sharp_Turn - Multiple sharp turn datasets
• 54: Test_Long_Flight (AD000001-060) - Full dataset

Test Data

GPS Analysis Results

• Mean distance: 120.8m
• Min distance: 24.2m
• Max distance: 268.6m

Identified Sharp Turns (>200m):

• AD000003 → AD000004: 202.2m
• AD000032 → AD000033: 220.6m
• AD000042 → AD000043: 234.2m
• AD000044 → AD000045: 230.2m
• AD000047 → AD000048: 268.6m (largest outlier)

Test Datasets

Test_Baseline: AD000001-030 (30 images, normal spacing) Test_Outlier_350m: AD000045-050 (6 images, 268.6m outlier) Test_Sharp_Turn_A: AD000042, AD000044, AD000045, AD000046 (skip 043) Test_Sharp_Turn_B: AD000032-035 (220m jump) Test_Sharp_Turn_C: AD000003, AD000009 (5-frame gap) Test_Long_Flight: AD000001-060 (all 60 images, all variations)

Acceptance Criteria Coverage

AC	Requirement	Test Specs	Status
AC-1	80% < 50m error	31, 43, 51, 54	✓ Covered
AC-2	60% < 20m error	32, 43, 51, 54	✓ Covered
AC-3	350m outlier robust	33, 43, 52, 54	✓ Covered
AC-4	Sharp turn <5% overlap	34, 43, 53, 54, 55	✓ Covered
AC-5	Multi-fragment connection	35, 39, 43, 56	✓ Covered
AC-6	User input after 3 failures	36, 43	✓ Covered
AC-7	<5s per image	37, 43, 51, 54	✓ Covered
AC-8	Real-time + refinement	38, 43	✓ Covered
AC-9	Registration >95%	39, 43, 51, 54	✓ Covered
AC-10	MRE <1.0px	40, 43	✓ Covered

Test Execution Strategy

Phase 1: Component Integration (01-16)

• Validate each component individually
• Verify interfaces and dependencies
• Establish baseline performance metrics

Phase 2: System Integration (21-25)

• Test multi-component interactions
• Validate end-to-end flows
• Verify error handling across components

Phase 3: Acceptance Testing (31-43)

• Validate all acceptance criteria
• Use GPS-analyzed real data
• Measure against requirements

Phase 4: Special Scenarios (51-54)

• Test specific GPS-identified situations
• Validate outliers and sharp turns
• Full system validation

Success Criteria Summary

Integration Tests: All components pass individual tests, interfaces work correctly System Tests: Multi-component flows work, errors handled properly Acceptance Tests: All 10 ACs met with real data Overall: System meets all requirements, ready for deployment

Test Metrics to Track

• Accuracy: Mean error, RMSE, percentiles
• Performance: Processing time per image, total time
• Reliability: Registration rate, success rate
• Quality: MRE, confidence scores
• Robustness: Outlier handling, error recovery

Notes

• All test specs follow standard format (Integration vs Acceptance)
• GPS-analyzed datasets based on actual test data coordinates
• Specifications ready for QA team implementation
• No code included per requirement
• Tests cover all components and all acceptance criteria