gps-denied-onboard/docs/03_tests/32_accuracy_50m_varied_terrain_spec.md

Acceptance Test: 80% Photos <50m Error - Varied Terrain

Summary

Validate AC-1 accuracy requirement (80% of photos within 50m error) across different terrain types including agricultural fields, mixed vegetation, and urban edges.

Linked Acceptance Criteria

AC-1: The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS.

Preconditions

• ASTRAL-Next system fully deployed and operational
• Satellite reference data downloaded for test region
• TensorRT models loaded (SuperPoint, LightGlue, AnyLoc, LiteSAM)
• Ground truth GPS coordinates available for validation
• Test datasets covering varied terrain types

Test Data

• Primary Dataset: AD000001-AD000060 (varied terrain across 60 images)
• Terrain Types: Agricultural fields, tree lines, mixed vegetation, roads
• Ground Truth: coordinates.csv
• Camera Parameters: 400m altitude, 25mm focal length, 26MP resolution

Test Steps

Step 1: Initialize System with Starting GPS

Action: Start flight processing with first image GPS coordinate (48.275292, 37.385220) Expected Result:

• System initializes successfully
• Satellite tiles downloaded for operational area
• L1, L2, L3 layers ready
• Status: INITIALIZED

Step 2: Process Agricultural Field Segment (AD000001-015)

Action: Process images over predominantly agricultural terrain Expected Result:

• L1 sequential tracking maintains continuity
• SuperPoint detects field boundaries and crop variations
• LiteSAM achieves cross-view matching despite seasonal differences
• Mean error <40m for this segment
• Status: PROCESSING

Step 3: Process Mixed Vegetation Segment (AD000016-030)

Action: Process images with mixed terrain features Expected Result:

• L2 global place recognition active during transitions
• AnyLoc retrieval successful using DINOv2 features
• Factor graph optimization smooths trajectory
• Mean error <45m for this segment
• Status: PROCESSING

Step 4: Process Complex Terrain with Sharp Turns (AD000031-060)

Action: Process remaining images including sharp turns and outliers Expected Result:

• L2 recovers from sharp turns (AD000042-043, AD000032-033)
• Robust cost functions handle AD000047-048 outlier (268.6m)
• Multiple map fragments merged successfully
• Mean error <50m for challenging segments
• Status: PROCESSING

Step 5: Calculate Accuracy Metrics

Action: Compare estimated GPS coordinates with ground truth Expected Result:

Total images: 60
Error <50m: ≥48 images (80%)
Error <20m: ≥36 images (60%)
Mean error: <40m
Median error: <35m
Max error: <150m (excluding known outliers)

Step 6: Validate Terrain-Specific Performance

Action: Analyze accuracy by terrain type Expected Result:

• Agricultural fields: 75-85% <50m
• Mixed vegetation: 80-90% <50m
• Road intersections: 85-95% <50m
• Overall: ≥80% <50m
• Status: COMPLETED

Pass/Fail Criteria

PASS if:

• ≥80% of images (48/60) have error <50m
• No systematic bias across terrain types
• System completes without fatal errors
• Factor graph converges (final MRE <1.5px)

FAIL if:

• <80% of images meet 50m threshold

• 3 terrain types show <70% accuracy

• System crashes or hangs
• Catastrophic tracking loss without recovery

Performance Requirements

• Processing time: <5 seconds per image average
• Total flight time: <5 minutes for 60 images
• Memory usage: <8GB on RTX 3070
• CPU usage: <80% average

Notes

• Varied terrain test provides more comprehensive validation than baseline
• Different terrain types stress different system components
• AC-1 threshold of 80% allows for difficult scenarios while maintaining operational utility