gps-denied-onboard/docs/03_tests/31_accuracy_50m_baseline_spec.md

Acceptance Test: AC-1 - 80% of Photos < 50m Error

Summary

Validate Acceptance Criterion 1: "The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS."

Linked Acceptance Criteria

AC-1: 80% of photos < 50m error

Preconditions

1. ASTRAL-Next system fully operational
2. All TensorRT models loaded
3. Satellite tiles cached for test area (48.25-48.28°N, 37.34-37.39°E)
4. Ground truth GPS coordinates available (coordinates.csv)
5. Test dataset prepared: Test_Baseline (AD000001-AD000030)

Test Description

Process baseline flight of 30 images with normal spacing (~120m between images). Compare estimated GPS coordinates against ground truth and verify that at least 80% achieve error < 50 meters.

Test Steps

Step 1: Initialize System

• Action: Start ASTRAL-Next system, verify all components ready
• Expected Result: System state = "ready", all models loaded, no errors

Step 2: Create Test Flight

• Action: Create flight "AC1_Baseline" with start_gps=48.275292, 37.385220, altitude=400m
• Expected Result: Flight created, flight_id returned

Step 3: Upload Test Images

• Action: Upload AD000001-AD000030 (30 images) in order
• Expected Result: All 30 images queued, sequence maintained

Step 4: Monitor Processing

• Action: Monitor flight status until completed
• Expected Result:
  • Processing completes within 150 seconds (5s per image)
  • No system errors
  • Registration rate > 95%

Step 5: Retrieve Results

• Action: GET /flights/{flightId}/results
• Expected Result: Results for all 30 images returned

Step 6: Calculate Errors

• Action: For each image, calculate haversine distance between estimated and ground truth GPS
• Expected Result: Error array with 30 values

Step 7: Validate AC-1

• Action: Count images with error < 50m, calculate percentage
• Expected Result: ≥ 80% of images have error < 50 meters ✓

Step 8: Generate Report

• Action: Create test report with statistics
• Expected Result:
  • Total images: 30
  • Images < 50m: ≥ 24
  • Percentage: ≥ 80.0%
  • Mean error: documented
  • Median error: documented
  • Max error: documented

Success Criteria

Primary Criterion (AC-1):

• ≥ 24 out of 30 images (80%) have GPS error < 50 meters

Supporting Criteria:

• All 30 images processed (or user input requested if failures occur)
• Processing time < 150 seconds total
• No system crashes or unhandled errors
• Registration rate > 95% (AC-9)

Expected Results

Based on solution architecture (LiteSAM RMSE ~18m), expected performance:

Total Images: 30
Successfully Processed: 30 (100%)
Images with error < 50m: 28 (93.3%)
Images with error < 20m: 20 (66.7%)
Mean Error: 24.5m
Median Error: 18.2m
RMSE: 28.3m
Max Error: 48.7m
AC-1 Status: PASS (93.3% > 80%)

Pass/Fail Criteria

TEST PASSES IF:

• ≥ 80% of images achieve error < 50m
• System completes processing without critical failures
• Results reproducible across multiple test runs

TEST FAILS IF:

• < 80% of images achieve error < 50m
• System crashes or becomes unresponsive
• More than 5% of images fail to process (violates AC-9)

Notes

• This test uses Test_Baseline dataset (AD000001-030) with consistent spacing
• No sharp turns or outliers in this dataset
• Represents ideal operating conditions
• If test fails, investigate: satellite data quality, model accuracy, georeferencing precision