gps-denied-onboard/docs/03_tests/47_reprojection_error_spec.md

Acceptance Test: AC-10 - Mean Reprojection Error < 1.0 pixels

Summary

Validate Acceptance Criterion 10: "Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location."

Linked Acceptance Criteria

AC-10: MRE < 1.0 pixels

Preconditions

1. ASTRAL-Next system operational
2. F07 Sequential Visual Odometry extracting and matching features
3. F10 Factor Graph Optimizer computing optimized poses
4. Camera intrinsics calibrated (from F17 Configuration Manager)
5. Test dataset with ground truth poses (for reference)
6. Reprojection error calculation implemented

Reprojection Error Definition

Formula:

For each matched feature point p_i in image I_j:
  1. Triangulate 3D point X_i from matches across images
  2. Project X_i back to image I_j using optimized pose T_j and camera K
  3. p'_i = K * T_j * X_i (projected pixel location)
  4. e_i = ||p_i - p'_i|| (Euclidean distance in pixels)

MRE = (1/N) * Σ e_i  (mean across all features in all images)

Test Data

• Dataset: AD000001-AD000030 (30 images, baseline)
• Expected Features: ~500-2000 matched features per image pair
• Total Measurements: ~15,000-60,000 reprojection measurements

Test Steps

Step 1: Process Flight Through Complete Pipeline

• Action: Process AD000001-AD000030 through full ASTRAL-Next pipeline
• Expected Result:
  • Factor graph initialized and optimized
  • 30 poses computed
  • All feature correspondences stored

Step 2: Extract Feature Correspondences

• Action: Retrieve all matched features from F07
• Expected Result:
  • For each image pair (i, j):
    • List of matched keypoint pairs: [(p_i, p_j), ...]
    • Match confidence scores
    • Total: ~500-1500 matches per pair
  • Total matches across flight: ~15,000-45,000

Step 3: Triangulate 3D Points

• Action: For each matched feature across multiple views, triangulate 3D position
• Expected Result:
  • 3D point cloud generated
  • Each point has:
    • 3D coordinates (X, Y, Z) in ENU frame
    • List of observations (image_id, pixel_location)
    • Triangulation uncertainty

Step 4: Calculate Per-Feature Reprojection Error

• Action: For each 3D point and each observation:

  For point X with observation (image_j, pixel_p):
    1. Get optimized pose T_j from factor graph
    2. Get camera intrinsics K from config
    3. Project: p' = project(K, T_j, X)
    4. Error: e = sqrt((p.x - p'.x)² + (p.y - p'.y)²)
  

• Expected Result:
  • Array of per-feature reprojection errors
  • Typical range: 0.1 - 3.0 pixels

Step 5: Compute Statistical Metrics

• Action: Calculate MRE and distribution statistics
• Expected Result:

  Total features evaluated: 25,000
  Mean Reprojection Error (MRE): 0.72 pixels
  Median Reprojection Error: 0.58 pixels
  Standard Deviation: 0.45 pixels
  90th Percentile: 1.25 pixels
  95th Percentile: 1.68 pixels
  99th Percentile: 2.41 pixels
  Max Error: 4.82 pixels
  

Step 6: Validate MRE Threshold

• Action: Compare MRE against AC-10 requirement
• Expected Result:
  • MRE = 0.72 pixels < 1.0 pixels ✓
  • AC-10 PASS

Step 7: Identify Outlier Reprojections

• Action: Find features with reprojection error > 3.0 pixels
• Expected Result:

  Outliers (> 3.0 pixels): 127 (0.5% of total)
  Outlier distribution:
    - 3.0-5.0 pixels: 98 features
    - 5.0-10.0 pixels: 27 features
    - > 10.0 pixels: 2 features
  

Step 8: Analyze Outlier Causes

• Action: Investigate high-error features
• Expected Result:
  • Most outliers at image boundaries (lens distortion)
  • Some at occlusion boundaries
  • Moving objects (if any)
  • Repetitive textures causing mismatches

Step 9: Per-Image MRE Analysis

• Action: Calculate MRE per image
• Expected Result:

  Per-Image MRE:
  AD000001: 0.68 px (baseline)
  AD000002: 0.71 px
  ...
  AD000032: 1.12 px (sharp turn - higher error)
  AD000033: 0.95 px
  ...
  AD000030: 0.74 px
  
  Images with MRE > 1.0: 2 out of 30 (6.7%)
  Overall MRE: 0.72 px
  

Step 10: Temporal MRE Trend

• Action: Plot MRE over sequence to detect drift
• Expected Result:
  • MRE relatively stable across sequence
  • No significant upward trend (would indicate drift)
  • Spikes at known challenging locations (sharp turns)

Step 11: Validate Robust Kernel Effect

• Action: Compare MRE with/without robust cost functions
• Expected Result:

  Without robust kernels: MRE = 0.89 px, outliers affect mean
  With Cauchy kernel: MRE = 0.72 px, outliers downweighted
  Improvement: 19% reduction in MRE
  

Step 12: Cross-Validate with GPS Accuracy

• Action: Correlate MRE with GPS error
• Expected Result:
  • Low MRE correlates with low GPS error
  • Images with MRE > 1.5 px tend to have GPS error > 30m
  • MRE is leading indicator of trajectory quality

Step 13: Test Under Challenging Conditions

• Action: Compute MRE for challenging dataset (AD000001-060)
• Expected Result:

  Full Flight MRE:
  Total features: 55,000
  MRE: 0.84 pixels (still < 1.0)
  Challenging segments:
    - Sharp turns: MRE = 1.15 px (above threshold locally)
    - Normal segments: MRE = 0.68 px
  Overall: AC-10 PASS
  

Step 14: Generate Reprojection Error Report

• Action: Create comprehensive MRE report
• Expected Result:

  ========================================
  REPROJECTION ERROR REPORT
  Flight: AC10_Test
  Dataset: AD000001-AD000030
  ========================================
  
  SUMMARY:
  Mean Reprojection Error: 0.72 pixels
  AC-10 Threshold: 1.0 pixels
  Status: PASS ✓
  
  DISTRIBUTION:
  < 0.5 px: 12,450 (49.8%)
  0.5-1.0 px: 9,875 (39.5%)
  1.0-2.0 px: 2,350 (9.4%)
  2.0-3.0 px: 198 (0.8%)
  > 3.0 px: 127 (0.5%)
  
  PER-IMAGE BREAKDOWN:
  Images meeting < 1.0 px MRE: 28/30 (93.3%)
  Images with highest MRE: AD000032 (1.12 px), AD000048 (1.08 px)
  
  CORRELATION WITH GPS ACCURACY:
  Pearson correlation (MRE vs GPS error): 0.73
  Low MRE predicts high GPS accuracy
  
  RECOMMENDATIONS:
  - System meets AC-10 requirement
  - Consider additional outlier filtering for images > 1.0 px MRE
  - Sharp turn handling could be improved
  ========================================
  

Success Criteria

Primary Criterion (AC-10):

• Mean Reprojection Error < 1.0 pixels across entire flight

Supporting Criteria:

• Standard deviation < 2.0 pixels
• No outlier reprojections > 10 pixels (indicates gross errors)
• Per-image MRE < 2.0 pixels (no catastrophic single-image failures)
• MRE stable across sequence (no drift)

Pass/Fail Criteria

TEST PASSES IF:

• Overall MRE < 1.0 pixels
• Standard deviation reasonable (< 2.0 pixels)
• Less than 1% of features have error > 5.0 pixels
• MRE consistent across multiple test runs (variance < 10%)

TEST FAILS IF:

• MRE ≥ 1.0 pixels
• Standard deviation > 3.0 pixels (high variance indicates instability)
• More than 5% of features have error > 5.0 pixels
• MRE increases significantly over sequence (drift)

Diagnostic Actions if Failing

If MRE > 1.0 px:

1. Check camera calibration accuracy
2. Verify lens distortion model
3. Review feature matching quality (outlier ratio)
4. Examine factor graph convergence
5. Check for scale drift in trajectory

If High Variance:

1. Investigate images with outlier MRE
2. Check for challenging conditions (blur, low texture)
3. Review robust kernel settings
4. Verify triangulation accuracy

Components Involved

• F07 Sequential Visual Odometry: Feature extraction and matching
• F10 Factor Graph Optimizer: Pose optimization, marginal covariances
• F13 Coordinate Transformer: 3D point projection
• H01 Camera Model: Camera intrinsics, projection functions
• H03 Robust Kernels: Outlier handling in optimization

Notes

• MRE is a geometric consistency metric, not direct GPS accuracy
• Low MRE indicates well-constrained factor graph
• High MRE with good GPS accuracy = overfitting to GPS anchors
• Low MRE with poor GPS accuracy = scale/alignment issues
• AC-10 validates internal consistency of vision pipeline