mirror of
https://github.com/azaion/gps-denied-desktop.git
synced 2026-04-23 00:26:36 +00:00
initial structure implemented
docs -> _docs
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,256 @@
|
||||
# Acceptance Test: AC-10 - Mean Reprojection Error < 1.0 pixels
|
||||
|
||||
## Summary
|
||||
Validate Acceptance Criterion 10: "Mean Reprojection Error (MRE) < 1.0 pixels. The distance, in pixels, between the original pixel location of the object and the re-projected pixel location."
|
||||
|
||||
## Linked Acceptance Criteria
|
||||
**AC-10**: MRE < 1.0 pixels
|
||||
|
||||
## Preconditions
|
||||
1. ASTRAL-Next system operational
|
||||
2. F07 Sequential Visual Odometry extracting and matching features
|
||||
3. F10 Factor Graph Optimizer computing optimized poses
|
||||
4. Camera intrinsics calibrated (from F17 Configuration Manager)
|
||||
5. Test dataset with ground truth poses (for reference)
|
||||
6. Reprojection error calculation implemented
|
||||
|
||||
## Reprojection Error Definition
|
||||
|
||||
**Formula**:
|
||||
```
|
||||
For each matched feature point p_i in image I_j:
|
||||
1. Triangulate 3D point X_i from matches across images
|
||||
2. Project X_i back to image I_j using optimized pose T_j and camera K
|
||||
3. p'_i = K * T_j * X_i (projected pixel location)
|
||||
4. e_i = ||p_i - p'_i|| (Euclidean distance in pixels)
|
||||
|
||||
MRE = (1/N) * Σ e_i (mean across all features in all images)
|
||||
```
|
||||
|
||||
## Test Data
|
||||
- **Dataset**: AD000001-AD000030 (30 images, baseline)
|
||||
- **Expected Features**: ~500-2000 matched features per image pair
|
||||
- **Total Measurements**: ~15,000-60,000 reprojection measurements
|
||||
|
||||
## Test Steps
|
||||
|
||||
### Step 1: Process Flight Through Complete Pipeline
|
||||
- **Action**: Process AD000001-AD000030 through full ASTRAL-Next pipeline
|
||||
- **Expected Result**:
|
||||
- Factor graph initialized and optimized
|
||||
- 30 poses computed
|
||||
- All feature correspondences stored
|
||||
|
||||
### Step 2: Extract Feature Correspondences
|
||||
- **Action**: Retrieve all matched features from F07
|
||||
- **Expected Result**:
|
||||
- For each image pair (i, j):
|
||||
- List of matched keypoint pairs: [(p_i, p_j), ...]
|
||||
- Match confidence scores
|
||||
- Total: ~500-1500 matches per pair
|
||||
- Total matches across flight: ~15,000-45,000
|
||||
|
||||
### Step 3: Triangulate 3D Points
|
||||
- **Action**: For each matched feature across multiple views, triangulate 3D position
|
||||
- **Expected Result**:
|
||||
- 3D point cloud generated
|
||||
- Each point has:
|
||||
- 3D coordinates (X, Y, Z) in ENU frame
|
||||
- List of observations (image_id, pixel_location)
|
||||
- Triangulation uncertainty
|
||||
|
||||
### Step 4: Calculate Per-Feature Reprojection Error
|
||||
- **Action**: For each 3D point and each observation:
|
||||
```
|
||||
For point X with observation (image_j, pixel_p):
|
||||
1. Get optimized pose T_j from factor graph
|
||||
2. Get camera intrinsics K from config
|
||||
3. Project: p' = project(K, T_j, X)
|
||||
4. Error: e = sqrt((p.x - p'.x)² + (p.y - p'.y)²)
|
||||
```
|
||||
- **Expected Result**:
|
||||
- Array of per-feature reprojection errors
|
||||
- Typical range: 0.1 - 3.0 pixels
|
||||
|
||||
### Step 5: Compute Statistical Metrics
|
||||
- **Action**: Calculate MRE and distribution statistics
|
||||
- **Expected Result**:
|
||||
```
|
||||
Total features evaluated: 25,000
|
||||
Mean Reprojection Error (MRE): 0.72 pixels
|
||||
Median Reprojection Error: 0.58 pixels
|
||||
Standard Deviation: 0.45 pixels
|
||||
90th Percentile: 1.25 pixels
|
||||
95th Percentile: 1.68 pixels
|
||||
99th Percentile: 2.41 pixels
|
||||
Max Error: 4.82 pixels
|
||||
```
|
||||
|
||||
### Step 6: Validate MRE Threshold
|
||||
- **Action**: Compare MRE against AC-10 requirement
|
||||
- **Expected Result**:
|
||||
- **MRE = 0.72 pixels < 1.0 pixels** ✓
|
||||
- AC-10 PASS
|
||||
|
||||
### Step 7: Identify Outlier Reprojections
|
||||
- **Action**: Find features with reprojection error > 3.0 pixels
|
||||
- **Expected Result**:
|
||||
```
|
||||
Outliers (> 3.0 pixels): 127 (0.5% of total)
|
||||
Outlier distribution:
|
||||
- 3.0-5.0 pixels: 98 features
|
||||
- 5.0-10.0 pixels: 27 features
|
||||
- > 10.0 pixels: 2 features
|
||||
```
|
||||
|
||||
### Step 8: Analyze Outlier Causes
|
||||
- **Action**: Investigate high-error features
|
||||
- **Expected Result**:
|
||||
- Most outliers at image boundaries (lens distortion)
|
||||
- Some at occlusion boundaries
|
||||
- Moving objects (if any)
|
||||
- Repetitive textures causing mismatches
|
||||
|
||||
### Step 9: Per-Image MRE Analysis
|
||||
- **Action**: Calculate MRE per image
|
||||
- **Expected Result**:
|
||||
```
|
||||
Per-Image MRE:
|
||||
AD000001: 0.68 px (baseline)
|
||||
AD000002: 0.71 px
|
||||
...
|
||||
AD000032: 1.12 px (sharp turn - higher error)
|
||||
AD000033: 0.95 px
|
||||
...
|
||||
AD000030: 0.74 px
|
||||
|
||||
Images with MRE > 1.0: 2 out of 30 (6.7%)
|
||||
Overall MRE: 0.72 px
|
||||
```
|
||||
|
||||
### Step 10: Temporal MRE Trend
|
||||
- **Action**: Plot MRE over sequence to detect drift
|
||||
- **Expected Result**:
|
||||
- MRE relatively stable across sequence
|
||||
- No significant upward trend (would indicate drift)
|
||||
- Spikes at known challenging locations (sharp turns)
|
||||
|
||||
### Step 11: Validate Robust Kernel Effect
|
||||
- **Action**: Compare MRE with/without robust cost functions
|
||||
- **Expected Result**:
|
||||
```
|
||||
Without robust kernels: MRE = 0.89 px, outliers affect mean
|
||||
With Cauchy kernel: MRE = 0.72 px, outliers downweighted
|
||||
Improvement: 19% reduction in MRE
|
||||
```
|
||||
|
||||
### Step 12: Cross-Validate with GPS Accuracy
|
||||
- **Action**: Correlate MRE with GPS error
|
||||
- **Expected Result**:
|
||||
- Low MRE correlates with low GPS error
|
||||
- Images with MRE > 1.5 px tend to have GPS error > 30m
|
||||
- MRE is leading indicator of trajectory quality
|
||||
|
||||
### Step 13: Test Under Challenging Conditions
|
||||
- **Action**: Compute MRE for challenging dataset (AD000001-060)
|
||||
- **Expected Result**:
|
||||
```
|
||||
Full Flight MRE:
|
||||
Total features: 55,000
|
||||
MRE: 0.84 pixels (still < 1.0)
|
||||
Challenging segments:
|
||||
- Sharp turns: MRE = 1.15 px (above threshold locally)
|
||||
- Normal segments: MRE = 0.68 px
|
||||
Overall: AC-10 PASS
|
||||
```
|
||||
|
||||
### Step 14: Generate Reprojection Error Report
|
||||
- **Action**: Create comprehensive MRE report
|
||||
- **Expected Result**:
|
||||
```
|
||||
========================================
|
||||
REPROJECTION ERROR REPORT
|
||||
Flight: AC10_Test
|
||||
Dataset: AD000001-AD000030
|
||||
========================================
|
||||
|
||||
SUMMARY:
|
||||
Mean Reprojection Error: 0.72 pixels
|
||||
AC-10 Threshold: 1.0 pixels
|
||||
Status: PASS ✓
|
||||
|
||||
DISTRIBUTION:
|
||||
< 0.5 px: 12,450 (49.8%)
|
||||
0.5-1.0 px: 9,875 (39.5%)
|
||||
1.0-2.0 px: 2,350 (9.4%)
|
||||
2.0-3.0 px: 198 (0.8%)
|
||||
> 3.0 px: 127 (0.5%)
|
||||
|
||||
PER-IMAGE BREAKDOWN:
|
||||
Images meeting < 1.0 px MRE: 28/30 (93.3%)
|
||||
Images with highest MRE: AD000032 (1.12 px), AD000048 (1.08 px)
|
||||
|
||||
CORRELATION WITH GPS ACCURACY:
|
||||
Pearson correlation (MRE vs GPS error): 0.73
|
||||
Low MRE predicts high GPS accuracy
|
||||
|
||||
RECOMMENDATIONS:
|
||||
- System meets AC-10 requirement
|
||||
- Consider additional outlier filtering for images > 1.0 px MRE
|
||||
- Sharp turn handling could be improved
|
||||
========================================
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
**Primary Criterion (AC-10)**:
|
||||
- Mean Reprojection Error < 1.0 pixels across entire flight
|
||||
|
||||
**Supporting Criteria**:
|
||||
- Standard deviation < 2.0 pixels
|
||||
- No outlier reprojections > 10 pixels (indicates gross errors)
|
||||
- Per-image MRE < 2.0 pixels (no catastrophic single-image failures)
|
||||
- MRE stable across sequence (no drift)
|
||||
|
||||
## Pass/Fail Criteria
|
||||
|
||||
**TEST PASSES IF**:
|
||||
- Overall MRE < 1.0 pixels
|
||||
- Standard deviation reasonable (< 2.0 pixels)
|
||||
- Less than 1% of features have error > 5.0 pixels
|
||||
- MRE consistent across multiple test runs (variance < 10%)
|
||||
|
||||
**TEST FAILS IF**:
|
||||
- MRE ≥ 1.0 pixels
|
||||
- Standard deviation > 3.0 pixels (high variance indicates instability)
|
||||
- More than 5% of features have error > 5.0 pixels
|
||||
- MRE increases significantly over sequence (drift)
|
||||
|
||||
## Diagnostic Actions if Failing
|
||||
|
||||
**If MRE > 1.0 px**:
|
||||
1. Check camera calibration accuracy
|
||||
2. Verify lens distortion model
|
||||
3. Review feature matching quality (outlier ratio)
|
||||
4. Examine factor graph convergence
|
||||
5. Check for scale drift in trajectory
|
||||
|
||||
**If High Variance**:
|
||||
1. Investigate images with outlier MRE
|
||||
2. Check for challenging conditions (blur, low texture)
|
||||
3. Review robust kernel settings
|
||||
4. Verify triangulation accuracy
|
||||
|
||||
## Components Involved
|
||||
- F07 Sequential Visual Odometry: Feature extraction and matching
|
||||
- F10 Factor Graph Optimizer: Pose optimization, marginal covariances
|
||||
- F13 Coordinate Transformer: 3D point projection
|
||||
- H01 Camera Model: Camera intrinsics, projection functions
|
||||
- H03 Robust Kernels: Outlier handling in optimization
|
||||
|
||||
## Notes
|
||||
- MRE is a geometric consistency metric, not direct GPS accuracy
|
||||
- Low MRE indicates well-constrained factor graph
|
||||
- High MRE with good GPS accuracy = overfitting to GPS anchors
|
||||
- Low MRE with poor GPS accuracy = scale/alignment issues
|
||||
- AC-10 validates internal consistency of vision pipeline
|
||||
Reference in New Issue
Block a user