add tests

gen_tests updated
solution.md updated
This commit is contained in:
Oleksandr Bezdieniezhnykh
2025-11-24 22:57:46 +02:00
parent f50006d100
commit 4f8c18a066
49 changed files with 7209 additions and 3 deletions
@@ -0,0 +1,104 @@
# Acceptance Test: Single 350m Outlier Robustness
## Summary
Validate AC-3 requirement that the system handles a single outlier image with up to 350m position deviation caused by aircraft tilt without trajectory divergence.
## Linked Acceptance Criteria
**AC-3**: The system should correctly continue the work even in the presence of up to 350 meters of an outlier photo between 2 consecutive pictures en route. This could happen due to tilt of the plane.
## Preconditions
- ASTRAL-Next system operational
- Factor graph optimizer configured with robust cost functions (Cauchy/Huber kernel)
- Test dataset with known large position jump
- Ground truth available for validation
## Test Data
- **Dataset**: AD000045-AD000050 (6 images)
- **Outlier**: AD000047 → AD000048 (268.6m jump, closest to 350m requirement)
- **Ground Truth**: coordinates.csv
- **Purpose**: Verify robust handling of tilt-induced position jumps
## Test Steps
### Step 1: Establish Baseline Trajectory
**Action**: Process AD000045-AD000046 to establish stable tracking
**Expected Result**:
- L1 sequential tracking successful
- Factor graph initialized with first 2 poses
- Baseline trajectory established
- Status: TRACKING
### Step 2: Process Outlier Frame
**Action**: Process AD000047 (before the 268.6m jump)
**Expected Result**:
- Frame processed normally
- Factor graph updated
- Position estimate within expected range
- Status: TRACKING
### Step 3: Encounter Large Position Jump
**Action**: Process AD000048 (268.6m from AD000047)
**Expected Result**:
- L1 reports low confidence or matching failure
- Large translation vector detected (>>120m expected)
- Robust cost function activates (error penalty changes from quadratic to linear/log)
- Factor graph does NOT diverge
- Status: OUTLIER_DETECTED
### Step 4: Verify Robust Handling
**Action**: Examine factor graph's treatment of the outlier constraint
**Expected Result**:
- Outlier factor down-weighted (residual weight <0.3)
- OR outlier treated as rotation+normal translation
- Trajectory maintains consistency with other constraints
- No catastrophic error propagation
- Status: OUTLIER_HANDLED
### Step 5: Continue Processing Post-Outlier
**Action**: Process AD000049-AD000050
**Expected Result**:
- L2 global relocalization may activate to verify position
- L1 tracking resumes successfully
- Factor graph optimization converges
- Post-outlier trajectory accurate (<50m error)
- Status: RECOVERED
### Step 6: Validate Final Trajectory
**Action**: Compare entire sequence with ground truth
**Expected Result**:
```
AD000045: Error <50m
AD000046: Error <50m
AD000047: Error <50m
AD000048: Error may be 50-100m (outlier frame)
AD000049: Error <50m (recovered)
AD000050: Error <50m (recovered)
Overall: System continues operation successfully
```
## Pass/Fail Criteria
**PASS if**:
- System processes all 6 images without crashing
- Outlier frame (AD000048) does not cause trajectory divergence
- Post-outlier frames (049-050) achieve <50m error
- Factor graph converges with MRE <2.0px
- No manual intervention required
**FAIL if**:
- System crashes on outlier frame
- Trajectory diverges (errors >200m persist for >2 frames)
- Factor graph fails to converge
- Manual user input required for recovery
## Technical Validation
- **Robust Kernel Active**: Verify Cauchy/Huber kernel applied to outlier factor
- **Residual Analysis**: Outlier residual >3σ from mean
- **Weight Analysis**: Outlier factor weight <0.3 vs normal ~1.0
- **Convergence**: Final optimization converges in <50 iterations
## Notes
- Test uses 268.6m outlier (real data) as proxy for 350m requirement
- Robust M-estimation is critical for AC-3 compliance
- Single outlier is easier than multiple; see AC-3 test 35 for multiple outliers