add clarification to research methodology by including a step for solution comparison and user consultation

2026-04-22 11:36:36 +00:00 · 2026-03-17 18:43:57 +02:00
parent d764250f9a
commit b419e2c04a
35 changed files with 6030 additions and 0 deletions
@@ -0,0 +1,96 @@
+# Validation Log
+
+## Validation Scenario 1: Normal Flight (500 images, 60-80% overlap, mild turns)
+
+### Expected Based on Conclusions
+- XFeat VO: ~15ms/frame → total VO time for 500 images: ~7.5s
+- LiteSAM satellite matching (overlapped): ~200ms/frame
+- Total processing: ~100s (under 5s/image budget)
+- Most images get satellite anchors → HIGH confidence
+- VRAM peak: ~900MB (XFeat 200 + DINOv2 300 + LiteSAM 400)
+
+### Actual Validation Results
+Consistent with SatLoc-Fusion results on similar setup. XFeat handles consecutive frames well. Time budget is well within 5s AC.
+
+### Counterexamples
+None for normal flight.
+
+---
+
+## Validation Scenario 2: Flight over outdated satellite imagery area (eastern Ukraine conflict zone)
+
+### Expected Based on Conclusions
+- DINOv2 coarse retrieval: semantic matching should still identify approximate area despite 1-3 year imagery age
+- LiteSAM fine matching: likely degraded if buildings destroyed/rebuilt. Hit rate could drop 10-20pp from baseline.
+- Many frames may be VO-only → drift accumulates
+- Drift monitoring triggers warnings at 100m, user input at 200m
+
+### Actual Validation Results
+System degrades gracefully. VO chain continues providing relative positioning. Satellite anchors become sparse. Confidence reporting reflects this via exponential decay formula.
+
+### Counterexamples
+If entire flight is over heavily changed terrain, ALL satellite matches may fail. System falls back to pure VO + user manual anchoring. This is handled by the segment manager but accuracy degrades significantly.
+
+---
+
+## Validation Scenario 3: LiteSAM startup failure (weights corrupted or unavailable)
+
+### Expected Based on Conclusions
+- SHA256 checksum verification catches corruption at startup
+- System falls back to EfficientLoFTR (or SP+LG) for satellite fine matching
+- Warning logged, system continues
+
+### Actual Validation Results
+Fallback mechanism ensures system availability. EfficientLoFTR has proven quality (CVPR 2024).
+
+### Counterexamples
+If BOTH LiteSAM and fallback fail to load → system should still start but without satellite matching (VO-only mode). Not currently handled — SHOULD add this graceful degradation.
+
+---
+
+## Validation Scenario 4: Sharp turn with 5+ disconnected segments
+
+### Expected Based on Conclusions
+- Each segment tracks independently with VO
+- Satellite anchoring attempts run for each segment
+- ANCHORED segments check for nearby FLOATING segments
+- With XFeat VO at 15ms, segment transitions are detected quickly
+
+### Actual Validation Results
+Strategy works for 2-3 segments. With 5+ segments, reconnection order matters. Should process segments in proximity order. If satellite imagery is outdated for the area, many segments remain FLOATING.
+
+### Counterexamples
+All segments FLOATING in a poor satellite imagery area. User must manually anchor at least one image per segment. Current system handles this but UX could be improved — suggest a "batch anchor" endpoint.
+
+---
+
+## Validation Scenario 5: iSAM2 exception during optimization
+
+### Expected Based on Conclusions
+- IndeterminantLinearSystemException caught
+- Skip problematic factor, retry with relaxed noise
+- Pipeline continues
+
+### Actual Validation Results
+Error handling prevents crash. Position for affected frame derived from VO only.
+
+### Counterexamples
+If exception happens on first frame's prior factor → entire optimization fails. Need special handling for initial factor.
+
+---
+
+## Review Checklist
+- [x] Draft conclusions consistent with fact cards
+- [x] No important dimensions missed
+- [x] No over-extrapolation
+- [x] Conclusions actionable/verifiable
+- [x] LiteSAM hit rate correctly attributed to proper dataset
+- [x] VO regression identified and fix proposed
+- [x] Security CVEs addressed with version pinning
+- [ ] Issue: Need to add EfficientLoFTR fallback and graceful degradation for model loading failures
+- [ ] Issue: Need to add iSAM2 error handling for initial factor edge case
+
+## Conclusions Requiring Revision
+1. Add graceful degradation when ALL matchers fail to load (VO-only mode)
+2. Add special iSAM2 error handling for initial prior factor
+3. Consider "batch anchor" API endpoint for multi-segment manual anchoring UX