mirror of
https://github.com/azaion/gps-denied-desktop.git
synced 2026-04-23 01:46:37 +00:00
add tests
gen_tests updated solution.md updated
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# Acceptance Test: 80% Photos <50m Error - Varied Terrain
|
||||
|
||||
## Summary
|
||||
Validate AC-1 accuracy requirement (80% of photos within 50m error) across different terrain types including agricultural fields, mixed vegetation, and urban edges.
|
||||
|
||||
## Linked Acceptance Criteria
|
||||
**AC-1**: The system should find out the GPS of centers of 80% of the photos from the flight within an error of no more than 50 meters in comparison to the real GPS.
|
||||
|
||||
## Preconditions
|
||||
- ASTRAL-Next system fully deployed and operational
|
||||
- Satellite reference data downloaded for test region
|
||||
- TensorRT models loaded (SuperPoint, LightGlue, AnyLoc, LiteSAM)
|
||||
- Ground truth GPS coordinates available for validation
|
||||
- Test datasets covering varied terrain types
|
||||
|
||||
## Test Data
|
||||
- **Primary Dataset**: AD000001-AD000060 (varied terrain across 60 images)
|
||||
- **Terrain Types**: Agricultural fields, tree lines, mixed vegetation, roads
|
||||
- **Ground Truth**: coordinates.csv
|
||||
- **Camera Parameters**: 400m altitude, 25mm focal length, 26MP resolution
|
||||
|
||||
## Test Steps
|
||||
|
||||
### Step 1: Initialize System with Starting GPS
|
||||
**Action**: Start flight processing with first image GPS coordinate (48.275292, 37.385220)
|
||||
**Expected Result**:
|
||||
- System initializes successfully
|
||||
- Satellite tiles downloaded for operational area
|
||||
- L1, L2, L3 layers ready
|
||||
- Status: INITIALIZED
|
||||
|
||||
### Step 2: Process Agricultural Field Segment (AD000001-015)
|
||||
**Action**: Process images over predominantly agricultural terrain
|
||||
**Expected Result**:
|
||||
- L1 sequential tracking maintains continuity
|
||||
- SuperPoint detects field boundaries and crop variations
|
||||
- LiteSAM achieves cross-view matching despite seasonal differences
|
||||
- Mean error <40m for this segment
|
||||
- Status: PROCESSING
|
||||
|
||||
### Step 3: Process Mixed Vegetation Segment (AD000016-030)
|
||||
**Action**: Process images with mixed terrain features
|
||||
**Expected Result**:
|
||||
- L2 global place recognition active during transitions
|
||||
- AnyLoc retrieval successful using DINOv2 features
|
||||
- Factor graph optimization smooths trajectory
|
||||
- Mean error <45m for this segment
|
||||
- Status: PROCESSING
|
||||
|
||||
### Step 4: Process Complex Terrain with Sharp Turns (AD000031-060)
|
||||
**Action**: Process remaining images including sharp turns and outliers
|
||||
**Expected Result**:
|
||||
- L2 recovers from sharp turns (AD000042-043, AD000032-033)
|
||||
- Robust cost functions handle AD000047-048 outlier (268.6m)
|
||||
- Multiple map fragments merged successfully
|
||||
- Mean error <50m for challenging segments
|
||||
- Status: PROCESSING
|
||||
|
||||
### Step 5: Calculate Accuracy Metrics
|
||||
**Action**: Compare estimated GPS coordinates with ground truth
|
||||
**Expected Result**:
|
||||
```
|
||||
Total images: 60
|
||||
Error <50m: ≥48 images (80%)
|
||||
Error <20m: ≥36 images (60%)
|
||||
Mean error: <40m
|
||||
Median error: <35m
|
||||
Max error: <150m (excluding known outliers)
|
||||
```
|
||||
|
||||
### Step 6: Validate Terrain-Specific Performance
|
||||
**Action**: Analyze accuracy by terrain type
|
||||
**Expected Result**:
|
||||
- Agricultural fields: 75-85% <50m
|
||||
- Mixed vegetation: 80-90% <50m
|
||||
- Road intersections: 85-95% <50m
|
||||
- Overall: ≥80% <50m
|
||||
- Status: COMPLETED
|
||||
|
||||
## Pass/Fail Criteria
|
||||
|
||||
**PASS if**:
|
||||
- ≥80% of images (48/60) have error <50m
|
||||
- No systematic bias across terrain types
|
||||
- System completes without fatal errors
|
||||
- Factor graph converges (final MRE <1.5px)
|
||||
|
||||
**FAIL if**:
|
||||
- <80% of images meet 50m threshold
|
||||
- >3 terrain types show <70% accuracy
|
||||
- System crashes or hangs
|
||||
- Catastrophic tracking loss without recovery
|
||||
|
||||
## Performance Requirements
|
||||
- Processing time: <5 seconds per image average
|
||||
- Total flight time: <5 minutes for 60 images
|
||||
- Memory usage: <8GB on RTX 3070
|
||||
- CPU usage: <80% average
|
||||
|
||||
## Notes
|
||||
- Varied terrain test provides more comprehensive validation than baseline
|
||||
- Different terrain types stress different system components
|
||||
- AC-1 threshold of 80% allows for difficult scenarios while maintaining operational utility
|
||||
|
||||
Reference in New Issue
Block a user