mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 20:31:12 +00:00
Oleksandr Bezdieniezhnykh
174 lines
8.8 KiB
Markdown
174 lines
8.8 KiB
Markdown
# Blackbox Tests
|
|
|
|
## Positive Scenarios
|
|
|
|
### FT-P-01: Still-Image Frame Center Geolocation
|
|
|
|
**Summary**: Validate that the system estimates WGS84 frame centers for the provided 60-image nadir dataset.
|
|
|
|
**Traces to**: AC-1.1, AC-1.2, AC-6.3, AC-8.1
|
|
|
|
**Category**: Position Accuracy
|
|
|
|
**Preconditions**:
|
|
- Offline satellite cache fixture is available for the sample area.
|
|
- Expected results are loaded from `input_data/expected_results/results_report.md`.
|
|
|
|
**Input data**: `project_60_still_images`, `expected_frame_centers`
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Submit `AD000001.jpg` through `AD000060.jpg` with height/camera metadata | System emits one WGS84 estimate per processed image |
|
|
| 2 | Compare each estimate to the mapped expected coordinate | Per-frame error is reported in meters |
|
|
|
|
**Expected outcome**: At least 80% of images are within 50 m and at least 50% are within 20 m.
|
|
|
|
**Max execution time**: 15 minutes for the 60-image replay on the local replay environment.
|
|
|
|
---
|
|
|
|
### FT-P-02: Position Confidence Output Contract
|
|
|
|
**Summary**: Validate that every emitted position estimate includes confidence and source-label fields required by the public contract.
|
|
|
|
**Traces to**: AC-1.3, AC-1.4, AC-4.4, AC-4.5
|
|
|
|
**Category**: Position Confidence
|
|
|
|
**Preconditions**:
|
|
- Same fixture setup as FT-P-01.
|
|
|
|
**Input data**: `project_60_still_images`, `expected_frame_centers`
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Submit the 60-image replay | System emits estimates frame-by-frame, not batched |
|
|
| 2 | Inspect public output fields | Each estimate contains WGS84 coordinate, 95% covariance semi-major axis, source label, and `last_satellite_anchor_age_ms` |
|
|
| 3 | Submit a later correction for a prior frame if available | System emits updated estimate with timestamp and covariance without corrupting newer estimates |
|
|
|
|
**Expected outcome**: 100% of emitted estimates include required confidence fields; no `horiz_accuracy` equivalent under-reports the 95% covariance semi-major axis.
|
|
|
|
**Max execution time**: 15 minutes.
|
|
|
|
---
|
|
|
|
### FT-P-03: BASALT VIO Replay With Synchronized Video/Telemetry
|
|
|
|
**Summary**: Validate that BASALT + safety/anchor wrapper can process synchronized nadir video, IMU, and trajectory telemetry and produce frame-by-frame estimates with honest confidence.
|
|
|
|
**Traces to**: AC-1.3, AC-2.1a, AC-2.2, AC-4.1, AC-4.2
|
|
|
|
**Category**: VO / IMU Propagation
|
|
|
|
**Preconditions**:
|
|
- Derkachi replay fixture is mounted from `input_data/flight_derkachi/`.
|
|
- `flight_derkachi.mp4` is readable as cropped nadir video: 880 x 720, 30 fps, approximately 490.07 s.
|
|
- `data_imu.csv` contains monotonic 10 Hz `Time`, `timestamp(ms)`, `SCALED_IMU2.*`, and `GLOBAL_POSITION_INT.*` fields for 4,900 rows.
|
|
- Camera intrinsics, lens distortion, and camera-to-body transform are either pinned or the run is marked as calibration-limited.
|
|
- Public synchronized dataset slice remains useful for calibrated final comparison. Strongest candidates: MUN-FRL, ALTO, EPFL fixed-wing, Kagaru; EuRoC/UZH FPV are proxy-only.
|
|
|
|
**Input data**: `derkachi_video_telemetry`, `public_nadir_vio_candidates`
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Validate Derkachi video/telemetry alignment | Harness accepts the fixture only if MP4 duration and CSV duration differ by <=250 ms and there are exactly 3 video frames per telemetry row |
|
|
| 2 | Replay synchronized video frames and IMU stream | System emits frame-by-frame `vo_extrapolated` or `satellite_anchored` estimates without batching |
|
|
| 3 | Compare output trajectory to `GLOBAL_POSITION_INT` lat/lon/alt/heading | Error, covariance, source label, and anchor age are reported per segment |
|
|
| 4 | Compare calibrated public/representative replay against ground truth when available | BASALT + wrapper does not materially under-report uncertainty relative to error |
|
|
| 5 | Compare against OpenVINS reference replay when available | BASALT + wrapper does not materially under-report uncertainty relative to error |
|
|
|
|
**Expected outcome**: Derkachi replay is accepted as a synchronized representative fixture and produces continuous estimates for >95% of normal overlapping frames. Absolute geolocation and covariance pass/fail thresholds are calibration-gated until camera intrinsics, distortion, and camera-to-body transform are pinned. For calibrated datasets, VO homography MRE is <1.0 px where homography validation is applicable.
|
|
|
|
**Max execution time**: Dataset-dependent, but replay must report per-frame latency.
|
|
|
|
---
|
|
|
|
### FT-P-04: Satellite Service And Anchor Verification
|
|
|
|
**Summary**: Validate that relocalization uses global retrieval plus local verification and emits only verified satellite anchors.
|
|
|
|
**Traces to**: AC-2.1b, AC-2.2, AC-3.2, AC-3.3, AC-8.6
|
|
|
|
**Category**: Satellite Anchor
|
|
|
|
**Preconditions**:
|
|
- AerialVL/ALTO/VPAir-style public dataset slice or project satellite-cache fixture is available.
|
|
- VPR chunks and descriptors are precomputed.
|
|
|
|
**Input data**: Public aerial localization slice, cache fixture
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Trigger cold-start or relocalization query | System searches CPU FAISS top-K chunks |
|
|
| 2 | Present top-K candidates to local verification | System runs ALIKED/DISK+LightGlue and RANSAC |
|
|
| 3 | Inspect emitted anchor decision | Accepted anchors include source label, MRE, inlier count, covariance, and tile provenance |
|
|
|
|
**Expected outcome**: Cross-domain satellite-anchor MRE is <2.5 px for accepted anchors; rejected candidates do not produce `satellite_anchored` estimates.
|
|
|
|
**Max execution time**: Must be measured as part of performance tests.
|
|
|
|
## Negative Scenarios
|
|
|
|
### FT-N-01: Repetitive Or Low-Texture Imagery
|
|
|
|
**Summary**: Validate that visually ambiguous images do not produce confident false satellite anchors.
|
|
|
|
**Traces to**: AC-1.4, AC-3.1, AC-NEW-4, AC-8.6
|
|
|
|
**Category**: False Position Prevention
|
|
|
|
**Input data**: Repetitive agricultural or low-texture frames from project/public data.
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Submit ambiguous frame or sequence | System either emits degraded `vo_extrapolated`/`dead_reckoned` output or rejects low-confidence anchor |
|
|
| 2 | Inspect anchor and confidence outputs | No anchor is accepted unless local verification and covariance gates pass |
|
|
|
|
**Expected outcome**: 0 confident `satellite_anchored` outputs for candidates that fail local verification, freshness, or Mahalanobis gates.
|
|
|
|
**Max execution time**: 15 minutes per fixture.
|
|
|
|
---
|
|
|
|
### FT-N-02: GPS Spoofing During Total Visual Blackout
|
|
|
|
**Summary**: Validate that spoofed GPS is not promoted during total camera occlusion/visual blackout and that output degrades honestly before unusable frames reach VIO.
|
|
|
|
**Traces to**: AC-3.5, AC-5.2, AC-NEW-2, AC-NEW-8
|
|
|
|
**Category**: Spoofing / Blackout
|
|
|
|
**Input data**: ArduPilot Plane SITL spoofing trace with camera blackout/total-occlusion frames.
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Start normal replay with trusted visual/satellite anchor | System emits normal estimates |
|
|
| 2 | Inject full visual blackout/total occlusion and spoofed `GPS_RAW_INT` | Camera gate sets `usable_for_vio=false`, BASALT is bypassed for occluded frames, and system switches to `dead_reckoned` within <=1 processed frame or <=400 ms |
|
|
| 3 | Continue blackout beyond thresholds | IMU-only covariance grows monotonically; system degrades fix type and emits failsafe status at specified covariance/time thresholds |
|
|
|
|
**Expected outcome**: Spoofed GPS is ignored; total occlusion never feeds BASALT as a usable VIO frame; `fix_type=0`, `horiz_accuracy=999.0`, and `VISUAL_BLACKOUT_FAILSAFE` are emitted when covariance >500 m or blackout >30 s.
|
|
|
|
**Max execution time**: 10 minutes per SITL scenario.
|
|
|
|
---
|
|
|
|
### FT-N-03: Invalid Or Stale Satellite Cache
|
|
|
|
**Summary**: Validate cache freshness, integrity, and provenance gates.
|
|
|
|
**Traces to**: AC-8.2, AC-8.3, AC-NEW-6, AC-NEW-7
|
|
|
|
**Category**: Cache Integrity
|
|
|
|
**Input data**: `cache_integrity_fixtures`
|
|
|
|
| Step | Consumer Action | Expected System Response |
|
|
|------|-----------------|--------------------------|
|
|
| 1 | Replay with stale tile manifest | Tile is rejected or down-confidence weighted; no stale tile emits `satellite_anchored` |
|
|
| 2 | Replay with hash-mismatched or unsigned manifest | Cache fixture is rejected and security event is logged |
|
|
| 3 | Replay generated tile with weak parent-pose covariance | Tile is not promoted beyond allowed trust level |
|
|
|
|
**Expected outcome**: 0 invalid/stale/cache-poisoning fixtures produce trusted anchors or trusted basemap tiles.
|
|
|
|
**Max execution time**: 15 minutes.
|