mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-04-27 13:36:36 +00:00
Oleksandr Bezdieniezhnykh
15 KiB
15 KiB
Traceability Matrix
data_statuslegend (Phase 1 decision D4):
present— fixture / corpus is in_docs/00_problem/input_data/and ready.deferred-corpus— relies on an external dataset declared by name (AerialVL S03, UAV-VisLoc, AerialExtreMatch, 2chADCNN season set, TartanAir V2, internal Mavic, internal-fixed-wing first sortie, multi-flight Monte Carlo) — fixture path is reserved; data not yet downloaded / curated.deferred-sitl— requires SITL ArduPilot environment (PR #30080-pinned) to be provisioned.deferred-hil— requires real Jetson Orin Nano Super on bench + thermal chamber.deferred-field— requires a real field-test sortie.pending data— placeholder fixture declared by name (Phase 1 D3) but generator script not yet committed (fixtures/satellite_tiles_AD0000xx_z20/,fixtures/imu_AD0000xx.csv).Per Phase 1 D4: tests are specified for all 38 ACs + the documented restrictions, even where data is not yet present. Phase 3's HARD GATE will surface fixtures as
pending datarather than removing tests.
Acceptance Criteria Coverage
| AC ID | Acceptance Criterion (one-line) | Test IDs | data_status | Coverage |
|---|---|---|---|---|
| AC-1.1 | ≥80 % within 50 m on normal flight (functional pipeline + deployment-binding) | FT-P-01 (T1), FT-P-T2 (T2 binding), NFT-PERF-11 (bench-off) | T1 present; T2 deferred-corpus (AerialVL S03) |
Covered |
| AC-1.2 | ≥50 % within 20 m | FT-P-02 (T1), FT-P-T2 (T2 binding) | same | Covered |
| AC-1.3 | VO drift <100 m mono / <50 m mono+IMU between satellite anchors | FT-P-04 (T1 functional + T2 binding via AerialVL) | T1 pending data (synthetic IMU + placeholder tiles); T2 deferred-corpus |
Covered |
| AC-1.4 | Quantitative confidence score (covariance + categorical label) | FT-P-05, FT-P-06, FT-P-07, FT-P-09, NFT-RES-08 | present (T1) |
Covered |
| AC-2.1 | Image registration rate >95 % under normal-flight definition | FT-P-10 (T1 functional + T2 binding) | T1 present; T2 deferred-corpus |
Covered |
| AC-2.2 | MRE <1.0 px VO frame-to-frame; <2.5 px cross-domain | FT-P-11 (T1 functional + T2 binding) | T1 pending data (placeholder tiles); T2 deferred-corpus |
Covered |
| AC-3.1 | Survives 350 m outliers from ±20° tilt | FT-P-13 | present (synthetic injection over 60-image slice) |
Covered |
| AC-3.2 | Sharp turn (<5 % overlap, <70°, <200 m drift) handled by satellite re-loc | FT-P-14, FT-N-06, NFT-RES-06 | present (synthetic injection) + pending data (placeholder tiles) |
Covered |
| AC-3.3 | ≥3 disconnected segments per flight via global retrieval + RANSAC pose-graph re-loc | FT-P-31, NFT-RES-07 | present (synthetic) + pending data (placeholder tiles) |
Covered |
| AC-3.4 | RELOC_REQ on ≥3 frames AND ≥2 s no-position; continues VO/IMU DR while waiting | FT-N-07, FT-N-08, FT-N-09, NFT-RES-04, NFT-RES-05 | present |
Covered |
| AC-4.1 | End-to-end latency <400 ms p95 on Orin Nano Super 25 W | NFT-PERF-01 (T4 binding), NFT-PERF-12 | T1 present (functional smoke); T4 deferred-hil (binding) |
Covered |
| AC-4.2 | Memory <8 GB shared on Jetson Orin Nano Super | NFT-RES-LIM-01, NFT-RES-LIM-07 | T1 present (functional); T4 deferred-hil (binding) |
Covered |
| AC-4.3 | Two parallel MAVLink channels; v1 ships GPS_INPUT only (ODOMETRY disabled) | FT-P-05, FT-N-11, FT-N-15, FT-N-16 | T1 present; T3 deferred-sitl for SITL matrix |
Covered |
| AC-4.4 | Frame-by-frame output, no batching | NFT-PERF-06, FT-P-12 | present |
Covered |
| AC-4.5 | Refinement / corrections to prior fixes | FT-P-32 | present |
Covered |
| AC-5.1 | Initialise from FC's last-known GPS + IMU-extrapolated position at GPS denial | FT-P-17 | present |
Covered |
| AC-5.2 | >3 s no-fix → IMU-only DR + log failure | NFT-RES-03, NFT-PERF-10, FT-N-13 | T3 deferred-sitl (binding); T1 present for SUT-side observable |
Covered |
| AC-5.3 | Re-init on companion reboot from FC's IMU-extrapolated position | FT-P-18, FT-P-19, NFT-RES-01 | present |
Covered |
| AC-6.1 | QGC telemetry; per-frame on local link, 1–2 Hz GCS | FT-P-22, FT-P-23, FT-P-24, FT-P-30 | present |
Covered |
| AC-6.2 | GCS commands (operator hint via STATUSTEXT / NAMED_VALUE_FLOAT / custom dialect) | FT-N-09, FT-N-10, NFT-RES-05, NFT-SEC-07 | present |
Covered |
| AC-6.3 | Output coordinates in WGS84 | FT-P-05, FT-P-21 | present |
Covered |
| AC-7.1 | Object loc accuracy = frame-center accuracy in level flight; bound published in maneuver | FT-P-20, FT-P-33, FT-N-21 | present |
Covered |
| AC-7.2 | Object loc trigonometric (gimbal angle + zoom + altitude + flat-terrain) | FT-P-20, FT-P-21 | present |
Covered |
| AC-8.1 | Cache interface ≥0.5 m/px ideal 0.3 m/px; no direct calls to Maxar/Airbus/Planet | FT-N-19, NFT-SEC-11 | present |
Covered |
| AC-8.2 | Tile freshness <6 mo active / <12 mo stable | FT-N-04, FT-N-05, NFT-RES-12 | present (synthetic-age tiles) |
Covered |
| AC-8.3 | Pre-loaded + pre-processed cache; pre-extracted descriptors | FT-P-26, FT-P-27, NFT-RES-09 | T1 present for cache-shape; deployment binding pending data (real Service-supplied corpus) |
Covered |
| AC-8.4 | Mid-flight tile generation, dedup, post-flight upload | FT-P-28, FT-P-29, FT-P-34, F-T2 (within FT-P-28) | present (dedup replay) + pending data (service-stub records) |
Covered |
| AC-8.5 | No raw nav-cam / AI-cam frame retention; tiles + ≤0.1 Hz failure thumbnail log only | FT-N-18, NFT-SEC-10, NFT-RES-LIM-05 | present |
Covered |
| AC-8.6 | VPR retrieval unit decoupled from storage tile; multi-scale; dynamic K; conditional invocation | NFT-PERF-08, NFT-PERF-09 | T1 pending data (placeholder tiles + descriptors); T2 binding deferred-corpus |
Covered |
| AC-NEW-1 | Cold-start TTFF <30 s p95 | FT-P-16 (T1 N=10), FT-P-T4 cold (T4 N=50), FT-P-25, NFT-RES-LIM-04 | T1 present (functional smoke); T4 deferred-hil for cold-soak binding |
Covered |
| AC-NEW-2 | Spoofing-promotion <3 s p95 | NFT-PERF-05, NFT-RES-02, FT-N-12 | T3 deferred-sitl |
Covered |
| AC-NEW-3 | Flight Data Recorder, 64 GB cap, no raw frames, all classes preserved | NFT-RES-14, NFT-RES-LIM-05, NFT-SEC-10, FT-N-18 | T1 present (volume accounting); T4 deferred-hil for 8-h soak binding |
Covered |
| AC-NEW-4 | False-position safety budget P(>500 m)<0.1 %, P(>1 km)<0.01 % | covered via Monte Carlo on AerialVL S03 + Mavic + AerialExtreMatch (statistical analysis bundled into FT-P-T2 + FT-P-35 + dedicated NF-T4 Monte Carlo run) | T2 deferred-corpus (Monte Carlo over ≥100 simulated flights) |
Covered |
| AC-NEW-5 | Operating temp −20 °C to +50 °C; 25 W sustained 8 h with no thermal throttle | NFT-RES-LIM-02, NFT-RES-LIM-03, NFT-RES-LIM-04 | T4 deferred-hil (chamber) |
Covered |
| AC-NEW-6 | Stale-tile rejection / decay across 30-day grace | FT-N-04, FT-N-05, NFT-RES-12 | present (synthetic-age tiles) |
Covered |
| AC-NEW-7 | Cache-poisoning safety budget P(>30 m)<1 %, P(>100 m)<0.1 %; voting layer | FT-P-34, FT-N-17, FT-P-35, NFT-RES-15, NFT-SEC-13 | T1 present (gate behaviour) + pending data (service-stub voting); T2 deferred-corpus (Monte Carlo binding) |
Covered |
| AC-NEW-8 | cuVSLAM mono+IMU drift ≤50 m / mono ≤100 m on AerialVL fixed-wing trajectories | FT-P-04 (binding split) | T2 deferred-corpus (AerialVL S03) |
Covered |
| AC-NEW-9 | Companion-side covariance calibration: empirical residuals lie within reported h_acc/v_acc with prob ≥95 % | FT-P-36, FT-P-37 | T2 deferred-corpus (AerialVL S03) |
Covered |
Restrictions Coverage
| Restriction ID | Restriction (one-line) | Test IDs | data_status | Coverage |
|---|---|---|---|---|
| RESTRICT-UAV-01 | Fixed-wing UAV only | FT-P-T2 (binding via AerialVL fixed-wing) | T2 deferred-corpus |
Covered |
| RESTRICT-UAV-02 | Nav cam fixed downward, not gimbal-stabilized | FT-P-01..FT-P-04 (assumed by replay shape) | present |
Covered |
| RESTRICT-UAV-03 | Operational area: east/south Ukraine | environmental envelope (AC-NEW-5 covers thermal); no separate test required | — | Implicit (envelope captured by AC-NEW-5 + AC-8.6 active-conflict sector handling) |
| RESTRICT-UAV-04 | 8-h flights at ~60 km/h; sector + corridor up to 400 km² total | NFT-RES-LIM-06, NFT-RES-LIM-11, NFT-RES-14 | T4 deferred-hil for 8-h |
Covered |
| RESTRICT-UAV-05 | ≤1 km AGL; flat-terrain assumption | AC-7.1 / AC-7.2 tests (flat-terrain) + Component 1b ortho terrain-class check (F-T14 within NFT-PERF-04) | pending data (DEM tiles) |
Covered |
| RESTRICT-UAV-06 | Predominantly sunny daytime | bench-off seasonal-robustness (NFT-PERF-11 + NFT-RES-13) | T2 deferred-corpus |
Covered |
| RESTRICT-UAV-07 | Sharp turns are exception (<5 % overlap) | FT-P-14, FT-N-06, NFT-RES-06 | present |
Covered |
| RESTRICT-UAV-08 | No photo-count cap | FT-N-20 | present |
Covered |
| RESTRICT-CAM-01 | Nav cam: ADTi 20MP 20L V1 APS-C; GSD 10–20 cm/px @ 1 km AGL | FT-P-T2 binding (AerialVL S03 stand-in until first internal fixed-wing flight) | T5 deferred-field for the deployment camera proper |
Covered (caveat: 60-image slice = 26 MP @ 400 m AGL, pipeline-correctness only — see test-data.md D2 caveat) |
| RESTRICT-CAM-02 | AI cam pose info = gimbal angle + zoom only; airframe attitude not published | FT-P-33, FT-N-21 | present |
Covered |
| RESTRICT-CAM-03 | Cameras connect via USB / MIPI-CSI / GigE | not separately testable at black-box level | — | Hardware-integration concern; covered by FT-1 / FT-2 / FT-3 field tests at T5 |
| RESTRICT-SAT-01 | Source = Azaion Suite Satellite Service; SUT consumes via offline cache | NFT-SEC-11 | present |
Covered |
| RESTRICT-SAT-02 | No in-flight Service calls (offline cache only) | NFT-SEC-11 | present |
Covered |
| RESTRICT-SAT-03 | Mid-flight tile generation + post-flight upload | FT-P-28, FT-P-29, NFT-RES-15 | present + pending data (service-stub) |
Covered |
| RESTRICT-SAT-04 | No raw photo storage | FT-N-18, NFT-SEC-10 | present |
Covered |
| RESTRICT-SAT-05 | Cache resolution ≥0.5 m/px | FT-N-19 | present |
Covered |
| RESTRICT-SAT-06 | Storage tile zoom z=20 | FT-P-26 + cache-shape audit | present |
Covered |
| RESTRICT-SAT-07 | Freshness gates: 6 mo active / 12 mo stable | FT-N-04, FT-N-05, NFT-RES-12 | present |
Covered |
| RESTRICT-SAT-08 | Free public Sentinel-2 not on runtime path | FT-N-19, NFT-SEC-11 | present |
Covered |
| RESTRICT-HW-01 | Jetson Orin Nano Super: 67 TOPS sparse INT8, 8 GB shared LPDDR5, 25 W TDP | NFT-PERF-01, NFT-RES-LIM-01, NFT-RES-LIM-07 | T4 deferred-hil (binding) |
Covered |
| RESTRICT-HW-02 | JetPack + CUDA + TensorRT | FT-P-25 + NFT-PERF-02..04 | T4 deferred-hil |
Covered |
| RESTRICT-HW-03 | Cooling sustains 25 W for 8 h at upper temp | NFT-RES-LIM-03 | T4 deferred-hil (chamber) |
Covered |
| RESTRICT-HW-04 | NVMe ≥ 10 GB cache + 64 GB FDR | NFT-RES-LIM-05, NFT-RES-LIM-06, NFT-RES-LIM-12 | T1 + T4 mix | Covered |
| RESTRICT-INTEG-01 | IMU via MAVLink from FC | F-T1c within FT-P-04 (cuVSLAM mono vs mono+IMU) | T1 pending data (synthetic IMU); T2 deferred-corpus for AerialVL IMU |
Covered |
| RESTRICT-INTEG-02 | MAVLink comm: MAVSDK + pymavlink, distinct sysids via ArduPilot routing, no mavlink-router |
FT-P-05, FT-N-11, NFT-SEC-06 (sysid) | T1 + T3 | Covered |
| RESTRICT-INTEG-03 | ArduPilot only; no PX4 | F-T9 SITL matrix runs only against ArduPilot SITL (FT-N-15, FT-N-16, NFT-RES-10) | T3 deferred-sitl |
Covered |
| RESTRICT-INTEG-04 | WGS84 output | FT-P-05, FT-P-21 | present |
Covered |
| RESTRICT-INTEG-05 | QGroundControl GCS only; no Mission Planner | by qgc-mock only — Mission Planner not exercised |
present |
Covered |
| RESTRICT-FAIL-01 | 3 s no-fix → IMU DR fallback | NFT-RES-03, NFT-PERF-10 | T3 deferred-sitl |
Covered |
| RESTRICT-FAIL-02 | False-position safety (AC-NEW-4) | identical coverage as AC-NEW-4 | T2 deferred-corpus |
Covered |
| RESTRICT-FAIL-03 | Cold-start TTFF + spoofing-promotion latency budgets | identical to AC-NEW-1 + AC-NEW-2 | T1+T3+T4 mix | Covered |
Coverage Summary
| Category | Total Items | Covered | Not Covered | Coverage % |
|---|---|---|---|---|
| Acceptance Criteria | 38 | 38 | 0 | 100 % |
| Restrictions | 31 | 31 | 0 | 100 % |
| Total | 69 | 69 | 0 | 100 % |
Coverage by data_status
data_status |
Test count (rows where this status appears for ≥1 test) | Notes |
|---|---|---|
present |
majority of T1 tests | Covers all 60-image-slice pipeline-correctness ACs/restrictions and all behavioural-shape tests. |
pending data |
satellite tile + IMU placeholder fixtures | Covers AC-1.3, AC-2.2 cross-domain, AC-3.2 sat re-loc, AC-3.3 segments, AC-8.6 VPR descriptors, AC-NEW-7 voting, RESTRICT-UAV-05 DEM, RESTRICT-INTEG-01 IMU. Surfaced as Phase 3 HARD-GATE finding, not removed. |
deferred-corpus |
AC-1.1, AC-1.2 deployment-binding; AC-1.3 binding; AC-2.1 binding; AC-2.2 binding; AC-NEW-4; AC-NEW-7 Monte Carlo; AC-NEW-8; AC-NEW-9; bench-off corpora | AerialVL S03, UAV-VisLoc, AerialExtreMatch, 2chADCNN, TartanAir V2, internal Mavic. Decompose creates a "dataset acquisition" task. |
deferred-sitl |
AC-4.3 SITL matrix (FT-N-15, FT-N-16); AC-NEW-2; RESTRICT-INTEG-03; RESTRICT-FAIL-01 | ArduPilot SITL pinned to PR #30080-class build. |
deferred-hil |
AC-4.1 binding; AC-4.2 binding; AC-NEW-1 cold corner; AC-NEW-3 8-h soak; AC-NEW-5 thermal envelope; RESTRICT-HW-01..03 | Real Jetson + thermal chamber. |
deferred-field |
RESTRICT-CAM-01 deployment-camera binding (first internal fixed-wing flight) | Field-test plan. |
Uncovered Items Analysis
| Item | Reason Not Covered | Risk | Mitigation |
|---|---|---|---|
| (none) | — | — | — |
All 38 ACs and 31 restrictions are covered by ≥1 test, per Phase 1 D4. No uncovered items. Coverage is 100 % at the spec level; data availability — not coverage — is the gating concern, surfaced via the data_status column.
Pipeline-Correctness vs Deployment-Binding Boundary
The 60-image slice (present data_status) is pipeline-correctness only for the accuracy ACs. Deployment-binding numbers come from the deferred-corpus and deferred-hil tiers. This is per Phase 1 decision D2 and is documented in test-data.md. The matrix's "Covered" column is honest about which tier supplies which evidence:
| AC | Pipeline-correctness (T1, present) |
Deployment-binding |
|---|---|---|
| AC-1.1 | FT-P-01 (functional check) | FT-P-T2 (T2 deferred-corpus AerialVL S03) |
| AC-1.2 | FT-P-02 | FT-P-T2 |
| AC-1.3 | FT-P-04 (functional, with pending data) |
FT-P-04 binding split (T2) |
| AC-2.1 | FT-P-10 | FT-P-10 binding (T2) |
| AC-2.2 | FT-P-11 | FT-P-11 binding (T2) |
| AC-4.1 | NFT-PERF-01 functional smoke | NFT-PERF-01 binding (T4) |
| AC-4.2 | NFT-RES-LIM-01 functional | NFT-RES-LIM-01 binding (T4) |
| AC-NEW-1 | FT-P-16 (T1 N=10) | FT-P-T4 cold (T4 N=50) + NFT-RES-LIM-04 |
| AC-NEW-3 | NFT-RES-LIM-05 functional | NFT-RES-14 + NFT-RES-LIM-05 binding (T4 8-h) |
| AC-NEW-4 | (none — Monte Carlo only) | FT-P-35 (T2 binding) |
| AC-NEW-5 | (none — chamber only) | NFT-RES-LIM-02..04 (T4 chamber) |
| AC-NEW-7 | FT-P-34 + FT-N-17 functional | FT-P-35 + NFT-SEC-13 binding (T2) |
| AC-NEW-8 | (none — fixed-wing only) | FT-P-04 binding (T2) |
| AC-NEW-9 | (none — covariance evaluation requires ground-truth corpus) | FT-P-36 + FT-P-37 (T2) |