mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 19:31:12 +00:00
Oleksandr Bezdieniezhnykh
cab7b5d020
- Modified the Docker Compose configuration to include an input root for replay tests and added an environment variable for enabling SITL. - Enhanced documentation for various testing processes, including the addition of a Runtime Completeness Decomposition Gate and clarifications on internal module testing requirements. - Updated the implementation completeness report to reflect the current state and added new test cases for performance and resilience scenarios. Co-authored-by: Cursor <cursoragent@cursor.com>
9.3 KiB
9.3 KiB
Traceability Matrix
Acceptance Criteria Coverage
| AC ID | Acceptance Criterion Summary | Test IDs | Coverage |
|---|---|---|---|
| AC-1.1 | >=80% frame centers within 50 m | FT-P-01, FT-P-04 | Covered |
| AC-1.2 | >=50% frame centers within 20 m | FT-P-01, FT-P-04 | Covered |
| AC-1.3 | Drift and anchor age reporting | FT-P-02, FT-P-03, NFT-RES-02 | Covered |
| AC-1.4 | Quantitative confidence and source label | FT-P-02, FT-N-01, FT-P-03 | Covered |
| AC-2.1a | VO registration >95% on normal segments | FT-P-03, NFT-PERF-02 | Covered |
| AC-2.1b | Satellite-anchor registration measured separately | FT-P-04 | Covered |
| AC-2.2 | MRE <1 px VO and <2.5 px satellite anchor | FT-P-03, FT-P-04 | Covered |
| AC-3.1 | Handles up to 350 m outliers | FT-N-01, NFT-RES-02 | Covered |
| AC-3.2 | Sharp turn relocalization | FT-P-04, NFT-RES-02, NFT-PERF-03 | Covered |
| AC-3.3 | >=3 disconnected segments via retrieval/relocalization | NFT-RES-02, FT-P-04 | Covered |
| AC-3.4 | Relocalization request after loss threshold | NFT-RES-02 | Covered |
| AC-3.5 | Total visual blackout/occlusion + GPS spoofing degraded mode | FT-N-02, NFT-RES-01, NFT-SEC-03 | Covered |
| AC-4.1 | End-to-end latency <400 ms p95 | NFT-PERF-01, NFT-PERF-02, NFT-PERF-03 | Covered |
| AC-4.2 | Memory below 8 GB shared | NFT-RES-LIM-01, NFT-RES-LIM-05 | Covered |
| AC-4.3 | MAVLink GPS_INPUT output for ArduPilot |
FT-N-02, NFT-SEC-03 | Covered |
| AC-4.4 | Frame-by-frame streaming | FT-P-02, NFT-PERF-01 | Covered |
| AC-4.5 | Previous estimate corrections | FT-P-02 | Covered |
| AC-5.1 | Initialize from last trusted FC state | NFT-RES-03, NFT-PERF-04 | Covered |
| AC-5.2 | >3 s no-estimate fallback behavior | FT-N-02, NFT-RES-01 | Covered |
| AC-5.3 | Re-initialize after companion reboot | NFT-RES-03 | Covered |
| AC-6.1 | QGC status at 1-2 Hz | FT-N-02, NFT-SEC-03 | Covered |
| AC-6.2 | Ground station command ingress | FT-P-04, NFT-RES-02 | Covered |
| AC-6.3 | WGS84 output | FT-P-01, FT-P-02 | Covered |
| AC-7.1 | Object localization accuracy consistent with frame center in level flight | FT-P-01 | Covered at system boundary; detailed AI-camera tests deferred to component specs |
| AC-7.2 | Object coordinates from UAV position, gimbal angle, zoom, altitude | FT-P-01 | Covered at system boundary; detailed AI-camera tests deferred to component specs |
| AC-8.1 | Cache imagery 0.5 m/px minimum, 0.3 m/px ideal | FT-P-04, NFT-SEC-04 | Covered |
| AC-8.2 | Tile freshness thresholds | FT-N-03, NFT-RES-04, NFT-SEC-01 | Covered |
| AC-8.3 | Preloaded/preprocessed offline cache | NFT-SEC-04, NFT-RES-LIM-03 | Covered |
| AC-8.4 | Mid-flight tile generation and write-back | NFT-SEC-02, NFT-RES-LIM-04 | Covered |
| AC-8.5 | No raw frame retention | NFT-RES-LIM-04 | Covered |
| AC-8.6 | VPR retrieval chunks, multi-scale, dynamic K | FT-P-04, NFT-PERF-03 | Covered |
| AC-NEW-1 | Cold start first fix <30 s | NFT-PERF-04, NFT-RES-LIM-05, NFT-RES-03 | Covered |
| AC-NEW-2 | Spoofing promotion <3 s | FT-N-02, NFT-SEC-03 | Covered |
| AC-NEW-3 | FDR <=64 GB and payload retention | NFT-RES-LIM-04 | Covered |
| AC-NEW-4 | False-position safety budget | FT-N-01, NFT-SEC-01 | Covered |
| AC-NEW-5 | Thermal envelope and no throttle | NFT-RES-LIM-02 | Covered |
| AC-NEW-6 | Imagery freshness enforcement | FT-N-03, NFT-RES-04 | Covered |
| AC-NEW-7 | Cache-poisoning safety budget | FT-N-03, NFT-SEC-02 | Covered |
| AC-NEW-8 | Pre-VIO total occlusion gate, IMU-only blackout propagation, and spoofing degraded-mode budget | FT-N-02, NFT-RES-01 | Covered |
Restrictions Coverage
| Restriction ID | Restriction Summary | Test IDs | Coverage |
|---|---|---|---|
| R-UAV-01 | Fixed-wing UAV mission profile and 8-hour operations | NFT-RES-LIM-02, NFT-RES-LIM-04 | Covered |
| R-CAM-01 | Fixed downward navigation camera | FT-P-01, FT-P-03 | Covered |
| R-CAM-02 | ADTi 20MP 20L V1 and calibration assumptions | FT-P-01, NFT-PERF-01 | Covered |
| R-SAT-01 | Offline-only Satellite Service cache, no in-flight provider fetch | NFT-SEC-04 | Covered |
| R-SAT-02 | Cache resolution/freshness/metadata conventions | FT-N-03, NFT-RES-LIM-03 | Covered |
| R-HW-01 | Jetson Orin Nano Super 8 GB / 25 W | NFT-RES-LIM-01, NFT-RES-LIM-02 | Covered |
| R-SENSOR-01 | FC IMU available; original still-image sample lacks synchronized IMU; Derkachi fixture provides video/IMU/GPS trajectory but calibration is pending | FT-P-03, NFT-PERF-02 | Covered through Derkachi representative replay plus public/calibrated dataset plan |
| R-MAV-01 | MAVLink, ArduPilot only, GPS_INPUT via pymavlink | FT-N-02, NFT-SEC-03 | Covered |
| R-GCS-01 | QGroundControl supported GCS | FT-N-02, NFT-SEC-03 | Covered |
| R-SAFETY-01 | False-position, cold-start, spoofing, and failsafe constraints | FT-N-01, FT-N-02, NFT-PERF-04, NFT-RES-01 | Covered |
Cycle 1 Implementation-Learned Test Coverage
| Task AC ID | Task Acceptance Criterion Summary | Test IDs | Coverage |
|---|---|---|---|
| AZ-233 AC-1 | Docker/replay environment starts or reports clear blocked prerequisites | NFT-RES-INFRA, NFT-RES-LIM-INFRA | Covered |
| AZ-233 AC-2 | External dependency stubs are deterministic and record interactions | NFT-SEC-INFRA, NFT-RES-INFRA | Covered |
| AZ-233 AC-3 | Runner executes blackbox, performance, resilience, security, and resource-limit groups | FT-P-01, NFT-PERF-INFRA, NFT-RES-INFRA, NFT-SEC-INFRA, NFT-RES-LIM-INFRA | Covered |
| AZ-233 AC-4 | CSV and Markdown evidence reports are generated with required fields | FT-P-01, NFT-PERF-INFRA, NFT-RES-INFRA, NFT-SEC-INFRA, NFT-RES-LIM-INFRA | Covered |
| AZ-234 AC-1 | Still-image WGS84 error is reported against expected coordinates | FT-P-01 | Covered |
| AZ-234 AC-2 | Confidence output contract fields are validated | FT-P-02 | Covered |
| AZ-234 AC-3 | Replay latency and dropped-frame metrics are recorded | NFT-PERF-INFRA, NFT-PERF-01 | Covered |
| AZ-235 AC-1 | Derkachi fixture alignment is validated before replay | FT-P-03 | Covered |
| AZ-235 AC-2 | Synchronized replay emits frame-by-frame estimates or explicit degradation | FT-P-03 | Covered |
| AZ-235 AC-3 | VIO latency, completion, memory, and calibration status are reported | NFT-PERF-02 | Covered |
| AZ-236 AC-1 | Verified anchors include retrieval, matching, geometry, freshness, and provenance evidence | FT-P-04 | Covered |
| AZ-236 AC-2 | Unsafe cache or low-texture candidates are rejected | FT-N-01, FT-N-03, NFT-SEC-INFRA | Covered |
| AZ-236 AC-3 | Flight-mode missing-cache behavior does not fetch external satellite data | NFT-SEC-04, NFT-SEC-INFRA | Covered |
| AZ-236 AC-4 | Cache and trigger-path metrics are reported | NFT-PERF-03, NFT-RES-04, NFT-RES-LIM-03 | Covered |
| AZ-237 AC-1 | Blackout transitions to dead reckoning within threshold | FT-N-02, NFT-RES-01 | Covered |
| AZ-237 AC-2 | Degraded covariance and no-fix/failsafe thresholds are enforced | FT-N-02, NFT-RES-01 | Covered |
| AZ-237 AC-3 | Spoofed or unauthorized MAVLink inputs are rejected | NFT-SEC-03 | Covered |
| AZ-237 AC-4 | QGC and FDR degraded-mode evidence is visible | FT-N-02, NFT-SEC-03, NFT-RES-INFRA | Covered |
| AZ-238 AC-1 | Disconnected segments trigger relocalization or degraded status | NFT-RES-02 | Covered |
| AZ-238 AC-2 | Companion restart first-output and FDR evidence are recorded | NFT-RES-03 | Covered |
| AZ-238 AC-3 | Cold-start trials report first-fix timing or blocked prerequisite | NFT-PERF-04, NFT-RES-LIM-05 | Covered |
| AZ-238 AC-4 | Cold-start resource spikes are captured where measurable | NFT-RES-LIM-05 | Covered |
| AZ-239 AC-1 | Jetson memory budget is measured on target hardware | NFT-RES-LIM-01, NFT-RES-LIM-INFRA | Covered |
| AZ-239 AC-2 | Thermal/power endurance is validated or blocked with reason | NFT-RES-LIM-02, NFT-RES-LIM-INFRA | Covered |
| AZ-239 AC-3 | FDR rollover behavior is validated | NFT-RES-LIM-04 | Covered |
| AZ-239 AC-4 | Resource/endurance evidence artifacts are complete | NFT-RES-LIM-01, NFT-RES-LIM-02, NFT-RES-LIM-04, NFT-RES-LIM-INFRA | Covered |
Coverage Summary
| Category | Total Items | Covered | Not Covered | Coverage % |
|---|---|---|---|---|
| Acceptance Criteria | 39 | 39 | 0 | 100% |
| Restrictions | 10 | 10 | 0 | 100% |
| Total | 49 | 49 | 0 | 100% |
Uncovered Items Analysis
| Item | Reason Not Covered | Risk | Mitigation |
|---|---|---|---|
| None | All current AC and restriction groups have black-box coverage | N/A | Continue to Phase 3 validation gate |
Data Coverage Caveats
- Current project data fully supports still-image frame-center checks for 60 mapped images.
- Derkachi project data supports synchronized video/IMU/GPS trajectory replay for FT-P-03 and NFT-PERF-02.
- Derkachi project data is calibration-limited: raw camera intrinsics, lens distortion, and camera-to-body transform are still required before final absolute accuracy thresholds can be treated as production acceptance.
- Phase 3 must validate camera calibration inputs and public/calibrated dataset acquisition before FT-P-03, FT-P-04, and NFT-PERF-02 can be used for final signoff.
- Cycle 1 Docker replay smoke evidence currently passes blackbox, performance, resilience, and security infrastructure scenarios; Jetson resource evidence remains a target-hardware release gate and is reported as blocked on local runners.