mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 18:11:13 +00:00
c19c76481c
Enhanced the SKILL.md file to enforce conciseness rules for the state file, specifying acceptable content and file size limits. Updated the autodev state to reflect the transition to the planning phase, including changes to the current step and sub-step details. Revised acceptance criteria to clarify validation requirements and external dependencies, ensuring alignment with the latest research findings. Added a new overlay for Mode B revisions to track changes and decisions made during the assessment process.
596 lines
26 KiB
Markdown
596 lines
26 KiB
Markdown
# Blackbox Tests
|
||
|
||
All tests run from the `e2e-runner` container against the SUT through public boundaries only (frame source, FC inbound stream, tile cache mount, FC outbound observed via SITL, GCS observed via mavproxy-listener, FDR via post-run filesystem read). Two FC adapters parameterize every test that touches the FC contract: `ardupilot` and `inav`. Two `VioStrategy` modes parameterize Tier-1 product correctness tests: `okvis2` (production-default) and `klt_ransac` (mandatory simple-baseline). `vins_mono` is parameterized only when the research build is under test.
|
||
|
||
## Positive Scenarios
|
||
|
||
### FT-P-01: Still-image satellite-anchor frame-center accuracy
|
||
|
||
**Summary**: Validates the canonical satellite-anchor frame-center geolocation pipeline against the 60-image GT set.
|
||
**Traces to**: AC-1.1, AC-1.2
|
||
**Category**: Position Accuracy
|
||
|
||
**Preconditions**:
|
||
- `tile-cache-fixture` mounted at `/var/azaion/tile-cache`.
|
||
- SUT cold-started with no prior state; configured for the FC adapter under test.
|
||
|
||
**Input data**: `still-image-set-60` (per `test-data.md`).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | For each image `AD0000NN.jpg` in order, write the frame to the SUT's frame-source path and wait up to 5 s for the corresponding outbound `GPS_INPUT` (AP) / `MSP2_SENSOR_GPS` (iNav) message at the SITL listener | One outbound message per input image; payload includes WGS84 lat/lon |
|
||
| 2 | Compute Vincenty geodesic distance between estimated lat/lon and `coordinates.csv` GT row for that image | Per-image error ≤ 50 m for ≥80% of images, ≤ 20 m for ≥50% |
|
||
| 3 | Capture per-image error to `e2e-results/run-${RUN_ID}/ft-p-01.csv` | CSV produced with one row per image |
|
||
|
||
**Expected outcome**: aggregate `pass_count(error≤50m) ≥ 48` AND `pass_count(error≤20m) ≥ 30` (matching the rule in `expected_results/results_report.md`).
|
||
**Max execution time**: 5 min (60 images × ~5 s including SITL round trip).
|
||
|
||
---
|
||
|
||
### FT-P-02: Derkachi VIO drift between satellite anchors
|
||
|
||
**Summary**: Validates cumulative drift between consecutive satellite-anchored fixes during the Derkachi flight replay.
|
||
**Traces to**: AC-1.3
|
||
**Category**: Position Accuracy
|
||
|
||
**Preconditions**:
|
||
- `tile-cache-fixture` mounted (covers Derkachi route).
|
||
- SUT cold-started; FC adapter under test connected via SITL; `data_imu.csv` replayed at 10 Hz into FC IMU stream.
|
||
|
||
**Input data**: `derkachi-fixture` video at 30 fps + IMU CSV at 10 Hz.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start synchronized video + IMU replay (3 video frames per IMU row) | SUT begins emitting estimates at the SUT's runtime cadence |
|
||
| 2 | At each frame whose outbound estimate carries `source_label = satellite_anchored`, record the propagated centre estimate of the prior visual-only segment AND the new anchor centre | Two values per anchor pair captured |
|
||
| 3 | Compute per-anchor-pair drift = ‖propagated_centre − next_anchor_centre‖. Bin by `last_satellite_anchor_age_ms`. | Bins populated; CSV emitted |
|
||
|
||
**Expected outcome**: Across all anchor pairs, at least 95% satisfy `drift < 100 m` (visual-only) AND `drift < 50 m` (when CombinedImuFactor IMU fusion is active in C5). Drift distribution monotonically grows with anchor age, with no anomalous spike.
|
||
**Max execution time**: 10 min (8 min replay + parsing).
|
||
|
||
---
|
||
|
||
### FT-P-03: Estimate output schema and source-label semantics
|
||
|
||
**Summary**: Validates the SUT's outbound estimate carries every required field with correct types and the source label is one of the three allowed values.
|
||
**Traces to**: AC-1.4, AC-4.3
|
||
**Category**: Position Accuracy / FC Contract
|
||
|
||
**Preconditions**:
|
||
- One image from `still-image-set-60` already loaded into the cache fixture.
|
||
- SUT cold-started.
|
||
|
||
**Input data**: any single image (default `AD000001.jpg`).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Push the image to the frame source | SUT emits one outbound `GPS_INPUT` (AP) / `MSP2_SENSOR_GPS` (iNav) AND one out-of-band channel message (MAVLink `STATUSTEXT` or `NAMED_VALUE_FLOAT` per AC-4.3) carrying the source label |
|
||
| 2 | Read the SITL-side fields | Schema match: `lat`, `lon`, `cov_semi_major_m`, `last_satellite_anchor_age_ms` present and well-typed |
|
||
| 3 | Read the out-of-band label channel | Label ∈ `{satellite_anchored, visual_propagated, dead_reckoned}` |
|
||
|
||
**Expected outcome**: Schema check passes AND label is in the allowed set.
|
||
**Max execution time**: 30 s.
|
||
|
||
---
|
||
|
||
### FT-P-04: Derkachi frame-to-frame registration success rate
|
||
|
||
**Summary**: Validates frame-to-frame registration succeeds for ≥95% of "normal" segments of the Derkachi flight.
|
||
**Traces to**: AC-2.1a
|
||
**Category**: Image Processing
|
||
|
||
**Preconditions**:
|
||
- SUT cold-started; FC adapter and VioStrategy both parameterized.
|
||
|
||
**Input data**: `derkachi-fixture` (full duration). "Normal" segments derived per AC-2.1a: nadir ±10° bank/pitch (estimated from `SCALED_IMU2`-derived attitude), ≥40% inferred prior-frame overlap (heuristic from frame-to-frame translation magnitude).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay the Derkachi fixture | SUT emits per-frame registration-success metric (exposed via `NAMED_VALUE_FLOAT` or in FDR per AC-NEW-3) |
|
||
| 2 | After replay, compute success-ratio over normal segments only | Success ratio ≥ 0.95 |
|
||
|
||
**Expected outcome**: ≥95% on normal segments. Sharp-turn segments (excluded from this denominator) are exercised separately by FT-N-02.
|
||
**Max execution time**: 12 min.
|
||
|
||
---
|
||
|
||
### FT-P-05: Satellite-anchor cross-domain registration
|
||
|
||
**Summary**: Validates the satellite-anchor (UAV→satellite cross-domain) matcher succeeds with the cross-domain MRE budget.
|
||
**Traces to**: AC-2.1b, AC-2.2
|
||
**Category**: Image Processing
|
||
|
||
**Preconditions**:
|
||
- `tile-cache-fixture` includes the still-image footprints.
|
||
- SUT cold-started.
|
||
|
||
**Input data**: `still-image-set-60` plus `still-image-sat-refs-2` (for the 2 images with paired `_gmaps.png`).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | For each still image, push to frame source | One satellite-anchor result per image |
|
||
| 2 | Read per-frame MRE (via FDR or `NAMED_VALUE_FLOAT`) | MRE recorded |
|
||
| 3 | Aggregate per-image accuracy AND MRE distribution | All images: MRE < 2.5 px; ≥80% within 50 m of GT; ≥50% within 20 m of GT |
|
||
|
||
**Expected outcome**: AC-1.1, AC-1.2, AC-2.1b, AC-2.2 all satisfied.
|
||
**Max execution time**: 5 min.
|
||
|
||
---
|
||
|
||
### FT-P-06: Mean Reprojection Error budgets (frame-to-frame + cross-domain)
|
||
|
||
**Summary**: Validates the two MRE budgets are honored.
|
||
**Traces to**: AC-2.2
|
||
|
||
**Preconditions**: Same as FT-P-04 + FT-P-05.
|
||
|
||
**Input data**: `derkachi-fixture` (frame-to-frame MRE) + `still-image-set-60` (cross-domain MRE).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Run FT-P-04 and FT-P-05 in sequence; collect per-frame MRE from both runs | MRE values captured |
|
||
| 2 | Aggregate by domain (frame-to-frame vs satellite-anchored) | Distribution per domain |
|
||
|
||
**Expected outcome**: Frame-to-frame MRE < 1.0 px (95th percentile); cross-domain MRE < 2.5 px (95th percentile).
|
||
**Max execution time**: piggybacks on FT-P-04 / FT-P-05.
|
||
|
||
---
|
||
|
||
### FT-P-07: Sharp-turn recovery via satellite reference
|
||
|
||
**Summary**: Validates that frames during sharp turns may fail frame-to-frame but recover via satellite-reference re-localization.
|
||
**Traces to**: AC-3.2
|
||
|
||
**Preconditions**:
|
||
- Sharp-turn segment of the Derkachi flight identified by gyro_z spikes in `SCALED_IMU2`. (If Derkachi has no sharp turn meeting AC-3.2 thresholds, fall back to a synthetic gyro overlay; flag in FDR.)
|
||
|
||
**Input data**: `derkachi-fixture` filtered to sharp-turn segment(s).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay sharp-turn segment | SUT emits `source_label = visual_propagated` or `dead_reckoned` during turn |
|
||
| 2 | After turn, observe next satellite-anchor attempt | Recovery: `source_label = satellite_anchored` returns within 3 frames of turn end; drift ≤ 200 m, heading change handled |
|
||
|
||
**Expected outcome**: Recovery within 3 frames; <200 m drift; <70° heading change handled.
|
||
**Max execution time**: 5 min (per turn segment, multiple per replay).
|
||
|
||
---
|
||
|
||
### FT-P-08: Multi-segment satellite-reference re-localization
|
||
|
||
**Summary**: Validates ≥3 disconnected segments per flight handled via satellite-reference re-localization.
|
||
**Traces to**: AC-3.3
|
||
|
||
**Preconditions**:
|
||
- `multi-segment-derkachi` synthetic fixture generated with 3+ blackout windows.
|
||
|
||
**Input data**: `multi-segment-derkachi`.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay with injected blackout windows | SUT emits `dead_reckoned` during each blackout |
|
||
| 2 | At end of each blackout, observe re-localization | `source_label` returns to `satellite_anchored` within 3 frames; trajectory continuity preserved (no >100 m jump) |
|
||
|
||
**Expected outcome**: All 3+ segments re-localized successfully; no trajectory jump exceeds 100 m.
|
||
**Max execution time**: 12 min.
|
||
|
||
---
|
||
|
||
### FT-P-09-AP: ArduPilot Plane GPS_INPUT contract conformance + signing
|
||
|
||
**Summary**: Validates `GPS_INPUT` reaches AP SITL, AP EKF accepts it as primary GPS, and MAVLink 2.0 message signing handshake completes per D-C8-9.
|
||
**Traces to**: AC-4.3 (AP), D-C8-9, AC-NEW-2 (precondition)
|
||
**Category**: FC Contract / Security
|
||
|
||
**Preconditions**:
|
||
- `ardupilot-plane-sitl` running with `GPS_TYPE=14`.
|
||
- `mavlink-passkey` loaded as Docker secret into SUT.
|
||
|
||
**Input data**: `derkachi-fixture` (any 60 s segment).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start SUT with FC adapter `ardupilot` | Signing handshake completes within 5 s; signed channel established |
|
||
| 2 | Replay 60 s of Derkachi | SUT emits signed `GPS_INPUT` at the configured rate |
|
||
| 3 | Read AP `EK3_SRC1_POSXY` parameter via MAVPROXY | Value reads `3` (GPS source) |
|
||
| 4 | Read AP-side GPS health via `GPS_RAW_INT` | Fix type ≥ 3 (3D fix), HDOP within nominal |
|
||
|
||
**Expected outcome**: Signing handshake succeeds; AP EKF on GPS source-set; GPS_RAW_INT shows healthy fix.
|
||
**Max execution time**: 90 s.
|
||
|
||
---
|
||
|
||
### FT-P-09-iNav: iNav MSP2_SENSOR_GPS contract conformance
|
||
|
||
**Summary**: Validates `MSP2_SENSOR_GPS` reaches iNav SITL and iNav GPS provider accepts it as the sole source.
|
||
**Traces to**: AC-4.3 (iNav)
|
||
**Category**: FC Contract
|
||
|
||
**Preconditions**:
|
||
- `inav-sitl` running with GPS provider configured to MSP per `docs/SITL/SITL.md`.
|
||
|
||
**Input data**: `derkachi-fixture` (any 60 s segment).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start SUT with FC adapter `inav` | TCP connection to `inav-sitl:5760` established |
|
||
| 2 | Replay 60 s of Derkachi | SUT emits `MSP2_SENSOR_GPS` (ID 0x1F03) frames at 5 Hz |
|
||
| 3 | Read iNav GPS state via MSP query | `gpsSol.fixType` ≥ 3, `gpsSol.numSat` reflects emitted value, provider=MSP |
|
||
|
||
**Expected outcome**: iNav GPS state reflects emitted frames; no fallback to internal GPS.
|
||
**Max execution time**: 90 s.
|
||
|
||
---
|
||
|
||
### FT-P-10: GTSAM smoothing-loop look-back accuracy (IT-11)
|
||
|
||
**Summary**: Validates the smoothing-loop's past-keyframe pose estimates improve over raw single-shot estimates (Mode B Fact #107). NOT validated as FC-side retroactive correction.
|
||
**Traces to**: AC-4.5 (revised scope per Mode B), Mode B Fact #107
|
||
**Category**: Position Accuracy / Internal smoothing
|
||
|
||
**Preconditions**:
|
||
- SUT cold-started; FDR enabled.
|
||
|
||
**Input data**: `derkachi-fixture` full replay.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay Derkachi end-to-end | FDR contains per-keyframe (a) raw single-shot pose at first emission, (b) smoothed pose at iSAM2 convergence |
|
||
| 2 | After replay, parse FDR; for each past keyframe compute distance(raw, GT) and distance(smoothed, GT) | Per-keyframe pair extracted |
|
||
| 3 | Aggregate across keyframes | smoothed_error < raw_error for ≥80% of keyframes; mean improvement ≥ 5 m |
|
||
|
||
**Expected outcome**: Internal smoothing improves past-keyframe accuracy; FC-side retroactive correction NOT exercised (out of scope per Mode B revision A6).
|
||
**Max execution time**: 12 min.
|
||
|
||
---
|
||
|
||
### FT-P-11: Cold-start initialization from FC EKF
|
||
|
||
**Summary**: Validates SUT initialization from FC EKF's last valid GPS + IMU-extrapolated position at GPS denial.
|
||
**Traces to**: AC-5.1
|
||
**Category**: Startup
|
||
|
||
**Preconditions**:
|
||
- `cold-boot-fixture` provides a frozen FC pose snapshot.
|
||
- SUT not running.
|
||
|
||
**Input data**: `cold-boot-fixture`.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start `ardupilot-plane-sitl` (or `inav-sitl`) with the frozen-pose snapshot loaded | SITL EKF reflects the snapshot pose |
|
||
| 2 | Start SUT | SUT queries FC EKF; reads pose; initializes |
|
||
| 3 | Push first nav-camera frame | First outbound estimate's lat/lon within ±50 m of the FC EKF snapshot pose |
|
||
|
||
**Expected outcome**: First emitted estimate uses FC EKF's pose as prior, within ±50 m tolerance.
|
||
**Max execution time**: 60 s.
|
||
|
||
---
|
||
|
||
### FT-P-12: GCS downsample at 1-2 Hz
|
||
|
||
**Summary**: Validates position estimates + confidence stream to the GCS (via `mavproxy-listener`) at 1-2 Hz.
|
||
**Traces to**: AC-6.1
|
||
**Category**: GCS / Telemetry
|
||
|
||
**Preconditions**:
|
||
- `mavproxy-listener` running and capturing to `.tlog`.
|
||
|
||
**Input data**: `derkachi-fixture` 60 s segment.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start replay | SUT emits to FC at runtime cadence (~3 Hz) AND to GCS at 1-2 Hz |
|
||
| 2 | After replay, parse `.tlog` for SUT-emitted GCS messages over the 60 s window | GCS rate within [1, 2] Hz inclusive |
|
||
|
||
**Expected outcome**: GCS-side rate observed in [1, 2] Hz over the window.
|
||
**Max execution time**: 90 s.
|
||
|
||
---
|
||
|
||
### FT-P-13: GCS command path (operator re-loc hint)
|
||
|
||
**Summary**: Validates that GCS-originated commands (via standard MAVLink) can carry operator re-loc hints to the SUT.
|
||
**Traces to**: AC-6.2
|
||
**Category**: GCS / Telemetry
|
||
|
||
**Preconditions**:
|
||
- `mavproxy-listener` configured to send commands.
|
||
- SUT in `dead_reckoned` state (e.g. mid-blackout from FT-N-03 setup).
|
||
|
||
**Input data**: synthesized `STATUSTEXT` carrying re-loc hint from MAVPROXY.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | While SUT is in `dead_reckoned`, send re-loc-hint STATUSTEXT from MAVPROXY | SUT acknowledges the hint via FDR log entry |
|
||
| 2 | Push next nav-camera frame after hint | Next satellite-anchor attempt uses hint as a search prior |
|
||
|
||
**Expected outcome**: Hint received; next anchor attempt biases search; no rejection.
|
||
**Max execution time**: 60 s.
|
||
|
||
---
|
||
|
||
### FT-P-14: WGS84 output coordinate system
|
||
|
||
**Summary**: Validates output coordinates are in WGS84 (latitude/longitude in degrees as per ArduPilot/iNav GPS convention scaled to 1e-7).
|
||
**Traces to**: AC-6.3
|
||
|
||
**Preconditions**: any FT-P-01 / FT-P-02 run.
|
||
|
||
**Input data**: any.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Capture one outbound `GPS_INPUT` / `MSP2_SENSOR_GPS` from SITL | Lat/lon present; values in valid WGS84 range; scaled per protocol convention |
|
||
|
||
**Expected outcome**: Coordinates parse as WGS84 within Earth bounds.
|
||
**Max execution time**: 30 s.
|
||
|
||
---
|
||
|
||
### FT-P-15: Tile cache schema and resolution floor
|
||
|
||
**Summary**: Validates the tile cache manifest carries every required field and tiles meet the ≥0.5 m/px floor.
|
||
**Traces to**: AC-8.1, RESTRICT-SAT-2 (manifest schema)
|
||
|
||
**Preconditions**: `tile-cache-fixture` mounted.
|
||
|
||
**Input data**: tile cache.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | The SUT exposes a one-time cache-load self-check at startup; observe via FDR | Each tile manifest entry has CRS, tile matrix, dimension, lat-adjusted m/px, capture date, source, compression |
|
||
| 2 | Inspect m/px values | All ≥ 0.5 m/px; reject below floor |
|
||
|
||
**Expected outcome**: All loaded tiles pass schema check and resolution floor.
|
||
**Max execution time**: 30 s.
|
||
|
||
---
|
||
|
||
### FT-P-16: Pre-loaded cache (offline-only interface)
|
||
|
||
**Summary**: Validates the SUT loads tiles from the local cache only, with no in-flight Service calls.
|
||
**Traces to**: AC-8.3, RESTRICT-SAT-1
|
||
|
||
**Preconditions**: `tile-cache-fixture` mounted; `e2e-net` `internal: true` enforced (no internet egress).
|
||
|
||
**Input data**: `derkachi-fixture` 60 s segment.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start replay | SUT serves tiles from `/var/azaion/tile-cache` only |
|
||
| 2 | Observe network egress counter on `gps-denied-onboard` container | All egress to non-`e2e-net` destinations is 0 (paired with NFT-SEC-02) |
|
||
|
||
**Expected outcome**: 0 external egress; replay completes against local cache.
|
||
**Max execution time**: 90 s.
|
||
|
||
---
|
||
|
||
### FT-P-17: Mid-flight tile generation
|
||
|
||
**Summary**: Validates the SUT continuously orthorectifies nav-camera frames into basemap-projected tiles, deduplicates them, and stores them locally for landing-time upload.
|
||
**Traces to**: AC-8.4
|
||
|
||
**Preconditions**: empty `mid-flight-tile-output` directory in the FDR volume; mock-suite-sat-service running.
|
||
|
||
**Input data**: `derkachi-fixture` 5 min segment.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start replay | SUT generates and writes tiles to FDR's `mid-flight-tile-output/` |
|
||
| 2 | After replay, read tiles | ≥1 tile per ~3 s of high-quality nav frames; each tile carries quality metadata sufficient for the Service voting layer (per Mode B Fact #105) |
|
||
| 3 | Simulate landing event; SUT uploads to `mock-suite-sat-service` | Mock service receives all tiles with HTTP 202 |
|
||
|
||
**Expected outcome**: Tiles produced + deduplicated + uploaded with quality metadata.
|
||
**Max execution time**: 8 min.
|
||
|
||
---
|
||
|
||
### FT-P-18: No raw nav/AI-cam frame retention (storage policy)
|
||
|
||
**Summary**: Validates that no raw nav-camera or AI-camera frames are retained except the ≤0.1 Hz failed-tile-gen thumbnail log.
|
||
**Traces to**: AC-8.5
|
||
|
||
**Preconditions**: `derkachi-fixture` replay just completed.
|
||
|
||
**Input data**: post-replay state of FDR + tile cache.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Walk the FDR + tile cache for any file matching nav-camera raw-frame pattern (JPEG/RAW with original dimensions) | Only the failed-tile-gen thumbnail log files present (≤0.1 Hz cadence) |
|
||
| 2 | Verify thumbnail log is bounded by AC-NEW-3 FDR budget | Total thumbnail log < 1 GB over 8 h (NFT-LIM-02 cross-check) |
|
||
|
||
**Expected outcome**: 0 unauthorized raw frames retained.
|
||
**Max execution time**: 30 s (filesystem walk).
|
||
|
||
---
|
||
|
||
### FT-P-19: Satellite relocalization scale-ratio + scene-change
|
||
|
||
**Summary**: Validates UAV-frame ground footprint at deployment altitude is retrievable from cache regardless of internal tiling. Scene-change subset is reduced-confidence (PARTIAL — see traceability matrix).
|
||
**Traces to**: AC-8.6 (scale-ratio FULL; scene-change PARTIAL)
|
||
|
||
**Preconditions**: `tile-cache-fixture` mounted with multi-zoom-level coverage.
|
||
|
||
**Input data**: `still-image-set-60` (scale-ratio); the 2 paired `_gmaps.png` images (scene-change subset).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | For each still image, query cache top-K=10 retrieval | Top-K result includes a tile whose centre is within 100 m of the image's true centre (scale-ratio satisfied) |
|
||
| 2 | For the 2 paired images, run cross-domain matcher against the `_gmaps.png` reference | Scale-ratio match succeeds; scene-change behavior recorded (PARTIAL — full coverage requires a labeled change-pair dataset, deferred under D-PROJ-3) |
|
||
|
||
**Expected outcome**: Scale-ratio passes for 60/60; scene-change recorded as PARTIAL.
|
||
**Max execution time**: 5 min.
|
||
|
||
---
|
||
|
||
## Negative Scenarios
|
||
|
||
### FT-N-01: 350 m outlier injection tolerance
|
||
|
||
**Summary**: Validates the system tolerates up to 350 m outliers between two consecutive frames with airframe tilt up to ±20°.
|
||
**Traces to**: AC-3.1, RESTRICT-CAM-1 (nadir camera, tilt limits)
|
||
|
||
**Preconditions**: SUT running on `derkachi-fixture`; `outlier-injection-derkachi` injector primed in `medium` density.
|
||
|
||
**Input data**: `outlier-injection-derkachi` (medium).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start replay with injector active (every 10th frame replaced by far-away tile crop) | SUT detects outlier; rejects from anchor; estimate continues from prior valid state |
|
||
| 2 | Compare per-frame outbound estimate vs GT for non-outlier frames | Error_after_outlier ≤ error_before_outlier + 50 m; covariance grows monotonically across the outlier event |
|
||
|
||
**Expected outcome**: Outliers rejected; estimate degrades at most by 50 m drift; covariance monotonic.
|
||
**Max execution time**: 12 min.
|
||
|
||
---
|
||
|
||
### FT-N-02: Sharp-turn frame-to-frame failure expected
|
||
|
||
**Summary**: Negative twin of FT-P-07 — validates that during a sharp turn, frame-to-frame may LEGITIMATELY fail, and the system labels accordingly.
|
||
**Traces to**: AC-3.2 (negative path)
|
||
|
||
**Preconditions**: Same as FT-P-07.
|
||
|
||
**Input data**: sharp-turn segment of Derkachi (or synthetic gyro overlay).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay sharp-turn segment | During turn frames: `source_label` ∈ `{visual_propagated, dead_reckoned}`; covariance grows |
|
||
| 2 | After turn, observe label transition | Label returns to `satellite_anchored` once next anchor succeeds |
|
||
|
||
**Expected outcome**: Sharp-turn frames correctly mark themselves as not-satellite-anchored; recovery exercised in FT-P-07.
|
||
**Max execution time**: 5 min.
|
||
|
||
---
|
||
|
||
### FT-N-03: Extended outage triggers operator re-loc request
|
||
|
||
**Summary**: Validates that on ≥3 consecutive frames AND ≥2 s without estimate, the SUT requests operator re-loc via telemetry and continues dead-reckoned propagation.
|
||
**Traces to**: AC-3.4
|
||
|
||
**Preconditions**: `derkachi-fixture` + 3-frame outage injector primed.
|
||
|
||
**Input data**: synthetic outage on Derkachi.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Trigger 3-consecutive-frame failure (corrupt frames) | SUT fails to produce estimates for 3+ frames |
|
||
| 2 | Wait ≥2 s | STATUSTEXT containing `OPERATOR_RELOC_REQUEST` emitted to `mavproxy-listener` |
|
||
| 3 | During outage, observe FC outbound | Estimates labeled `dead_reckoned` continue; FC uses last-known + IMU extrapolation |
|
||
|
||
**Expected outcome**: Re-loc request emitted; dead-reckoned estimates continue.
|
||
**Max execution time**: 60 s.
|
||
|
||
---
|
||
|
||
### FT-N-04: Visual blackout + spoofed GPS combined failsafe
|
||
|
||
**Summary**: Validates the AC-3.5 + AC-NEW-8 combined failsafe: switch label, reject spoof, propagate from last trusted state, monotonic covariance, STATUSTEXT.
|
||
**Traces to**: AC-3.5, AC-NEW-8
|
||
|
||
**Preconditions**: `blackout-spoof-derkachi` injector primed for 5 s, 15 s, 35 s windows; FC inbound stream patched to inject spoofed GPS.
|
||
|
||
**Input data**: `blackout-spoof-derkachi` (each window run as a sub-case).
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Begin blackout window AND inject spoofed GPS in same temporal window | Within ≤1 frame OR ≤400 ms: `source_label = dead_reckoned`; spoofed GPS rejected from estimator input; covariance grows monotonically |
|
||
| 2 | Observe `horiz_accuracy` field in outbound `GPS_INPUT` (AP) | `horiz_accuracy` ≥ 95% covariance semi-major axis (no under-reporting) |
|
||
| 3 | Observe GCS stream | `VISUAL_BLACKOUT_IMU_ONLY` STATUSTEXT at 1-2 Hz throughout blackout |
|
||
| 4 | For 35 s window only | Per AC-NEW-8: when 95% covariance crosses 100 m → fix-quality degraded; when crosses 500 m OR blackout exceeds 30 s → `horiz_accuracy=999.0` AND `VISUAL_BLACKOUT_FAILSAFE` STATUSTEXT |
|
||
| 5 | End blackout; restore FC GPS-health | Recovery only after FC GPS-health stable + non-spoofed for ≥10 s AND a visual/satellite consistency check succeeds |
|
||
|
||
**Expected outcome**: All four steps' assertions pass for each window.
|
||
**Max execution time**: 5 min (3 windows × ~90 s each).
|
||
|
||
---
|
||
|
||
### FT-N-05: Stale-tile rejection on freshness violation
|
||
|
||
**Summary**: Validates that tiles violating AC-8.2 freshness window are rejected (or downgraded so they cannot produce a `satellite_anchored` label).
|
||
**Traces to**: AC-8.2, AC-NEW-6
|
||
|
||
**Preconditions**: `synth-age-tile-set` (`synth-age-7mo` for active-conflict, `synth-age-13mo` for rear) mounted instead of fresh fixture.
|
||
|
||
**Input data**: `still-image-set-60` against the aged cache.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Replay against `synth-age-7mo` (configure SUT for active-conflict sector) | SUT either rejects load OR loads but never emits `satellite_anchored` from these tiles |
|
||
| 2 | Replay against `synth-age-13mo` (configure SUT for rear sector) | Same: reject or non-`satellite_anchored` only |
|
||
|
||
**Expected outcome**: 0 frames emit `satellite_anchored` from aged tiles.
|
||
**Max execution time**: 5 min.
|
||
|
||
---
|
||
|
||
### FT-N-06: Mid-flight tile freshness (current-timestamped)
|
||
|
||
**Summary**: Validates that mid-flight-generated tiles are timestamped as current and treated as fresh per AC-NEW-6.
|
||
**Traces to**: AC-NEW-6 (positive sub-case)
|
||
|
||
**Preconditions**: empty `mid-flight-tile-output`.
|
||
|
||
**Input data**: `derkachi-fixture` 5 min segment.
|
||
|
||
**Steps**:
|
||
|
||
| Step | Consumer Action | Expected System Response |
|
||
|------|----------------|------------------------|
|
||
| 1 | Start replay | SUT generates mid-flight tiles |
|
||
| 2 | Inspect each generated tile's manifest entry | `capture_date` is within ±60 s of generation wall-clock; treated as fresh by the freshness gate |
|
||
|
||
**Expected outcome**: All mid-flight tiles current-timestamped and fresh.
|
||
**Max execution time**: 6 min.
|