gps-denied-onboard/_docs/02_document/tests/test-data.md

# Test Data Management

## Seed Data Sets

| Data Set | Description | Used by Tests | How Loaded | Cleanup |
|----------|-------------|---------------|-----------|---------|
| `still-image-set-60` | 60 nadir aerial images `AD000001-60.jpg` from `_docs/00_problem/input_data/` with WGS84 frame-center GT in `coordinates.csv` and per-image accuracy table in `expected_results/position_accuracy.csv`. Captured at 400 m AGL with ADTi 20MP 20L V1 (per `data_parameters.md`). Slow cadence (~1 per 2-3 s), so suitable for satellite-anchor frame-center tests, NOT frame-to-frame VIO. | FT-P-01, FT-P-03, FT-P-05, FT-P-06, FT-P-15, FT-P-19, NFT-RES-03 (Monte Carlo), NFT-PERF-04 | Bind-mounted from `_docs/00_problem/input_data/` to `/test-data` in `e2e-runner` (read-only) | None — read-only fixture |
| `still-image-sat-refs-2` | Two paired Google Maps reference images `AD000001_gmaps.png`, `AD000002_gmaps.png`. Insufficient for full satellite-anchor coverage of the 60-image set; supplements the tile-cache fixture for AC-2.1b cross-validation only. | FT-P-05 (subset), FT-P-19 | Same as above | Same |
| `derkachi-fixture` | Cropped nadir flight footage `flight_derkachi/flight_derkachi.mp4` (H.264, 880×720, 30 fps, ~490.07 s = 14,700 frames) plus synchronized FC telemetry `flight_derkachi/data_imu.csv` (4,900 rows @ 10 Hz, columns `timestamp(ms)`, `Time`, `SCALED_IMU2.*`, `GLOBAL_POSITION_INT.*`). Three video frames per telemetry row. The `GLOBAL_POSITION_INT` columns are the trajectory ground truth. | FT-P-02, FT-P-04, FT-P-07, FT-P-10, FT-N-01 (synth on top), FT-N-02, FT-N-03 (synth), FT-N-04 (synth), NFT-PERF-01, NFT-PERF-02, NFT-RES-01, NFT-RES-02, NFT-RES-03 (Monte Carlo), NFT-RES-04, NFT-LIM-02 (8 h synth load loop) | Same bind mount as above | Same |
| `tile-cache-fixture` | Pre-built FAISS HNSW index + tile filesystem covering: (a) the 60 still-image footprints at 0.3-0.5 m/px, (b) the Derkachi route bbox at the same resolution. Built once per CI run by `tests/fixtures/tile-cache-builder/` from the `_gmaps.png` references and from a curated public-data subset (when D-PROJ-3 is resolved — until then, stub-tile content for footprints not paired with `_gmaps.png`). Tile manifest schema per `restrictions.md` § Satellite Imagery. | FT-P-01, FT-P-05, FT-P-15, FT-P-16, FT-P-17, FT-P-19, FT-N-05, FT-N-06, NFT-LIM-03, NFT-PERF-01, NFT-PERF-04, NFT-SEC-01 (poisoning test), NFT-SEC-02 (egress) | Built into named Docker volume `tile-cache-fixture`; mounted read-only into SUT at `/var/azaion/tile-cache` | Volume removed at teardown |
| `synth-age-tile-set` | Two clones of the tile-cache-fixture with manifest `capture_date` field synthetically aged: `synth-age-7mo` (>6 mo, exceeds AC-8.2 active-conflict threshold) and `synth-age-13mo` (>12 mo, exceeds rear threshold). Tile pixels unchanged; only manifest dates differ. | FT-N-05, FT-N-06 | Built from `tile-cache-fixture` by date-mutating script in `tests/fixtures/age-injector/` | Volume removed at teardown |
| `outlier-injection-derkachi` | Synthetic adversarial overlay on `derkachi-fixture`: every Nth frame replaced by a random crop from a far-away tile (>350 m offset, per AC-3.1) to inject a visual outlier. Three injection densities: `light` (1 in 100), `medium` (1 in 10), `heavy` (1 in 3). Generated at runtime by `tests/fixtures/injectors/outlier.py`. | FT-N-01 | Generated at scenario start, written to `tmpfs` in `e2e-runner`, mounted into SUT as a derived frame source | Auto-cleared at teardown (tmpfs) |
| `blackout-spoof-derkachi` | Synthetic overlay on `derkachi-fixture`: pure-black frames inserted in 5 s / 15 s / 35 s windows AND simultaneous spoofed-GPS injection on the FC inbound stream. Spoof pattern: realistic-looking GPS jumps the trajectory 200-500 m in `north_east_random_direction`. Three windows produce three sub-scenarios per AC-NEW-8. Generated at runtime. | FT-N-04, NFT-RES-04 | Same | Same |
| `multi-segment-derkachi` | Synthetic overlay: 3+ blackout segments distributed across the Derkachi flight to exercise satellite-reference re-localization (AC-3.3) without spoofing. Generated at runtime. | FT-P-08 | Same | Same |
| `cold-boot-fixture` | The state needed to validate AC-NEW-1: a frozen FC pose (`GLOBAL_POSITION_INT` snapshot at flight-resume time) + the tile-cache-fixture + a blank FDR. Test cold-boots the SUT and measures TTFF. | NFT-PERF-03 (AC-NEW-1) | The frozen FC pose is a JSON fixture in `tests/fixtures/cold-boot/`; SUT is restarted (`docker compose restart gps-denied-onboard`) and TTFF is measured from container-ready event to first valid `GPS_INPUT` / `MSP2_SENSOR_GPS` arrival at SITL | Container restart only |
| `mavlink-passkey` | A test-only MAVLink 2.0 signing passkey (32-byte hex). Used for D-C8-9 ArduPilot-track signing channel. NEVER reused outside test environment; checked-in as `tests/fixtures/secrets/mavlink-test-passkey.txt` with explicit comment "TEST ONLY". | FT-P-09 (AP track), NFT-SEC-03 | Loaded via Docker secret into SUT environment | None — fixture file |
| `cve-jpeg-fixture` | Crafted JPEG that triggers CVE-2025-53644 (uninitialized stack pointer → heap buffer write) in OpenCV 4.10/4.11. The pinned ≥4.12.0 must process it without crash and either decode safely or reject. | NFT-SEC-04 | Local-data-only fixture file at `tests/fixtures/security/cve-2025-53644.jpg` (sourced from public PoC, license-checked) | None — fixture file |

## Data Isolation Strategy

Each `pytest` test case runs against a fresh `gps-denied-onboard` container (`docker compose restart` between tests, OR `--forked` pytest mode that brings a clean compose stack per case for hermetic-critical tests). The `tile-cache-fixture` and `input-data` mounts are read-only so cross-contamination between tests is impossible at the SUT-input layer. The `fdr-output` volume is reset between tests (`docker volume rm` + recreate) so each test sees a blank FDR.

For Tier-2 (Jetson hardware), the same isolation discipline applies but at the systemd-service level: `systemctl restart gps-denied-onboard.service` between tests, `/var/azaion/fdr` is wiped between tests.

Synthetic-injection fixtures (`outlier-injection-derkachi`, `blackout-spoof-derkachi`, `multi-segment-derkachi`, `synth-age-tile-set`) are generated into per-test tmpfs and never written back to a persistent volume.

## Input Data Mapping

| Input Data File | Source Location | Description | Covers Scenarios |
|-----------------|----------------|-------------|-----------------|
| `AD000001.jpg` ... `AD000060.jpg` | `_docs/00_problem/input_data/` | 60 nadir still images, ADTi 20MP @ 400 m AGL | FT-P-01, FT-P-03, FT-P-05, FT-P-06, FT-P-15, FT-P-19, NFT-PERF-04, NFT-RES-03 |
| `coordinates.csv` | `_docs/00_problem/input_data/` | 60-row WGS84 frame-center GT (image, lat, lon) | Same as above |
| `AD000001_gmaps.png`, `AD000002_gmaps.png` | `_docs/00_problem/input_data/` | Google Maps satellite reference for images 1-2 | FT-P-05, FT-P-19 |
| `data_parameters.md` | `_docs/00_problem/input_data/` | AGL height (400 m) + camera model | All — global metadata |
| `flight_derkachi/flight_derkachi.mp4` | `_docs/00_problem/input_data/flight_derkachi/` | H.264 nadir video, 880×720 @ 30 fps, ~490 s | FT-P-02, FT-P-04, FT-P-07, FT-P-10, FT-N-01..04, NFT-PERF-01..04, NFT-RES-01..04, NFT-LIM-02 |
| `flight_derkachi/data_imu.csv` | `_docs/00_problem/input_data/flight_derkachi/` | 4,900 rows @ 10 Hz of `SCALED_IMU2` + `GLOBAL_POSITION_INT` | Same as above |
| `flight_derkachi/README.md` | `_docs/00_problem/input_data/flight_derkachi/` | Fixture metadata | Documentation only |
| `expected_results/results_report.md` | `_docs/00_problem/input_data/expected_results/` | Pass/fail rules + still-image and Derkachi mappings | All FT-P / FT-N scenarios that load this fixture |
| `expected_results/position_accuracy.csv` | `_docs/00_problem/input_data/expected_results/` | Per-image accuracy threshold flags | FT-P-01, NFT-RES-03 |

## Expected Results Mapping

This table closes the gap between each test scenario and the quantifiable expected result it asserts on. Comparison methods follow `.cursor/skills/test-spec/templates/expected-results.md`. The `Expected Result Source` column points at the canonical source of truth for the assertion.

### Position accuracy

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| FT-P-01 | `still-image-set-60` + `tile-cache-fixture` | `pass_count(error≤50m) ≥ 48` (≥80% of 60) AND `pass_count(error≤20m) ≥ 30` (≥50% of 60) | `threshold_min` on aggregate counts; per-image error via `numeric_tolerance` against Vincenty geodesic distance to GT in `coordinates.csv` | ±50 m / ±20 m | `expected_results/results_report.md` § Pass/Fail Rules + `expected_results/position_accuracy.csv` |
| FT-P-02 | `derkachi-fixture` | At each anchor frame, `‖propagated_centre − next_anchor_centre‖ < 100 m` (visual-only) AND `< 50 m` (IMU-fused). Drift binned by `last_satellite_anchor_age_ms`. | `threshold_max` per anchor pair, then aggregate rule `≥95% of anchor pairs satisfy` | < 100 m / < 50 m | AC-1.3 + Derkachi `GLOBAL_POSITION_INT` GT |
| FT-P-03 | `still-image-set-60` (any 1 image) | Estimate output schema fields present: `lat:float`, `lon:float`, `cov_semi_major_m:float`, `source_label ∈ {satellite_anchored, visual_propagated, dead_reckoned}`, `last_satellite_anchor_age_ms:int` | `schema_match` (presence + type) AND `set_contains` (label) | N/A | AC-1.4 + AC-4.3 |
| FT-P-19 | `tile-cache-fixture` + `still-image-sat-refs-2` | Scale-ratio: any UAV-frame footprint at 400 m AGL retrievable from cache (FAISS top-K=10 includes a tile with center within 100 m of true position). Scene-change subset (PARTIAL — flag-marked, see traceability matrix). | `set_contains` (top-K result includes correct tile) | top-K hit | AC-8.6 |

### Image processing quality

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| FT-P-04 | `derkachi-fixture` | Frame-to-frame registration succeeds for `≥95%` of "normal" segments (defined per AC-2.1a: nadir ±10° bank/pitch from `data_imu.csv` `SCALED_IMU2` quaternion-derived attitude estimate, ≥40% inferred prior-frame overlap). Sharp-turn frames excluded from this denominator. | `threshold_min` on success ratio | ≥95% | AC-2.1a |
| FT-P-05 | `still-image-set-60` (with `_gmaps.png` subset for ground-truth match) | Satellite-anchor registration succeeds AND satisfies AC-1.1/1.2 accuracy AND MRE < 2.5 px | `threshold_max` MRE | < 2.5 px | AC-2.1b + AC-2.2 |
| FT-P-06 | `derkachi-fixture` (frame-to-frame) AND `still-image-set-60` (sat-anchor) | Mean Reprojection Error: `< 1.0 px` frame-to-frame, `< 2.5 px` satellite-anchored cross-domain | `threshold_max` per shape | < 1.0 / < 2.5 px | AC-2.2 |

### Resilience

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| FT-N-01 | `outlier-injection-derkachi` | Up to 350 m offset in a single frame is rejected as outlier; estimate continues from prior valid state with grown covariance; airframe tilt up to ±20° handled | Per-injected-outlier: `error_after_outlier ≤ error_before_outlier + 50 m` AND `covariance_growth_monotonic` | ±50 m drift budget | AC-3.1 |
| FT-N-02 | `derkachi-fixture` (sharp-turn segment, identified via `SCALED_IMU2` gyro_z spikes) | Sharp-turn frames may fail frame-to-frame registration; recovery via satellite-reference re-localization within next 3 frames | Boolean recovery within 3 frames | N/A | AC-3.2 |
| FT-P-08 | `multi-segment-derkachi` | ≥3 disconnected segments handled; satellite-reference re-localization succeeds at each gap; trajectory remains continuous (no >100 m jump) | `threshold_max` discontinuity | < 100 m | AC-3.3 |
| FT-N-03 | `derkachi-fixture` + synthetic 3-frame outage injector | After ≥3 consecutive frames AND ≥2 s without estimate: STATUSTEXT containing `OPERATOR_RELOC_REQUEST` emitted to GCS via `mavproxy-listener`; estimates labeled `dead_reckoned` continue | `regex` on STATUSTEXT + `set_contains` on labels | regex | AC-3.4 |
| FT-N-04 | `blackout-spoof-derkachi` (5 s / 15 s / 35 s windows) | Within ≤1 frame OR ≤400 ms: label switches to `dead_reckoned`; spoofed GPS rejected; covariance grows monotonically; `horiz_accuracy` not under-reported; `VISUAL_BLACKOUT_IMU_ONLY` STATUSTEXT at 1-2 Hz | `threshold_max` switch latency + `regex` STATUSTEXT + monotonic check | ≤400 ms | AC-3.5 |

### FC contract & startup

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| FT-P-09-AP | `derkachi-fixture` + `mavlink-passkey` + `ardupilot-plane-sitl` | `GPS_INPUT` messages reach AP SITL; AP EKF accepts them as `EK3_SRC1_POSXY=3` (GPS); MAVLink 2.0 signing handshake completes (D-C8-9); messages without valid signature are rejected | `exact` (AP source-set state via param read) + `boolean` (signing handshake success) + `exact` (rejection of unsigned in NFT-SEC-03) | N/A | AC-4.3 + D-C8-9 |
| FT-P-09-iNav | `derkachi-fixture` + `inav-sitl` | `MSP2_SENSOR_GPS` (ID 0x1F03) messages reach iNav SITL via TCP 5760; iNav GPS provider state shows `provider=MSP` and fix is acquired | `exact` on iNav GPS provider state via MSP read | N/A | AC-4.3 + Source #4 |
| FT-P-10 | `derkachi-fixture` | Per Mode B Fact #107: GTSAM iSAM2 smoothed past-keyframe pose estimates differ from raw single-shot estimates AND smoothed estimates are closer to `GLOBAL_POSITION_INT` GT than raw (IT-11). NOT validated as FC-side retroactive correction (out of scope per Mode B revision). | `numeric_tolerance` improvement check | smoothed_error < raw_error | AC-4.5 (revised) + Mode B Fact #107 |
| FT-P-11 | `cold-boot-fixture` + `ardupilot-plane-sitl` | On boot, SUT initializes from FC EKF's last valid GPS + IMU-extrapolated position | `numeric_tolerance` initial-pose-vs-FC-pose | ±50 m | AC-5.1 |
| NFT-RES-01 | `derkachi-fixture` + 4 s outage injector | After >3 s without estimate, FC falls back to IMU-only dead reckoning; SUT emits a `NO_ESTIMATE_TIMEOUT` failure log | `boolean` on FC EKF source-set transition + `regex` on log | N/A | AC-5.2 |
| NFT-RES-02 | `derkachi-fixture` + container restart mid-replay | After companion reboot, SUT re-initializes from FC's current IMU-extrapolated position; first emitted `GPS_INPUT` / `MSP2_SENSOR_GPS` is within ±100 m of FC's IMU-extrapolated pose at boot-complete time | `numeric_tolerance` pose at first emit | ±100 m | AC-5.3 |

### Performance

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| NFT-PERF-01 (Tier-2 only) | `derkachi-fixture` resampled to 3 Hz on Jetson Orin Nano Super | End-to-end latency (camera capture → GPS to FC) | `threshold_max` p95 | ≤ 400 ms | AC-4.1 + D-CROSS-LATENCY-1 |
| NFT-PERF-02 (Tier-1+2) | `derkachi-fixture` | Estimates emitted frame-by-frame (no batching > 1 frame); inter-emit interval p95 ≤ inter-frame interval × 1.05 | `threshold_max` p95 inter-emit | ≤ 350 ms (at 3 Hz target) | AC-4.4 |
| NFT-PERF-03 (Tier-2 only) | `cold-boot-fixture` | Cold-start TTFF: from container-ready to first valid `GPS_INPUT` / `MSP2_SENSOR_GPS` | `threshold_max` p95 over 50 cold boots | < 30 s | AC-NEW-1 |
| NFT-PERF-04 | `still-image-set-60` + spoofed FC GPS injection in `ardupilot-plane-sitl` | Spoofing-promotion latency: from FC GPS-denial / spoof signal to SUT estimate becoming AP primary position source | `threshold_max` p95 over 50 trials per FC | < 3 s | AC-NEW-2 |

### Resource limits

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| NFT-LIM-01 (Tier-2) | `derkachi-fixture` 8 h replay loop | Memory `< 8 GB shared` on Jetson Orin Nano Super throughout | `threshold_max` peak RSS over duration | ≤ 8 GB | AC-4.2 |
| NFT-LIM-02 (Tier-1) | 8 h Derkachi replay loop | FDR ≤ `64 GB`; no payload class silently dropped without a logged rollover | `threshold_max` total FDR size + `regex` on rollover-event presence | ≤ 64 GB | AC-NEW-3 |
| NFT-LIM-03 | `tile-cache-fixture` plus exercised manifests/overviews/indices | Cache budget `≤ 10 GB` for the ~400 km² operational area unless solution defines a separate descriptor budget | `threshold_max` total cache size | ≤ 10 GB | RESTRICT-SAT-2 + AC-8.3 |
| NFT-LIM-04 (Tier-2) | `derkachi-fixture` 8 h | CPU/GPU/temp/throttle telemetry recorded; no thermal throttling at 25 W TDP at the upper temp envelope (deferred to chamber for AC-NEW-5) | `threshold_max` throttle event count = 0 (workstation thermal-day) | 0 events | RESTRICT-HW-1 + AC-NEW-5 (Tier-2 partial) |

### Security

| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
| NFT-SEC-01 | Synthetic over-confidence injection: deflate covariance ×1.5-3 in 3 trial flights, observe AC-NEW-7 cache-poisoning behavior at the `mock-suite-sat-service` ingest | Per flight: `P(geo-misalign > 30 m) < 1%`, `P(> 100 m) < 0.1%` of written tiles. PARTIAL — multi-flight Monte Carlo (≥100 flights per AC text) is reduced-confidence with current single Derkachi fixture; trace flag in matrix. | `threshold_max` on probability | < 1% / < 0.1% | AC-NEW-7 |
| NFT-SEC-02 | Network egress probe from SUT container | All non-`e2e-net` egress attempts blocked by Docker `internal: true`; per-attempt logged as security event in SUT log | `exact` (egress count = 0) + `regex` (security-event log emission) | N/A | RESTRICT-SAT-1 + AC-8.1 |
| NFT-SEC-03 | `ardupilot-plane-sitl` + un-signed MAVLink GPS_INPUT injection | AP SITL rejects unsigned messages on the signed channel; SUT-emitted (signed) messages pass; SBOM check confirms passkey configuration | `exact` (AP rejection of unsigned) + `boolean` (SBOM passkey present) | N/A | D-C8-9 + Mode B Fact #109 + AC-NEW-2 |
| NFT-SEC-04 | `cve-jpeg-fixture` fed to SUT image pipeline (C1 + C4 paths) | OpenCV ≥4.12.0 either decodes safely or rejects the file; no crash, no buffer overflow detected by AddressSanitizer | `boolean` on no-crash + ASan clean | N/A | D-CROSS-CVE-1 + Mode B Fact #112 |

## External Dependency Mocks

| External Service | Mock/Stub | How Provided | Behavior |
|-----------------|-----------|-------------|----------|
| Azaion Suite Satellite Service (ingest API for AC-NEW-7 voting layer) | `mock-suite-sat-service` Docker service | Local FastAPI stub returning canned tile-publish-acknowledgement responses with deterministic IDs; logs every received tile + per-tile quality metadata to a file the e2e-runner reads back | Returns 202 Accepted on every well-formed publish; returns 400 on malformed; never simulates real voting (the project's role is to publish, the Service's role is to vote per Mode B Fact #105 / D-PROJ-2) |
| ArduPilot Plane FC | `ardupilot-plane-sitl` Docker service | Open-source SITL build of ArduPilot Plane stable; configured with `GPS_TYPE=14` per Source #2 to accept MAVLink GPS_INPUT | Real ArduPilot EKF behavior; we observe but do not patch |
| iNav FC | `inav-sitl` Docker service | Open-source iNav SITL; GPS provider configured to MSP per `docs/SITL/SITL.md` | Real iNav GPS subsystem behavior; we observe but do not patch |
| QGroundControl GCS | `mavproxy-listener` Docker service | Passive MAVLink listener that forwards SUT → GCS stream into a `.tlog` file the e2e-runner parses | Captures all STATUSTEXT, NAMED_VALUE_FLOAT, downsampled position frames for assertions |
| AI camera (AC-7.x) | NOT MOCKED — out of scope per Phase 1 gate | N/A | NOT COVERED in current matrix — see traceability matrix |

## Data Validation Rules

| Data Type | Validation | Invalid Examples | Expected System Behavior |
|-----------|-----------|-----------------|------------------------|
| Nav-camera frame | Resolution within ADTi spec (~5472×3648 production, downscaled equivalents allowed in Tier-1 Docker) | 0×0 frame, corrupt JPEG (CVE fixture), wrong color depth | Reject frame, log invalid-input event, do NOT advance estimator state |
| FC IMU sample | `SCALED_IMU2` fields present; timestamp monotonic; non-zero accelerometer norm | Missing field, backwards timestamp, NaN | Reject sample, log invalid-input event, propagate estimator from prior valid state |
| Satellite tile manifest | Required fields per `restrictions.md`: CRS, tile matrix, dimension, lat-adjusted m/px, capture date, source, compression. m/px ≥ 0.5. capture_date within AC-8.2 freshness window. | Missing capture_date, m/px = 1.0 (below floor), capture_date older than freshness threshold | Reject tile load OR downgrade to non-`satellite_anchored` source label per AC-NEW-6 |
| Spoofed FC GPS | (FC-side input the SUT detects) | GPS jump >200 m between consecutive 5 Hz frames; FC GPS-health flag toggled to spoofed | SUT switches estimator label to `dead_reckoned`, stops promoting FC GPS, continues per AC-NEW-8 |
| MAVLink GPS_INPUT outbound | Honest covariance — `horiz_accuracy` ≥ estimator's 95% covariance semi-major axis | Under-reported covariance | This is a defect (AC-NEW-4) — fail NFT-PERF-04 if observed |
| MAVLink message signature | MAVLink 2.0 signed on AP wired channel per D-C8-9 | Unsigned message on signed channel | AP-side rejection (NFT-SEC-03 expected behavior) |