# Test Infrastructure

**Task**: AZ-406_test_infrastructure
**Name**: Blackbox Test Infrastructure Bootstrap (Tier-1 Docker + Tier-2 Jetson harness scaffold)
**Description**: Scaffold the e2e blackbox test project — `e2e/` directory, pytest runner, docker-compose.test.yml, mock services, fixture builders, secrets handling, CSV reporter. This is the foundation every blackbox test depends on.
**Complexity**: 5 points
**Dependencies**: None
**Component**: Blackbox Tests (epic AZ-262 / E-BBT)
**Tracker**: AZ-406
**Epic**: AZ-262 (E-BBT)

## Problem

The product (`gps-denied-onboard`) needs a behavioral verification layer that drives the SUT exclusively through its declared public boundaries (frame source, FC inbound, tile cache mount, FC outbound via SITL, GCS via mavproxy, FDR filesystem). Without a unified test harness — Docker compose for Tier-1, Jetson runner harness for Tier-2, fixture builders, mock Suite Sat Service, MAVLink listener, CSV reporter — every individual blackbox / performance / resilience / security / resource-limit task would re-invent its own scaffolding. This task delivers that shared scaffold once.

## Outcome

- A single `cd e2e/docker && docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-runner` command brings up the full Tier-1 environment and runs the full pytest suite (when test tasks land).
- A single `./e2e/jetson/run-tier2.sh --fc-adapter <ardupilot|inav> --vio-strategy <okvis2|klt_ransac>` runs the Tier-2 hardware-loop with the same CSV reporter contract.
- The matrix dimensions `FC_ADAPTER × VIO_STRATEGY × build_kind` (per `environment.md` § CI runner mapping) are first-class CI parameters; CI YAML scaffold provided.
- Every external dependency named in `environment.md` (`ardupilot-plane-sitl`, `inav-sitl`, `mock-suite-sat-service`, `mavproxy-listener`) is provisioned by the compose file and reachable inside `e2e-net`.
- Egress isolation (`internal: true` on `e2e-net`) is enforced by default, satisfying RESTRICT-SAT-1 / NFT-SEC-02 at the network layer.
- A single `pytest`-based runner discovers tests under `e2e/tests/`, parameterizes by `FC_ADAPTER` + `VIO_STRATEGY`, and emits the CSV report at `e2e-results/run-${RUN_ID}/report.csv` with the columns specified in `environment.md` § Reporting.
- Fixture builders for `tile-cache-fixture`, `synth-age-tile-set`, `outlier-injection-derkachi`, `blackout-spoof-derkachi`, `multi-segment-derkachi`, `cold-boot-fixture`, `cve-jpeg-fixture`, `mavlink-passkey` exist as separate Dockerized helpers under `tests/fixtures/`.

## Test Project Folder Layout

```
e2e/
├── docker/
│   ├── docker-compose.test.yml         # Tier-1 entrypoint
│   ├── docker-compose.tier2-bridge.yml # optional override for Jetson-attached SITLs
│   └── secrets/
│       └── mavlink_passkey             # Docker-secret mount target (test passkey)
├── jetson/
│   ├── run-tier2.sh                    # Tier-2 entrypoint
│   ├── tier2.service                   # systemd unit template for SUT lifecycle
│   ├── tegrastats_parser.py            # parse tegrastats → per-sample CSV rows
│   └── jtop_parser.py                  # parse jetson-stats jtop API → per-sample CSV
├── runner/
│   ├── Dockerfile                      # e2e-runner image (Python 3.12 + pytest 8.x)
│   ├── requirements.txt                # pytest, pymavlink, msp_gps_toy bridge, opencv-python>=4.12.0, numpy, scipy, geopy
│   ├── conftest.py                     # session/module/function fixtures, FC_ADAPTER/VIO_STRATEGY parameterization
│   ├── reporting/
│   │   ├── csv_reporter.py             # pytest plugin emitting environment.md § Reporting columns
│   │   └── evidence_bundler.py         # collects .tlog, FDR archives, profiler traces, screenshots
│   └── helpers/
│       ├── frame_source_replay.py      # replay images / video to V4L2 file source
│       ├── imu_replay.py               # replay data_imu.csv at 10 Hz to FC inbound
│       ├── sitl_observer.py            # AP/iNav state-read helpers (param read, GPS_RAW_INT, MSP queries)
│       ├── mavproxy_tlog_reader.py     # parse .tlog from mavproxy-listener
│       ├── fdr_reader.py               # post-run filesystem read of FDR archive
│       └── geo.py                      # Vincenty / WGS84 geodesic helpers
├── fixtures/
│   ├── tile-cache-builder/             # builds tile-cache-fixture from input_data + curated public subset
│   ├── age-injector/                   # mutates tile manifest dates → synth-age-tile-set
│   ├── injectors/
│   │   ├── outlier.py                  # outlier-injection-derkachi
│   │   ├── blackout_spoof.py           # blackout-spoof-derkachi (5/15/35 s windows)
│   │   ├── multi_segment.py            # multi-segment-derkachi
│   │   └── cold_boot.py                # cold-boot-fixture (frozen FC pose JSON)
│   ├── secrets/
│   │   └── mavlink-test-passkey.txt    # 32-byte hex; "TEST ONLY"
│   ├── security/
│   │   └── cve-2025-53644.jpg          # crafted JPEG for NFT-SEC-04 (license-checked PoC)
│   └── mock-suite-sat/                 # FastAPI stub for mock-suite-sat-service
│       ├── Dockerfile
│       └── app.py                      # 202 on well-formed publish; 400 on malformed
└── tests/
    ├── positive/                       # FT-P-* scenarios
    ├── negative/                       # FT-N-* scenarios
    ├── performance/                    # NFT-PERF-*
    ├── resilience/                     # NFT-RES-*
    ├── security/                       # NFT-SEC-*
    └── resource_limit/                 # NFT-LIM-*
```

### Layout Rationale

- `e2e/docker/` and `e2e/jetson/` separate the two execution tiers, mirroring `environment.md` § Two-tier execution profile. Each tier has its own entrypoint script — the runner image and CSV-reporter contract are shared.
- `e2e/runner/helpers/` keeps reusable boundary-driving primitives (frame replay, IMU replay, SITL observers, FDR reader, MAVLink listener) out of individual test modules — every blackbox task imports from here, not from the SUT.
- `e2e/fixtures/` holds *fixture builders*, not the data itself. Heavy fixture content (Derkachi video, 60 still images) is bind-mounted from `_docs/00_problem/input_data/` per `test-data.md`.
- `e2e/tests/<category>/` mirrors the `_docs/02_document/tests/*-tests.md` grouping so a reader can map any spec scenario to its test file.

## Mock Services

| Mock Service | Replaces | Endpoints | Behavior |
|---|---|---|---|
| `mock-suite-sat-service` | Azaion Suite Satellite Service ingest API | `POST /tiles` (publish), `GET /tiles/audit` (test-side audit retrieval) | Returns 202 on well-formed publish; 400 on malformed; logs every received tile + per-tile quality metadata to `/audit/<run-id>.jsonl`; `GET /audit` reads the log back. Deterministic; same input → same response. |
| `ardupilot-plane-sitl` | ArduPilot Plane FC | UDP 14550 (MAVLink) | Real `ardupilot/ardupilot-sitl:plane-stable`; `GPS_TYPE=14`. Tests OBSERVE, do not patch. |
| `inav-sitl` | iNav FC | TCP 5760 (MSP2) | Real `inavflight/inav-sitl:9.0.0`; GPS provider configured to MSP per `docs/SITL/SITL.md`. Tests OBSERVE. |
| `mavproxy-listener` | QGroundControl GCS | UDP 14551 | Passive MAVLink listener; captures SUT → GCS stream into `/var/log/tlogs/<run-id>.tlog` for assertions. |

### Mock Control Surface

The mock-suite-sat-service exposes:

- `POST /mock/config` — test-time behavior control (e.g., simulate downtime, inject 400 errors for negative-path scenarios)
- `GET /mock/audit` — returns received tiles + their declared quality metadata for assertion
- `POST /mock/reset` — clears audit log between tests for isolation

The two SITL services (ArduPilot, iNav) are NOT control-surface mocks — they are real flight-controller stacks running in simulation. Tests interact via standard MAVLink / MSP2 protocols.

## Docker Test Environment

### docker-compose.test.yml structure

The full structure is defined in `_docs/02_document/tests/environment.md` § Docker Environment. The test infrastructure task implements that structure verbatim with the following behaviors:

- All services on the `e2e-net` bridge network with `internal: true` (no external connectivity — RESTRICT-SAT-1 / NFT-SEC-02).
- Volumes: `tile-cache-fixture` (RO mount into SUT), `fdr-output` (RW from SUT, RO from runner), `input-data` (RO bind from `_docs/00_problem/input_data/`), `expected-results` (RO bind from `_docs/00_problem/input_data/expected_results/`).
- `fdr-output` sized exactly 64 GB via Docker `--storage-opt size=64g` to enforce AC-NEW-3 capacity at the volume layer (NFT-LIM-02 cross-checks rotation behavior).
- `MAVLINK_SIGNING_PASSKEY_FILE` injected as a Docker secret from `e2e/docker/secrets/mavlink_passkey`.
- `FC_ADAPTER` and `VIO_STRATEGY` pulled from environment, default `ardupilot` + `okvis2`. CI matrix sets these per job.

### Networks and Volumes

| Network / Volume | Type | Purpose |
|---|---|---|
| `e2e-net` | bridge, `internal: true` | All test traffic; enforces no-external-egress at network layer |
| `tile-cache-fixture` | named volume | Pre-built FAISS HNSW index + tile filesystem; built once per CI run |
| `fdr-output` | named volume, 64 GB cap | Per-flight FDR write target |
| `input-data` | bind mount | RO bind of `_docs/00_problem/input_data/` |
| `expected-results` | bind mount | RO bind of `_docs/00_problem/input_data/expected_results/` |

### Tier-2 Bridge

Tier-2 runs the SUT on real Jetson hardware with `systemctl start gps-denied-onboard.service`. SITL containers (ArduPilot, iNav) run either on the same Jetson (constrained CPU sharing) OR on a paired x86 host on the same network — `docker-compose.tier2-bridge.yml` provisions the SITL-only subset with the same `e2e-net` definition so the runner observes the same boundaries.

## Test Runner Configuration

**Framework**: pytest 8.x
**Plugins**:
- `pytest-csv` — CSV emission per `environment.md` § Reporting (one row per test)
- `pytest-xdist` — parallel test execution within a tier (Tier-1 only; Tier-2 runs serially due to single-Jetson constraint)
- `pytest-timeout` — per-test wall-clock budget enforcement (matches the per-scenario `Max execution time` in test specs)
- `pymavlink` — MAVLink ground side
- `msp_gps_toy` (Rust binary, called via subprocess) — MSP2 ground side
- `opencv-python>=4.12.0` — frame source replay (CVE-mitigated per D-CROSS-CVE-1)
- `numpy` + `scipy` + `geopy` (Vincenty) — geodesic-distance assertions in WGS84
- `pytest-forked` — `--forked` mode for hermetic-critical scenarios

**Entry point**: `pytest e2e/tests/ --csv=e2e-results/run-${RUN_ID}/report.csv --csv-columns="test_id,test_name,traces_to,fc_adapter,vio_strategy,tier,started_at_utc,execution_time_ms,result,error_message,evidence_paths"`

### Fixture Strategy

| Fixture | Scope | Purpose |
|---|---|---|
| `fc_adapter` | session | parametrized over `{ardupilot, inav}`; selects which SITL to bind |
| `vio_strategy` | session | parametrized over `{okvis2, klt_ransac}` (production); `vins_mono` only on research-build sessions |
| `tile_cache` | session | mounts `tile-cache-fixture` once per session |
| `clean_sut` | function | `docker compose restart gps-denied-onboard` between tests; resets `fdr-output` |
| `clean_sut_forked` | function (--forked) | full `docker compose down/up` per test; for hermetic-critical scenarios |
| `mavproxy_tlog` | function | starts a fresh `.tlog` capture window for the duration of the test |
| `fdr_reader` | function | post-run helper for parsing the FDR archive |
| `sitl_observer` | function | AP/iNav state-read helper |

### Parameterization

Every test file is parameterized by `(fc_adapter, vio_strategy)` unless the spec explicitly skips one or both. The conftest skip-rules:

- AC-7.x scenarios: skipped on every run (NOT COVERED per traceability matrix; `pytest.skip(reason="AC-7.x deferred — see traceability matrix")`).
- `vins_mono`: skipped on production-build runs (`pytest.skip(reason="vins_mono is research-build-only per D-C1-1-SUB-A")`).
- Tier-2-only scenarios (NFT-PERF-01, NFT-LIM-01, NFT-PERF-03, NFT-LIM-04): skipped on Tier-1 with `pytest.skip(reason="Tier-2 only — Jetson hardware required")`.
- Chamber-only scenarios (AC-NEW-5 hot-soak portion): skipped on Tier-2 workstation-thermal runs; gated by `--enable-chamber` flag.

## Test Data Fixtures

| Data Set | Source | Format | Used By |
|---|---|---|---|
| `still-image-set-60` | bind mount from `_docs/00_problem/input_data/` | JPEG + CSV GT | FT-P-01, FT-P-03, FT-P-05, FT-P-06, FT-P-15, FT-P-19, NFT-RES-03, NFT-PERF-04 |
| `still-image-sat-refs-2` | same | PNG | FT-P-05, FT-P-19 |
| `derkachi-fixture` | bind mount | MP4 + CSV | FT-P-02, FT-P-04, FT-P-07, FT-P-10, FT-N-01..04, NFT-PERF-01..02, NFT-RES-01..04, NFT-LIM-02, NFT-LIM-04 |
| `tile-cache-fixture` | named volume built by `tests/fixtures/tile-cache-builder/` | FAISS HNSW + tile filesystem | FT-P-01, FT-P-05, FT-P-15..17, FT-P-19, FT-N-05..06, NFT-LIM-03, NFT-PERF-01, NFT-PERF-04, NFT-SEC-01..02 |
| `synth-age-tile-set` | built from `tile-cache-fixture` by `age-injector/` | tile filesystem with mutated manifest dates | FT-N-05, FT-N-06 |
| `outlier-injection-derkachi` | runtime-generated by `injectors/outlier.py` | tmpfs frame source | FT-N-01 |
| `blackout-spoof-derkachi` | runtime-generated by `injectors/blackout_spoof.py` | tmpfs + spoof injector on FC inbound | FT-N-04, NFT-RES-04 |
| `multi-segment-derkachi` | runtime-generated by `injectors/multi_segment.py` | tmpfs frame source | FT-P-08 |
| `cold-boot-fixture` | static JSON fixture in `tests/fixtures/cold-boot/` | JSON pose snapshot | FT-P-11, NFT-PERF-03 |
| `mavlink-passkey` | `tests/fixtures/secrets/mavlink-test-passkey.txt` | 32-byte hex | FT-P-09-AP, NFT-SEC-03 |
| `cve-jpeg-fixture` | `tests/fixtures/security/cve-2025-53644.jpg` | crafted JPEG | NFT-SEC-04 |

### Data Isolation

Per `test-data.md` § Data Isolation Strategy:

- Each test runs against a fresh SUT container (`docker compose restart` between tests, OR `--forked` pytest mode for hermetic-critical scenarios).
- `tile-cache-fixture` and `input-data` mounts are read-only — cross-contamination at the SUT input layer is impossible.
- `fdr-output` volume is reset between tests (`docker volume rm` + recreate).
- Synthetic-injection fixtures generate to per-test tmpfs; never written back to a persistent volume.
- For Tier-2: same isolation discipline at the systemd-service level (`systemctl restart`); `/var/azaion/fdr` wiped between tests.

## Test Reporting

**Format**: CSV (one row per test) — exactly per `environment.md` § Reporting.

**Columns**: `test_id, test_name, traces_to, fc_adapter, vio_strategy, tier, started_at_utc, execution_time_ms, result, error_message, evidence_paths`

**Output path**: `e2e-results/run-${RUN_ID}/report.csv` plus a per-run bundle of evidence at `e2e-results/run-${RUN_ID}/evidence/` (assembled by `evidence_bundler.py` from `.tlog` files, FDR archives, screenshots, profiler traces, tegrastats CSV, jtop CSV).

## Acceptance Criteria

**AC-1: Test environment starts (Tier-1)**
Given a clean checkout of the repo
When `cd e2e/docker && docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-runner` is executed
Then all services in `environment.md` § Docker Environment start, the e2e-runner image builds, and pytest discovers ≥1 test file (sample test in `e2e/tests/positive/test_smoke.py`).

**AC-2: Mock services respond**
Given the test environment is running
When the e2e-runner POSTs a well-formed tile-publish JSON to `mock-suite-sat-service`
Then the service responds 202 and records the tile in its audit log; subsequent `GET /mock/audit` returns the recorded entry.

**AC-3: SITL services accept SUT output**
Given the test environment is running with a placeholder SUT that emits one valid `GPS_INPUT` (AP) AND one valid `MSP2_SENSOR_GPS` (iNav)
When the e2e-runner reads `EK3_SRC1_POSXY` from `ardupilot-plane-sitl` AND queries iNav GPS state via MSP from `inav-sitl`
Then both SITLs reflect the test-injected GPS source as primary.

**AC-4: CSV report generated with required columns**
Given the test runner executes
When the test run completes
Then `e2e-results/run-${RUN_ID}/report.csv` exists with exactly the columns from `environment.md` § Reporting, and a per-run evidence bundle exists at `e2e-results/run-${RUN_ID}/evidence/`.

**AC-5: Egress isolation enforced**
Given the test environment is running with `e2e-net.internal: true`
When the e2e-runner attempts a TCP connect to `1.1.1.1:443` from inside the SUT container
Then the connection fails (network-layer block); no DNS resolution succeeds for non-`e2e-net` names.

**AC-6: Tier-2 runner harness contract**
Given a Jetson Orin Nano Super with the SUT installed via systemd
When `./e2e/jetson/run-tier2.sh --fc-adapter ardupilot --vio-strategy okvis2 --duration 5min` is executed
Then the same CSV report format is produced at `e2e-results/run-${RUN_ID}/report.csv`, with `tier=tier2-jetson` for every row, and `tegrastats` + `jtop` per-sample CSVs land in the evidence bundle.

**AC-7: Fixture builders are reproducible**
Given a clean Docker volume state
When `tests/fixtures/tile-cache-builder/build.sh` runs
Then the same tile-cache-fixture content is produced bit-for-bit twice in a row (same FAISS index, same tile manifest hashes); same idempotency for `age-injector` and the static JSON fixtures.

**AC-8: Parameterization matrix coverage**
Given the conftest sets up `(fc_adapter, vio_strategy, tier, build_kind)` parameterization
When CI runs the standard matrix
Then every produced report row has well-defined values for `fc_adapter`, `vio_strategy`, `tier`; `vins_mono` rows appear only on `build_kind=research` runs; `Tier-2`-only test_ids are SKIP on Tier-1 with the expected reason string.

**AC-9: Skips per traceability matrix**
Given the e2e-runner starts
When the discoverer encounters a test mapped to AC-7.1, AC-7.2, AC-NEW-5 chamber portion, AC-8.6 scene-change subset, RESTRICT-CAM-2, or RESTRICT-HW-2 chamber portion
Then those test rows show `result=SKIP` (or `XFAIL` for AC-8.6 scene-change PARTIAL) with an `error_message` referencing the traceability-matrix mitigation entry.

## Constraints

- **Public-boundary discipline**: the e2e-runner image MUST NOT import any module from the SUT source tree. The only legal interaction surfaces are MAVLink / MSP2 / HTTP / filesystem — same as a real consumer would have.
- **OpenCV pin**: the runner image's OpenCV version MUST be `>=4.12.0` (D-CROSS-CVE-1); pinned in `e2e/runner/requirements.txt`.
- **MAVLink-passkey provenance**: the test passkey is a checked-in fixture explicitly labeled "TEST ONLY"; the production passkey path is `/run/secrets/mavlink_passkey` per environment.md and is never the test fixture.
- **Tier separation**: Tier-1 and Tier-2 produce IDENTICAL CSV row formats so the same downstream tooling (badge generators, regression detectors) can consume both.
- **No internal state probes**: no test may read SUT internal state (GTSAM iSAM2 graph, FAISS in-memory index, internal Python/C++ buffers, logger handles). Only public boundaries + FDR archive + SITL observation are legal evidence sources.

## Risks & Mitigation

**Risk 1: Tier-1 runner image build slow / large**
- *Risk*: pulling `tensorrt`, `gtsam`, `faiss-gpu`, `opencv-python>=4.12.0` plus dev dependencies into a single image bloats the e2e-runner build to ≥30 min and ≥10 GB.
- *Mitigation*: the e2e-runner image is **separate from the SUT image** — the runner only needs ground-side libs (`pymavlink`, `msp_gps_toy`, `opencv-python`, `numpy`, `scipy`, `geopy`, `pytest`). The SUT image is what carries the heavy ML stack. Keep the runner image lean (target ≤2 GB).

**Risk 2: SITL containers flaky / non-deterministic timing**
- *Risk*: `ardupilot-plane-sitl` and `inav-sitl` boot times vary; tests may race the SITL's parameter-init phase.
- *Mitigation*: conftest fixture polls SITL readiness via a known parameter read (e.g., `EKF_ENABLE`) before any test runs. Failure to reach readiness within 60 s fails the SITL fixture, not the individual test.

**Risk 3: Mock Suite Sat Service drift from D-PROJ-2 contract**
- *Risk*: when the real Suite Sat Service ingest contract ships (D-PROJ-2), the mock may diverge silently.
- *Mitigation*: the mock's request/response schema is sourced from the contract sketch in `_docs/_process_leftovers/2026-05-09_satellite-provider-design-tasks.md`; a contract test in NFT-SEC-01 asserts the mock's accepted-fields match that sketch. When the real endpoint ships, the mock is retired (per F7 in traceability matrix).

**Risk 4: `--storage-opt size=64g` not portable**
- *Risk*: Docker's `--storage-opt size=64g` for volumes requires specific storage drivers (overlay2 with xfs backing); may not work on all CI runners.
- *Mitigation*: fallback strategy in the docker-compose: if the volume cannot be size-capped at the Docker layer, the SUT enforces the cap internally per AC-NEW-3 and NFT-LIM-02 verifies via volume-size sampling. CI runner config flagged in the runner README.

**Risk 5: `cve-jpeg-fixture` license / distribution**
- *Risk*: PoC JPEG for CVE-2025-53644 may have unclear redistribution terms.
- *Mitigation*: license-checked at fixture-import time; if license unclear, the fixture is generated programmatically following the published PoC structure (no copyrighted bytes); generation script is itself part of `tests/fixtures/security/`.

## Document Dependencies

- `_docs/02_document/tests/environment.md` — full Docker environment spec (services, networks, volumes, secrets, ports)
- `_docs/02_document/tests/test-data.md` — fixture sources, formats, isolation strategy, validation rules
- `_docs/02_document/tests/traceability-matrix.md` — AC coverage map (drives the SKIP / XFAIL rules in conftest)
- `_docs/02_document/tests/blackbox-tests.md` + `performance-tests.md` + `resilience-tests.md` + `security-tests.md` + `resource-limit-tests.md` — list of test categories that the `e2e/tests/<category>/` folders mirror

## Excluded

- The SUT (`gps-denied-onboard`) container build — owned by the BUILD-side epics (E-CC-CONF / E-BOOT and per-component epics). The test infrastructure references the SUT image but does NOT build it.
- Individual test scenario implementations (FT-P-*, FT-N-*, NFT-*) — owned by the per-scenario tasks decomposed in Step 3.
- The Suite Sat Service real endpoint contract — owned by the parent suite (D-PROJ-2); the mock here mirrors a sketch only.
- The thermal chamber AC-NEW-5 hot-soak run — physical hardware, deferred per traceability matrix.
- The AI-camera fixture (AC-7.x) — out of scope per Phase 1 gate.