mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 19:01:14 +00:00
Oleksandr Bezdieniezhnykh
59d9116d36
Bootstraps the public-boundary blackbox test harness owned by epic
AZ-262 (E-BBT). Establishes the e2e/ directory tree at the repo root,
fully separated from src/gps_denied_onboard/** and from the in-process
tests/** tree, and commits to the contracts every subsequent test
ticket (AZ-407..AZ-446) builds against.
Tier-1 (workstation Docker):
- docker/docker-compose.test.yml wires SUT + ArduPilot SITL + iNav SITL
+ mock Suite Sat Service + mavproxy listener + e2e-runner onto one
e2e-net bridge with internal: true (enforces RESTRICT-SAT-1 /
NFT-SEC-02 egress isolation at the network layer).
- docker/docker-compose.tier2-bridge.yml override disables the in-
compose SUT so Tier-2 pairs SITLs + mock + runner on an x86 host
while the SUT runs natively on the Jetson under systemd.
Tier-2 (Jetson):
- jetson/run-tier2.sh + tier2.service systemd unit + tegrastats /
jtop parsers feed per-sample telemetry into the evidence bundle.
Runner image (e2e/runner/):
- Dockerfile + requirements.txt install ONLY ground-side libs
(pymavlink, opencv-python>=4.12, numpy/scipy/geopy/pyproj, httpx,
orjson, pydantic, structlog, pytest 8.x). The runner deliberately
does NOT install the SUT package.
- conftest.py implements the AC-9 skip-rule mapping (tier2_only,
chamber_only, vins_mono, deferred_ac) tied to environment.md
parametrize axes.
- reporting/csv_reporter.py is a pytest plugin emitting one row per
test with the exact 11-column schema from environment.md §
Reporting (test_id, test_name, traces_to, fc_adapter, vio_strategy,
tier, started_at_utc, execution_time_ms, result, error_message,
evidence_paths). XFAIL surfaced only when a test carries
@pytest.mark.deferred_ac(verdict="xfail", reason=...).
- reporting/evidence_bundler.py exposes the attach_evidence fixture
that copies per-test artifacts (.tlog, FDR archives, screenshots,
tegrastats / jtop CSVs) into the run bundle and records relative
paths into the reporter's evidence_paths column.
- helpers/{frame_source_replay,imu_replay,sitl_observer,
mavproxy_tlog_reader,fdr_reader}.py declare the public surfaces
(concrete implementations owned by AZ-407 / AZ-408 / AZ-416 /
AZ-417 / AZ-441 per the dependency table); helpers/geo.py ships
today (no downstream task dep) — WGS84 distance / forward-bearing
/ offset via pyproj with NaN rejection.
Mock Suite Sat Service (e2e/fixtures/mock-suite-sat/):
- FastAPI app: POST /tiles (ingest contract from D-PROJ-2 follow-up),
GET /tiles/audit + /mock/audit (per-run read-back), POST
/mock/config (force-status, response delay), POST /mock/reset
(clears audit between tests), GET /mock/health.
Fixture scaffolds (e2e/fixtures/{tile-cache-builder, age-injector,
injectors, cold-boot, secrets, security}/):
- Public surfaces only. Concrete builders land in AZ-407 (static
fixtures), AZ-408 (runtime synthetic injection), AZ-419 (cold-boot
fixture), AZ-439 (CVE-2025-53644 JPEG generator).
Test tree (e2e/tests/{positive,negative,performance,resilience,
security,resource_limit}/):
- Mirror of the test-spec category grouping in
_docs/02_document/tests/*-tests.md.
- tests/positive/test_smoke.py is the AC-1 harness-boot smoke run
inside the e2e-runner image once Docker brings everything up.
Out-of-container unit tests (e2e/_unit_tests/):
- Exercises the harness internals (CSV reporter plugin lifecycle,
conftest skip rules, helper modules, parsers, mock app, compose
YAML structural contract, public-boundary enforcement) without
Docker / SITL. 97 unit tests, all passing.
Build / config:
- pyproject.toml: testpaths extended with e2e/_unit_tests; pythonpath
extended with e2e; fastapi>=0.111,<0.120 added to dev extras for the
mock-app TestClient unit test.
AC coverage:
- AC-1 (Tier-1 boot) → compose YAML test + directory layout
+ smoke test (Docker-bound)
- AC-2 (mock services) → 6 FastAPI TestClient unit tests
- AC-3 (SITLs accept output) → contract present; concrete check
deferred to AZ-416 / AZ-417
- AC-4 (CSV columns) → in-process plugin lifecycle test
emits the exact 11-column schema
- AC-5 (egress isolation) → static config test + runtime probe
in Docker-bound smoke
- AC-6 (Tier-2 contract) → tegrastats + jtop parser unit tests
+ jetson/* layout test; full Tier-2
contract is AZ-444
- AC-7 (fixture reproducibility) → deferred to AZ-407 per task spec
- AC-8 (parametrize matrix) → vins_mono skip-rule cases +
tests/positive/test_smoke
- AC-9 (skip semantics) → 9 conftest skip-rule unit tests
Module layout entry for blackbox_tests was added in 2026-05-16
preparatory commit d7a17a8 so this diff stays focused on the harness
scaffold. AZ-406 advances to In Testing on commit.
Co-authored-by: Cursor <cursoragent@cursor.com>
68 lines
3.8 KiB
Markdown
68 lines
3.8 KiB
Markdown
# Blackbox Test Harness (`e2e/`)
|
|
|
|
This directory is the **public-boundary** test harness for `gps-denied-onboard`. It is owned by the `blackbox_tests` cross-cutting entry in `_docs/02_document/module-layout.md` and implements task **AZ-406** (Test Infrastructure Bootstrap) plus its downstream test-task siblings (AZ-407..AZ-446).
|
|
|
|
The harness runs in two execution tiers (`environment.md` § Two-tier execution profile):
|
|
|
|
- **Tier-1** — workstation Docker. `cd e2e/docker && docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-runner`
|
|
- **Tier-2** — Jetson Orin Nano Super hardware loop. `./e2e/jetson/run-tier2.sh --fc-adapter <ardupilot|inav> --vio-strategy <okvis2|klt_ransac>`
|
|
|
|
Both tiers emit the same CSV report format (one row per test) per `environment.md` § Reporting.
|
|
|
|
## Layout
|
|
|
|
```
|
|
e2e/
|
|
├── docker/ Tier-1 entrypoint (docker-compose.test.yml + Tier-2 bridge override + secrets mount)
|
|
├── jetson/ Tier-2 entrypoint (run-tier2.sh + systemd unit + tegrastats/jtop parsers)
|
|
├── runner/ e2e-runner image (Dockerfile, conftest, pytest plugins, helpers, requirements)
|
|
├── fixtures/ Fixture builders (tile-cache, age-injector, injectors/, mock-suite-sat, secrets, security)
|
|
├── tests/ Pytest target — `positive/`, `negative/`, `performance/`, `resilience/`, `security/`, `resource_limit/`
|
|
└── _unit_tests/ Out-of-container unit tests for the harness internals (run as part of the project test suite)
|
|
```
|
|
|
|
## Public-Boundary Discipline (hard rule)
|
|
|
|
The e2e-runner image **MUST NOT** import any module from the SUT source tree (`src/gps_denied_onboard/**`). The only legal interaction surfaces are:
|
|
|
|
- MAVLink (ArduPilot SITL — UDP 14550)
|
|
- MSP2 (iNav SITL — TCP 5760)
|
|
- HTTP/JSON (mock-suite-sat-service — port 8080)
|
|
- Filesystem read of the FDR archive after a run (`fdr-output` volume)
|
|
|
|
This rule is enforced by:
|
|
|
|
1. The runner `Dockerfile` building from a base image that does NOT install the SUT package.
|
|
2. Layout discipline: no `import gps_denied_onboard.*` in any file under `e2e/`.
|
|
3. Compose `e2e-net.internal: true` — no external network egress (RESTRICT-SAT-1, NFT-SEC-02).
|
|
|
|
See `_docs/02_document/tests/environment.md` for the full per-service spec.
|
|
|
|
## RUN_ID and report paths
|
|
|
|
Each invocation must set `RUN_ID` (defaults to `local-${USER}-${EPOCH}` in development; CI sets it from the workflow run id). Reports land at:
|
|
|
|
- `e2e-results/run-${RUN_ID}/report.csv`
|
|
- `e2e-results/run-${RUN_ID}/evidence/` (per-run `.tlog`, FDR archives, screenshots, profiler traces, tegrastats CSV, jtop CSV)
|
|
|
|
The `e2e-results/` directory is gitignored.
|
|
|
|
## How to add a new blackbox scenario
|
|
|
|
1. Decompose the scenario into a task spec under `_docs/02_tasks/todo/`.
|
|
2. Implement the test under the appropriate `e2e/tests/<category>/` folder.
|
|
3. The conftest's session-scoped `(fc_adapter, vio_strategy)` parameterization automatically applies — opt out with `@pytest.mark.parametrize` overrides.
|
|
4. Trace the scenario to the AC/RESTRICT IDs it exercises via the `traces_to` pytest marker — the CSV reporter emits this verbatim.
|
|
|
|
## How to add a new fixture builder
|
|
|
|
Fixture builders live under `e2e/fixtures/` and may be standalone Python modules (for runtime injectors) or Dockerized helpers (for tile-cache / mock-suite-sat). Each builder must:
|
|
|
|
- Be reproducible — given the same input, produce bit-identical output.
|
|
- Document its output volume / path in `_docs/02_document/tests/test-data.md`.
|
|
- Have a corresponding unit test under `e2e/_unit_tests/fixtures/`.
|
|
|
|
## Out-of-container unit tests
|
|
|
|
The harness's internal Python — CSV reporter, helpers, parsers, mock app, conftest skip rules — is unit-tested under `e2e/_unit_tests/`. These tests do NOT require Docker, SITL, or any external service and run as part of the project's main pytest invocation (`testpaths` extension in `pyproject.toml`).
|