2 Commits

Author	SHA1	Message	Date
Oleksandr Bezdieniezhnykh	6599d828d2	[AZ-407] [AZ-444] [AZ-445] Batch 68: fixtures, Tier-2 harness, NFR reporter ... Three blackbox-harness tasks landed together — all depend only on AZ-406 and unblock the FT-* / NFT-* scenario tasks scheduled for batches 69+. AZ-407 — Static fixture builders (3pt): * tile-cache-builder/{builder.py, Dockerfile, build.sh} produces a deterministic tile-cache-fixture Docker volume from _docs/00_problem/input_data/. Reproducibility primitives: sorted iteration, frozen PIL JPEG settings, FAISS HNSW32 built single- threaded with seeded stub descriptors. * age-injector/{age_injector.py, inject.sh} clones the volume and shifts capture_date by N×30.44 days; tile JPEG bytes preserved bit-identical. Emits synth-age-7mo + synth-age-13mo volumes. * cold-boot/cold_boot_fixture.json: frozen FC pose snapshot at Derkachi sector centre, schema v1. * secrets/mavlink-test-passkey.txt: 64-hex with required `# TEST ONLY` header line per AC-5. Passkey-equality test now compares the secret line after stripping the header. * security/cve-2025-53644.jpg: synthetic 158-byte malformed JPEG (truncated SOS marker). OpenCV 4.11.x rejects gracefully with imdecode → None. AZ-439 will sharpen for ASan instrumentation. * Top-level Makefile with `make fixtures` / `make fixtures-*` / `make e2e-tier1*` / `make unit-tests` targets. AZ-444 — Tier-2 Jetson harness wrapper (5pt): * run-tier2.sh rewritten as orchestrator. Detects local (aarch64 + TIER2_HOST=localhost) vs remote (ssh into TIER2_HOST). New flags: -k/--selector, --build-kind production|asan, --reflash (gated behind TIER2_REFLASH_ACK=1 two-key gate), --dry-run. * tier2-on-jetson.sh (new) — on-device delegate. Verifies gps-denied-onboard{,-asan}.service health; restarts with 5s tolerance; spawns tegrastats + jtop parallel samplers; tails ASan unit's journal in asan mode; drives docker compose with TIER=tier2-jetson; forwards SELECTOR to pytest -k. * docker/run-tier1.sh (new) — selector-parity sibling. * AC-1 (selector parity) and AC-6 (reflash gating) unit-tested via --dry-run output assertions. AC-2/AC-3/AC-4/AC-5 are hardware- loop ACs verified by the Tier-2 runtime smoke (no Jetson in the unit-test layer). AZ-445 — CSV reporter + evidence bundler refinements (2pt): * reporting/nfr_recorder.py (new) — pytest plugin. Provides the `nfr_recorder` fixture with record_metric(name, value, ac_id) and partial(ac_id, reason). At session end emits: - per-nfr/<scenario_id>.json (AC-1) - traceability-status.json with every AC ID parsed from traceability-matrix.md, classified Covered/PARTIAL/NOT COVERED with source scenario IDs (AC-2) - regression-baseline.json with all numeric metrics (AC-3) * csv_reporter.py extended — `_outcome_to_result` consults the aggregator; rows flip PASS → PARTIAL when an AC was marked PARTIAL by nfr_recorder (AC-4). Graceful fallback when aggregator isn't registered (unit-test contexts). * conftest.py registers nfr_recorder in pytest_plugins. * New --traceability-matrix CLI flag seeds the NOT COVERED rows. Build / config: * pyproject.toml dev extras: added Pillow>=10.4,<13.0 for the tile-cache-builder unit test (broad enough to keep torchvision's Pillow 12 pin happy; the production builder runs inside its own Docker image with its own pin). * Updated test_directory_layout.py to cover 10 new files + replaced the byte-equal passkey assertion with the header-stripping variant. Test results: * 157 focused tests pass (was 97 in batch 67; +60 new across this batch). No regressions. Module-layout / spec drift: * AZ-407 spec text says `tests/fixtures/...`; module-layout blackbox_tests entry (commit d7a17a8) authoritatively places the harness under `e2e/`. Implementation followed the layout entry. * AZ-444 spec mentions `e2e/tier2/run-tier2.sh`; AZ-406 placed it at `e2e/jetson/run-tier2.sh`. Kept at `e2e/jetson/` for consistency. * Cold-boot README ownership: corrected from AZ-419 to AZ-407 per AZ-419's own Dependencies field. Specs archived to _docs/02_tasks/done/. Jira tickets transitioned to In Testing on commit. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-16 17:18:01 +03:00
Oleksandr Bezdieniezhnykh	59d9116d36	[AZ-406] Blackbox test harness bootstrap (Tier-1 + Tier-2 scaffold) ... Bootstraps the public-boundary blackbox test harness owned by epic AZ-262 (E-BBT). Establishes the e2e/ directory tree at the repo root, fully separated from src/gps_denied_onboard/** and from the in-process tests/** tree, and commits to the contracts every subsequent test ticket (AZ-407..AZ-446) builds against. Tier-1 (workstation Docker): - docker/docker-compose.test.yml wires SUT + ArduPilot SITL + iNav SITL + mock Suite Sat Service + mavproxy listener + e2e-runner onto one e2e-net bridge with internal: true (enforces RESTRICT-SAT-1 / NFT-SEC-02 egress isolation at the network layer). - docker/docker-compose.tier2-bridge.yml override disables the in- compose SUT so Tier-2 pairs SITLs + mock + runner on an x86 host while the SUT runs natively on the Jetson under systemd. Tier-2 (Jetson): - jetson/run-tier2.sh + tier2.service systemd unit + tegrastats / jtop parsers feed per-sample telemetry into the evidence bundle. Runner image (e2e/runner/): - Dockerfile + requirements.txt install ONLY ground-side libs (pymavlink, opencv-python>=4.12, numpy/scipy/geopy/pyproj, httpx, orjson, pydantic, structlog, pytest 8.x). The runner deliberately does NOT install the SUT package. - conftest.py implements the AC-9 skip-rule mapping (tier2_only, chamber_only, vins_mono, deferred_ac) tied to environment.md parametrize axes. - reporting/csv_reporter.py is a pytest plugin emitting one row per test with the exact 11-column schema from environment.md § Reporting (test_id, test_name, traces_to, fc_adapter, vio_strategy, tier, started_at_utc, execution_time_ms, result, error_message, evidence_paths). XFAIL surfaced only when a test carries @pytest.mark.deferred_ac(verdict="xfail", reason=...). - reporting/evidence_bundler.py exposes the attach_evidence fixture that copies per-test artifacts (.tlog, FDR archives, screenshots, tegrastats / jtop CSVs) into the run bundle and records relative paths into the reporter's evidence_paths column. - helpers/{frame_source_replay,imu_replay,sitl_observer, mavproxy_tlog_reader,fdr_reader}.py declare the public surfaces (concrete implementations owned by AZ-407 / AZ-408 / AZ-416 / AZ-417 / AZ-441 per the dependency table); helpers/geo.py ships today (no downstream task dep) — WGS84 distance / forward-bearing / offset via pyproj with NaN rejection. Mock Suite Sat Service (e2e/fixtures/mock-suite-sat/): - FastAPI app: POST /tiles (ingest contract from D-PROJ-2 follow-up), GET /tiles/audit + /mock/audit (per-run read-back), POST /mock/config (force-status, response delay), POST /mock/reset (clears audit between tests), GET /mock/health. Fixture scaffolds (e2e/fixtures/{tile-cache-builder, age-injector, injectors, cold-boot, secrets, security}/): - Public surfaces only. Concrete builders land in AZ-407 (static fixtures), AZ-408 (runtime synthetic injection), AZ-419 (cold-boot fixture), AZ-439 (CVE-2025-53644 JPEG generator). Test tree (e2e/tests/{positive,negative,performance,resilience, security,resource_limit}/): - Mirror of the test-spec category grouping in _docs/02_document/tests/*-tests.md. - tests/positive/test_smoke.py is the AC-1 harness-boot smoke run inside the e2e-runner image once Docker brings everything up. Out-of-container unit tests (e2e/_unit_tests/): - Exercises the harness internals (CSV reporter plugin lifecycle, conftest skip rules, helper modules, parsers, mock app, compose YAML structural contract, public-boundary enforcement) without Docker / SITL. 97 unit tests, all passing. Build / config: - pyproject.toml: testpaths extended with e2e/_unit_tests; pythonpath extended with e2e; fastapi>=0.111,<0.120 added to dev extras for the mock-app TestClient unit test. AC coverage: - AC-1 (Tier-1 boot) → compose YAML test + directory layout + smoke test (Docker-bound) - AC-2 (mock services) → 6 FastAPI TestClient unit tests - AC-3 (SITLs accept output) → contract present; concrete check deferred to AZ-416 / AZ-417 - AC-4 (CSV columns) → in-process plugin lifecycle test emits the exact 11-column schema - AC-5 (egress isolation) → static config test + runtime probe in Docker-bound smoke - AC-6 (Tier-2 contract) → tegrastats + jtop parser unit tests + jetson/* layout test; full Tier-2 contract is AZ-444 - AC-7 (fixture reproducibility) → deferred to AZ-407 per task spec - AC-8 (parametrize matrix) → vins_mono skip-rule cases + tests/positive/test_smoke - AC-9 (skip semantics) → 9 conftest skip-rule unit tests Module layout entry for blackbox_tests was added in 2026-05-16 preparatory commit d7a17a8 so this diff stays focused on the harness scaffold. AZ-406 advances to In Testing on commit. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-16 16:22:44 +03:00