azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 08:51:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-406] Blackbox test harness bootstrap (Tier-1 + Tier-2 scaffold)

Browse Source 
Bootstraps the public-boundary blackbox test harness owned by epic
AZ-262 (E-BBT). Establishes the e2e/ directory tree at the repo root,
fully separated from src/gps_denied_onboard/** and from the in-process
tests/** tree, and commits to the contracts every subsequent test
ticket (AZ-407..AZ-446) builds against.

Tier-1 (workstation Docker):
- docker/docker-compose.test.yml wires SUT + ArduPilot SITL + iNav SITL
  + mock Suite Sat Service + mavproxy listener + e2e-runner onto one
  e2e-net bridge with internal: true (enforces RESTRICT-SAT-1 /
  NFT-SEC-02 egress isolation at the network layer).
- docker/docker-compose.tier2-bridge.yml override disables the in-
  compose SUT so Tier-2 pairs SITLs + mock + runner on an x86 host
  while the SUT runs natively on the Jetson under systemd.

Tier-2 (Jetson):
- jetson/run-tier2.sh + tier2.service systemd unit + tegrastats /
  jtop parsers feed per-sample telemetry into the evidence bundle.

Runner image (e2e/runner/):
- Dockerfile + requirements.txt install ONLY ground-side libs
  (pymavlink, opencv-python>=4.12, numpy/scipy/geopy/pyproj, httpx,
  orjson, pydantic, structlog, pytest 8.x). The runner deliberately
  does NOT install the SUT package.
- conftest.py implements the AC-9 skip-rule mapping (tier2_only,
  chamber_only, vins_mono, deferred_ac) tied to environment.md
  parametrize axes.
- reporting/csv_reporter.py is a pytest plugin emitting one row per
  test with the exact 11-column schema from environment.md §
  Reporting (test_id, test_name, traces_to, fc_adapter, vio_strategy,
  tier, started_at_utc, execution_time_ms, result, error_message,
  evidence_paths). XFAIL surfaced only when a test carries
  @pytest.mark.deferred_ac(verdict="xfail", reason=...).
- reporting/evidence_bundler.py exposes the attach_evidence fixture
  that copies per-test artifacts (.tlog, FDR archives, screenshots,
  tegrastats / jtop CSVs) into the run bundle and records relative
  paths into the reporter's evidence_paths column.
- helpers/{frame_source_replay,imu_replay,sitl_observer,
  mavproxy_tlog_reader,fdr_reader}.py declare the public surfaces
  (concrete implementations owned by AZ-407 / AZ-408 / AZ-416 /
  AZ-417 / AZ-441 per the dependency table); helpers/geo.py ships
  today (no downstream task dep) — WGS84 distance / forward-bearing
  / offset via pyproj with NaN rejection.

Mock Suite Sat Service (e2e/fixtures/mock-suite-sat/):
- FastAPI app: POST /tiles (ingest contract from D-PROJ-2 follow-up),
  GET /tiles/audit + /mock/audit (per-run read-back), POST
  /mock/config (force-status, response delay), POST /mock/reset
  (clears audit between tests), GET /mock/health.

Fixture scaffolds (e2e/fixtures/{tile-cache-builder, age-injector,
injectors, cold-boot, secrets, security}/):
- Public surfaces only. Concrete builders land in AZ-407 (static
  fixtures), AZ-408 (runtime synthetic injection), AZ-419 (cold-boot
  fixture), AZ-439 (CVE-2025-53644 JPEG generator).

Test tree (e2e/tests/{positive,negative,performance,resilience,
security,resource_limit}/):
- Mirror of the test-spec category grouping in
  _docs/02_document/tests/*-tests.md.
- tests/positive/test_smoke.py is the AC-1 harness-boot smoke run
  inside the e2e-runner image once Docker brings everything up.

Out-of-container unit tests (e2e/_unit_tests/):
- Exercises the harness internals (CSV reporter plugin lifecycle,
  conftest skip rules, helper modules, parsers, mock app, compose
  YAML structural contract, public-boundary enforcement) without
  Docker / SITL. 97 unit tests, all passing.

Build / config:
- pyproject.toml: testpaths extended with e2e/_unit_tests; pythonpath
  extended with e2e; fastapi>=0.111,<0.120 added to dev extras for the
  mock-app TestClient unit test.

AC coverage:
- AC-1 (Tier-1 boot)         → compose YAML test + directory layout
                                + smoke test (Docker-bound)
- AC-2 (mock services)       → 6 FastAPI TestClient unit tests
- AC-3 (SITLs accept output) → contract present; concrete check
                                deferred to AZ-416 / AZ-417
- AC-4 (CSV columns)         → in-process plugin lifecycle test
                                emits the exact 11-column schema
- AC-5 (egress isolation)    → static config test + runtime probe
                                in Docker-bound smoke
- AC-6 (Tier-2 contract)     → tegrastats + jtop parser unit tests
                                + jetson/* layout test; full Tier-2
                                contract is AZ-444
- AC-7 (fixture reproducibility) → deferred to AZ-407 per task spec
- AC-8 (parametrize matrix)  → vins_mono skip-rule cases +
                                tests/positive/test_smoke
- AC-9 (skip semantics)      → 9 conftest skip-rule unit tests

Module layout entry for blackbox_tests was added in 2026-05-16
preparatory commit d7a17a8 so this diff stays focused on the harness
scaffold. AZ-406 advances to In Testing on commit.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-16 16:22:44 +03:00
parent d7a17a8248
commit 59d9116d36
72 changed files with 3515 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/runner/conftest.py
+214
View File
@@ -0,0 +1,214 @@
"""Top-level pytest conftest for the blackbox e2e harness.
Responsibilities:
1. Session-level parameterization over ``(fc_adapter, vio_strategy)``.
2. Skip-rule enforcement per the traceability matrix
(`_docs/02_document/tests/traceability-matrix.md`):
- AC-7.1, AC-7.2 → SKIP (deferred — no AI-camera fixture)
- RESTRICT-CAM-2 → SKIP (paired with AC-7.x)
- AC-NEW-5 chamber portion → SKIP unless --enable-chamber
- RESTRICT-HW-2 chamber portion → SKIP unless --enable-chamber
- Tier-2-only tests → SKIP on tier1-docker
- `vins_mono` parametrization → SKIP on production-build sessions
3. Wiring of the boundary-driving fixtures (`sitl_observer`,
`mavproxy_tlog`, `fdr_reader`, `mock_suite_sat_client`) consumed by
per-scenario tests.
The actual boundary-driving fixtures import helper modules from
``runner.helpers.*``. They are registered here but their implementations
live in the helpers package.
"""
from __future__ import annotations
import os
from collections.abc import Iterator
from pathlib import Path
import pytest
# ---------------------------------------------------------------------------
# Command-line options
# ---------------------------------------------------------------------------
def pytest_addoption(parser: pytest.Parser) -> None:
"""Harness-level options (not exposed to individual tests)."""
group = parser.getgroup("e2e-runner", "Blackbox e2e harness options")
group.addoption(
"--enable-chamber",
action="store_true",
default=False,
help="Enable thermal-chamber-gated tests (AC-NEW-5 hot-soak, RESTRICT-HW-2). "
"Requires the chamber-attached Jetson runner; default off.",
)
group.addoption(
"--build-kind",
action="store",
default=os.environ.get("BUILD_KIND", "production"),
choices=("production", "research"),
help="Selects which VIO strategies are valid: production excludes vins_mono.",
)
group.addoption(
"--evidence-out",
action="store",
default=os.environ.get("EVIDENCE_OUT", "/e2e-results/evidence"),
help="Directory the evidence bundler writes per-run artifacts to.",
)
group.addoption(
"--allow-no-skip-reason",
action="store_true",
default=False,
help="Allow @pytest.mark.deferred_ac without an explicit reason= kwarg. "
"Default off — every deferred AC must cite its traceability-matrix row.",
)
# ---------------------------------------------------------------------------
# Parameterization matrix
# ---------------------------------------------------------------------------
_FC_ADAPTERS = ("ardupilot", "inav")
_VIO_STRATEGIES = ("okvis2", "klt_ransac", "vins_mono")
def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
"""Parametrize tests that request the ``fc_adapter`` / ``vio_strategy`` fixtures.
Tests opt in by listing the fixture name in their signature. Tests that
explicitly do not depend on the matrix simply do not request the fixture.
"""
if "fc_adapter" in metafunc.fixturenames:
env_default = os.environ.get("FC_ADAPTER")
if env_default:
metafunc.parametrize("fc_adapter", [env_default], ids=[env_default])
else:
metafunc.parametrize("fc_adapter", _FC_ADAPTERS, ids=_FC_ADAPTERS)
if "vio_strategy" in metafunc.fixturenames:
env_default = os.environ.get("VIO_STRATEGY")
if env_default:
metafunc.parametrize("vio_strategy", [env_default], ids=[env_default])
else:
metafunc.parametrize("vio_strategy", _VIO_STRATEGIES, ids=_VIO_STRATEGIES)
# ---------------------------------------------------------------------------
# Skip-rule enforcement (deterministic; runs at collection time)
# ---------------------------------------------------------------------------
def pytest_collection_modifyitems(
config: pytest.Config, items: list[pytest.Item]
) -> None:
"""Apply traceability-matrix-driven skips before any test executes.
The mapping between AC / RESTRICT IDs and the SKIP reason strings is the
one declared in `_docs/02_document/tests/traceability-matrix.md` §
Uncovered Items Analysis. Any change to that matrix MUST be mirrored
here (and vice-versa) — the unit tests in
`e2e/_unit_tests/test_traceability_skip_rules.py` catch drift.
"""
tier = os.environ.get("TIER", "tier1-docker")
chamber_enabled = config.getoption("--enable-chamber")
build_kind = config.getoption("--build-kind")
skip_tier2 = pytest.mark.skip(reason="Tier-2 only — Jetson hardware required")
skip_chamber = pytest.mark.skip(
reason="Chamber-gated — run with --enable-chamber on the chamber-attached Jetson runner"
)
skip_research = pytest.mark.skip(
reason="vins_mono is research-build-only per D-C1-1-SUB-A"
)
for item in items:
# ----- Tier-2 only -----
if "tier2_only" in item.keywords and tier != "tier2-jetson":
item.add_marker(skip_tier2)
continue
# ----- Chamber only -----
if "chamber_only" in item.keywords and not chamber_enabled:
item.add_marker(skip_chamber)
continue
# ----- Research-build vs production matrix -----
# Skip vins_mono on production-build runs (the marker is set on the
# parametrize id, not the test fn — we check the param id).
if build_kind == "production":
call_params = getattr(item, "callspec", None)
if call_params is not None and call_params.params.get("vio_strategy") == "vins_mono":
item.add_marker(skip_research)
continue
# ----- Deferred-AC traceability-matrix skips -----
deferred = item.get_closest_marker("deferred_ac")
if deferred is not None:
reason = deferred.kwargs.get("reason")
if reason is None and not config.getoption("--allow-no-skip-reason"):
# Hard failure at collection — every deferred_ac MUST cite its
# matrix row to prevent silent coverage erosion.
item.add_marker(
pytest.mark.skip(
reason=(
"deferred_ac marker without reason= kwarg; cite the "
"traceability-matrix row that justifies the deferral, "
"or run with --allow-no-skip-reason for local debugging."
)
)
)
continue
verdict = deferred.kwargs.get("verdict", "skip").lower()
if verdict == "xfail":
item.add_marker(pytest.mark.xfail(reason=reason or "deferred AC (xfail)", strict=False))
else:
item.add_marker(
pytest.mark.skip(
reason=(
reason
or "deferred AC — see _docs/02_document/tests/traceability-matrix.md"
)
)
)
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="session")
def run_id() -> str:
return os.environ.get("RUN_ID", "local")
@pytest.fixture(scope="session")
def tier() -> str:
return os.environ.get("TIER", "tier1-docker")
@pytest.fixture(scope="session")
def evidence_dir(pytestconfig: pytest.Config, run_id: str) -> Path:
base = Path(pytestconfig.getoption("--evidence-out"))
target = base if base.name == "evidence" else base / "evidence"
target.mkdir(parents=True, exist_ok=True)
return target
@pytest.fixture(scope="session")
def mock_suite_sat_url() -> str:
return os.environ.get("MOCK_SUITE_SAT_URL", "http://mock-suite-sat-service:8080")
# ---------------------------------------------------------------------------
# Plugin registration
# ---------------------------------------------------------------------------
# The CSV reporter plugin is a separate module so the unit tests can exercise
# it directly without going through a real pytest run. It is registered via
# `pytest_plugins` so docker-compose's `--csv=...` flag binds to our column
# set rather than the upstream pytest-csv default.
pytest_plugins = [
"runner.reporting.csv_reporter",
"runner.reporting.evidence_bundler",
]