Files
Oleksandr Bezdieniezhnykh 59d9116d36 [AZ-406] Blackbox test harness bootstrap (Tier-1 + Tier-2 scaffold)
Bootstraps the public-boundary blackbox test harness owned by epic
AZ-262 (E-BBT). Establishes the e2e/ directory tree at the repo root,
fully separated from src/gps_denied_onboard/** and from the in-process
tests/** tree, and commits to the contracts every subsequent test
ticket (AZ-407..AZ-446) builds against.

Tier-1 (workstation Docker):
- docker/docker-compose.test.yml wires SUT + ArduPilot SITL + iNav SITL
  + mock Suite Sat Service + mavproxy listener + e2e-runner onto one
  e2e-net bridge with internal: true (enforces RESTRICT-SAT-1 /
  NFT-SEC-02 egress isolation at the network layer).
- docker/docker-compose.tier2-bridge.yml override disables the in-
  compose SUT so Tier-2 pairs SITLs + mock + runner on an x86 host
  while the SUT runs natively on the Jetson under systemd.

Tier-2 (Jetson):
- jetson/run-tier2.sh + tier2.service systemd unit + tegrastats /
  jtop parsers feed per-sample telemetry into the evidence bundle.

Runner image (e2e/runner/):
- Dockerfile + requirements.txt install ONLY ground-side libs
  (pymavlink, opencv-python>=4.12, numpy/scipy/geopy/pyproj, httpx,
  orjson, pydantic, structlog, pytest 8.x). The runner deliberately
  does NOT install the SUT package.
- conftest.py implements the AC-9 skip-rule mapping (tier2_only,
  chamber_only, vins_mono, deferred_ac) tied to environment.md
  parametrize axes.
- reporting/csv_reporter.py is a pytest plugin emitting one row per
  test with the exact 11-column schema from environment.md §
  Reporting (test_id, test_name, traces_to, fc_adapter, vio_strategy,
  tier, started_at_utc, execution_time_ms, result, error_message,
  evidence_paths). XFAIL surfaced only when a test carries
  @pytest.mark.deferred_ac(verdict="xfail", reason=...).
- reporting/evidence_bundler.py exposes the attach_evidence fixture
  that copies per-test artifacts (.tlog, FDR archives, screenshots,
  tegrastats / jtop CSVs) into the run bundle and records relative
  paths into the reporter's evidence_paths column.
- helpers/{frame_source_replay,imu_replay,sitl_observer,
  mavproxy_tlog_reader,fdr_reader}.py declare the public surfaces
  (concrete implementations owned by AZ-407 / AZ-408 / AZ-416 /
  AZ-417 / AZ-441 per the dependency table); helpers/geo.py ships
  today (no downstream task dep) — WGS84 distance / forward-bearing
  / offset via pyproj with NaN rejection.

Mock Suite Sat Service (e2e/fixtures/mock-suite-sat/):
- FastAPI app: POST /tiles (ingest contract from D-PROJ-2 follow-up),
  GET /tiles/audit + /mock/audit (per-run read-back), POST
  /mock/config (force-status, response delay), POST /mock/reset
  (clears audit between tests), GET /mock/health.

Fixture scaffolds (e2e/fixtures/{tile-cache-builder, age-injector,
injectors, cold-boot, secrets, security}/):
- Public surfaces only. Concrete builders land in AZ-407 (static
  fixtures), AZ-408 (runtime synthetic injection), AZ-419 (cold-boot
  fixture), AZ-439 (CVE-2025-53644 JPEG generator).

Test tree (e2e/tests/{positive,negative,performance,resilience,
security,resource_limit}/):
- Mirror of the test-spec category grouping in
  _docs/02_document/tests/*-tests.md.
- tests/positive/test_smoke.py is the AC-1 harness-boot smoke run
  inside the e2e-runner image once Docker brings everything up.

Out-of-container unit tests (e2e/_unit_tests/):
- Exercises the harness internals (CSV reporter plugin lifecycle,
  conftest skip rules, helper modules, parsers, mock app, compose
  YAML structural contract, public-boundary enforcement) without
  Docker / SITL. 97 unit tests, all passing.

Build / config:
- pyproject.toml: testpaths extended with e2e/_unit_tests; pythonpath
  extended with e2e; fastapi>=0.111,<0.120 added to dev extras for the
  mock-app TestClient unit test.

AC coverage:
- AC-1 (Tier-1 boot)         → compose YAML test + directory layout
                                + smoke test (Docker-bound)
- AC-2 (mock services)       → 6 FastAPI TestClient unit tests
- AC-3 (SITLs accept output) → contract present; concrete check
                                deferred to AZ-416 / AZ-417
- AC-4 (CSV columns)         → in-process plugin lifecycle test
                                emits the exact 11-column schema
- AC-5 (egress isolation)    → static config test + runtime probe
                                in Docker-bound smoke
- AC-6 (Tier-2 contract)     → tegrastats + jtop parser unit tests
                                + jetson/* layout test; full Tier-2
                                contract is AZ-444
- AC-7 (fixture reproducibility) → deferred to AZ-407 per task spec
- AC-8 (parametrize matrix)  → vins_mono skip-rule cases +
                                tests/positive/test_smoke
- AC-9 (skip semantics)      → 9 conftest skip-rule unit tests

Module layout entry for blackbox_tests was added in 2026-05-16
preparatory commit d7a17a8 so this diff stays focused on the harness
scaffold. AZ-406 advances to In Testing on commit.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-16 16:22:44 +03:00

164 lines
5.6 KiB
Python

"""Mock Suite Satellite Service — FastAPI ingest stub for blackbox tests.
Endpoints:
POST /tiles — main ingest. Returns 202 on well-formed tile,
400 on malformed; appends to the run audit log.
GET /tiles/audit — read-back of the per-run audit log (JSONL).
POST /mock/config — test-time behaviour control (force 5xx, simulate downtime).
GET /mock/audit — alias of /tiles/audit with optional ?run_id filter.
POST /mock/reset — clears the audit log between tests for isolation.
GET /mock/health — Docker healthcheck.
The accepted ingest schema is the contract sketch from
`_docs/_process_leftovers/2026-05-09_satellite-provider-design-tasks.md`.
NFT-SEC-01 asserts the schema's accepted-fields match that sketch.
"""
from __future__ import annotations
import os
import time
import uuid
from pathlib import Path
from typing import Annotated, Literal
import orjson
from fastapi import FastAPI, HTTPException, Query
from fastapi.responses import ORJSONResponse, PlainTextResponse
from pydantic import BaseModel, Field, ValidationError
AUDIT_ROOT = Path(os.environ.get("MOCK_SUITE_SAT_AUDIT_PATH", "/audit"))
AUDIT_ROOT.mkdir(parents=True, exist_ok=True)
app = FastAPI(
title="mock-suite-sat-service",
version="0.1.0",
description="Deterministic stub of the parent Suite Satellite Service.",
default_response_class=ORJSONResponse,
)
# ---------------------------------------------------------------------------
# Behaviour control (test-only)
# ---------------------------------------------------------------------------
class _MockConfig(BaseModel):
force_status: int | None = Field(default=None, description="Force this status on every ingest.")
simulated_latency_ms: int = 0
_config = _MockConfig()
# ---------------------------------------------------------------------------
# Ingest schema (mirror of the contract sketch — keep them in sync)
# ---------------------------------------------------------------------------
class TileQualityMetadata(BaseModel):
capture_utc: str
source_provider: Literal["maxar", "planet", "sentinel-2", "skywatch", "operator-supplied"]
resolution_m_per_px: float = Field(gt=0, le=10.0)
cloud_coverage_pct: float = Field(ge=0, le=100)
geo_accuracy_m: float = Field(ge=0)
class TilePublishRequest(BaseModel):
tile_id: str = Field(min_length=8, max_length=128)
bbox_wgs84: tuple[float, float, float, float]
zoom_level: int = Field(ge=10, le=22)
descriptor_sha256: str = Field(min_length=64, max_length=64)
payload_size_bytes: int = Field(gt=0)
quality: TileQualityMetadata
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _run_audit_path(run_id: str) -> Path:
safe = "".join(c for c in run_id if c.isalnum() or c in "-_") or "default"
return AUDIT_ROOT / f"{safe}.jsonl"
def _append_audit(run_id: str, entry: dict[str, object]) -> None:
entry = {**entry, "received_at_unix": time.time(), "entry_id": str(uuid.uuid4())}
path = _run_audit_path(run_id)
with path.open("ab") as fh:
fh.write(orjson.dumps(entry))
fh.write(b"\n")
# ---------------------------------------------------------------------------
# Routes
# ---------------------------------------------------------------------------
@app.get("/mock/health")
def health() -> dict[str, str]:
return {"status": "ok"}
@app.post("/tiles", status_code=202)
def publish_tile(
request: TilePublishRequest,
run_id: Annotated[str, Query(alias="run_id")] = "default",
) -> dict[str, object]:
if _config.simulated_latency_ms > 0:
time.sleep(_config.simulated_latency_ms / 1000.0)
if _config.force_status is not None and _config.force_status >= 400:
raise HTTPException(
status_code=_config.force_status,
detail=f"forced status by /mock/config (current force_status={_config.force_status})",
)
_append_audit(
run_id,
{
"tile_id": request.tile_id,
"bbox_wgs84": list(request.bbox_wgs84),
"zoom_level": request.zoom_level,
"descriptor_sha256": request.descriptor_sha256,
"payload_size_bytes": request.payload_size_bytes,
"quality": request.quality.model_dump(),
},
)
return {"accepted": True, "tile_id": request.tile_id, "run_id": run_id}
@app.exception_handler(ValidationError)
def on_validation_error(_request, exc: ValidationError) -> ORJSONResponse: # type: ignore[no-untyped-def]
return ORJSONResponse(status_code=400, content={"detail": exc.errors()})
@app.get("/tiles/audit")
@app.get("/mock/audit")
def get_audit(run_id: Annotated[str, Query(alias="run_id")] = "default") -> ORJSONResponse:
path = _run_audit_path(run_id)
if not path.exists():
return ORJSONResponse(content={"run_id": run_id, "entries": []})
entries = []
with path.open("rb") as fh:
for line in fh:
line = line.strip()
if not line:
continue
entries.append(orjson.loads(line))
return ORJSONResponse(content={"run_id": run_id, "entries": entries})
@app.post("/mock/config")
def update_config(config: _MockConfig) -> _MockConfig:
global _config
_config = config
return _config
@app.post("/mock/reset")
def reset(run_id: Annotated[str, Query(alias="run_id")] = "default") -> PlainTextResponse:
path = _run_audit_path(run_id)
if path.exists():
path.unlink()
return PlainTextResponse("reset")