gps-denied-onboard/_docs/03_implementation/batch_109_cycle3_report.md

Batch 109 — Cycle 3 — AZ-840 e2e orchestrator test

Date: 2026-05-23 Tasks: AZ-840 (C4 — Epic AZ-835). Story points: 3 (per the task spec). Jira status: AZ-840 In Progress → In Testing at commit step.

Why this batch exists

Epic AZ-835 (real-flight e2e validation) needs a single Tier-2 test that proves the 7-step pipeline runs from (tlog, video, calibration) to a horizontal-error verdict without operator hand-curation between steps. Steps 3-5 were delivered by AZ-839 (C3 — operator_pre_flight_setup); steps 1-2-6-7 are this batch.

The AZ-839 batch 108b follow-up note explicitly anticipated this batch: "AZ-840 will additionally need to feed the airborne replay binary a config that points at the same cache_root ... the cleanest path is for AZ-840 to write an effective YAML at runtime from the same override recipe used here."

What this batch ships

A driver module + unit test suite + Tier-2 integration test:

• tests/e2e/replay/_e2e_orchestrator.py — wraps the AZ-699 verdict-report path with the AZ-839 C3 fixture's PopulatedC6Cache. Public surface:
  • OrchestratorStep enum — failure-step labels per AC-5.
  • OrchestrationFailure(step, message) exception — wraps every step failure with the step name in the message prefix.
  • OrchestrationReport dataclass — verdict, distribution, paths, wall-clock measurements per AC-4.
  • write_effective_replay_config — small helper that overlays c6_tile_cache.root_dir onto the static operator YAML.
  • read_calibration_acquisition_method — mirror of AZ-699's helper so the report writer keeps the same shape.
  • run_e2e_orchestration — the AC-1 entry point wiring validate → write_config → airborne subprocess → parse JSONL → load tlog GT → compute distribution → render report.
• tests/e2e/replay/test_e2e_orchestrator_unit.py — 17 unit tests covering each of the 7 steps' failure modes plus the happy path. The runner is injected (subprocess.run default) so unit tests stage synthetic JSONL output without touching the airborne binary. load_tlog_ground_truth is monkeypatched to return a synthetic 3-row series.
• tests/e2e/replay/test_az835_e2e_real_flight.py:: test_az840_e2e_real_flight_orchestration — Tier-2 + RUN_REPLAY_E2E gated test that consumes the C3 fixture + Derkachi inputs and asserts the verdict markdown is written, the threshold-hit share table is present, and the 15-min budget held.

AC coverage

AC	Description	Coverage
AC-1	Steps 1-7 end-to-end on Tier-2 from a fresh tlog/video	test_az840_e2e_real_flight_orchestration (Tier-2-gated); 17 unit tests prove the orchestrator structure
AC-2	Verdict report exists either PASS or FAIL	test_run_e2e_orchestration_writes_report_even_on_fail_verdict + integration assertion report_path.is_file()
AC-3	Reuses C3 fixture (operator_pre_flight_setup)	Integration test consumes the fixture; effective config overlay points at populated_cache.cache_root
AC-4	15-min wall-time soft target on the Derkachi clip	_DEFAULT_MAX_SECONDS = 900.0 passed as subprocess.run timeout; integration asserts replay_subprocess_seconds <= 900
AC-5	Mid-pipeline failure fails LOUD with a clear step prefix	OrchestratorStep enum + 8 step-specific failure unit tests (validate/write_config/airborne × 3/parse × 2/gt)
AC-6	Gated by RUN_REPLAY_E2E=1 + Tier-2 marker	_orchestrator_skip_reason() checks env vars + binary + video size; @pytest.mark.tier2 decorator
AC-7	AZ-699 verdict test continues to pass	No changes to test_derkachi_real_tlog.py; same real_flight_validation_<date>.md report path convention
AC-8	Unit-tested orchestration helper without Tier-2 inputs	17 unit tests covering config write (4) + calibration parse (3) + run helper (10) — all use mocked subprocess + GT loader

Test run results

$ .venv/bin/pytest tests/e2e/replay/ -v --tb=short --timeout=60
============================ 45 passed, 10 skipped, 3 warnings in 0.78s ============

Breakdown:

• 17 new orchestrator unit tests pass.
• 11 AZ-839 driver unit tests still pass (no driver changes).
• 14 helper unit tests (test_helpers.py) still pass.
• 3 derkachi-1min mode-agnostic AST tests still pass.
• 10 skips: 1 new Tier-2 (this AZ-840 integration), 6 RUN_REPLAY_E2E gated AZ-404 cases, 1 AC-8 D-PROJ-2 placeholder, 1 Tier-2 AZ-699, 1 Tier-2 AZ-839 integration. None are regressions; the tier2 gate trips off-Jetson.

Design notes

--auto-trim ownership

The orchestrator passes --auto-trim unconditionally so AZ-405 / AZ-698 active-flight-cut + tlog/video sync (Epic step 1) runs inside the airborne binary every time. The Epic narrative does not separate trim from the airborne pipeline; collapsing them into a single subprocess invocation matches AZ-699 and avoids duplicating the trim path.

clip_duration_s parity with AZ-699

run_e2e_orchestration computes clip_duration_s = ground_truth[-1].t_s - ground_truth[0].t_s exactly as test_derkachi_real_tlog.py does. This means both verdict reports name the same clip duration even when the trimmed video is shorter than the ground-truth window — a deliberate choice: the report header documents what the verdict covers, not what the binary processed.

Effective config write — single source of truth

write_effective_replay_config materialises the same override recipe AZ-839 uses in-memory, but on disk so the airborne subprocess sees the cache_root the fixture chose. Field-level merge: every other block in the operator YAML is preserved verbatim; only c6_tile_cache.root_dir and c6_tile_cache.faiss_index_path are overwritten. The static operator YAML on disk is never touched.

Failure surface = step prefix

OrchestrationFailure always prefixes its message with [<step>]. CI log scrapers and pytest's traceback printer both surface the prefix on the first line; AC-5 ("clear error pointing at the failing step") holds without requiring the test to inspect the exception object. The step is also exposed as exc.step for programmatic assertions.

Files changed

• tests/e2e/replay/_e2e_orchestrator.py (new, 656 LOC).
• tests/e2e/replay/test_e2e_orchestrator_unit.py (new, 660+ LOC).
• tests/e2e/replay/test_az835_e2e_real_flight.py (new, 156 LOC).

No src/ changes, no operator-config YAML changes, no AZ-839 driver changes. AZ-840 is purely additive at the test layer.

Code review (self-review)

Verdict: PASS_WITH_WARNINGS.

Phase	Result
1. Context loading	Re-read gps_compare.py, accuracy_report.py, replay_input.py, cli/replay.py, test_derkachi_real_tlog.py. Emission schema (emitted_at, position_wgs84) is the same shape gps-denied-replay writes.
2. Spec compliance	All 8 AZ-840 ACs covered; AC-7 holds by inspection (no AZ-699 changes).
3. Code quality	All public types have docstrings; failure messages name the upstream exception via repr so OSError / subprocess.TimeoutExpired carry through. Runner kw-args mirror subprocess.run signature 1:1.
4. Security quick-scan	Effective config write goes to a tmp file the test owns; no secrets in the YAML overlay (override is two string fields). Subprocess env is opt-in (None defaults to os.environ).
5. Performance scan	Unit tests run in 0.51 s. Tier-2 wall-clock cap is 900 s, enforced by the subprocess timeout.
6. Cross-task consistency	clip_duration_s and report_path match AZ-699 exactly so a single Jetson run produces the same markdown shape.
7. Architecture compliance	Orchestrator lives entirely under tests/e2e/replay/; no src/ writes. C3 fixture's invariants (PopulatedC6Cache.cache_root is the single source of truth) propagate via write_effective_replay_config.

Findings

ID	Severity	Description	Disposition
F1	Low	_default_tile_decoder in conftest.py (carried from batch 108) — still raw TIFF. Not in the AZ-840 path; AZ-840 doesn't change tile decoding.	Defer; no AZ-840 ticket.
F2	Low	_resolve_replay_descriptor_dim is NetVLAD-only (carried from batch 108). AZ-840 doesn't change descriptors.	Defer; no AZ-840 ticket.
F3	Low	--pace asap is hardcoded in _run_replay_subprocess argv; the AZ-699 test passes --pace asap too, so behaviour is identical. If a future test wants a real-time pace, the runner kwarg is the seam.	Document; no ticket.
F4	Low	_run_replay_subprocess does not stream stdout/stderr; failures surface only after the subprocess exits. For 15-min runs this means the operator sees no progress until the budget expires. AZ-699 has the same shape.	Document; consider an AZ-* if the budget grows.

Notes for follow-up

• AZ-840 lands the orchestrator test as Tier-2-gated. Verifying the Tier-2 path actually runs on the Jetson harness is the next gating step before Epic AZ-835 can flip from "covered by unit tests" to "covered by Tier-2 integration".
• _e2e_orchestrator.py is intentionally kept under tests/ rather than promoted to src/. If a second consumer of the same orchestration shape appears (e.g. AZ-833 mock-suite-sat parity test), the move to a shared helper module under src/gps_denied_onboard/replay/ is the right next step; for now the test-only location matches the helper's only consumer.
• AZ-841 (Tier-2 unxfail follow-up) and AZ-842 (replay protocol
  • orchestrator docs) sit downstream — both should reference this batch report in their planning sections.