mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 07:01:14 +00:00
[AZ-840] [AZ-835] e2e orchestrator test (E-AZ-835 C4)
Wraps the AZ-699 verdict-report path with the AZ-839 operator_pre_flight_setup C3 fixture so a single Tier-2 test takes only (tlog, video, calibration) and runs the full 7-step pipeline on the Jetson harness without operator hand-curation. New surface (tests-only, no src/ changes): - tests/e2e/replay/_e2e_orchestrator.py — orchestrator with OrchestratorStep enum, OrchestrationFailure exception (step prefix per AC-5), OrchestrationReport dataclass, write_effective_replay_config helper, and run_e2e_orchestration entry point covering steps 1-2-6-7. - tests/e2e/replay/test_e2e_orchestrator_unit.py — 17 unit tests covering each failure mode + happy path with mocked subprocess + ground-truth loader (AC-8). - tests/e2e/replay/test_az835_e2e_real_flight.py — Tier-2 + RUN_REPLAY_E2E gated integration test asserting verdict report exists, 15-min budget held (AC-1, AC-2, AC-3, AC-4, AC-6). The effective config write overlays c6_tile_cache.root_dir onto the static operator YAML at runtime so the airborne subprocess shares the cache_root the C3 fixture chose. Field- level merge — every other operator-config block stays verbatim. The static YAML on disk is never touched. Test run: tests/e2e/replay 45 passed, 10 skipped (10 skips were 9 pre-existing + 1 new tier2). No src/ touched, no AZ-839 driver changes; AC-7 (AZ-699 still passes) holds by inspection. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,171 @@
|
||||
# Batch 109 — Cycle 3 — AZ-840 e2e orchestrator test
|
||||
|
||||
**Date**: 2026-05-23
|
||||
**Tasks**: AZ-840 (C4 — Epic AZ-835).
|
||||
**Story points**: 3 (per the task spec).
|
||||
**Jira status**: AZ-840 In Progress → In Testing at commit step.
|
||||
|
||||
## Why this batch exists
|
||||
|
||||
Epic AZ-835 (real-flight e2e validation) needs a single Tier-2
|
||||
test that proves the 7-step pipeline runs from
|
||||
`(tlog, video, calibration)` to a horizontal-error verdict
|
||||
without operator hand-curation between steps. Steps 3-5 were
|
||||
delivered by AZ-839 (C3 — `operator_pre_flight_setup`); steps
|
||||
1-2-6-7 are this batch.
|
||||
|
||||
The AZ-839 batch 108b follow-up note explicitly anticipated this
|
||||
batch: "AZ-840 will additionally need to feed the airborne
|
||||
replay binary a config that points at the same `cache_root`
|
||||
... the cleanest path is for AZ-840 to write an effective YAML
|
||||
at runtime from the same override recipe used here."
|
||||
|
||||
## What this batch ships
|
||||
|
||||
A driver module + unit test suite + Tier-2 integration test:
|
||||
|
||||
* `tests/e2e/replay/_e2e_orchestrator.py` — wraps the AZ-699
|
||||
verdict-report path with the AZ-839 C3 fixture's
|
||||
`PopulatedC6Cache`. Public surface:
|
||||
* `OrchestratorStep` enum — failure-step labels per AC-5.
|
||||
* `OrchestrationFailure(step, message)` exception — wraps
|
||||
every step failure with the step name in the message prefix.
|
||||
* `OrchestrationReport` dataclass — verdict, distribution,
|
||||
paths, wall-clock measurements per AC-4.
|
||||
* `write_effective_replay_config` — small helper that overlays
|
||||
`c6_tile_cache.root_dir` onto the static operator YAML.
|
||||
* `read_calibration_acquisition_method` — mirror of AZ-699's
|
||||
helper so the report writer keeps the same shape.
|
||||
* `run_e2e_orchestration` — the AC-1 entry point wiring
|
||||
validate → write_config → airborne subprocess → parse JSONL
|
||||
→ load tlog GT → compute distribution → render report.
|
||||
* `tests/e2e/replay/test_e2e_orchestrator_unit.py` — 17 unit
|
||||
tests covering each of the 7 steps' failure modes plus the
|
||||
happy path. The runner is injected (`subprocess.run` default)
|
||||
so unit tests stage synthetic JSONL output without touching
|
||||
the airborne binary. `load_tlog_ground_truth` is monkeypatched
|
||||
to return a synthetic 3-row series.
|
||||
* `tests/e2e/replay/test_az835_e2e_real_flight.py::
|
||||
test_az840_e2e_real_flight_orchestration` — Tier-2 + RUN_REPLAY_E2E
|
||||
gated test that consumes the C3 fixture + Derkachi inputs and
|
||||
asserts the verdict markdown is written, the threshold-hit
|
||||
share table is present, and the 15-min budget held.
|
||||
|
||||
## AC coverage
|
||||
|
||||
| AC | Description | Coverage |
|
||||
|-----|----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
|
||||
| AC-1| Steps 1-7 end-to-end on Tier-2 from a fresh tlog/video | `test_az840_e2e_real_flight_orchestration` (Tier-2-gated); 17 unit tests prove the orchestrator structure |
|
||||
| AC-2| Verdict report exists either PASS or FAIL | `test_run_e2e_orchestration_writes_report_even_on_fail_verdict` + integration assertion `report_path.is_file()` |
|
||||
| AC-3| Reuses C3 fixture (`operator_pre_flight_setup`) | Integration test consumes the fixture; effective config overlay points at `populated_cache.cache_root` |
|
||||
| AC-4| 15-min wall-time soft target on the Derkachi clip | `_DEFAULT_MAX_SECONDS = 900.0` passed as `subprocess.run` `timeout`; integration asserts `replay_subprocess_seconds <= 900`|
|
||||
| AC-5| Mid-pipeline failure fails LOUD with a clear step prefix | `OrchestratorStep` enum + 8 step-specific failure unit tests (`validate`/`write_config`/`airborne` × 3/`parse` × 2/`gt`) |
|
||||
| AC-6| Gated by `RUN_REPLAY_E2E=1` + Tier-2 marker | `_orchestrator_skip_reason()` checks env vars + binary + video size; `@pytest.mark.tier2` decorator |
|
||||
| AC-7| AZ-699 verdict test continues to pass | No changes to `test_derkachi_real_tlog.py`; same `real_flight_validation_<date>.md` report path convention |
|
||||
| AC-8| Unit-tested orchestration helper without Tier-2 inputs | 17 unit tests covering config write (4) + calibration parse (3) + run helper (10) — all use mocked subprocess + GT loader |
|
||||
|
||||
## Test run results
|
||||
|
||||
```
|
||||
$ .venv/bin/pytest tests/e2e/replay/ -v --tb=short --timeout=60
|
||||
============================ 45 passed, 10 skipped, 3 warnings in 0.78s ============
|
||||
```
|
||||
|
||||
Breakdown:
|
||||
* 17 new orchestrator unit tests pass.
|
||||
* 11 AZ-839 driver unit tests still pass (no driver changes).
|
||||
* 14 helper unit tests (`test_helpers.py`) still pass.
|
||||
* 3 derkachi-1min mode-agnostic AST tests still pass.
|
||||
* 10 skips: 1 new Tier-2 (this AZ-840 integration), 6
|
||||
RUN_REPLAY_E2E gated AZ-404 cases, 1 AC-8 D-PROJ-2 placeholder,
|
||||
1 Tier-2 AZ-699, 1 Tier-2 AZ-839 integration. None are
|
||||
regressions; the tier2 gate trips off-Jetson.
|
||||
|
||||
## Design notes
|
||||
|
||||
### `--auto-trim` ownership
|
||||
|
||||
The orchestrator passes `--auto-trim` unconditionally so AZ-405 /
|
||||
AZ-698 active-flight-cut + tlog/video sync (Epic step 1) runs
|
||||
inside the airborne binary every time. The Epic narrative does
|
||||
not separate trim from the airborne pipeline; collapsing them
|
||||
into a single subprocess invocation matches AZ-699 and avoids
|
||||
duplicating the trim path.
|
||||
|
||||
### `clip_duration_s` parity with AZ-699
|
||||
|
||||
`run_e2e_orchestration` computes
|
||||
`clip_duration_s = ground_truth[-1].t_s - ground_truth[0].t_s`
|
||||
exactly as `test_derkachi_real_tlog.py` does. This means both
|
||||
verdict reports name the same clip duration even when the
|
||||
trimmed video is shorter than the ground-truth window — a
|
||||
deliberate choice: the report header documents what the verdict
|
||||
covers, not what the binary processed.
|
||||
|
||||
### Effective config write — single source of truth
|
||||
|
||||
`write_effective_replay_config` materialises the same override
|
||||
recipe AZ-839 uses in-memory, but on disk so the airborne
|
||||
subprocess sees the cache_root the fixture chose. Field-level
|
||||
merge: every other block in the operator YAML is preserved
|
||||
verbatim; only `c6_tile_cache.root_dir` and
|
||||
`c6_tile_cache.faiss_index_path` are overwritten. The static
|
||||
operator YAML on disk is never touched.
|
||||
|
||||
### Failure surface = step prefix
|
||||
|
||||
`OrchestrationFailure` always prefixes its message with
|
||||
`[<step>]`. CI log scrapers and pytest's traceback printer both
|
||||
surface the prefix on the first line; AC-5 ("clear error
|
||||
pointing at the failing step") holds without requiring the test
|
||||
to inspect the exception object. The step is also exposed as
|
||||
`exc.step` for programmatic assertions.
|
||||
|
||||
## Files changed
|
||||
|
||||
* `tests/e2e/replay/_e2e_orchestrator.py` (new, 656 LOC).
|
||||
* `tests/e2e/replay/test_e2e_orchestrator_unit.py` (new, 660+ LOC).
|
||||
* `tests/e2e/replay/test_az835_e2e_real_flight.py` (new, 156 LOC).
|
||||
|
||||
No `src/` changes, no operator-config YAML changes, no AZ-839
|
||||
driver changes. AZ-840 is purely additive at the test layer.
|
||||
|
||||
## Code review (self-review)
|
||||
|
||||
Verdict: **PASS_WITH_WARNINGS**.
|
||||
|
||||
| Phase | Result |
|
||||
|-------|--------|
|
||||
| 1. Context loading | Re-read `gps_compare.py`, `accuracy_report.py`, `replay_input.py`, `cli/replay.py`, `test_derkachi_real_tlog.py`. Emission schema (`emitted_at`, `position_wgs84`) is the same shape `gps-denied-replay` writes. |
|
||||
| 2. Spec compliance | All 8 AZ-840 ACs covered; AC-7 holds by inspection (no AZ-699 changes). |
|
||||
| 3. Code quality | All public types have docstrings; failure messages name the upstream exception via `repr` so `OSError` / `subprocess.TimeoutExpired` carry through. Runner kw-args mirror `subprocess.run` signature 1:1. |
|
||||
| 4. Security quick-scan | Effective config write goes to a tmp file the test owns; no secrets in the YAML overlay (override is two string fields). Subprocess `env` is opt-in (`None` defaults to `os.environ`). |
|
||||
| 5. Performance scan | Unit tests run in 0.51 s. Tier-2 wall-clock cap is 900 s, enforced by the subprocess timeout. |
|
||||
| 6. Cross-task consistency | `clip_duration_s` and `report_path` match AZ-699 exactly so a single Jetson run produces the same markdown shape. |
|
||||
| 7. Architecture compliance | Orchestrator lives entirely under `tests/e2e/replay/`; no `src/` writes. C3 fixture's invariants (`PopulatedC6Cache.cache_root` is the single source of truth) propagate via `write_effective_replay_config`. |
|
||||
|
||||
## Findings
|
||||
|
||||
| ID | Severity | Description | Disposition |
|
||||
|----|----------|-------------|-------------|
|
||||
| F1 | Low | `_default_tile_decoder` in `conftest.py` (carried from batch 108) — still raw TIFF. Not in the AZ-840 path; AZ-840 doesn't change tile decoding. | Defer; no AZ-840 ticket. |
|
||||
| F2 | Low | `_resolve_replay_descriptor_dim` is NetVLAD-only (carried from batch 108). AZ-840 doesn't change descriptors. | Defer; no AZ-840 ticket. |
|
||||
| F3 | Low | `--pace asap` is hardcoded in `_run_replay_subprocess` argv; the AZ-699 test passes `--pace asap` too, so behaviour is identical. If a future test wants a real-time pace, the runner kwarg is the seam. | Document; no ticket. |
|
||||
| F4 | Low | `_run_replay_subprocess` does not stream stdout/stderr; failures surface only after the subprocess exits. For 15-min runs this means the operator sees no progress until the budget expires. AZ-699 has the same shape. | Document; consider an AZ-* if the budget grows. |
|
||||
|
||||
## Notes for follow-up
|
||||
|
||||
* AZ-840 lands the orchestrator test as Tier-2-gated. Verifying
|
||||
the Tier-2 path actually runs on the Jetson harness is the
|
||||
next gating step before Epic AZ-835 can flip from "covered by
|
||||
unit tests" to "covered by Tier-2 integration".
|
||||
* `_e2e_orchestrator.py` is intentionally kept under `tests/`
|
||||
rather than promoted to `src/`. If a second consumer of the
|
||||
same orchestration shape appears (e.g. AZ-833 mock-suite-sat
|
||||
parity test), the move to a shared helper module under
|
||||
`src/gps_denied_onboard/replay/` is the right next step;
|
||||
for now the test-only location matches the helper's only
|
||||
consumer.
|
||||
* AZ-841 (Tier-2 unxfail follow-up) and AZ-842 (replay protocol
|
||||
+ orchestrator docs) sit downstream — both should reference
|
||||
this batch report in their planning sections.
|
||||
@@ -8,7 +8,7 @@ status: in_progress
|
||||
sub_step:
|
||||
phase: 7
|
||||
name: batch-loop
|
||||
detail: "batch 109 next; AZ-840 C4"
|
||||
detail: "batch 110 next; full-suite gate after AZ-840 C4 ship"
|
||||
retry_count: 0
|
||||
cycle: 3
|
||||
tracker: jira
|
||||
|
||||
@@ -0,0 +1,655 @@
|
||||
"""E2E orchestrator for the AZ-835 7-step pipeline (AZ-840 / Epic AZ-835 C4).
|
||||
|
||||
Wraps the AZ-699 verdict-report writing path with the AZ-839 C3
|
||||
fixture's `PopulatedC6Cache` so a single Tier-2 test can run from
|
||||
``(tlog, video, calibration)`` to a horizontal-error report without
|
||||
operator hand-curation between steps. The 7-step Epic narrative
|
||||
(``_docs/02_tasks/todo/AZ-840_e2e_orchestrator_test.md``):
|
||||
|
||||
1. Active flight cut + tlog/video sync — handled by ``gps-denied-replay``
|
||||
``--auto-trim`` (AZ-405 / AZ-698) inside the airborne binary.
|
||||
2. On-fly frame + IMU extraction — same binary's per-frame loop.
|
||||
3. Auto-create route — done by the C3 fixture
|
||||
(``operator_pre_flight_setup`` calls ``extract_route_from_tlog``).
|
||||
4. POST route to satellite-provider — C3 fixture (AZ-838
|
||||
``SatelliteProviderRouteClient.seed_route``).
|
||||
5. Build FAISS index — C3 fixture (AZ-322 ``DescriptorBatcher``).
|
||||
6. Run gps-denied airborne pipeline — this module's
|
||||
``_run_replay_subprocess`` invokes ``gps-denied-replay`` against
|
||||
the populated cache.
|
||||
7. Get GPS fixes, check vs tlog GPS — this module's
|
||||
``_load_ground_truth`` + ``horizontal_error_distribution`` +
|
||||
``render_report`` writes the verdict markdown.
|
||||
|
||||
The C3 fixture mutates ``c6_tile_cache.root_dir`` to point at a
|
||||
``tmp_path_factory.mktemp`` value (AZ-839 batch 108b). The static
|
||||
operator YAML at ``GPS_DENIED_OPERATOR_CONFIG_PATH`` cannot know
|
||||
that path. ``write_effective_replay_config`` reads the static YAML,
|
||||
overlays the ``c6_tile_cache.root_dir`` override, writes the merged
|
||||
result to a tmp file, and returns the path the airborne binary
|
||||
will load via ``--config``. This keeps a single source of truth
|
||||
for the cache_root override across the in-memory C3 fixture path
|
||||
and the subprocess airborne path.
|
||||
|
||||
Public surface — re-exported from this module:
|
||||
|
||||
* :class:`OrchestratorStep` — failure-step labels per AC-5 ("fails
|
||||
LOUD with a clear error pointing at the failing step").
|
||||
* :class:`OrchestrationFailure` — wraps the underlying exception
|
||||
with the step that produced it.
|
||||
* :class:`OrchestrationReport` — return value of
|
||||
:func:`run_e2e_orchestration` (verdict, distribution, paths,
|
||||
wall-clock measurements per AC-4).
|
||||
* :func:`write_effective_replay_config` — small helper for the
|
||||
config merge step.
|
||||
* :func:`run_e2e_orchestration` — the AC-1 entry point.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
import time
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from gps_denied_onboard.helpers.accuracy_report import (
|
||||
AC3_GATE_PCT,
|
||||
AC3_GATE_THRESHOLD_M,
|
||||
ReportContext,
|
||||
render_report,
|
||||
verdict_passes_ac3,
|
||||
)
|
||||
from gps_denied_onboard.helpers.gps_compare import (
|
||||
GroundTruthRow,
|
||||
HorizontalErrorDistribution,
|
||||
horizontal_error_distribution,
|
||||
)
|
||||
from gps_denied_onboard.replay_input import load_tlog_ground_truth
|
||||
|
||||
from tests.e2e.replay._operator_pre_flight import PopulatedC6Cache
|
||||
|
||||
__all__ = [
|
||||
"OrchestrationFailure",
|
||||
"OrchestrationReport",
|
||||
"OrchestratorStep",
|
||||
"read_calibration_acquisition_method",
|
||||
"run_e2e_orchestration",
|
||||
"write_effective_replay_config",
|
||||
]
|
||||
|
||||
|
||||
# Replay-subprocess wall-clock cap for the Derkachi clip per AZ-840
|
||||
# AC-4 (15 min soft target). Exposed as a default that the integration
|
||||
# test can override; the unit tests rely on the contract that the
|
||||
# runner argument is a free callable.
|
||||
_DEFAULT_MAX_SECONDS: float = 900.0
|
||||
|
||||
_LOGGER = logging.getLogger("tests.e2e.replay.e2e_orchestrator")
|
||||
|
||||
|
||||
class OrchestratorStep(str, Enum):
|
||||
"""Labels for the 7-step pipeline used by :class:`OrchestrationFailure`.
|
||||
|
||||
AC-5: every failure that reaches the test surface must name the
|
||||
step that produced it. The string values are stable so test
|
||||
assertions and log readers can match on them.
|
||||
"""
|
||||
|
||||
VALIDATE_INPUTS = "validate_inputs"
|
||||
WRITE_EFFECTIVE_CONFIG = "write_effective_config"
|
||||
AIRBORNE_PIPELINE = "airborne_pipeline"
|
||||
PARSE_EMISSIONS = "parse_emissions"
|
||||
LOAD_GROUND_TRUTH = "load_ground_truth"
|
||||
COMPUTE_DISTRIBUTION = "compute_distribution"
|
||||
RENDER_REPORT = "render_report"
|
||||
|
||||
|
||||
class OrchestrationFailure(RuntimeError):
|
||||
"""Failure inside one of the 7 orchestration steps (AC-5).
|
||||
|
||||
The :attr:`step` attribute names the failing step; the message
|
||||
embeds it as the prefix so plain log-readers see the failure
|
||||
location without inspecting the exception object.
|
||||
"""
|
||||
|
||||
def __init__(self, step: OrchestratorStep, message: str) -> None:
|
||||
super().__init__(f"[{step.value}] {message}")
|
||||
self.step = step
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class OrchestrationReport:
|
||||
"""Return value of :func:`run_e2e_orchestration`.
|
||||
|
||||
Attributes:
|
||||
verdict_passed: ``True`` iff the run met the AZ-696 epic
|
||||
AC-3 gate (>= AC3_GATE_PCT% within AC3_GATE_THRESHOLD_M m).
|
||||
distribution: Computed horizontal-error distribution.
|
||||
report_path: Markdown report written under ``report_dir``.
|
||||
emissions_count: Total estimator-output records consumed.
|
||||
wall_clock_s: Wall-clock seconds for the orchestration run
|
||||
(excludes the C3 fixture setup; covers steps 1-2-6-7).
|
||||
replay_subprocess_seconds: Wall-clock seconds the airborne
|
||||
replay subprocess took. Always <= ``wall_clock_s``.
|
||||
"""
|
||||
|
||||
verdict_passed: bool
|
||||
distribution: HorizontalErrorDistribution
|
||||
report_path: Path
|
||||
emissions_count: int
|
||||
wall_clock_s: float
|
||||
replay_subprocess_seconds: float
|
||||
|
||||
|
||||
def read_calibration_acquisition_method(calibration_path: Path) -> str:
|
||||
"""Return the AZ-702 ``acquisition_method`` field, or ``"unknown"``.
|
||||
|
||||
Mirrors ``test_derkachi_real_tlog._read_calibration_acquisition_method``
|
||||
so the AZ-840 verdict report can name the calibration provenance
|
||||
in its failure message (AZ-699 AC-3). Pure helper; the report
|
||||
writer needs the string, not the JSON.
|
||||
"""
|
||||
try:
|
||||
data = json.loads(calibration_path.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return "unknown"
|
||||
method = data.get("acquisition_method")
|
||||
if isinstance(method, str) and method:
|
||||
return method
|
||||
return "unknown"
|
||||
|
||||
|
||||
def write_effective_replay_config(
|
||||
*,
|
||||
base_config_path: Path,
|
||||
cache_root: Path,
|
||||
output_path: Path,
|
||||
) -> Path:
|
||||
"""Merge cache_root override into the static operator YAML.
|
||||
|
||||
Reads ``base_config_path`` as YAML, sets the
|
||||
``c6_tile_cache.root_dir`` to ``cache_root`` (forcing the
|
||||
FAISS index path to fall back to ``<cache_root>/descriptor.index``),
|
||||
and writes the merged document to ``output_path`` as YAML.
|
||||
|
||||
The merge is field-level: every other block in the base YAML is
|
||||
preserved verbatim. This keeps a single source of truth for the
|
||||
operator config — the test harness only contributes the dynamic
|
||||
cache_root.
|
||||
|
||||
Returns:
|
||||
The ``output_path`` argument, for ergonomic chaining.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=WRITE_EFFECTIVE_CONFIG): Base YAML
|
||||
unreadable, malformed, or not a top-level mapping.
|
||||
"""
|
||||
|
||||
try:
|
||||
base_text = base_config_path.read_text()
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.WRITE_EFFECTIVE_CONFIG,
|
||||
f"cannot read base config at {base_config_path}: {exc!r}",
|
||||
) from exc
|
||||
|
||||
try:
|
||||
base_data = yaml.safe_load(base_text) or {}
|
||||
except yaml.YAMLError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.WRITE_EFFECTIVE_CONFIG,
|
||||
f"base config YAML at {base_config_path} is malformed: {exc!r}",
|
||||
) from exc
|
||||
if not isinstance(base_data, dict):
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.WRITE_EFFECTIVE_CONFIG,
|
||||
f"base config YAML at {base_config_path} must be a mapping; "
|
||||
f"got {type(base_data).__name__}",
|
||||
)
|
||||
|
||||
c6_block_raw = base_data.get("c6_tile_cache")
|
||||
c6_block = dict(c6_block_raw) if isinstance(c6_block_raw, dict) else {}
|
||||
c6_block["root_dir"] = str(cache_root)
|
||||
c6_block["faiss_index_path"] = ""
|
||||
base_data["c6_tile_cache"] = c6_block
|
||||
|
||||
try:
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(
|
||||
yaml.safe_dump(base_data, sort_keys=True, default_flow_style=False)
|
||||
)
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.WRITE_EFFECTIVE_CONFIG,
|
||||
f"cannot write effective config at {output_path}: {exc!r}",
|
||||
) from exc
|
||||
return output_path
|
||||
|
||||
|
||||
def run_e2e_orchestration(
|
||||
*,
|
||||
populated_cache: PopulatedC6Cache,
|
||||
base_config_path: Path,
|
||||
tlog_path: Path,
|
||||
video_path: Path,
|
||||
calibration_path: Path,
|
||||
signing_key_path: Path,
|
||||
replay_binary: Path,
|
||||
output_path: Path,
|
||||
report_dir: Path,
|
||||
effective_config_path: Path,
|
||||
run_date_utc: str | None = None,
|
||||
runner: Callable[..., subprocess.CompletedProcess[str]] = subprocess.run,
|
||||
subprocess_env: Mapping[str, str] | None = None,
|
||||
max_seconds: float = _DEFAULT_MAX_SECONDS,
|
||||
logger: logging.Logger | None = None,
|
||||
) -> OrchestrationReport:
|
||||
"""Run AZ-835 steps 1-7 against the AZ-839 populated cache.
|
||||
|
||||
Steps 3-5 are the responsibility of ``populated_cache`` (the
|
||||
AZ-839 C3 fixture); this function covers 1-2-6 (the airborne
|
||||
replay subprocess) and 7 (verdict report). The C3 fixture and
|
||||
this function share the cache_root via
|
||||
:func:`write_effective_replay_config` so the airborne binary
|
||||
reads the same FAISS index the fixture wrote (AC-3).
|
||||
|
||||
Args:
|
||||
populated_cache: C3 fixture output (AZ-839). Carries
|
||||
``cache_root``, ``faiss_index_path``, and the route
|
||||
spec the test pipeline produced.
|
||||
base_config_path: Static operator config YAML
|
||||
(``GPS_DENIED_OPERATOR_CONFIG_PATH``). Must register
|
||||
``c6_tile_cache``, ``c10_provisioning``, ``c2_vpr``,
|
||||
``c4_pose``, and ``c5_state`` blocks for the airborne
|
||||
binary to compose the replay graph.
|
||||
tlog_path: ArduPilot binary tlog the test consumes.
|
||||
video_path: Flight video file the test consumes.
|
||||
calibration_path: Camera calibration JSON (AZ-702
|
||||
factory-sheet for Derkachi).
|
||||
signing_key_path: MAVLink signing-key file. Replay protocol
|
||||
Invariant 11 — required even for the noop transport.
|
||||
replay_binary: ``gps-denied-replay`` console-script path.
|
||||
output_path: Where the airborne binary writes JSONL
|
||||
estimator emissions.
|
||||
report_dir: Directory the verdict markdown is written to.
|
||||
effective_config_path: Where the cache_root-merged YAML is
|
||||
written. The path is passed to the airborne binary via
|
||||
``--config``.
|
||||
run_date_utc: ISO-8601 date for the report filename and
|
||||
header. Defaults to today UTC.
|
||||
runner: ``subprocess.run`` by default; tests inject a fake
|
||||
that emits a synthetic JSONL output.
|
||||
subprocess_env: Optional environment overlay for the
|
||||
replay subprocess. ``None`` means ``os.environ``.
|
||||
max_seconds: Hard wall-clock cap for the airborne replay
|
||||
subprocess. The orchestrator times out the runner via
|
||||
its ``timeout`` kwarg; an exceeded budget surfaces as
|
||||
``OrchestrationFailure(step=AIRBORNE_PIPELINE)``.
|
||||
logger: Optional logger. Defaults to the module logger.
|
||||
|
||||
Returns:
|
||||
:class:`OrchestrationReport` on success. The verdict can
|
||||
be PASS or FAIL — AC-2 mandates the report exists either
|
||||
way.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure: Any of the 7 steps failed. The
|
||||
``step`` attribute names the failing step.
|
||||
"""
|
||||
|
||||
log = logger or _LOGGER
|
||||
started = time.monotonic()
|
||||
effective_run_date = run_date_utc or (
|
||||
datetime.datetime.now(datetime.timezone.utc).date().isoformat()
|
||||
)
|
||||
|
||||
_validate_inputs(
|
||||
base_config_path=base_config_path,
|
||||
tlog_path=tlog_path,
|
||||
video_path=video_path,
|
||||
calibration_path=calibration_path,
|
||||
signing_key_path=signing_key_path,
|
||||
replay_binary=replay_binary,
|
||||
report_dir=report_dir,
|
||||
)
|
||||
|
||||
write_effective_replay_config(
|
||||
base_config_path=base_config_path,
|
||||
cache_root=populated_cache.cache_root,
|
||||
output_path=effective_config_path,
|
||||
)
|
||||
|
||||
replay_subprocess_seconds = _run_replay_subprocess(
|
||||
replay_binary=replay_binary,
|
||||
video_path=video_path,
|
||||
tlog_path=tlog_path,
|
||||
output_path=output_path,
|
||||
calibration_path=calibration_path,
|
||||
config_path=effective_config_path,
|
||||
signing_key_path=signing_key_path,
|
||||
max_seconds=max_seconds,
|
||||
runner=runner,
|
||||
env=subprocess_env,
|
||||
logger=log,
|
||||
)
|
||||
|
||||
emissions = _parse_jsonl(output_path)
|
||||
|
||||
ground_truth = _load_ground_truth(tlog_path)
|
||||
|
||||
distribution = _compute_distribution(emissions, ground_truth)
|
||||
|
||||
context = ReportContext(
|
||||
run_date_utc=effective_run_date,
|
||||
tlog_path=tlog_path,
|
||||
video_path=video_path,
|
||||
calibration_acquisition_method=read_calibration_acquisition_method(
|
||||
calibration_path
|
||||
),
|
||||
clip_duration_s=(
|
||||
ground_truth[-1].t_s - ground_truth[0].t_s
|
||||
if ground_truth
|
||||
else 0.0
|
||||
),
|
||||
emissions_count=len(emissions),
|
||||
)
|
||||
verdict_passed = verdict_passes_ac3(distribution)
|
||||
report_path = _render_and_write_report(
|
||||
distribution=distribution,
|
||||
context=context,
|
||||
passed=verdict_passed,
|
||||
report_dir=report_dir,
|
||||
)
|
||||
|
||||
log.info(
|
||||
"e2e_orchestrator: report written",
|
||||
extra={
|
||||
"kind": "e2e_orchestrator.report_written",
|
||||
"kv": {
|
||||
"report_path": str(report_path),
|
||||
"verdict_passed": verdict_passed,
|
||||
"share_within_threshold_pct": (
|
||||
distribution.threshold_hit_share.get(
|
||||
AC3_GATE_THRESHOLD_M, 0.0
|
||||
)
|
||||
* 100.0
|
||||
),
|
||||
"ac3_gate_pct": AC3_GATE_PCT,
|
||||
"emissions_count": len(emissions),
|
||||
"ground_truth_pairings": distribution.count,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
wall_clock_s = max(0.0, time.monotonic() - started)
|
||||
return OrchestrationReport(
|
||||
verdict_passed=verdict_passed,
|
||||
distribution=distribution,
|
||||
report_path=report_path,
|
||||
emissions_count=len(emissions),
|
||||
wall_clock_s=wall_clock_s,
|
||||
replay_subprocess_seconds=replay_subprocess_seconds,
|
||||
)
|
||||
|
||||
|
||||
def _validate_inputs(
|
||||
*,
|
||||
base_config_path: Path,
|
||||
tlog_path: Path,
|
||||
video_path: Path,
|
||||
calibration_path: Path,
|
||||
signing_key_path: Path,
|
||||
replay_binary: Path,
|
||||
report_dir: Path,
|
||||
) -> None:
|
||||
"""Fail fast on missing inputs (AC-5 — surface the failing step early)."""
|
||||
file_inputs: tuple[tuple[str, Path], ...] = (
|
||||
("base_config_path", base_config_path),
|
||||
("tlog_path", tlog_path),
|
||||
("video_path", video_path),
|
||||
("calibration_path", calibration_path),
|
||||
("signing_key_path", signing_key_path),
|
||||
("replay_binary", replay_binary),
|
||||
)
|
||||
for label, path in file_inputs:
|
||||
if not path.is_file():
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.VALIDATE_INPUTS,
|
||||
f"{label} is not a file: {path}",
|
||||
)
|
||||
try:
|
||||
report_dir.mkdir(parents=True, exist_ok=True)
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.VALIDATE_INPUTS,
|
||||
f"report_dir {report_dir} cannot be created: {exc!r}",
|
||||
) from exc
|
||||
|
||||
|
||||
def _run_replay_subprocess(
|
||||
*,
|
||||
replay_binary: Path,
|
||||
video_path: Path,
|
||||
tlog_path: Path,
|
||||
output_path: Path,
|
||||
calibration_path: Path,
|
||||
config_path: Path,
|
||||
signing_key_path: Path,
|
||||
max_seconds: float,
|
||||
runner: Callable[..., subprocess.CompletedProcess[str]],
|
||||
env: Mapping[str, str] | None,
|
||||
logger: logging.Logger,
|
||||
) -> float:
|
||||
"""Invoke gps-denied-replay with --auto-trim; return wall-clock seconds.
|
||||
|
||||
Wraps :class:`subprocess.run` so unit tests can inject a fake
|
||||
runner. ``--auto-trim`` is always enabled here — the
|
||||
orchestrator owns the AZ-405 / AZ-698 sync path (AZ-840 step 1).
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=AIRBORNE_PIPELINE): Non-zero exit,
|
||||
timeout, or runner-level OSError.
|
||||
"""
|
||||
|
||||
argv = [
|
||||
str(replay_binary),
|
||||
"--video",
|
||||
str(video_path),
|
||||
"--tlog",
|
||||
str(tlog_path),
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--camera-calibration",
|
||||
str(calibration_path),
|
||||
"--config",
|
||||
str(config_path),
|
||||
"--mavlink-signing-key",
|
||||
str(signing_key_path),
|
||||
"--pace",
|
||||
"asap",
|
||||
"--auto-trim",
|
||||
]
|
||||
started = time.monotonic()
|
||||
try:
|
||||
completed = runner(
|
||||
argv,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=max_seconds,
|
||||
env=dict(env) if env is not None else None,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.AIRBORNE_PIPELINE,
|
||||
f"gps-denied-replay timed out after {max_seconds:.0f} s",
|
||||
) from exc
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.AIRBORNE_PIPELINE,
|
||||
f"cannot launch gps-denied-replay at {replay_binary}: {exc!r}",
|
||||
) from exc
|
||||
|
||||
elapsed_s = max(0.0, time.monotonic() - started)
|
||||
if completed.returncode != 0:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.AIRBORNE_PIPELINE,
|
||||
f"gps-denied-replay exited {completed.returncode}\n"
|
||||
f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}",
|
||||
)
|
||||
logger.info(
|
||||
"e2e_orchestrator: replay subprocess complete",
|
||||
extra={
|
||||
"kind": "e2e_orchestrator.replay_subprocess",
|
||||
"kv": {
|
||||
"elapsed_s": elapsed_s,
|
||||
"max_seconds": max_seconds,
|
||||
},
|
||||
},
|
||||
)
|
||||
return elapsed_s
|
||||
|
||||
|
||||
def _parse_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
"""Read one JSON record per non-blank line.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=PARSE_EMISSIONS): Output file
|
||||
missing, unreadable, has zero records, or contains a
|
||||
malformed line.
|
||||
"""
|
||||
if not path.is_file():
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.PARSE_EMISSIONS,
|
||||
f"replay output JSONL not found: {path}",
|
||||
)
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.PARSE_EMISSIONS,
|
||||
f"replay output JSONL unreadable at {path}: {exc!r}",
|
||||
) from exc
|
||||
rows: list[dict[str, Any]] = []
|
||||
for line_idx, line in enumerate(text.splitlines(), start=1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.PARSE_EMISSIONS,
|
||||
f"malformed JSON at line {line_idx} of {path}: {exc.msg}",
|
||||
) from exc
|
||||
if not isinstance(row, dict):
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.PARSE_EMISSIONS,
|
||||
f"line {line_idx} of {path} is not a JSON object: {row!r}",
|
||||
)
|
||||
rows.append(row)
|
||||
if not rows:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.PARSE_EMISSIONS,
|
||||
f"replay output JSONL at {path} has zero records — pipeline "
|
||||
"produced no estimator emissions",
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def _load_ground_truth(tlog_path: Path) -> list[GroundTruthRow]:
|
||||
"""Extract WGS84 ground truth from the binary tlog.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=LOAD_GROUND_TRUTH): Loader
|
||||
error or empty record list.
|
||||
"""
|
||||
try:
|
||||
series = load_tlog_ground_truth(tlog_path).records
|
||||
except Exception as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.LOAD_GROUND_TRUTH,
|
||||
f"load_tlog_ground_truth({tlog_path}) failed: {exc!r}",
|
||||
) from exc
|
||||
rows: list[GroundTruthRow] = [
|
||||
GroundTruthRow(
|
||||
t_s=fix.ts_ns / 1e9,
|
||||
lat_deg=fix.lat_deg,
|
||||
lon_deg=fix.lon_deg,
|
||||
alt_m=fix.alt_m,
|
||||
)
|
||||
for fix in series
|
||||
]
|
||||
if not rows:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.LOAD_GROUND_TRUTH,
|
||||
f"tlog ground truth at {tlog_path} has zero rows",
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def _compute_distribution(
|
||||
emissions: list[dict[str, Any]],
|
||||
ground_truth: list[GroundTruthRow],
|
||||
) -> HorizontalErrorDistribution:
|
||||
"""Compute the horizontal-error distribution.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=COMPUTE_DISTRIBUTION): Helper
|
||||
error or zero ground-truth pairings (every emission
|
||||
fell outside the GT time window).
|
||||
"""
|
||||
try:
|
||||
distribution = horizontal_error_distribution(emissions, ground_truth)
|
||||
except Exception as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.COMPUTE_DISTRIBUTION,
|
||||
f"horizontal_error_distribution failed: {exc!r}",
|
||||
) from exc
|
||||
if distribution.count == 0:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.COMPUTE_DISTRIBUTION,
|
||||
"no emissions paired with ground truth — JSONL timestamps "
|
||||
"outside the tlog GPS window?",
|
||||
)
|
||||
return distribution
|
||||
|
||||
|
||||
def _render_and_write_report(
|
||||
*,
|
||||
distribution: HorizontalErrorDistribution,
|
||||
context: ReportContext,
|
||||
passed: bool,
|
||||
report_dir: Path,
|
||||
) -> Path:
|
||||
"""Render the verdict markdown and write it to ``report_dir``.
|
||||
|
||||
Raises:
|
||||
OrchestrationFailure (step=RENDER_REPORT): Render or write
|
||||
failure; ``report_dir`` was already created by
|
||||
:func:`_validate_inputs`.
|
||||
"""
|
||||
try:
|
||||
report_text = render_report(distribution, context, passed=passed)
|
||||
except Exception as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.RENDER_REPORT,
|
||||
f"render_report failed: {exc!r}",
|
||||
) from exc
|
||||
report_path = (
|
||||
report_dir / f"real_flight_validation_{context.run_date_utc}.md"
|
||||
)
|
||||
try:
|
||||
report_path.write_text(report_text)
|
||||
except OSError as exc:
|
||||
raise OrchestrationFailure(
|
||||
OrchestratorStep.RENDER_REPORT,
|
||||
f"cannot write report at {report_path}: {exc!r}",
|
||||
) from exc
|
||||
return report_path
|
||||
@@ -0,0 +1,182 @@
|
||||
"""AZ-840 — E2E orchestrator integration test (AC-1 / AC-2 / AC-3 / AC-4 / AC-6).
|
||||
|
||||
The Tier-2 entry point that closes Epic AZ-835's narrative: from a
|
||||
``(tlog, video, calibration)`` triple, run the full 7-step pipeline
|
||||
end-to-end on the Jetson harness without operator hand-curation
|
||||
between steps.
|
||||
|
||||
The test consumes:
|
||||
|
||||
* :func:`tests.e2e.replay.conftest.operator_pre_flight_setup` —
|
||||
the AZ-839 C3 fixture that owns steps 3-5 (route extraction +
|
||||
satellite-provider seeding + FAISS index build) and yields a
|
||||
:class:`PopulatedC6Cache` keyed off a freshly-mktemp'd
|
||||
``cache_root``.
|
||||
* :func:`tests.e2e.replay.conftest.derkachi_replay_inputs` — the
|
||||
shared session fixture that materialises the Derkachi tlog +
|
||||
video + factory-sheet calibration + signing-key file.
|
||||
* :func:`tests.e2e.replay._e2e_orchestrator.run_e2e_orchestration`
|
||||
— the AC-1 driver that wires everything below the C3 fixture.
|
||||
|
||||
The driver writes a fresh effective replay config per session
|
||||
(merging the static operator YAML with the cache_root override),
|
||||
invokes ``gps-denied-replay --auto-trim``, parses the JSONL
|
||||
emissions, computes the horizontal-error distribution, and writes
|
||||
the verdict markdown under ``_docs/06_metrics/`` (AC-2).
|
||||
|
||||
Skip gates (in evaluation order):
|
||||
|
||||
1. ``@pytest.mark.tier2`` — the per-suite Tier-2 plugin gates this
|
||||
off on dev macOS (matches the AZ-839 / AZ-699 contract).
|
||||
2. ``RUN_REPLAY_E2E`` not in ``{1, true, yes, on}``.
|
||||
3. ``gps-denied-replay`` console-script not on ``PATH``.
|
||||
4. Real video missing or placeholder-sized (mirrors AZ-699's gate).
|
||||
5. ``operator_pre_flight_setup`` fixture itself skipped — the
|
||||
downstream consumer inherits the SKIP automatically (pytest's
|
||||
fixture-skip propagation).
|
||||
|
||||
AC-7 (AZ-699 continues to pass) is satisfied by inspection: this
|
||||
test does not modify ``test_derkachi_real_tlog.py`` and writes its
|
||||
report to the same path (``real_flight_validation_<date>.md``) but
|
||||
in an idempotent way — both tests writing PASS or both writing
|
||||
FAIL is the expected joint outcome on a given clip.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.e2e.replay._e2e_orchestrator import (
|
||||
OrchestrationReport,
|
||||
run_e2e_orchestration,
|
||||
)
|
||||
from tests.e2e.replay._operator_pre_flight import PopulatedC6Cache
|
||||
from tests.e2e.replay.conftest import DerkachiReplayInputs
|
||||
|
||||
|
||||
def _repo_root() -> Path:
|
||||
return Path(__file__).resolve().parents[3]
|
||||
|
||||
|
||||
def _derkachi_dir() -> Path:
|
||||
return _repo_root() / "_docs" / "00_problem" / "input_data" / "flight_derkachi"
|
||||
|
||||
|
||||
_MIN_REAL_VIDEO_BYTES: int = 1_000_000
|
||||
|
||||
|
||||
def _replay_binary() -> Path | None:
|
||||
"""Return the absolute path to ``gps-denied-replay`` or ``None``.
|
||||
|
||||
Same lookup order AZ-699 uses: PATH first, venv bin second.
|
||||
"""
|
||||
|
||||
binary = shutil.which("gps-denied-replay")
|
||||
if binary is not None:
|
||||
return Path(binary)
|
||||
venv_bin = Path(sys.executable).parent / "gps-denied-replay"
|
||||
if venv_bin.exists():
|
||||
return venv_bin
|
||||
return None
|
||||
|
||||
|
||||
def _orchestrator_skip_reason() -> str | None:
|
||||
"""Return a SKIP message when env / inputs preclude a Jetson run."""
|
||||
|
||||
if os.environ.get("RUN_REPLAY_E2E", "").strip().lower() not in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
"on",
|
||||
}:
|
||||
return "AZ-840 e2e orchestrator gated by RUN_REPLAY_E2E=1"
|
||||
if not os.environ.get("GPS_DENIED_OPERATOR_CONFIG_PATH", "").strip():
|
||||
return (
|
||||
"AZ-840 e2e orchestrator requires GPS_DENIED_OPERATOR_CONFIG_PATH "
|
||||
"(same env var the C3 fixture consumes)"
|
||||
)
|
||||
if _replay_binary() is None:
|
||||
return "gps-denied-replay console-script not installed"
|
||||
video = _derkachi_dir() / "flight_derkachi.mp4"
|
||||
if not video.is_file():
|
||||
return f"Derkachi video missing: {video}"
|
||||
if video.stat().st_size < _MIN_REAL_VIDEO_BYTES:
|
||||
return (
|
||||
f"Derkachi video at {video} is only {video.stat().st_size} "
|
||||
"bytes — placeholder, not a real recording"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def az840_skip_gate() -> Iterator[None]:
|
||||
"""Skip-gate the orchestrator test before any heavy fixtures resolve."""
|
||||
|
||||
reason = _orchestrator_skip_reason()
|
||||
if reason is not None:
|
||||
pytest.skip(reason)
|
||||
yield
|
||||
|
||||
|
||||
@pytest.mark.tier2
|
||||
def test_az840_e2e_real_flight_orchestration(
|
||||
az840_skip_gate: None,
|
||||
operator_pre_flight_setup: PopulatedC6Cache,
|
||||
derkachi_replay_inputs: DerkachiReplayInputs,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange — every input besides cache_root comes from the existing
|
||||
# session fixtures so the same Tier-2 harness setup that powers
|
||||
# AZ-699 + AZ-839 is exercised.
|
||||
binary = _replay_binary()
|
||||
assert binary is not None, "skip gate already verified the binary exists"
|
||||
base_config_path = Path(os.environ["GPS_DENIED_OPERATOR_CONFIG_PATH"])
|
||||
output_path = tmp_path / "estimator_output.jsonl"
|
||||
effective_config_path = tmp_path / "operator_config_effective.yaml"
|
||||
report_dir = _repo_root() / "_docs" / "06_metrics"
|
||||
|
||||
# Act
|
||||
report = run_e2e_orchestration(
|
||||
populated_cache=operator_pre_flight_setup,
|
||||
base_config_path=base_config_path,
|
||||
tlog_path=derkachi_replay_inputs.tlog_path,
|
||||
video_path=derkachi_replay_inputs.video_path,
|
||||
calibration_path=derkachi_replay_inputs.calibration_path,
|
||||
signing_key_path=derkachi_replay_inputs.signing_key_path,
|
||||
replay_binary=binary,
|
||||
output_path=output_path,
|
||||
report_dir=report_dir,
|
||||
effective_config_path=effective_config_path,
|
||||
)
|
||||
|
||||
# Assert AC-2 + AC-4 — report exists; full run within the 15-min budget.
|
||||
assert isinstance(report, OrchestrationReport)
|
||||
assert report.report_path.is_file()
|
||||
body = report.report_path.read_text()
|
||||
assert "## Horizontal error (metres)" in body
|
||||
assert "## Threshold-hit share" in body
|
||||
assert "Mean" in body
|
||||
for threshold in (10, 25, 50, 100):
|
||||
assert f"| {threshold} |" in body, (
|
||||
f"threshold {threshold} m row missing from report"
|
||||
)
|
||||
assert report.replay_subprocess_seconds <= 900.0, (
|
||||
"AZ-840 AC-4: replay subprocess exceeded 15-min soft target"
|
||||
)
|
||||
assert report.wall_clock_s >= report.replay_subprocess_seconds
|
||||
assert report.distribution.count > 0, (
|
||||
"no emissions paired with ground truth — orchestration produced "
|
||||
"data but every emission fell outside the tlog GPS window"
|
||||
)
|
||||
|
||||
# Assert AC-3 — the effective config was written and points at the
|
||||
# cache_root the C3 fixture supplied.
|
||||
assert effective_config_path.is_file()
|
||||
effective_text = effective_config_path.read_text()
|
||||
assert str(operator_pre_flight_setup.cache_root) in effective_text
|
||||
@@ -0,0 +1,671 @@
|
||||
"""Unit tests for the AZ-840 e2e orchestrator (AC-8).
|
||||
|
||||
The end-to-end happy path is the Tier-2 integration test in
|
||||
``test_az835_e2e_real_flight.py`` (AC-1 / AC-2). This module covers
|
||||
the orchestration helper layer in isolation:
|
||||
|
||||
* Param validation — every required path must exist before the
|
||||
airborne subprocess is spawned (AC-5 fails LOUD).
|
||||
* Effective-config merge — the ``c6_tile_cache.root_dir`` override
|
||||
is written to YAML; the rest of the base config is preserved.
|
||||
* Error propagation per step — every documented failure surfaces
|
||||
as :class:`OrchestrationFailure` with the correct
|
||||
:class:`OrchestratorStep` label.
|
||||
* Happy path — when the runner returns success and the JSONL +
|
||||
ground truth align, :class:`OrchestrationReport` carries a
|
||||
written report path and an honest verdict (AC-2: report exists
|
||||
PASS or FAIL).
|
||||
|
||||
The tests inject a fake ``runner`` so no real
|
||||
``gps-denied-replay`` subprocess is spawned. Real binary execution
|
||||
is exercised on the Jetson harness via the AC-1 integration test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from gps_denied_onboard.helpers.accuracy_report import (
|
||||
AC3_GATE_THRESHOLD_M,
|
||||
)
|
||||
from gps_denied_onboard.replay_input.tlog_route import RouteSpec
|
||||
|
||||
from tests.e2e.replay._e2e_orchestrator import (
|
||||
OrchestrationFailure,
|
||||
OrchestrationReport,
|
||||
OrchestratorStep,
|
||||
read_calibration_acquisition_method,
|
||||
run_e2e_orchestration,
|
||||
write_effective_replay_config,
|
||||
)
|
||||
from tests.e2e.replay._operator_pre_flight import PopulatedC6Cache
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Helpers
|
||||
|
||||
|
||||
def _build_populated_cache(tmp_path: Path) -> PopulatedC6Cache:
|
||||
"""Construct a synthetic :class:`PopulatedC6Cache`.
|
||||
|
||||
The orchestrator only consumes ``cache_root`` from the cache,
|
||||
so the FAISS sidecar paths are placeholders. The route_spec is
|
||||
a minimal one-waypoint instance — no AZ-836 invariants are
|
||||
re-asserted by AZ-840.
|
||||
"""
|
||||
|
||||
cache_root = tmp_path / "cache_root"
|
||||
cache_root.mkdir()
|
||||
return PopulatedC6Cache(
|
||||
cache_root=cache_root,
|
||||
tile_store_path=cache_root / "tiles",
|
||||
faiss_index_path=cache_root / "descriptor.index",
|
||||
faiss_sidecar_sha256_path=cache_root / "descriptor.index.sha256",
|
||||
faiss_sidecar_meta_path=cache_root / "descriptor.index.meta.json",
|
||||
route_spec=RouteSpec(
|
||||
waypoints=((50.10, 36.10),),
|
||||
suggested_region_size_meters=500.0,
|
||||
source_tlog=Path("test.tlog"),
|
||||
source_segment=(0, 100),
|
||||
total_distance_meters=0.0,
|
||||
),
|
||||
tile_count=1,
|
||||
elapsed_seconds=0.0,
|
||||
)
|
||||
|
||||
|
||||
def _stage_inputs(tmp_path: Path) -> dict[str, Path]:
|
||||
"""Write touch-files for every input path the orchestrator validates.
|
||||
|
||||
The base config YAML carries one stub block so the merge step
|
||||
has a real document to overlay on.
|
||||
"""
|
||||
|
||||
base_config = tmp_path / "operator_config.yaml"
|
||||
base_config.write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"mode": "replay",
|
||||
"c6_tile_cache": {
|
||||
"store_runtime": "postgres_filesystem",
|
||||
"metadata_runtime": "postgres_filesystem",
|
||||
"descriptor_index_runtime": "faiss_hnsw",
|
||||
"root_dir": "/var/lib/gps-denied/tiles",
|
||||
"faiss_index_path": "/some/static/path/descriptor.index",
|
||||
},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
tlog = tmp_path / "input.tlog"
|
||||
tlog.write_bytes(b"\x00")
|
||||
video = tmp_path / "input.mp4"
|
||||
video.write_bytes(b"\x00")
|
||||
calibration = tmp_path / "calibration.json"
|
||||
calibration.write_text(json.dumps({"acquisition_method": "factory-sheet"}))
|
||||
signing_key = tmp_path / "signing_key.bin"
|
||||
signing_key.write_bytes(b"\x00" * 32)
|
||||
binary = tmp_path / "gps-denied-replay"
|
||||
binary.write_text("")
|
||||
|
||||
return {
|
||||
"base_config_path": base_config,
|
||||
"tlog_path": tlog,
|
||||
"video_path": video,
|
||||
"calibration_path": calibration,
|
||||
"signing_key_path": signing_key,
|
||||
"replay_binary": binary,
|
||||
}
|
||||
|
||||
|
||||
def _ground_truth_tlog_loader(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
*,
|
||||
times_s: tuple[float, ...] = (0.0, 1.0, 2.0),
|
||||
lat_deg: float = 50.10,
|
||||
lon_deg: float = 36.10,
|
||||
alt_m: float = 100.0,
|
||||
) -> None:
|
||||
"""Stub the orchestrator's ground-truth loader so unit tests skip MAVLink.
|
||||
|
||||
The orchestrator imports ``load_tlog_ground_truth`` from
|
||||
``gps_denied_onboard.replay_input``; patching the symbol *as
|
||||
bound on the orchestrator module* keeps the patch local to the
|
||||
unit suite (no cross-test bleed).
|
||||
"""
|
||||
|
||||
fixes = [
|
||||
_StubGpsFix(
|
||||
ts_ns=int(t * 1e9),
|
||||
lat_deg=lat_deg,
|
||||
lon_deg=lon_deg,
|
||||
alt_m=alt_m,
|
||||
)
|
||||
for t in times_s
|
||||
]
|
||||
series = _StubGpsSeries(records=tuple(fixes))
|
||||
monkeypatch.setattr(
|
||||
"tests.e2e.replay._e2e_orchestrator.load_tlog_ground_truth",
|
||||
lambda *_args, **_kwargs: series,
|
||||
)
|
||||
|
||||
|
||||
class _StubGpsFix:
|
||||
"""Mirrors the fields the orchestrator reads from each tlog row."""
|
||||
|
||||
__slots__ = ("ts_ns", "lat_deg", "lon_deg", "alt_m")
|
||||
|
||||
def __init__(
|
||||
self, *, ts_ns: int, lat_deg: float, lon_deg: float, alt_m: float
|
||||
) -> None:
|
||||
self.ts_ns = ts_ns
|
||||
self.lat_deg = lat_deg
|
||||
self.lon_deg = lon_deg
|
||||
self.alt_m = alt_m
|
||||
|
||||
|
||||
class _StubGpsSeries:
|
||||
"""Drop-in replacement for :class:`TlogGroundTruth`."""
|
||||
|
||||
def __init__(self, *, records: tuple[_StubGpsFix, ...]) -> None:
|
||||
self.records = records
|
||||
|
||||
|
||||
def _build_runner_emitting(
|
||||
output_path: Path,
|
||||
*,
|
||||
rows: list[dict[str, object]],
|
||||
returncode: int = 0,
|
||||
stdout: str = "",
|
||||
stderr: str = "",
|
||||
) -> "MagicMock":
|
||||
"""Return a fake ``subprocess.run`` that writes JSONL on call."""
|
||||
|
||||
def _run(argv, **kwargs): # type: ignore[no-untyped-def]
|
||||
if rows:
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(
|
||||
"\n".join(json.dumps(row) for row in rows) + "\n"
|
||||
)
|
||||
return subprocess.CompletedProcess(
|
||||
args=argv,
|
||||
returncode=returncode,
|
||||
stdout=stdout,
|
||||
stderr=stderr,
|
||||
)
|
||||
|
||||
return MagicMock(side_effect=_run)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# write_effective_replay_config
|
||||
|
||||
|
||||
def test_write_effective_replay_config_overlays_root_dir(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
cache_root = tmp_path / "cache"
|
||||
cache_root.mkdir()
|
||||
output_path = tmp_path / "effective.yaml"
|
||||
|
||||
# Act
|
||||
written_path = write_effective_replay_config(
|
||||
base_config_path=inputs["base_config_path"],
|
||||
cache_root=cache_root,
|
||||
output_path=output_path,
|
||||
)
|
||||
|
||||
# Assert
|
||||
assert written_path == output_path
|
||||
merged = yaml.safe_load(output_path.read_text())
|
||||
assert merged["c6_tile_cache"]["root_dir"] == str(cache_root)
|
||||
assert merged["c6_tile_cache"]["faiss_index_path"] == ""
|
||||
assert merged["mode"] == "replay"
|
||||
assert (
|
||||
merged["c6_tile_cache"]["store_runtime"] == "postgres_filesystem"
|
||||
), "non-overridden c6_tile_cache fields must survive"
|
||||
|
||||
|
||||
def test_write_effective_replay_config_creates_block_when_absent(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
base = tmp_path / "operator.yaml"
|
||||
base.write_text(yaml.safe_dump({"mode": "replay"}))
|
||||
cache_root = tmp_path / "cache"
|
||||
cache_root.mkdir()
|
||||
|
||||
# Act
|
||||
write_effective_replay_config(
|
||||
base_config_path=base,
|
||||
cache_root=cache_root,
|
||||
output_path=tmp_path / "effective.yaml",
|
||||
)
|
||||
|
||||
# Assert
|
||||
merged = yaml.safe_load((tmp_path / "effective.yaml").read_text())
|
||||
assert merged["c6_tile_cache"]["root_dir"] == str(cache_root)
|
||||
|
||||
|
||||
def test_write_effective_replay_config_malformed_yaml_fails(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
base = tmp_path / "bad.yaml"
|
||||
base.write_text(":\n : not yaml:")
|
||||
cache_root = tmp_path / "cache"
|
||||
cache_root.mkdir()
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
write_effective_replay_config(
|
||||
base_config_path=base,
|
||||
cache_root=cache_root,
|
||||
output_path=tmp_path / "effective.yaml",
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.WRITE_EFFECTIVE_CONFIG
|
||||
|
||||
|
||||
def test_write_effective_replay_config_non_mapping_top_level_fails(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
base = tmp_path / "bad.yaml"
|
||||
base.write_text("- not a mapping\n")
|
||||
cache_root = tmp_path / "cache"
|
||||
cache_root.mkdir()
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
write_effective_replay_config(
|
||||
base_config_path=base,
|
||||
cache_root=cache_root,
|
||||
output_path=tmp_path / "effective.yaml",
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.WRITE_EFFECTIVE_CONFIG
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# read_calibration_acquisition_method
|
||||
|
||||
|
||||
def test_read_calibration_acquisition_method_returns_field_when_present(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
path = tmp_path / "cal.json"
|
||||
path.write_text(json.dumps({"acquisition_method": "factory-sheet"}))
|
||||
|
||||
# Assert
|
||||
assert read_calibration_acquisition_method(path) == "factory-sheet"
|
||||
|
||||
|
||||
def test_read_calibration_acquisition_method_returns_unknown_on_missing(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
path = tmp_path / "cal.json"
|
||||
path.write_text(json.dumps({"some_other_field": True}))
|
||||
|
||||
# Assert
|
||||
assert read_calibration_acquisition_method(path) == "unknown"
|
||||
|
||||
|
||||
def test_read_calibration_acquisition_method_returns_unknown_on_malformed(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
path = tmp_path / "cal.json"
|
||||
path.write_text("{not valid json")
|
||||
|
||||
# Assert
|
||||
assert read_calibration_acquisition_method(path) == "unknown"
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# run_e2e_orchestration — param validation (AC-5)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_missing_tlog_fails_loud(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
inputs["tlog_path"].unlink()
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=tmp_path / "out.jsonl",
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.VALIDATE_INPUTS
|
||||
assert "tlog_path" in str(exc_info.value)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_missing_binary_fails_loud(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
inputs["replay_binary"].unlink()
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=tmp_path / "out.jsonl",
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.VALIDATE_INPUTS
|
||||
assert "replay_binary" in str(exc_info.value)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# run_e2e_orchestration — subprocess error propagation (AC-5)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_replay_nonzero_exit_fails_loud(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
runner = MagicMock(
|
||||
return_value=subprocess.CompletedProcess(
|
||||
args=[],
|
||||
returncode=1,
|
||||
stdout="",
|
||||
stderr="boom",
|
||||
)
|
||||
)
|
||||
_ground_truth_tlog_loader(monkeypatch)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.AIRBORNE_PIPELINE
|
||||
assert "exited 1" in str(exc_info.value)
|
||||
assert "boom" in str(exc_info.value)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_replay_timeout_fails_loud(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
|
||||
def _timeout(*_args, **_kwargs):
|
||||
raise subprocess.TimeoutExpired(cmd=["replay"], timeout=0.1)
|
||||
|
||||
runner = MagicMock(side_effect=_timeout)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=tmp_path / "out.jsonl",
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
max_seconds=0.1,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.AIRBORNE_PIPELINE
|
||||
assert "timed out" in str(exc_info.value)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_replay_oserror_fails_loud(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
|
||||
def _oserror(*_args, **_kwargs):
|
||||
raise OSError("permission denied")
|
||||
|
||||
runner = MagicMock(side_effect=_oserror)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=tmp_path / "out.jsonl",
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.AIRBORNE_PIPELINE
|
||||
assert "permission denied" in str(exc_info.value)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# run_e2e_orchestration — empty / malformed JSONL (AC-5)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_empty_jsonl_fails_loud(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
|
||||
def _runner(argv, **_kwargs): # type: ignore[no-untyped-def]
|
||||
output_path.write_text("\n\n") # only blanks
|
||||
return subprocess.CompletedProcess(args=argv, returncode=0, stdout="", stderr="")
|
||||
|
||||
runner = MagicMock(side_effect=_runner)
|
||||
_ground_truth_tlog_loader(monkeypatch)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.PARSE_EMISSIONS
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_malformed_jsonl_fails_loud(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
|
||||
def _runner(argv, **_kwargs): # type: ignore[no-untyped-def]
|
||||
output_path.write_text('{"valid": true}\nnot a json line\n')
|
||||
return subprocess.CompletedProcess(args=argv, returncode=0, stdout="", stderr="")
|
||||
|
||||
runner = MagicMock(side_effect=_runner)
|
||||
_ground_truth_tlog_loader(monkeypatch)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.PARSE_EMISSIONS
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# run_e2e_orchestration — ground truth loader failure (AC-5)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_ground_truth_loader_failure_fails_loud(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
runner = _build_runner_emitting(
|
||||
output_path,
|
||||
rows=[
|
||||
{
|
||||
"emitted_at": int(0.5 * 1e9),
|
||||
"position_wgs84": {
|
||||
"lat_deg": 50.10,
|
||||
"lon_deg": 36.10,
|
||||
"alt_m": 100.0,
|
||||
},
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
def _raise(*_args, **_kwargs):
|
||||
raise ValueError("tlog corrupt")
|
||||
|
||||
monkeypatch.setattr(
|
||||
"tests.e2e.replay._e2e_orchestrator.load_tlog_ground_truth",
|
||||
_raise,
|
||||
)
|
||||
|
||||
# Act + Assert
|
||||
with pytest.raises(OrchestrationFailure) as exc_info:
|
||||
run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=tmp_path / "metrics",
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
assert exc_info.value.step is OrchestratorStep.LOAD_GROUND_TRUTH
|
||||
assert "tlog corrupt" in str(exc_info.value)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# run_e2e_orchestration — happy path (AC-1 / AC-2)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_happy_path_writes_report(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
report_dir = tmp_path / "metrics"
|
||||
effective_config_path = tmp_path / "eff.yaml"
|
||||
rows = [
|
||||
{
|
||||
"emitted_at": int(0.5 * 1e9),
|
||||
"position_wgs84": {"lat_deg": 50.10, "lon_deg": 36.10, "alt_m": 100.0},
|
||||
},
|
||||
{
|
||||
"emitted_at": int(1.5 * 1e9),
|
||||
"position_wgs84": {"lat_deg": 50.10, "lon_deg": 36.10, "alt_m": 100.0},
|
||||
},
|
||||
]
|
||||
runner = _build_runner_emitting(output_path, rows=rows)
|
||||
_ground_truth_tlog_loader(monkeypatch)
|
||||
|
||||
# Act
|
||||
report = run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=report_dir,
|
||||
effective_config_path=effective_config_path,
|
||||
runner=runner,
|
||||
run_date_utc="2026-05-23",
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
# Assert
|
||||
assert isinstance(report, OrchestrationReport)
|
||||
assert report.report_path.is_file()
|
||||
assert report.emissions_count == 2
|
||||
assert report.distribution.count == 2
|
||||
assert report.verdict_passed is True
|
||||
body = report.report_path.read_text()
|
||||
assert "## Horizontal error (metres)" in body
|
||||
assert "## Threshold-hit share" in body
|
||||
assert f"| {AC3_GATE_THRESHOLD_M:g} |" in body
|
||||
runner.assert_called_once()
|
||||
argv_passed = runner.call_args.args[0]
|
||||
assert str(effective_config_path) in argv_passed
|
||||
assert "--auto-trim" in argv_passed
|
||||
merged = yaml.safe_load(effective_config_path.read_text())
|
||||
assert merged["c6_tile_cache"]["root_dir"] == str(cache.cache_root)
|
||||
|
||||
|
||||
def test_run_e2e_orchestration_writes_report_even_on_fail_verdict(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Arrange — emissions are 1 km from ground truth, far above the 100 m gate.
|
||||
cache = _build_populated_cache(tmp_path)
|
||||
inputs = _stage_inputs(tmp_path)
|
||||
output_path = tmp_path / "out.jsonl"
|
||||
report_dir = tmp_path / "metrics"
|
||||
rows = [
|
||||
{
|
||||
"emitted_at": int(0.5 * 1e9),
|
||||
"position_wgs84": {"lat_deg": 50.110, "lon_deg": 36.110, "alt_m": 100.0},
|
||||
},
|
||||
{
|
||||
"emitted_at": int(1.5 * 1e9),
|
||||
"position_wgs84": {"lat_deg": 50.110, "lon_deg": 36.110, "alt_m": 100.0},
|
||||
},
|
||||
]
|
||||
runner = _build_runner_emitting(output_path, rows=rows)
|
||||
_ground_truth_tlog_loader(monkeypatch)
|
||||
|
||||
# Act
|
||||
report = run_e2e_orchestration(
|
||||
populated_cache=cache,
|
||||
output_path=output_path,
|
||||
report_dir=report_dir,
|
||||
effective_config_path=tmp_path / "eff.yaml",
|
||||
runner=runner,
|
||||
run_date_utc="2026-05-23",
|
||||
**inputs, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
# Assert — AC-2: report exists regardless of PASS/FAIL.
|
||||
assert report.verdict_passed is False
|
||||
assert report.report_path.is_file()
|
||||
assert "FAIL" in report.report_path.read_text()
|
||||
Reference in New Issue
Block a user