[AZ-835] Epic split (C1/C2) + workspace-boundary rule expansion

AZ-835 Epic (E2E real-flight validation pipeline, ~17 SP across
6 children C1-C6) supersedes AZ-777 Phase 3+ (bbox-based static
seed). Children C3-C6 deliberately not yet filed — will be
re-estimated after C1+C2 land from real RouteSpec shape and
Route API client ergonomics.

- AZ-836 (C1, 3 SP): TlogRouteExtractor — pure function over
  .tlog binary returning RouteSpec (waypoints + suggested
  region size). Deps: AZ-697 (load_tlog_ground_truth, done),
  AZ-279 (WGS converter, done).
- AZ-838 (C2, 3 SP): SatelliteProviderRouteClient + seed_route.py
  CLI mirror of seed_region.py. Hard-depends on AZ-836's
  RouteSpec dataclass.
- _dependencies_table.md updated with the three new rows.

Workspace-boundary rule expansion: codifies the sibling-repo
task-spec exception (the only permitted write into a sibling
repo) and the "External Systems Are Black Boxes" rule
(contract-only consumption of producer repos like
satellite-provider).

Bookkeeping: _autodev_state.md condensed to <30 lines per the
state.md conciseness rule; opencv-pin leftover replay
re-checked 2026-05-22 (gtsam still only 4.2, replay condition
unchanged).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-22 17:39:38 +03:00
parent b15454b9a9
commit 63c0217e3d
7 changed files with 329 additions and 7 deletions
+3
View File
@@ -185,6 +185,9 @@ are all declared and documented below under **Cycle Check**.
| AZ-702 | T6: Topotek KHP20S30 camera calibration (factory-sheet approximation) | 1 | None | AZ-696 |
| AZ-776 | Open-loop ESKF composition profile (c4_pose.enabled flag) | 3 | None | AZ-602 |
| AZ-777 | Derkachi e2e: wire EXISTING parent-suite satellite-provider into operator pre-flight fixture | 8 (override) | AZ-776 | AZ-602 |
| AZ-835 | E2E real-flight validation Epic: raw (tlog, video) → route-driven SP seeding → verdict | Epic (~17 child SP) | AZ-777 Phase 1 (reused); AZ-405; AZ-699; AZ-696; AZ-702 | (umbrella) |
| AZ-836 | C1: TlogRouteExtractor — active-segment trim + DP coarsen tlog GPS to ≤N waypoints | 3 | AZ-697, AZ-279 | AZ-835 |
| AZ-838 | C2: SatelliteProviderRouteClient + seed_route.py CLI — POST RouteSpec to SP, poll mapsReady | 3 | AZ-836; AZ-777 Phase 1; AZ-809 (soft) | AZ-835 |
## Notes
@@ -0,0 +1,92 @@
# End-to-end real-flight validation pipeline (Epic)
**Task**: AZ-835_e2e_real_flight_validation_epic
**Name**: End-to-end real-flight validation: raw (tlog, video) → route-driven satellite seeding → gps-denied verdict
**Description**: Drive the full gps-denied-onboard validation pipeline from raw operator inputs to a verdict. Given a `.tlog` binary + a flight video, the system automatically extracts the flight cut, syncs frames to IMU, builds the satellite imagery the descriptor stack needs (route-driven, not bbox-driven), runs the airborne pipeline, and reports the horizontal-error distribution against the tlog's own GPS ground truth. Supersedes AZ-777 Phase 3+ design.
**Complexity**: Epic — ~17 SP decomposed into 6 child tasks of ≤ 5 SP each (see decomposition table below)
**Dependencies**: AZ-777 Phase 1 (landed cycle 3 batch 105 — C11 contract adaptation + e2e-runner wiring); AZ-405 (tlog↔video auto-sync adapter); AZ-699 (verdict report writer); AZ-809 SOFT (Route API validation — landing AZ-809 before C2 lets the client consume RFC 7807 validator responses cleanly)
**Component**: cross-cutting — replay_input + new TlogRouteExtractor + new SatelliteProviderRouteClient + e2e fixtures + tests/e2e/replay
**Tracker**: AZ-835 (https://denyspopov.atlassian.net/browse/AZ-835)
**Originating directive**: user (2026-05-22) after AZ-777 Phase 2 deliverables landed — "In the end it should be full e2e flow. You give it a tlog + video, and the system does everything else."
Jira AZ-835 is the authoritative spec; this file mirrors the in-workspace-only sections that gps-denied-onboard implementers will need.
## Goal
A single pytest test takes only `(tlog, video, calibration)` as input and runs the full 7-step pipeline end-to-end on the Jetson harness, producing an honest PASS/FAIL verdict against the AZ-696 AC-3 threshold (≥ 80 % of emissions within 100 m).
## The 7-step pipeline
| # | Step | Existing? | Component / new code |
|---|------|-----------|----------------------|
| 1 | Extract active flight cut + sync with video | **Mostly existing** (AZ-405 `tlog_video_adapter.py`) | small extension for take-off/landing boundary detection if needed |
| 2 | On-fly frame + IMU extraction | **Existing** | `VideoFileFrameSource` + `TlogReplayFcAdapter` (no change) |
| 3 | Auto-create route from tlog GPS, coarsen to ≤ 10 pts | **New** | `TlogRouteExtractor` (Douglas-Peucker on `GLOBAL_POSITION_INT` rows) → `RouteSpec` |
| 4 | POST route to satellite-provider, get tiles | **New consumer** | `SatelliteProviderRouteClient` (POST `/api/satellite/route`, poll `mapsReady`) |
| 5 | Calc FAISS index from tiles | **Mostly existing** | C10 `DescriptorBatcher` runs; new fixture wires C11 → C10 trigger |
| 6 | Run gps-denied from all the info | **Existing** | `gps-denied-replay` console-script + airborne composition root |
| 7 | Get GPS fixes, check against tlog GPS | **Existing** | `helpers/accuracy_report.py` + `helpers/gps_compare.py` |
## Decomposition (6 child tasks)
| # | Title | Est | Depends |
|---|-------|-----|---------|
| C1 | `TlogRouteExtractor` — extract active segment + coarsen to N waypoints | 3 | — |
| C2 | `SatelliteProviderRouteClient` + `route_seed.py` CLI | 3 | AZ-809 (soft) |
| C3 | New `operator_pre_flight_setup` fixture (C1 + C2 + C11 + C10) — replaces placeholder, supersedes AZ-777 Phase 3 | 5 | C1, C2, AZ-777 Phase 1 |
| C4 | E2E test ingesting raw `(tlog, video)` and running steps 1-7 — extends/replaces AZ-699 verdict test | 3 | C3 |
| C5 | Un-xfail AZ-777 AC-4 + AC-5 tests | 1 | C4 |
| C6 | Docs: `replay_protocol.md` Invariant 12 + AZ-777 amendment + new-test README | 2 | C5 |
**Total ~17 SP**.
## Why route-driven seeding (not bbox)
- **Efficiency**: AZ-777 spec bbox = ~11400 tiles z15-z18 (~140 MB, 48% over budget). 10-point coarsened route with `regionSizeMeters=500` per point = ~50-100 unique tiles (~1.5 MB) for the same VPR descriptor lock area. **~100× reduction**.
- **Honesty**: bbox pre-commits to where the operator *might* fly. Route pre-commits to where they *did* fly. For real-flight validation, the latter is the right primitive.
- **Probe-confirmed**: Route API works end-to-end in ~15s for a 2-point route per 2026-05-22 black-box probe. Uses `lat`/`lon` already (no AZ-812 rename needed).
## Coordination with prior work
- **AZ-777** — Phase 1 + Phase 2 reused; Phase 3+ design **superseded** by this Epic when C3 lands.
- **AZ-699** — verdict-report-writing path preserved; C4 extends or wraps it.
- **AZ-405** — tlog↔video auto-sync adapter reused as-is for step 1.
- **AZ-702** — camera factory-sheet calibration unchanged.
- **AZ-696** — ≥ 80 % within 100 m threshold gate unchanged.
- **AZ-808** — Region-endpoint validation; not on this Epic's critical path (Route used, not Region).
- **AZ-809** — Route-endpoint validation; soft prereq for C2.
- **AZ-812** — Region rename to lat/lon; not on this Epic's critical path.
## Acceptance criteria (Epic-level)
**AC-1**: New pytest test gated by `RUN_REPLAY_E2E=1` + `@pytest.mark.tier2` takes only `(tlog, video, calibration)` and runs the full 7-step pipeline on Jetson.
**AC-2**: Step 1 auto-detects active flight cut from raw tlog (take-off → landing) without operator intervention.
**AC-3**: Step 3 produces ≤ 10 waypoints that materially follow the tlog GPS trajectory (DP tolerance documented in config).
**AC-4**: Step 4 succeeds against real satellite-provider on Jetson docker network, downloads route tiles from Google Maps, `mapsReady=true` within runtime budget.
**AC-5**: Step 5 builds FAISS HNSW index over route-seeded C6 cache; sidecar triple-consistency holds (AZ-306).
**AC-6**: Step 7 emits AZ-699 verdict report at `_docs/06_metrics/real_flight_validation_<YYYY-MM-DD>.md` with honest horizontal-error distribution — PASS or FAIL on AZ-696 AC-3 threshold, no xfail mask.
**AC-7**: End-to-end run ≤ 15 min on Tier-2 Jetson for the Derkachi clip (soft target for first delivery; hard NFR after first measurement).
**AC-8**: Docs: `replay_protocol.md` Invariant 12 sub-section + AZ-777 marked Phase 3+ superseded + new-test README.
## Out of scope
- Satellite-provider imagery-source migration to CC-BY (parent-suite ticket, TBD).
- FAISS / NetVLAD backbone replacement.
- Real-time tlog ingestion (this Epic operates on finished `.tlog` files).
- Multi-flight aggregate validation.
- ZERO modifications to `../satellite-provider/` (Route API consumed as-is).
- CI gating (test stays behind `RUN_REPLAY_E2E=1`).
## References
- Jira AZ-835: https://denyspopov.atlassian.net/browse/AZ-835
- Supersedes AZ-777 Phase 3+ design (AZ-777 Phase 1 + Phase 2 reused)
- Probe foundation: 2026-05-22 black-box probe of Route API confirmed end-to-end viability
- Related: AZ-405, AZ-696, AZ-699, AZ-702, AZ-777, AZ-808, AZ-809, AZ-812
@@ -0,0 +1,86 @@
# TlogRouteExtractor
**Task**: AZ-836_tlog_route_extractor
**Name**: TlogRouteExtractor: extract active flight segment + coarsen tlog GPS to ≤N waypoints (AZ-835 C1)
**Description**: First building block of Epic AZ-835. Pure, testable function that consumes a `.tlog` binary and returns a `RouteSpec` (≤ N waypoints + suggested per-waypoint coverage radius) suitable for posting to satellite-provider's `POST /api/satellite/route` endpoint (consumed by AZ-835 C2 / AZ-838).
**Complexity**: 3 SP
**Dependencies**: AZ-697 (`load_tlog_ground_truth` — done); AZ-279 (WGS converter — done); AZ-835 (parent Epic)
**Component**: `src/gps_denied_onboard/replay_input/tlog_route.py` (new module under `replay_input/`)
**Tracker**: AZ-836 (https://denyspopov.atlassian.net/browse/AZ-836)
**Parent Epic**: AZ-835
Jira AZ-836 is the authoritative spec; this file is the in-workspace mirror.
## Public surface
```python
from dataclasses import dataclass
from pathlib import Path
@dataclass(frozen=True, slots=True)
class RouteSpec:
waypoints: tuple[tuple[float, float], ...] # (lat, lon), 1..max_waypoints
suggested_region_size_meters: float # per-waypoint coverage radius
source_tlog: Path # provenance
source_segment: tuple[int, int] # (start_idx, end_idx) into tlog GPS rows
total_distance_meters: float # along-track distance of active segment
class RouteExtractionError(ReplayInputAdapterError): ...
def extract_route_from_tlog(
tlog: Path,
*,
max_waypoints: int = 10,
min_takeoff_speed_m_s: float = 2.0,
min_takeoff_altitude_agl_m: float = 5.0,
douglas_peucker_tolerance_m: float | None = None, # auto-computed if None
region_size_meters: float = 500.0,
) -> RouteSpec: ...
```
Reuses `replay_input.tlog_ground_truth.load_tlog_ground_truth()` for GPS extraction — no MAVLink re-parsing.
## Active-segment detection
Trim leading + trailing rows where horizontal speed < `min_takeoff_speed_m_s` AND altitude AGL < `min_takeoff_altitude_agl_m`. Both thresholds configurable. If trimmed segment has < 2 fixes, raise `RouteExtractionError` with the explicit threshold values — no silent fallback to the full tlog.
## Coarsening
Douglas-Peucker in WGS84 with great-circle distance metric. Use the existing `helpers.wgs_converter` or `helpers.gps_compare` meter conversion — do NOT reimplement (check both first; pick whichever has the right primitive).
When `douglas_peucker_tolerance_m is None`, auto-compute by binary-search over the tolerance until `len(result) <= max_waypoints`. Halt at convergence (delta < 1 m) or 32 iterations.
## Validation
- `max_waypoints >= 1` (raise `ValueError`).
- `region_size_meters > 0` (raise `ValueError`).
- At least 1 fix from `GLOBAL_POSITION_INT` (preferred) or `GPS_RAW_INT` (fallback); if neither, `RouteExtractionError` referencing missing message types (mirrors AZ-697).
- Missing tlog file → `RouteExtractionError` (not bare `FileNotFoundError`) so callers can catch one error class.
## Acceptance criteria
| # | Criterion |
|---|-----------|
| AC-1 | Real Derkachi tlog → RouteSpec with `len(waypoints) <= 10`; every waypoint inside lat 50.0808..50.0832, lon 36.1070..36.1134 |
| AC-2 | Active-segment trim filters pre-takeoff stationary frames (synthetic 5+ stationary leading fixes → `source_segment[0] > 0`) |
| AC-3 | `max_waypoints=2` → exactly 2 waypoints |
| AC-4 | `max_waypoints=100` on N<100 tlog → N waypoints (no coarsening below natural fix count) |
| AC-5 | Missing tlog → `RouteExtractionError` with path; not `FileNotFoundError` |
| AC-6 | Tlog with no GPS → `RouteExtractionError` naming missing message types |
| AC-7 | `RouteSpec` is `frozen=True`, `slots=True`, all provenance fields populated |
| AC-8 | Auto-tolerance binary-search converges within 32 iters on a 200-fix synthetic trajectory |
| AC-9 | No I/O beyond tlog read; logging at DEBUG only |
| AC-10 | Unit tests cover: Derkachi happy path, small/large max_waypoints, missing tlog, missing GPS, custom DP tolerance, custom region size, synthetic stationary-leading trim |
## Out of scope
- Posting to satellite-provider (AZ-838 / C2)
- Route visualization on a map (future, AZ-700-style)
- Multi-tlog aggregation
- Live-stream tlog ingestion
## References
- Parent Epic: AZ-835 — https://denyspopov.atlassian.net/browse/AZ-835
- Reference tlog: `_docs/00_problem/input_data/flight_derkachi/derkachi.tlog`
- Reuse: `src/gps_denied_onboard/replay_input/tlog_ground_truth.py` (AZ-697), `src/gps_denied_onboard/helpers/gps_compare.py`
@@ -0,0 +1,106 @@
# SatelliteProviderRouteClient + seed_route.py CLI
**Task**: AZ-838_satellite_provider_route_client
**Name**: SatelliteProviderRouteClient + seed_route.py CLI: POST tlog-derived route to satellite-provider (AZ-835 C2)
**Description**: Second building block of Epic AZ-835. Consumer-side HTTP client + CLI wrapper that takes a `RouteSpec` (from AZ-836 / C1) and registers it with satellite-provider's `POST /api/satellite/route` endpoint, polls until `mapsReady=true`, and returns the inventory size for downstream consumption.
**Complexity**: 3 SP
**Dependencies**: AZ-836 (C1, RouteSpec dataclass + extractor — hard code dep); AZ-777 Phase 1 (existing satellite-provider HTTP plumbing patterns + JWT handling — done); AZ-809 (Route API validation — SOFT prereq, client pre-emptively validates so it's correct without it); AZ-835 (parent Epic)
**Component**: new `src/gps_denied_onboard/satellite_provider/route_client.py` + new CLI `tests/fixtures/derkachi_c6/seed_route.py`
**Tracker**: AZ-838 (https://denyspopov.atlassian.net/browse/AZ-838)
**Parent Epic**: AZ-835
Jira AZ-838 is the authoritative spec; this file is the in-workspace mirror.
## Public surface
```python
import uuid
from dataclasses import dataclass
from gps_denied_onboard.replay_input.tlog_route import RouteSpec # AZ-836
@dataclass(frozen=True, slots=True)
class RouteSeedResult:
route_id: uuid.UUID
terminal_status: str
maps_ready: bool
tile_count: int
elapsed_ms: int
submitted_payload_sha256: str
class SatelliteProviderRouteError(Exception): ...
class RouteValidationError(SatelliteProviderRouteError): ... # 4xx + ProblemDetails
class RouteTransientError(SatelliteProviderRouteError): ... # 5xx / network / timeout
class RouteTerminalFailureError(SatelliteProviderRouteError): ... # mapsReady never reached
class SatelliteProviderRouteClient:
def __init__(self, base_url: str, jwt: str, *, tls_insecure: bool = False,
request_timeout_s: float = 30.0, poll_interval_s: float = 5.0,
poll_max_attempts: int = 60): ...
def seed_route(self, spec: RouteSpec, *, name: str | None = None) -> RouteSeedResult: ...
```
## Wire shape
No formal Route API contract doc exists in `../satellite-provider/_docs/02_document/contracts/api/` as of 2026-05-22. DTOs are the source of truth:
- `../satellite-provider/SatelliteProvider.Common/DTO/CreateRouteRequest.cs` (top-level)
- `../satellite-provider/SatelliteProvider.Common/DTO/RoutePoint.cs` (`[JsonPropertyName("lat")] Latitude`, `[JsonPropertyName("lon")] Longitude` — input/output naming asymmetry flagged in AZ-809 AC-10; consume `lat`/`lon` in the JSON)
- `../satellite-provider/SatelliteProvider.Common/DTO/GeoPoint.cs` (nested geofence point)
Probe 2026-05-22: 2-point route + `requestMaps=true` completes end-to-end in ~15 s.
## Behaviour
- **Pre-emptive validation** against AZ-809 rules — surface as `RouteValidationError` BEFORE HTTP POST:
- `points` non-empty AND `len(points) <= 100`
- `id` non-zero Guid
- `regionSizeMeters > 0` AND `<= 10000`
- `zoomLevel` in 15..18 (per AZ-777 Phase 2 bbox config)
- Each point's `lat` in -90..90, `lon` in -180..180
- **Submit** `POST /api/satellite/route` with `requestMaps=true`, `createTilesZip=false`.
- **Poll** `GET /api/satellite/route/{id}` every `poll_interval_s` up to `poll_max_attempts` until `mapsReady=true` OR terminal failure. Log cadence at INFO.
- **Return** `RouteSeedResult`; `tile_count` from a final `POST /api/satellite/tiles/inventory` enumerating the route's tile coverage (computed locally from waypoints + `regionSizeMeters`).
- **Raise** `RouteTerminalFailureError` on terminal failure (`.detail` = SP response JSON).
- **Raise** `RouteTransientError` on 5xx / network / timeout (`__cause__` = underlying `httpx` exception).
- **Raise** `RouteValidationError` on 4xx; parse RFC 7807 `errors` dict into `field_errors`.
## CLI (`tests/fixtures/derkachi_c6/seed_route.py`)
Mirrors `seed_region.py` (AZ-777 Phase 2):
- Env: `SATELLITE_PROVIDER_URL`, `SATELLITE_PROVIDER_API_KEY`, `SATELLITE_PROVIDER_TLS_INSECURE`, optional `--auto-mint-jwt` (uses `scripts/mint_dev_jwt.py`)
- Required: `--tlog <path>` (delegates to AZ-836's `extract_route_from_tlog`)
- Optional: `--max-waypoints` (10), `--region-size-meters` (500), `--name`, `--output-summary <path>`, `--dry-run`
- Exit codes: 0 success, 71 config malformed, 72 missing env, 73 SP unreachable, 74 4xx, 75 5xx / terminal failure, 76 inventory verification mismatch
## Acceptance criteria
| # | Criterion |
|---|-----------|
| AC-1 | POSTs wire shape exactly per `CreateRouteRequest.cs` + `RoutePoint.cs` + `GeoPoint.cs` |
| AC-2 | Polls `GET /api/satellite/route/{id}` until `mapsReady=true` OR terminal failure; respects `poll_max_attempts` + `poll_interval_s` |
| AC-3 | 4xx + RFC 7807 ProblemDetails → `RouteValidationError`; `field_errors` populated from `errors` dict |
| AC-4 | 5xx / network / timeout → `RouteTransientError`; `__cause__` = underlying `httpx` exc |
| AC-5 | Terminal failure → `RouteTerminalFailureError`; `.detail` = SP response JSON |
| AC-6 | Pre-emptive validation rejects (BEFORE HTTP POST): empty `points`, >100 `points`, missing/zero `id`, missing/zero `regionSizeMeters`, OOR `zoomLevel`, OOR lat/lon |
| AC-7 | `seed_route.py --dry-run --tlog <derkachi.tlog>`: extracts route, prints planned payload + sha256, exit 0, no HTTP |
| AC-8 | `seed_route.py --tlog <derkachi.tlog>` against Jetson SP: exit 0, prints `RouteSeedResult`, optional summary JSON |
| AC-9 | Unit tests (mocked HTTPX): happy path, 400+ProblemDetails, 500 transient, terminal failure, timeout, dry-run, missing env, all pre-emptive validation cases |
| AC-10 | Integration test gated by `RUN_E2E=1` + `SATELLITE_PROVIDER_URL`: Derkachi route seeded, `tile_count > 0`, `maps_ready=True` |
## Out of scope
- FAISS index from seeded tiles (AZ-835 C3 / C5)
- C6 cache population (AZ-835 C3 — new `operator_pre_flight_setup` fixture)
- Modifying satellite-provider source (Route API consumed as-is)
- Multi-route batching (one RouteSpec → one POST)
- Authentication beyond existing JWT pattern (AZ-494)
## References
- Parent Epic: AZ-835 — https://denyspopov.atlassian.net/browse/AZ-835
- Sibling: AZ-836 (C1) — RouteSpec source
- Mirror CLI: `tests/fixtures/derkachi_c6/seed_region.py` (AZ-777 Phase 2)
- HTTP patterns: `src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py` (AZ-316/777 Phase 1)
- DTOs (in `../satellite-provider/`): `SatelliteProvider.Common/DTO/{CreateRouteRequest,RoutePoint,GeoPoint}.cs`
- Soft prereq: AZ-809 (Route API validation in satellite-provider)