From 763d8b21ad7a94ddd26bbcbd615392096486b902 Mon Sep 17 00:00:00 2001 From: Oleksandr Bezdieniezhnykh Date: Fri, 29 May 2026 16:42:55 +0300 Subject: [PATCH] [AZ-962] [AZ-964] [AZ-965] operator_replay.yaml + Tier-2 wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AZ-962 SHIPPED — Tier-2 Jetson AZ-840 orchestrator test no longer SKIPs at the env-var gate. configs/operator_replay.yaml registers c6/c7/c10/c11 with sane defaults (backbones intentionally empty, see AZ-965); docker-compose.test.jetson.yml exports GPS_DENIED_OPERATOR_CONFIG_PATH=/opt/configs/operator_replay.yaml and bind-mounts ./configs:/opt/configs:ro. ENV_KEY_MAP gains SATELLITE_PROVIDER_URL → c11_tile_manager.satellite_provider_url and SATELLITE_PROVIDER_API_KEY → c11_tile_manager.service_api_key so secrets flow from .env.test and never sit in YAML. README drops the manual export step. 97/97 c11 + config unit tests stay green. Tier-2 re-run (4 failed / 48 passed / 1 skipped / 1 xfailed / 1 xpassed / 2 errors in 84.99s vs baseline 3 skipped — i.e. -2 skipped, +2 errors): AZ-840 orchestrator test moves from SKIP to ERROR with a deeper, real gate — IndexUnavailableError on FaissDescriptorIndex against a fresh c6_tile_cache.root_dir. AZ-964 (3 SP, todo/) filed for FAISS index bootstrap in the AZ-839 C3 fixture. AZ-965 (3 SP, todo/, blocked by AZ-964) filed for NetVLAD ONNX backbone provisioning — the next gate the orchestrator test will hit once FAISS clears. Cycle-4 e2e gate remains NOT GREEN: AZ-840 chain is now AZ-964 → AZ-965 → PASS; 60s smoke chain is AZ-963 → PASS. OKVIS2 deferral directive (2026-05-29) unchanged — still gated behind Derkachi e2e green, still NOT MET. Co-authored-by: Cursor --- _docs/02_tasks/_dependencies_table.md | 6 +- .../AZ-962_operator_config_jetson_wiring.md | 11 ++- ...faiss_index_bootstrap_for_az839_fixture.md | 80 ++++++++++++++++++ ...-965_netvlad_onnx_backbone_provisioning.md | 83 +++++++++++++++++++ _docs/_autodev_state.md | 2 +- configs/operator_replay.yaml | 66 +++++++++++++++ docker-compose.test.jetson.yml | 7 ++ src/gps_denied_onboard/config/loader.py | 11 +++ tests/e2e/replay/README.md | 12 ++- 9 files changed, 272 insertions(+), 6 deletions(-) rename _docs/02_tasks/{todo => done}/AZ-962_operator_config_jetson_wiring.md (81%) create mode 100644 _docs/02_tasks/todo/AZ-964_faiss_index_bootstrap_for_az839_fixture.md create mode 100644 _docs/02_tasks/todo/AZ-965_netvlad_onnx_backbone_provisioning.md create mode 100644 configs/operator_replay.yaml diff --git a/_docs/02_tasks/_dependencies_table.md b/_docs/02_tasks/_dependencies_table.md index c7935ee..f53d6b1 100644 --- a/_docs/02_tasks/_dependencies_table.md +++ b/_docs/02_tasks/_dependencies_table.md @@ -1,8 +1,8 @@ # Dependencies Table -**Date**: 2026-05-29 (cycle-4 Step 11 Validate — **Tier-2 Jetson e2e run NOT GREEN; AZ-962 + AZ-963 filed**: Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh` on Jetson AGX Orin (aarch64, JetPack 6.x). Result: **4 failed, 48 passed, 3 skipped, 1 xfailed, 1 xpassed in 90.59s**. Two distinct blockers surfaced. **Blocker 1**: the AZ-840 orchestrator test (`test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration`) — the test that should prove the full 7-step cycle-4 pipeline works — was SKIPPED, not PASSed. `docker-compose.test.jetson.yml` does not export `GPS_DENIED_OPERATOR_CONFIG_PATH` (despite the comment claiming the Jetson harness sets it) AND the `operator_replay.yaml` the README references does not exist anywhere in the repo. This means Epic AZ-835's "Done" status across AZ-836/AZ-838/AZ-839/AZ-840/AZ-842 was validated **by doc-content presence only**, not by end-to-end test execution — exactly the failure mode `meta-rule.mdc` warns against ("tests that pass by skipping the component they are supposed to exercise create false confidence"). Filed **AZ-962** (3 SP) to author the missing YAML + wire the env var into the compose so the orchestrator test can actually run. **Blocker 2**: 4 tests in `test_derkachi_1min.py` (60s smoke, AZ-265/AZ-404) regressed to FAIL with `EstimatorFatalError('eskf filter divergence on vio: mahalanobis²=212.311 > 100.0')` at frame 233. Root cause: AZ-895 made the CSV-driven path primary; the CSV path runs open-loop because the Derkachi fixture has no reference C6 tile cache (no satellite anchoring → C5 ESKF integrates open-loop → diverges in ~10s). Before AZ-895 the tlog path was primary and presumably exited cleanly even without anchoring; the deprecation didn't account for the runtime-semantic difference. Additionally `test_ac3_within_100m_80pct_of_ticks` XPASSed (was xfail, unexpectedly passed) — third silent-failure surface needing investigation. Filed **AZ-963** (3 SP) for triage + fix decision (options A–F documented in the spec). **AZ-842 caveat**: the AZ-840/AZ-842 "Done" tracker state I set earlier today (commits 10c2a1e / 2cc992d) was based on the prior misunderstanding that the cycle-4 pipeline was validated end-to-end; the user-skipped convention question would tilt toward "needs walk-back to In Testing" if convention (A) holds. Recorded as caveat in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. **Honest cycle-4 status**: NOT green. Cycle-4 todo/ now contains: AZ-835 Epic (stale Jira tracker; children done locally but orchestrator validation gap), AZ-943/AZ-951/AZ-952 OKVIS2 chain (still deferred per user 2026-05-29 directive until Derkachi e2e green — directive unchanged), AZ-962 + AZ-963 (newly filed). OKVIS2 remains deferred. Earlier same-day — **AZ-842 tracker drift fixed + wider Jira drift audit recorded as leftover**: AZ-842 was shipped 2026-05-29 in commit `42b1db6` (spec in `done/AZ-842_replay_protocol_and_orchestrator_docs.md`, Invariant 14 + cycle-4 redesign narrative landed in `replay_protocol.md` + `architecture.md` + `tests/e2e/replay/README.md`) but the Jira ticket was stuck in To Do. **Fixed**: To Do → In Progress → Done in Jira (read-back verified). **Wider audit triggered** when AZ-842 fix surfaced 10 more shipped tickets stuck in "In Testing" (AZ-836/838/839/840/894/895/896/899/900/901) and Epic AZ-835 stuck in "To Do" with all 5 children Done/deferred. Asked user A/B/C/D ("Done = shipped+tested" vs "Done = QA-accepted" convention question); user skipped — interpreted as "use judgment, don't block". Recorded in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md` per scope-discipline rule (out-of-scope bulk modifications need explicit user direction). **Corrected cycle-4 todo/ remainder**: there is **NO product work** left in cycle-4 `todo/` — only Epic AZ-835 (stale tracker state, all children done) + the OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) which is deferred per user 2026-05-29 directive until after Derkachi e2e green. My earlier eighth + ninth bump narratives that listed "AZ-899 + AZ-900 + AZ-901 = 3 SP cycle-4 todo/ remainder" were fiction — those three specs have been in `done/` the whole time. Cycle-4 product work is effectively complete pending the Derkachi e2e flight test and the relocated AZ-897 UI in `../ui`. Earlier same-day — **AZ-961 LANDED** (`ReportContext.tlog_path` → `ground_truth_path` rename + label fix): `ReportContext` field renamed; rendered report line now reads `- Ground truth: ` for both tlog and CSV runs; AZ-959's inline comment documenting the field-overload removed. Two new symmetric tests in `test_az699_report_writer.py` assert the canonical label for both input formats. All 4 call sites updated: `helpers/accuracy_report.py` (field + docstring + render label), `replay_api/app.py` (kwarg), `tests/unit/test_az699_report_writer.py` (fixture + 2 new tests), `tests/e2e/replay/_e2e_orchestrator.py` (production e2e), `tests/e2e/replay/test_derkachi_real_tlog.py` (e2e test). 62/62 tests green across the three relevant unit-test modules. AZ-961 moved todo/ → done/. **Cycle-4 implement-batch sequence summary**: AZ-959 (3pt) + AZ-960 (2pt) + AZ-961 (1pt) all landed today as a coherent CSV-replay-input chain unblocking the relocated AZ-897 UI in `../ui`. Cycle-4 todo/ remainder for next batches: AZ-842 (3, docs) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1) = 6 SP product. OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) still in todo/ but sequenced after the Derkachi e2e green per user 2026-05-29 directive. Earlier same-day — **AZ-960 LANDED** (`gps-denied-render-map` CSV-truth dispatch): `load_ground_truth_track` now dispatches on `truth_path.suffix` (`.csv` → `load_csv_ground_truth`, else → `load_tlog_ground_truth`); `_maybe_render_map` short-circuit removed (AZ-959 workaround gone); CSV-path replay jobs now ship with `map_html_url` populated in the API response. 44/44 unit tests green across `test_az700_render_map.py` + `test_az701_replay_api.py` (17 pre-existing render-map tests pass per AC-2 + 3 new AZ-960 tests covering ACs 1+3+4 + AZ-959 happy-path test extended to assert `map_html_url`). AZ-960 moved todo/ → done/. Next batch: AZ-961 (ReportContext.tlog_path rename + label fix). Earlier same-day — **AZ-960 + AZ-961 filed as AZ-959 follow-ups** per user 2026-05-29 directive ("File AZ-960 (render-map CSV dispatch) + AZ-961 (ReportContext rename) and continue with one of them next"): the two deferred items surfaced during AZ-959 implementation are now tracked. **AZ-960** (2pt, todo/, `gps-denied-render-map`: dispatch `--truth` loader on extension to unblock CSV-path map render; deps AZ-700 + AZ-894 + AZ-959; no epic — UX-completing follow-up). **AZ-961** (1pt, todo/, `accuracy_report`: rename `ReportContext.tlog_path` → `ground_truth_path` + label fix in rendered report; deps AZ-699 + AZ-959; no epic — cosmetic cleanup). Next implement batch picks AZ-960 first (closes the UI map gap; AZ-961 sequenced after to avoid re-conflict on `_maybe_render_report`'s kwargs). Earlier same-day — **AZ-959 LANDED** (`replay_api` `POST /replay` CSV-path extension): handler now accepts `(video, tlog)` OR `(video, csv)` multipart with XOR validation; `validate_csv_kind` rejects malformed schemas at the API boundary referencing `csv_replay_format.md`; `SubprocessReplayRunner.run` dispatches `--imu` vs `--tlog`; `_maybe_render_report` dispatches GT loader via `load_csv_ground_truth` / `load_tlog_ground_truth`; `ReplayInputs` DTO carries `tlog_path: Path | None` + `csv_path: Path | None` with XOR `__post_init__`; `JobStorage` reserves both `tlog_path` and `csv_path`; new `GET /static/example-csv` endpoint serves the AZ-896 reference CSV via `REPLAY_API_EXAMPLE_CSV_PATH` env or source-checkout fallback. 27/27 unit tests green (18 pre-existing tlog tests pass unchanged per AC-7 + 9 new tests covering ACs 1-6 + 2 `validate_csv_kind` unit cases). **Deferred items (NOT silently fixed, surfaced to user as end-of-turn notes)**: (a) `gps-denied-render-map` only consumes binary tlog truth → CSV-path jobs return `map_html_url=None` (deferred to AZ-700 follow-up); (b) `ReportContext.tlog_path` field is now overloaded as "ground-truth source path"; rendered report's `Tlog: ` line is cosmetically misleading for CSV runs (deferred to AZ-699 follow-up). AZ-959 moved todo/ → done/. Cycle-4 active-scope SP delta: −3 SP (3 → 0). Earlier same-day — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) -**Total Tasks**: 184 (143 product + 41 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +AZ-959 (1 product task, todo/, 3pt). AZ-897 was never in this table's row count (pre-existing gap — the cycle-4 AZ-89x specs land in todo/ but were not back-filled into the table rows; not fixing that here, out of scope). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 183 (142 product + 41 blackbox-test) — +AZ-951 + AZ-952 (2 product tasks, both backlog/, 3pt each). AZ-943 (5pt) moved todo/ → backlog/ (no count change). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 181 (140 product + 41 blackbox-test) → 182 (141 product) — +AZ-943 (1 product task, originally todo/, 5pt). AZ-944 + AZ-945 remain Jira-only at the time of this update (sibling tickets, local specs deferred to their own Implement turns); their Total-Tasks impact will be reconciled when their specs land. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 180 (139 product + 41 blackbox-test) — +AZ-908 (1 product task, backlog/, 3pt). Prior 2026-05-26 cycle-4 Step 9 bump: +AZ-899 + AZ-900 + AZ-901 (3 product tasks). AZ-841 moved todo/ → backlog/ (no count change; backlog tickets are still in the table). Prior 2026-05-23 refactor-run bump: 176 (135 product + 41 blackbox-test) — +AZ-844 (Epic, 0pt umbrella for refactor run 02) + AZ-845 + AZ-846 + AZ-847 (3 product tasks). Prior 2026-05-23 bump (Epic AZ-835 decomposition): 173 (132 product + 41 blackbox-test) = +AZ-835 (Epic) + AZ-836 (C1) + AZ-837 (test-stack hardening, not this Epic) + AZ-838 (C2) added 2026-05-22→2026-05-23 prior to that update; +AZ-839 (C3) + AZ-840 (C4) + AZ-841 (C5) + AZ-842 (C6) added in that update. AZ-777 stays in the table (now closed in Jira; spec at `done/AZ-777_derkachi_c6_reference_fixture.md` retains 8pt credit for Phases 1+2 shipped). Earlier counts: 165 (124 product + 41 blackbox-test) — AZ-317 retained in the table marked SUPERSEDED for audit; AZ-523 (C11 gate removal) + AZ-524 (C12 rename) added as 2 closed audit-trail tasks; AZ-526 = 2pt clock-helper hygiene; AZ-527 = 2pt c2 engine-dim helper hygiene; AZ-528 = 3pt c1_vio facade-spine hygiene; AZ-558 = 3pt MavlinkTransport routing follow-up; AZ-559 closed Won't Fix; AZ-589 + AZ-590 closed Won't Fix (kept in table as 0pt audit-trail rows); AZ-591 = 5pt cross-cutting compose_root bootstrap (todo/); AZ-592 = 5pt OKVIS2 Tier-2 placeholder (backlog/); AZ-593 = 5pt VINS-Mono Tier-2 placeholder (backlog/); AZ-618 = 0pt umbrella (split into AZ-619..AZ-624 on 2026-05-19); AZ-619..AZ-624 = 6 subtasks of AZ-618 covering Phase A..F of the airborne `pre_constructed` assembly, summing to 16pt actionable work; AZ-687 = 2pt replay-mode guard follow-up surfaced by AZ-618 Tier-2 run on 2026-05-19 -**Total Complexity Points**: 584 (451 product + 133 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +3pt AZ-959. AZ-897 (5pt) was never table-counted here, so no decrement at this layer; the in-repo cycle-4 effort still drops by 5pt at the active-scope layer (AZ-897 work is now executed in `../ui`). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 581 (448 product + 133 blackbox-test) — +3pt AZ-951 + 3pt AZ-952 = +6 product pts. AZ-943 stays counted at 5pt (moved todo/ → backlog/, not deleted). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 580 (447 product + 133 blackbox-test) — +5pt AZ-943. AZ-944 (3pt) + AZ-945 (3pt) sibling tickets are filed Jira-side but not yet imported as local specs; their +6pt will land when AZ-944 / AZ-945 specs are authored. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 570 (437 product + 133 blackbox-test) — +3pt AZ-908. Prior 2026-05-26 cycle-4 Step 9 bump: +1pt AZ-899 + 1pt AZ-900 + 1pt AZ-901 + 1pt AZ-842 rescope (2→3) = +4 product pts. Prior 2026-05-23 refactor-run bump: 563 (430 product + 133 blackbox-test) — +2pt AZ-845 + 2pt AZ-846 + 2pt AZ-847 = +6 product pts on top of prior reconciled total (AZ-844 epic itself is 0pt umbrella). Prior 2026-05-23 reconciled total: 557 (424 product + 133 blackbox-test) — +5pt AZ-839 + 3pt AZ-840 + 1pt AZ-841 + 2pt AZ-842 = +11 product pts on top of prior reconciled total. AZ-836 (3pt) + AZ-838 (3pt) were added 2026-05-22→2026-05-23 prior to that update; AZ-837 (test-stack hardening, not this Epic) is unaccounted in that delta and should be folded in at the next preamble reconciliation. Earlier baseline: 546 (413 product + 133 blackbox-test) — +3pt AZ-776 + 8pt AZ-777 (5→8 override 2026-05-21 cycle-3 batch 104; see `_docs/_process_leftovers/2026-05-21_az777_complexity_override.md` for rationale + the spec refresh that pulled e2e-runner wiring + C11 contract adapt + Derkachi catalog seed + fixture replacement + un-xfail into one ticket) — AZ-523 = 3pt, AZ-524 = 2pt, AZ-526 = 2pt, AZ-527 = 2pt, AZ-528 = 3pt, AZ-558 = 3pt, AZ-589 + AZ-590 retained at 5pt each but closed Won't Fix (treated as 0 effective pts going forward), AZ-591 = 5pt, AZ-592 = 5pt placeholder, AZ-593 = 5pt placeholder, AZ-618 = 0pt umbrella post-split, AZ-619 = 2pt, AZ-620 = 3pt, AZ-621 = 3pt, AZ-622 = 3pt, AZ-623 = 3pt, AZ-624 = 2pt, AZ-687 = 2pt +**Date**: 2026-05-29 (cycle-4 Step 11 Validate — **AZ-962 SHIPPED + AZ-964 + AZ-965 filed; cycle-4 e2e gate still NOT GREEN**: Implemented AZ-962 end-to-end same day. Changes: (a) `configs/operator_replay.yaml` authored (registers c6/c7/c10/c11 blocks, all defaults except `backbones: []` which is intentionally empty — see AZ-965); (b) `docker-compose.test.jetson.yml` exports `GPS_DENIED_OPERATOR_CONFIG_PATH=/opt/configs/operator_replay.yaml` + bind-mounts `./configs:/opt/configs:ro`; (c) `ENV_KEY_MAP` (`src/gps_denied_onboard/config/loader.py`) gained `SATELLITE_PROVIDER_URL` → `c11_tile_manager.satellite_provider_url` + `SATELLITE_PROVIDER_API_KEY` → `c11_tile_manager.service_api_key` so secrets stay out of the YAML and flow in from `.env.test`; (d) README `tests/e2e/replay/README.md` updated to drop the manual `export GPS_DENIED_OPERATOR_CONFIG_PATH=...` step. 97/97 c11+config unit tests still green. Tier-2 re-run on Jetson AGX Orin (`JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`): **4 failed / 48 passed / 1 skipped / 1 xfailed / 1 xpassed / 2 errors in 84.99s** (previously 4 failed / 48 passed / **3 skipped** / 1 xfailed / 1 xpassed — i.e. -2 skipped, +2 errors). AZ-962 AC-3 + AC-4 satisfied: the AZ-840 orchestrator test no longer SKIPs at the env-var gate; it now ERRORs at a deeper, real gate during fixture setup with `IndexUnavailableError: FaissDescriptorIndex: .index file missing at /tmp/pytest-of-root/pytest-0/operator_pre_flight_cache0/descriptor.index`. The same error also breaks `test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache`, confirming a fixture-wide bug. Root cause: `tests/e2e/replay/conftest.py:487` calls `build_descriptor_index(config)` against a fresh empty `c6_tile_cache.root_dir` (tmp dir per AZ-839 fixture invariant) — the FAISS factory requires an existing `.index` file. The `tile-init` compose service exists but writes its seed index to `/var/lib/gps-denied/tiles`, not the tmp dir the fixture overrides into. Filed **AZ-964** (3 SP, To Do, `todo/`) for the FAISS bootstrap fix (preferred option: invoke `scripts/mk_test_faiss_fixture.py` inline from the fixture against the override `root_dir`). Filed **AZ-965** (3 SP, To Do, `todo/`, blocked by AZ-964) for NetVLAD ONNX backbone provisioning (the next gate the test will hit after AZ-964 clears — `c10_provisioning.backbones` empty → empty-backbones SKIP). AZ-962 transitioned To Do → In Progress → Done in Jira (read-back verified, `status.name="Done"`). AZ-962 spec moved todo/ → done/. **Honest cycle-4 status**: still NOT GREEN. AZ-840 orchestrator chain is now AZ-964 → AZ-965 → orchestrator PASS; the 60s smoke regression chain is AZ-963 → 4 derkachi_1min tests PASS. OKVIS2 deferral directive (2026-05-29) unchanged — still gated behind Derkachi e2e green, still NOT MET. Cycle-4 active scope todo/: AZ-963 (3 SP) + AZ-964 (3 SP) + AZ-965 (3 SP) = 9 SP newly added; plus pre-existing OKVIS2 chain AZ-943+AZ-951+AZ-952 still deferred. **Earlier same-day** — Tier-2 Jetson e2e run NOT GREEN; AZ-962 + AZ-963 filed: Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh` on Jetson AGX Orin (aarch64, JetPack 6.x). Result: **4 failed, 48 passed, 3 skipped, 1 xfailed, 1 xpassed in 90.59s**. Two distinct blockers surfaced. **Blocker 1**: the AZ-840 orchestrator test (`test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration`) — the test that should prove the full 7-step cycle-4 pipeline works — was SKIPPED, not PASSed. `docker-compose.test.jetson.yml` does not export `GPS_DENIED_OPERATOR_CONFIG_PATH` (despite the comment claiming the Jetson harness sets it) AND the `operator_replay.yaml` the README references does not exist anywhere in the repo. This means Epic AZ-835's "Done" status across AZ-836/AZ-838/AZ-839/AZ-840/AZ-842 was validated **by doc-content presence only**, not by end-to-end test execution — exactly the failure mode `meta-rule.mdc` warns against ("tests that pass by skipping the component they are supposed to exercise create false confidence"). Filed **AZ-962** (3 SP) to author the missing YAML + wire the env var into the compose so the orchestrator test can actually run. **Blocker 2**: 4 tests in `test_derkachi_1min.py` (60s smoke, AZ-265/AZ-404) regressed to FAIL with `EstimatorFatalError('eskf filter divergence on vio: mahalanobis²=212.311 > 100.0')` at frame 233. Root cause: AZ-895 made the CSV-driven path primary; the CSV path runs open-loop because the Derkachi fixture has no reference C6 tile cache (no satellite anchoring → C5 ESKF integrates open-loop → diverges in ~10s). Before AZ-895 the tlog path was primary and presumably exited cleanly even without anchoring; the deprecation didn't account for the runtime-semantic difference. Additionally `test_ac3_within_100m_80pct_of_ticks` XPASSed (was xfail, unexpectedly passed) — third silent-failure surface needing investigation. Filed **AZ-963** (3 SP) for triage + fix decision (options A–F documented in the spec). **AZ-842 caveat**: the AZ-840/AZ-842 "Done" tracker state I set earlier today (commits 10c2a1e / 2cc992d) was based on the prior misunderstanding that the cycle-4 pipeline was validated end-to-end; the user-skipped convention question would tilt toward "needs walk-back to In Testing" if convention (A) holds. Recorded as caveat in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. **Honest cycle-4 status**: NOT green. Cycle-4 todo/ now contains: AZ-835 Epic (stale Jira tracker; children done locally but orchestrator validation gap), AZ-943/AZ-951/AZ-952 OKVIS2 chain (still deferred per user 2026-05-29 directive until Derkachi e2e green — directive unchanged), AZ-962 + AZ-963 (newly filed). OKVIS2 remains deferred. Earlier same-day — **AZ-842 tracker drift fixed + wider Jira drift audit recorded as leftover**: AZ-842 was shipped 2026-05-29 in commit `42b1db6` (spec in `done/AZ-842_replay_protocol_and_orchestrator_docs.md`, Invariant 14 + cycle-4 redesign narrative landed in `replay_protocol.md` + `architecture.md` + `tests/e2e/replay/README.md`) but the Jira ticket was stuck in To Do. **Fixed**: To Do → In Progress → Done in Jira (read-back verified). **Wider audit triggered** when AZ-842 fix surfaced 10 more shipped tickets stuck in "In Testing" (AZ-836/838/839/840/894/895/896/899/900/901) and Epic AZ-835 stuck in "To Do" with all 5 children Done/deferred. Asked user A/B/C/D ("Done = shipped+tested" vs "Done = QA-accepted" convention question); user skipped — interpreted as "use judgment, don't block". Recorded in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md` per scope-discipline rule (out-of-scope bulk modifications need explicit user direction). **Corrected cycle-4 todo/ remainder**: there is **NO product work** left in cycle-4 `todo/` — only Epic AZ-835 (stale tracker state, all children done) + the OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) which is deferred per user 2026-05-29 directive until after Derkachi e2e green. My earlier eighth + ninth bump narratives that listed "AZ-899 + AZ-900 + AZ-901 = 3 SP cycle-4 todo/ remainder" were fiction — those three specs have been in `done/` the whole time. Cycle-4 product work is effectively complete pending the Derkachi e2e flight test and the relocated AZ-897 UI in `../ui`. Earlier same-day — **AZ-961 LANDED** (`ReportContext.tlog_path` → `ground_truth_path` rename + label fix): `ReportContext` field renamed; rendered report line now reads `- Ground truth: ` for both tlog and CSV runs; AZ-959's inline comment documenting the field-overload removed. Two new symmetric tests in `test_az699_report_writer.py` assert the canonical label for both input formats. All 4 call sites updated: `helpers/accuracy_report.py` (field + docstring + render label), `replay_api/app.py` (kwarg), `tests/unit/test_az699_report_writer.py` (fixture + 2 new tests), `tests/e2e/replay/_e2e_orchestrator.py` (production e2e), `tests/e2e/replay/test_derkachi_real_tlog.py` (e2e test). 62/62 tests green across the three relevant unit-test modules. AZ-961 moved todo/ → done/. **Cycle-4 implement-batch sequence summary**: AZ-959 (3pt) + AZ-960 (2pt) + AZ-961 (1pt) all landed today as a coherent CSV-replay-input chain unblocking the relocated AZ-897 UI in `../ui`. Cycle-4 todo/ remainder for next batches: AZ-842 (3, docs) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1) = 6 SP product. OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) still in todo/ but sequenced after the Derkachi e2e green per user 2026-05-29 directive. Earlier same-day — **AZ-960 LANDED** (`gps-denied-render-map` CSV-truth dispatch): `load_ground_truth_track` now dispatches on `truth_path.suffix` (`.csv` → `load_csv_ground_truth`, else → `load_tlog_ground_truth`); `_maybe_render_map` short-circuit removed (AZ-959 workaround gone); CSV-path replay jobs now ship with `map_html_url` populated in the API response. 44/44 unit tests green across `test_az700_render_map.py` + `test_az701_replay_api.py` (17 pre-existing render-map tests pass per AC-2 + 3 new AZ-960 tests covering ACs 1+3+4 + AZ-959 happy-path test extended to assert `map_html_url`). AZ-960 moved todo/ → done/. Next batch: AZ-961 (ReportContext.tlog_path rename + label fix). Earlier same-day — **AZ-960 + AZ-961 filed as AZ-959 follow-ups** per user 2026-05-29 directive ("File AZ-960 (render-map CSV dispatch) + AZ-961 (ReportContext rename) and continue with one of them next"): the two deferred items surfaced during AZ-959 implementation are now tracked. **AZ-960** (2pt, todo/, `gps-denied-render-map`: dispatch `--truth` loader on extension to unblock CSV-path map render; deps AZ-700 + AZ-894 + AZ-959; no epic — UX-completing follow-up). **AZ-961** (1pt, todo/, `accuracy_report`: rename `ReportContext.tlog_path` → `ground_truth_path` + label fix in rendered report; deps AZ-699 + AZ-959; no epic — cosmetic cleanup). Next implement batch picks AZ-960 first (closes the UI map gap; AZ-961 sequenced after to avoid re-conflict on `_maybe_render_report`'s kwargs). Earlier same-day — **AZ-959 LANDED** (`replay_api` `POST /replay` CSV-path extension): handler now accepts `(video, tlog)` OR `(video, csv)` multipart with XOR validation; `validate_csv_kind` rejects malformed schemas at the API boundary referencing `csv_replay_format.md`; `SubprocessReplayRunner.run` dispatches `--imu` vs `--tlog`; `_maybe_render_report` dispatches GT loader via `load_csv_ground_truth` / `load_tlog_ground_truth`; `ReplayInputs` DTO carries `tlog_path: Path | None` + `csv_path: Path | None` with XOR `__post_init__`; `JobStorage` reserves both `tlog_path` and `csv_path`; new `GET /static/example-csv` endpoint serves the AZ-896 reference CSV via `REPLAY_API_EXAMPLE_CSV_PATH` env or source-checkout fallback. 27/27 unit tests green (18 pre-existing tlog tests pass unchanged per AC-7 + 9 new tests covering ACs 1-6 + 2 `validate_csv_kind` unit cases). **Deferred items (NOT silently fixed, surfaced to user as end-of-turn notes)**: (a) `gps-denied-render-map` only consumes binary tlog truth → CSV-path jobs return `map_html_url=None` (deferred to AZ-700 follow-up); (b) `ReportContext.tlog_path` field is now overloaded as "ground-truth source path"; rendered report's `Tlog: ` line is cosmetically misleading for CSV runs (deferred to AZ-699 follow-up). AZ-959 moved todo/ → done/. Cycle-4 active-scope SP delta: −3 SP (3 → 0). Earlier same-day — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) +**Total Tasks**: 186 (145 product + 41 blackbox-test) — 2026-05-29 cycle-4 Step 11 Validate (AZ-962 shipped + AZ-964 + AZ-965 filed): +AZ-964 (1 product task, todo/, 3pt) + AZ-965 (1 product task, todo/, 3pt). AZ-962 (already counted at +3pt earlier today) stays counted; status moved todo/ → done/. Prior 2026-05-29 cycle-4 Step 11 first count bump (AZ-962 + AZ-963 filed): 184 (143 product) → 184 stays at the count layer — that prior preamble bump implicitly added AZ-962 + AZ-963 but the explicit row counts didn't get reconciled (pre-existing gap; not fixing here, out of scope). Prior same-day 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +AZ-959 (1 product task, todo/, 3pt). AZ-897 was never in this table's row count (pre-existing gap — the cycle-4 AZ-89x specs land in todo/ but were not back-filled into the table rows; not fixing that here, out of scope). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 183 (142 product + 41 blackbox-test) — +AZ-951 + AZ-952 (2 product tasks, both backlog/, 3pt each). AZ-943 (5pt) moved todo/ → backlog/ (no count change). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 181 (140 product + 41 blackbox-test) → 182 (141 product) — +AZ-943 (1 product task, originally todo/, 5pt). AZ-944 + AZ-945 remain Jira-only at the time of this update (sibling tickets, local specs deferred to their own Implement turns); their Total-Tasks impact will be reconciled when their specs land. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 180 (139 product + 41 blackbox-test) — +AZ-908 (1 product task, backlog/, 3pt). Prior 2026-05-26 cycle-4 Step 9 bump: +AZ-899 + AZ-900 + AZ-901 (3 product tasks). AZ-841 moved todo/ → backlog/ (no count change; backlog tickets are still in the table). Prior 2026-05-23 refactor-run bump: 176 (135 product + 41 blackbox-test) — +AZ-844 (Epic, 0pt umbrella for refactor run 02) + AZ-845 + AZ-846 + AZ-847 (3 product tasks). Prior 2026-05-23 bump (Epic AZ-835 decomposition): 173 (132 product + 41 blackbox-test) = +AZ-835 (Epic) + AZ-836 (C1) + AZ-837 (test-stack hardening, not this Epic) + AZ-838 (C2) added 2026-05-22→2026-05-23 prior to that update; +AZ-839 (C3) + AZ-840 (C4) + AZ-841 (C5) + AZ-842 (C6) added in that update. AZ-777 stays in the table (now closed in Jira; spec at `done/AZ-777_derkachi_c6_reference_fixture.md` retains 8pt credit for Phases 1+2 shipped). Earlier counts: 165 (124 product + 41 blackbox-test) — AZ-317 retained in the table marked SUPERSEDED for audit; AZ-523 (C11 gate removal) + AZ-524 (C12 rename) added as 2 closed audit-trail tasks; AZ-526 = 2pt clock-helper hygiene; AZ-527 = 2pt c2 engine-dim helper hygiene; AZ-528 = 3pt c1_vio facade-spine hygiene; AZ-558 = 3pt MavlinkTransport routing follow-up; AZ-559 closed Won't Fix; AZ-589 + AZ-590 closed Won't Fix (kept in table as 0pt audit-trail rows); AZ-591 = 5pt cross-cutting compose_root bootstrap (todo/); AZ-592 = 5pt OKVIS2 Tier-2 placeholder (backlog/); AZ-593 = 5pt VINS-Mono Tier-2 placeholder (backlog/); AZ-618 = 0pt umbrella (split into AZ-619..AZ-624 on 2026-05-19); AZ-619..AZ-624 = 6 subtasks of AZ-618 covering Phase A..F of the airborne `pre_constructed` assembly, summing to 16pt actionable work; AZ-687 = 2pt replay-mode guard follow-up surfaced by AZ-618 Tier-2 run on 2026-05-19 +**Total Complexity Points**: 590 (457 product + 133 blackbox-test) — 2026-05-29 cycle-4 Step 11 Validate (AZ-962 shipped + AZ-964 + AZ-965 filed): +3pt AZ-964 + 3pt AZ-965 = +6 product pts. AZ-962 stays counted at +3pt (added earlier today). Prior same-day 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +3pt AZ-959. AZ-897 (5pt) was never table-counted here, so no decrement at this layer; the in-repo cycle-4 effort still drops by 5pt at the active-scope layer (AZ-897 work is now executed in `../ui`). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 581 (448 product + 133 blackbox-test) — +3pt AZ-951 + 3pt AZ-952 = +6 product pts. AZ-943 stays counted at 5pt (moved todo/ → backlog/, not deleted). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 580 (447 product + 133 blackbox-test) — +5pt AZ-943. AZ-944 (3pt) + AZ-945 (3pt) sibling tickets are filed Jira-side but not yet imported as local specs; their +6pt will land when AZ-944 / AZ-945 specs are authored. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 570 (437 product + 133 blackbox-test) — +3pt AZ-908. Prior 2026-05-26 cycle-4 Step 9 bump: +1pt AZ-899 + 1pt AZ-900 + 1pt AZ-901 + 1pt AZ-842 rescope (2→3) = +4 product pts. Prior 2026-05-23 refactor-run bump: 563 (430 product + 133 blackbox-test) — +2pt AZ-845 + 2pt AZ-846 + 2pt AZ-847 = +6 product pts on top of prior reconciled total (AZ-844 epic itself is 0pt umbrella). Prior 2026-05-23 reconciled total: 557 (424 product + 133 blackbox-test) — +5pt AZ-839 + 3pt AZ-840 + 1pt AZ-841 + 2pt AZ-842 = +11 product pts on top of prior reconciled total. AZ-836 (3pt) + AZ-838 (3pt) were added 2026-05-22→2026-05-23 prior to that update; AZ-837 (test-stack hardening, not this Epic) is unaccounted in that delta and should be folded in at the next preamble reconciliation. Earlier baseline: 546 (413 product + 133 blackbox-test) — +3pt AZ-776 + 8pt AZ-777 (5→8 override 2026-05-21 cycle-3 batch 104; see `_docs/_process_leftovers/2026-05-21_az777_complexity_override.md` for rationale + the spec refresh that pulled e2e-runner wiring + C11 contract adapt + Derkachi catalog seed + fixture replacement + un-xfail into one ticket) — AZ-523 = 3pt, AZ-524 = 2pt, AZ-526 = 2pt, AZ-527 = 2pt, AZ-528 = 3pt, AZ-558 = 3pt, AZ-589 + AZ-590 retained at 5pt each but closed Won't Fix (treated as 0 effective pts going forward), AZ-591 = 5pt, AZ-592 = 5pt placeholder, AZ-593 = 5pt placeholder, AZ-618 = 0pt umbrella post-split, AZ-619 = 2pt, AZ-620 = 3pt, AZ-621 = 3pt, AZ-622 = 3pt, AZ-623 = 3pt, AZ-624 = 2pt, AZ-687 = 2pt Dependencies columns list only the tracker-ID portion (descriptive tail text in each task spec is omitted here for table-readability). The diff --git a/_docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md b/_docs/02_tasks/done/AZ-962_operator_config_jetson_wiring.md similarity index 81% rename from _docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md rename to _docs/02_tasks/done/AZ-962_operator_config_jetson_wiring.md index ed3ad92..2991f8a 100644 --- a/_docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md +++ b/_docs/02_tasks/done/AZ-962_operator_config_jetson_wiring.md @@ -1,11 +1,20 @@ # AZ-962 — Wire `GPS_DENIED_OPERATOR_CONFIG_PATH` + `operator_replay.yaml` into Tier-2 Jetson harness -**Status**: To Do (Jira) / `todo/` (local) +**Status**: Done (Jira) / `done/` (local) **Issue type**: Task **Complexity**: 3 SP **Cycle**: cycle-4 e2e closure follow-up **Jira**: https://denyspopov.atlassian.net/browse/AZ-962 **Filed**: 2026-05-29 during cycle-4 Tier-2 validation run +**Shipped**: 2026-05-29 (same day) + +## Closure note (2026-05-29) + +Shipped: `configs/operator_replay.yaml` authored (registers all 4 blocks c6/c7/c10/c11), `docker-compose.test.jetson.yml` exports `GPS_DENIED_OPERATOR_CONFIG_PATH=/opt/configs/operator_replay.yaml` and bind-mounts `./configs:/opt/configs:ro`, and `ENV_KEY_MAP` (`src/gps_denied_onboard/config/loader.py`) gained two entries for `SATELLITE_PROVIDER_URL` / `SATELLITE_PROVIDER_API_KEY` → `c11_tile_manager` so secrets stay out of the YAML and flow in from `.env.test`. README `tests/e2e/replay/README.md` updated to drop the manual `export GPS_DENIED_OPERATOR_CONFIG_PATH=...` step. + +Tier-2 re-run on Jetson AGX Orin (`JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`): 4 failed / 48 passed / 1 skipped / 1 xfailed / 1 xpassed / 2 errors in 84.99s. AC-3 satisfied — `test_az840_e2e_real_flight_orchestration` no longer SKIPs at the env-var gate. AC-4 satisfied — it now ERRORs at a deeper, real gate (`IndexUnavailableError: FaissDescriptorIndex: .index file missing at /tmp/pytest-of-root/pytest-0/operator_pre_flight_cache0/descriptor.index`) which is captured in a NEW follow-up ticket **AZ-964**. The empty-backbones gate that this spec originally flagged (c10 backbones) becomes the gate AFTER AZ-964 clears — filed as **AZ-965**. + +Net cycle-4 status remains NOT GREEN (orchestrator test still doesn't PASS, blocked by AZ-964 + AZ-965; ESKF divergence regression still blocked by AZ-963). AZ-962 itself is complete. ## Why diff --git a/_docs/02_tasks/todo/AZ-964_faiss_index_bootstrap_for_az839_fixture.md b/_docs/02_tasks/todo/AZ-964_faiss_index_bootstrap_for_az839_fixture.md new file mode 100644 index 0000000..c57e05a --- /dev/null +++ b/_docs/02_tasks/todo/AZ-964_faiss_index_bootstrap_for_az839_fixture.md @@ -0,0 +1,80 @@ +# AZ-964 — Bootstrap FAISS descriptor index for AZ-839 C3 fixture (`operator_pre_flight_cache`) + +**Status**: To Do (Jira) / `todo/` (local) +**Issue type**: Task +**Complexity**: 3 SP +**Cycle**: cycle-4 e2e closure follow-up +**Jira**: https://denyspopov.atlassian.net/browse/AZ-964 +**Filed**: 2026-05-29 (surfaced by AZ-962 Tier-2 re-run) + +## Why + +Discovered 2026-05-29 during the AZ-962 Tier-2 re-run on Jetson AGX Orin. With `GPS_DENIED_OPERATOR_CONFIG_PATH` + `operator_replay.yaml` now correctly wired (AZ-962 shipped), the AZ-840 orchestrator test (`tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration`) moved from SKIPped to ERRORed at a deeper, real gate during fixture setup: + +``` +gps_denied_onboard.components.c6_tile_cache.errors.IndexUnavailableError: +FaissDescriptorIndex: .index file missing at +/tmp/pytest-of-root/pytest-0/operator_pre_flight_cache0/descriptor.index +``` + +The same error also breaks `test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache`, confirming this is a fixture-wide problem, not specific to one test. + +## Root cause (read from code) + +`tests/e2e/replay/conftest.py::_build_operator_pre_flight_cache` (line 487): + +1. Overrides `c6_tile_cache.root_dir` to a fresh `/tmp/pytest-of-root/.../operator_pre_flight_cache0/` (per AC of AZ-839, the fixture creates a *new* cache each test). +2. Calls `build_descriptor_index(config)` — which constructs `FaissDescriptorIndex.from_config(config)`. +3. `FaissDescriptorIndex.__init__` calls `_load()` which **raises** `IndexUnavailableError` when no `.index` file exists at `c6_tile_cache.root_dir/descriptor.index`. +4. The fixture never gets to call `populate_c6_from_route` (which presumably creates the index downstream). + +The compose `tile-init` setup service exists and runs `scripts/mk_test_faiss_fixture.py` — but it writes a seed index to `/var/lib/gps-denied/tiles` (the `tile-data` volume), **not** to the tmp dir the fixture overrides into. So the fixture's override path always starts empty. + +## Goal + +Make `_build_operator_pre_flight_cache` succeed past the `build_descriptor_index(config)` call so the AZ-840 orchestrator test can actually exercise the 7-step pipeline (or fail at the next real gate — c10 backbones, AZ-965). + +## Scope + +One of (in preference order; pick during implementation): + +1. **Fixture seeds the index inline**: before calling `build_descriptor_index`, invoke `scripts/mk_test_faiss_fixture.py` programmatically (or in-process equivalent) against the override `root_dir`. Pure test-infra change. +2. **`populate_c6_from_route` creates the index if missing**: production code change so the descriptor-index factory tolerates a fresh `root_dir`. Larger blast radius — touches a shared factory. +3. **`FaissDescriptorIndex` supports an explicit `bootstrap=True` mode**: factory signal that this run intends to create a fresh index. Requires API design. + +Option (1) is the smallest, lowest-risk path and the natural extension of the `tile-init` pattern already in compose. **Recommended.** + +## Acceptance Criteria + +* **AC-1**: `_build_operator_pre_flight_cache` no longer ERRORs at `build_descriptor_index` when started against a fresh empty `c6_tile_cache.root_dir`. +* **AC-2**: `JETSON_SSH_ALIAS= bash scripts/run-tests-jetson.sh` no longer reports the `IndexUnavailableError` for `test_az840_e2e_real_flight_orchestration` **or** for `test_operator_pre_flight_setup_produces_populated_cache`. +* **AC-3**: If the AZ-840 orchestrator test now reaches the c10-backbone gate (`AZ-839 operator_pre_flight_setup: config has no c10_provisioning.backbones entries`), that's the expected next gate — AZ-965 handles it; AZ-964 is done. +* **AC-4**: `tests/unit` + `tests/e2e/replay/test_operator_pre_flight_*` continue to pass on Tier-1 (Colima). + +## Out of scope + +* c10 backbone provisioning (separate ticket — AZ-965). +* The 4 ESKF-divergence regression failures in `test_derkachi_1min.py` (separate ticket — AZ-963). +* Adding a reference C6 tile cache for the Derkachi fixture (large separate work). +* Re-opening AZ-840 / AZ-842 tracker state. + +## Dependencies + +* **Blocks**: AZ-840 (orchestrator test cannot run end-to-end until this clears). +* **Surfaced by**: AZ-962 (env-var + YAML wiring exposed the next gate). +* **Related**: AZ-839 (C3 fixture — this is its bug to own). + +## Estimate + +3 SP. Multi-step (locate the seed-index script, invoke it from the fixture before `build_descriptor_index`, verify on Tier-2), moderate risk (the seed script's assumptions might not match the fixture's override path layout). + +## References + +* Run log: 2026-05-29 Tier-2 Jetson AGX Orin (AZ-962 re-run), 84.99s, 4 failed / 48 passed / 1 skipped / 1 xfailed / 1 xpassed / 2 errors +* Test: `tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration` (ERROR) +* Test: `tests/e2e/replay/test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache` (ERROR) +* Fixture: `tests/e2e/replay/conftest.py:487` +* Faulting factory: `src/gps_denied_onboard/runtime_root/storage_factory.py:176` +* Faulting class: `src/gps_denied_onboard/components/c6_tile_cache/faiss_descriptor_index.py:107,430` +* Existing seed script: `scripts/mk_test_faiss_fixture.py` (invoked by `tile-init` compose service) +* AZ-962 spec: `_docs/02_tasks/done/AZ-962_operator_config_jetson_wiring.md` diff --git a/_docs/02_tasks/todo/AZ-965_netvlad_onnx_backbone_provisioning.md b/_docs/02_tasks/todo/AZ-965_netvlad_onnx_backbone_provisioning.md new file mode 100644 index 0000000..893ca08 --- /dev/null +++ b/_docs/02_tasks/todo/AZ-965_netvlad_onnx_backbone_provisioning.md @@ -0,0 +1,83 @@ +# AZ-965 — Provision NetVLAD ONNX backbone for AZ-839 `c10_provisioning` corpus + +**Status**: To Do (Jira) / `todo/` (local) +**Issue type**: Task +**Complexity**: 3 SP (5 SP if export/training required) +**Cycle**: cycle-4 e2e closure follow-up +**Jira**: https://denyspopov.atlassian.net/browse/AZ-965 +**Filed**: 2026-05-29 (forward-looked during AZ-962) + +## Why + +Forward-looked during AZ-962. The AZ-839 C3 fixture's `_build_replay_backbone_embedder` (`conftest.py:594-601`) calls `build_backbone_specs(config)` which reads `config.components['c10_provisioning'].backbones` (a tuple of `BackboneSpec`). When empty (the current state — no `.onnx` files ship in the repo), the fixture `pytest.skip`s with: + +``` +AZ-839 operator_pre_flight_setup: config has no c10_provisioning.backbones +entries — the e2e harness config must declare at least one backbone +(typically DINOv2-VPR or NetVLAD per AZ-321). +``` + +The AZ-962 YAML (`configs/operator_replay.yaml`) explicitly leaves the `backbones:` list empty with a TODO note pointing at this ticket. Right now (post-AZ-962) the AZ-840 orchestrator test ERRORs at the FAISS-index gate (AZ-964) **before** reaching the backbones gate — but once AZ-964 ships, this is the next blocker. + +## Goal + +Provision a NetVLAD `.onnx` model (per AZ-321's pinned backbone choice) and matching `BackboneSpec` entry in `configs/operator_replay.yaml` so `c10_provisioning.compile_engines_for_corpus` can compile at least one engine in the AZ-839 fixture. + +## Scope + +1. **Source a NetVLAD `.onnx`**: AZ-321 specifies NetVLAD as the C2 baseline. Either: + - Export from an existing PyTorch checkpoint our team owns; + - Pull a vetted public weights file (with license/provenance recorded in `_docs/03_ip_attribution/`); + - Train from scratch (out of scope for this ticket — file a follow-up if neither of the above works). +2. **Place the `.onnx` in the repo**: under a path that's bind-mounted into the Jetson container (e.g. `models/netvlad/netvlad.onnx`). Add to `.gitattributes` for git-lfs if >50 MiB. Verify size against existing checked-in models. +3. **Verify TensorRT compile**: run `c7_inference.PyTorchFp16Runtime.compile_engine` (or the relevant production code path) against the new `.onnx` on Jetson AGX Orin to confirm a `.engine` file is produced with a sensible descriptor dim (typically 4096 per AZ-321). +4. **Populate `configs/operator_replay.yaml`**: + + ```yaml + c10_provisioning: + workspace_mb: 4096 + backbones: + - model_name: netvlad + onnx_path: /opt/models/netvlad/netvlad.onnx + input_name: image + input_shape_chw: [3, 224, 224] + descriptor_dim: 4096 + ``` + + (Exact field names per `BackboneSpec` dataclass — verify in `src/gps_denied_onboard/components/c10_provisioning/`.) +5. **Wire `./models` bind-mount** into `docker-compose.test.jetson.yml`. +6. **Update `c2_vpr` block** in the YAML if `_resolve_replay_descriptor_dim` requires `c2_vpr.strategy='net_vlad'` (it does — see `conftest.py:658-666`). + +## Acceptance Criteria + +* **AC-1**: `models/netvlad/netvlad.onnx` (or equivalent path) exists in the repo with documented provenance + license. +* **AC-2**: `c7_inference` can compile this `.onnx` to a TensorRT `.engine` on Jetson AGX Orin (Tier-2) without errors. +* **AC-3**: `configs/operator_replay.yaml` declares the `netvlad` backbone in `c10_provisioning.backbones`. +* **AC-4**: `JETSON_SSH_ALIAS= bash scripts/run-tests-jetson.sh` no longer SKIPs `test_az840_e2e_real_flight_orchestration` with the empty-backbones message. +* **AC-5**: The AZ-840 orchestrator test either PASSes (and the AZ-699 verdict report lands at `_docs/06_metrics/real_flight_validation_.md`) or fails with a NEW error filed as a separate follow-up ticket. +* **AC-6**: License/provenance recorded in `_docs/03_ip_attribution/` per project convention. + +## Out of scope + +* DINOv2-VPR or other alternative backbones (NetVLAD is AZ-321's pinned baseline). +* MegaLoc / MixVPR / UltraVPR (these require a descriptor-dim resolver change — out of conftest scope). +* The 4 ESKF-divergence regression failures (AZ-963). +* Reference C6 tile cache for the Derkachi fixture (large separate work). + +## Dependencies + +* **Blocked by**: AZ-964 (FAISS index bootstrap — the orchestrator test ERRORs there before reaching this gate; clearing AZ-964 first surfaces the empty-backbones gate cleanly). +* **Blocks**: AZ-840 (orchestrator test cannot PASS end-to-end without a real backbone). +* **Related**: AZ-321 (defines NetVLAD as the C2 baseline), AZ-839 (C3 fixture). + +## Estimate + +3 SP if a usable `.onnx` already exists in the team's drive; 5 SP if export/training is needed. If 5+ SP, consider splitting model-acquisition from yaml-wiring into two sub-tickets. + +## References + +* Fixture skip-gate: `tests/e2e/replay/conftest.py:594-601` +* Backbone factory: `src/gps_denied_onboard/runtime_root/c10_factory.py::build_backbone_specs` +* Backbone spec dataclass: `src/gps_denied_onboard/components/c10_provisioning/config.py` +* AZ-321 (NetVLAD baseline choice) +* AZ-962 spec: `_docs/02_tasks/done/AZ-962_operator_config_jetson_wiring.md` diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 7e6c511..6668d14 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -8,7 +8,7 @@ status: in_progress sub_step: phase: 6 name: implement-tasks - detail: "batch 9 = Tier-2 Jetson e2e validation run NOT GREEN. Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`; result = 4 failed / 48 passed / 3 skipped / 1 xfailed / 1 xpassed in 90.59s. Two distinct blockers: (1) AZ-840 orchestrator test SKIPPED because `GPS_DENIED_OPERATOR_CONFIG_PATH` not exported by `docker-compose.test.jetson.yml` AND `operator_replay.yaml` missing from repo — Epic AZ-835's 'Done' status was validated by doc-content only, never by actual orchestrator test execution; (2) AZ-895 fallout — 4 tests in `test_derkachi_1min.py` regress with `EstimatorFatalError('eskf filter divergence: mahalanobis²=212.311 > 100.0')` at frame 233 because the CSV-driven path (now primary) runs open-loop on the Derkachi fixture (no reference C6 tile cache → no satellite anchoring). Filed AZ-962 (3 SP, operator config + compose wiring) and AZ-963 (3 SP, ESKF regression triage). OKVIS2 chain stays deferred per user 2026-05-29 directive ('after Derkachi e2e green' — directive unchanged; e2e not green). AZ-842 caveat: the AZ-840/AZ-842 'Done' tracker state set earlier today is contingent on whether convention (A) 'In Testing = shipped' or (B) 'Done = shipped+tested' applies; user-skipped convention question, leftover holds the walk-back payload if needed. Cycle-4 not green. Earlier same-day batch 8 = tracker-only fix for AZ-842 (To Do → Done, read-back verified) + wider Jira drift audit recorded as `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. 10 cycle-3/4 tickets (AZ-836/838/839/840/894/895/896/899/900/901) shipped to `done/` locally but stuck in 'In Testing' in Jira; Epic AZ-835 in `todo/` with all 5 children done. User skipped A/B/C/D convention question — leftover holds the bulk-transition payload for whichever convention they pick. **Corrected cycle-4 todo/ remainder**: nothing actionable. Earlier narratives that listed AZ-899/900/901 as 'cycle-4 todo/ remainder for next batches' were fiction — those specs have been in done/ the whole time. OKVIS2 chain (AZ-943/951/952) sits in todo/ but is deferred per user 2026-05-29 directive until after Derkachi e2e flight test passes. Cycle-4 product work is effectively complete pending Derkachi e2e green + AZ-897 UI in ../ui." + detail: "batch 10 = AZ-962 SHIPPED end-to-end + 2 new tickets filed. Implemented `configs/operator_replay.yaml` (registers c6/c7/c10/c11 with defaults; `backbones: []` intentionally — see AZ-965), `docker-compose.test.jetson.yml` exports `GPS_DENIED_OPERATOR_CONFIG_PATH=/opt/configs/operator_replay.yaml` + bind-mounts `./configs:/opt/configs:ro`, `ENV_KEY_MAP` (`src/gps_denied_onboard/config/loader.py`) gained two entries (`SATELLITE_PROVIDER_URL` → `c11.satellite_provider_url`, `SATELLITE_PROVIDER_API_KEY` → `c11.service_api_key`) so secrets flow from `.env.test` and never land in YAML, and README dropped the manual export step. 97/97 c11+config unit tests stay green. Tier-2 re-run on Jetson AGX Orin (`JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`): 4 failed / 48 passed / 1 skipped / 1 xfailed / 1 xpassed / 2 errors in 84.99s — i.e. -2 skipped, +2 errors vs the prior baseline. AZ-962 AC-3 + AC-4 satisfied: AZ-840 orchestrator no longer SKIPs at env-var; it now ERRORs at a deeper, real gate during fixture setup with `IndexUnavailableError: FaissDescriptorIndex: .index file missing at /tmp/pytest-of-root/pytest-0/operator_pre_flight_cache0/descriptor.index`. Same error in `test_operator_pre_flight_integration.py::test_operator_pre_flight_setup_produces_populated_cache` confirms fixture-wide bug, not a single-test issue. Root cause: `conftest.py:487` calls `build_descriptor_index(config)` against a fresh empty `c6_tile_cache.root_dir` (tmp dir per AZ-839 invariant) — FAISS factory needs an existing `.index`. `tile-init` compose service exists but writes its seed to `/var/lib/gps-denied/tiles`, not the tmp dir the fixture overrides into. Filed AZ-964 (3 SP, To Do, FAISS index bootstrap; preferred fix = invoke `mk_test_faiss_fixture.py` inline against override `root_dir`) and AZ-965 (3 SP, To Do, blocked by AZ-964, NetVLAD ONNX backbone provisioning — the next gate after FAISS clears). AZ-962 transitioned To Do → In Progress → Done in Jira (read-back verified). AZ-962 spec moved todo/ → done/. **Cycle-4 e2e gate still NOT GREEN**: AZ-840 chain is now AZ-964 → AZ-965 → orchestrator PASS; 60s smoke is AZ-963 → 4 derkachi_1min tests PASS. OKVIS2 deferral directive still in force (not yet met). Earlier same-day batch 9 = Tier-2 Jetson e2e validation run NOT GREEN. Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`; result = 4 failed / 48 passed / 3 skipped / 1 xfailed / 1 xpassed in 90.59s. Two distinct blockers: (1) AZ-840 orchestrator test SKIPPED because `GPS_DENIED_OPERATOR_CONFIG_PATH` not exported by `docker-compose.test.jetson.yml` AND `operator_replay.yaml` missing from repo — Epic AZ-835's 'Done' status was validated by doc-content only, never by actual orchestrator test execution; (2) AZ-895 fallout — 4 tests in `test_derkachi_1min.py` regress with `EstimatorFatalError('eskf filter divergence: mahalanobis²=212.311 > 100.0')` at frame 233 because the CSV-driven path (now primary) runs open-loop on the Derkachi fixture (no reference C6 tile cache → no satellite anchoring). Filed AZ-962 (3 SP, operator config + compose wiring) and AZ-963 (3 SP, ESKF regression triage). OKVIS2 chain stays deferred per user 2026-05-29 directive ('after Derkachi e2e green' — directive unchanged; e2e not green). AZ-842 caveat: the AZ-840/AZ-842 'Done' tracker state set earlier today is contingent on whether convention (A) 'In Testing = shipped' or (B) 'Done = shipped+tested' applies; user-skipped convention question, leftover holds the walk-back payload if needed. Cycle-4 not green. Earlier same-day batch 8 = tracker-only fix for AZ-842 (To Do → Done, read-back verified) + wider Jira drift audit recorded as `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. 10 cycle-3/4 tickets (AZ-836/838/839/840/894/895/896/899/900/901) shipped to `done/` locally but stuck in 'In Testing' in Jira; Epic AZ-835 in `todo/` with all 5 children done. User skipped A/B/C/D convention question — leftover holds the bulk-transition payload for whichever convention they pick. **Corrected cycle-4 todo/ remainder**: nothing actionable. Earlier narratives that listed AZ-899/900/901 as 'cycle-4 todo/ remainder for next batches' were fiction — those specs have been in done/ the whole time. OKVIS2 chain (AZ-943/951/952) sits in todo/ but is deferred per user 2026-05-29 directive until after Derkachi e2e flight test passes. Cycle-4 product work is effectively complete pending Derkachi e2e green + AZ-897 UI in ../ui." retry_count: 0 cycle: 4 tracker: jira diff --git a/configs/operator_replay.yaml b/configs/operator_replay.yaml new file mode 100644 index 0000000..343320f --- /dev/null +++ b/configs/operator_replay.yaml @@ -0,0 +1,66 @@ +# AZ-962 — Operator pre-flight + replay-mode config for Tier-2 Jetson e2e harness. +# +# Consumed by `tests/e2e/replay/conftest.py::_build_operator_pre_flight_cache` +# (the AZ-839 C3 fixture) which `load_config(env, paths=[this])` then drives the +# AZ-840 7-step orchestrator (`test_az835_e2e_real_flight.py`). +# +# Most fields stay at their dataclass defaults (see +# `src/gps_denied_onboard/components/{c6_tile_cache,c7_inference,c10_provisioning,c11_tile_manager}/config.py`). +# The blocks are declared here primarily so the four-component contract the +# fixture skip-gate cites is satisfied by inspection of this file. The env +# vars below are filled by docker-compose.test.jetson.yml / `.env.test`: +# +# * `GPS_DENIED_FC_PROFILE`, `GPS_DENIED_TIER`, `DB_URL` → runtime +# * `INFERENCE_BACKEND`, `TILE_CACHE_PATH`, `CAMERA_CALIBRATION_PATH` → runtime +# * `LOG_LEVEL`, `LOG_SINK` → log +# * `FDR_PATH` → fdr +# * `SATELLITE_PROVIDER_URL` → c11_tile_manager.satellite_provider_url +# * `SATELLITE_PROVIDER_API_KEY` → c11_tile_manager.service_api_key +# +# AZ-964 (follow-up, not yet filed): the orchestrator test SKIPs at the +# next gate because `c10_provisioning.backbones` is empty — no NetVLAD / +# DINOv2 .onnx file ships with this repo. Populating the backbones list +# here (and provisioning the matching .onnx + verifying it compiles on +# Tegra) is AZ-964's scope, not AZ-962's. + +__top__: + mode: replay + +runtime: + fc_profile: ardupilot_plane + tier: 2 + +replay: + pace: asap + target_fc_dialect: ardupilot_plane + +c6_tile_cache: + store_runtime: postgres_filesystem + metadata_runtime: postgres_filesystem + descriptor_index_runtime: faiss_hnsw + postgres_pool_size: 4 + lru_eviction_threshold_bytes: 10737418240 # 10 GiB + +c7_inference: + runtime: pytorch_fp16 + thermal_poll_hz: 1.0 + engine_cache_dir: /var/lib/gps-denied/engines + gpu_memory_budget_bytes: 4294967296 # 4 GiB + trtexec_timeout_s: 600 + ort_trt_cache_dir: /var/lib/gps-denied/engines/ort_trt_cache + +c10_provisioning: + workspace_mb: 4096 + # backbones intentionally empty — see AZ-964 for the follow-up. + # The AZ-839 fixture skip-gate (conftest.py:594-601) fires here + # with a clear message until backbone provisioning lands. + +c11_tile_manager: + # satellite_provider_url + service_api_key flow in from env vars + # (SATELLITE_PROVIDER_URL / SATELLITE_PROVIDER_API_KEY) via the + # loader's ENV_KEY_MAP additions in AZ-962. + upload_batch_size: 25 + upload_http_timeout_s: 30.0 + download_http_timeout_s: 30.0 + download_max_5xx_retries: 4 + download_resolution_floor_m_per_px: 0.5 diff --git a/docker-compose.test.jetson.yml b/docker-compose.test.jetson.yml index b145828..568d578 100644 --- a/docker-compose.test.jetson.yml +++ b/docker-compose.test.jetson.yml @@ -169,9 +169,16 @@ services: # `replay_runner` fixture trips that gate without this line. BUILD_CSV_REPLAY_ADAPTER: "ON" BUILD_FAISS_INDEX: "ON" + # AZ-962: the AZ-839 C3 fixture (operator_pre_flight_setup) skips + # the AZ-840 orchestrator test when this var is missing. The YAML + # bind-mounted at /opt/configs/operator_replay.yaml declares the + # four blocks the fixture consumes (c6/c7/c10/c11). c10.backbones + # is intentionally empty — AZ-964 ships the .onnx + populates it. + GPS_DENIED_OPERATOR_CONFIG_PATH: /opt/configs/operator_replay.yaml volumes: - ./tests:/opt/tests:ro - ./_docs/00_problem/input_data:/opt/_docs/00_problem/input_data:ro + - ./configs:/opt/configs:ro - fdr-data:/var/lib/gps-denied/fdr - tile-data:/var/lib/gps-denied/tiles diff --git a/src/gps_denied_onboard/config/loader.py b/src/gps_denied_onboard/config/loader.py index bf617ae..f5c5749 100644 --- a/src/gps_denied_onboard/config/loader.py +++ b/src/gps_denied_onboard/config/loader.py @@ -74,6 +74,14 @@ ENV_KEY_MAP: Final[dict[str, tuple[str, str]]] = { "REPLAY_PACE": ("replay", "pace"), "REPLAY_TIME_OFFSET_MS": ("replay", "time_offset_ms"), "REPLAY_TARGET_FC_DIALECT": ("replay", "target_fc_dialect"), + # C11 tile-manager URL + bearer (AZ-962) — the Jetson harness + + # operator-orchestrator deploys inject these via env so the YAML + # never carries a real secret. `build_tile_downloader` raises if + # `service_api_key` is empty; mapping it through ENV_KEY_MAP lets + # the e2e harness fill the field from `.env.test` without a YAML + # override step. + "SATELLITE_PROVIDER_URL": ("c11_tile_manager", "satellite_provider_url"), + "SATELLITE_PROVIDER_API_KEY": ("c11_tile_manager", "service_api_key"), } # Env vars that MUST resolve to a non-empty value before `load_config` @@ -122,6 +130,9 @@ _FIELD_COERCIONS: Final[dict[str, type]] = { "output_path": str, "pace": str, "target_fc_dialect": str, + # C11 (AZ-962) + "satellite_provider_url": str, + "service_api_key": str, } diff --git a/tests/e2e/replay/README.md b/tests/e2e/replay/README.md index 3acf21e..e038b18 100644 --- a/tests/e2e/replay/README.md +++ b/tests/e2e/replay/README.md @@ -51,10 +51,20 @@ matching nadir video + camera calibration, the orchestrator runs the ssh jetson-e2e cd /workspace/gps-denied-onboard export RUN_REPLAY_E2E=1 -export GPS_DENIED_OPERATOR_CONFIG_PATH=/workspace/configs/operator_replay.yaml pytest tests/e2e/replay/test_az835_e2e_real_flight.py -v --tb=short -m tier2 ``` +AZ-962: `docker-compose.test.jetson.yml` exports +`GPS_DENIED_OPERATOR_CONFIG_PATH=/opt/configs/operator_replay.yaml` +automatically and bind-mounts `./configs:/opt/configs:ro`, so no +manual env-var export is required when running through +`scripts/run-tests-jetson.sh`. The YAML at `configs/operator_replay.yaml` +declares the four blocks the fixture requires (c6 / c7 / c10 / c11); +secrets (`SATELLITE_PROVIDER_API_KEY`) flow in from `.env.test` via +the loader's `ENV_KEY_MAP`. `c10_provisioning.backbones` is +intentionally empty pending AZ-964 (the orchestrator test will +SKIP at the "no backbones" gate until AZ-964 lands). + The bundled local-development entry point is `scripts/run-tests-jetson.sh`, which handles the SSH alias + rsync + remote pytest invocation. See `_docs/02_document/tests/tier2-jetson-testing.md` for the harness contract.