diff --git a/_docs/02_tasks/_dependencies_table.md b/_docs/02_tasks/_dependencies_table.md index cbb53e8..c7935ee 100644 --- a/_docs/02_tasks/_dependencies_table.md +++ b/_docs/02_tasks/_dependencies_table.md @@ -1,6 +1,6 @@ # Dependencies Table -**Date**: 2026-05-29 (cycle-4 Step 10 Implement — **AZ-842 tracker drift fixed + wider Jira drift audit recorded as leftover**: AZ-842 was shipped 2026-05-29 in commit `42b1db6` (spec in `done/AZ-842_replay_protocol_and_orchestrator_docs.md`, Invariant 14 + cycle-4 redesign narrative landed in `replay_protocol.md` + `architecture.md` + `tests/e2e/replay/README.md`) but the Jira ticket was stuck in To Do. **Fixed**: To Do → In Progress → Done in Jira (read-back verified). **Wider audit triggered** when AZ-842 fix surfaced 10 more shipped tickets stuck in "In Testing" (AZ-836/838/839/840/894/895/896/899/900/901) and Epic AZ-835 stuck in "To Do" with all 5 children Done/deferred. Asked user A/B/C/D ("Done = shipped+tested" vs "Done = QA-accepted" convention question); user skipped — interpreted as "use judgment, don't block". Recorded in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md` per scope-discipline rule (out-of-scope bulk modifications need explicit user direction). **Corrected cycle-4 todo/ remainder**: there is **NO product work** left in cycle-4 `todo/` — only Epic AZ-835 (stale tracker state, all children done) + the OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) which is deferred per user 2026-05-29 directive until after Derkachi e2e green. My earlier eighth + ninth bump narratives that listed "AZ-899 + AZ-900 + AZ-901 = 3 SP cycle-4 todo/ remainder" were fiction — those three specs have been in `done/` the whole time. Cycle-4 product work is effectively complete pending the Derkachi e2e flight test and the relocated AZ-897 UI in `../ui`. Earlier same-day — **AZ-961 LANDED** (`ReportContext.tlog_path` → `ground_truth_path` rename + label fix): `ReportContext` field renamed; rendered report line now reads `- Ground truth: ` for both tlog and CSV runs; AZ-959's inline comment documenting the field-overload removed. Two new symmetric tests in `test_az699_report_writer.py` assert the canonical label for both input formats. All 4 call sites updated: `helpers/accuracy_report.py` (field + docstring + render label), `replay_api/app.py` (kwarg), `tests/unit/test_az699_report_writer.py` (fixture + 2 new tests), `tests/e2e/replay/_e2e_orchestrator.py` (production e2e), `tests/e2e/replay/test_derkachi_real_tlog.py` (e2e test). 62/62 tests green across the three relevant unit-test modules. AZ-961 moved todo/ → done/. **Cycle-4 implement-batch sequence summary**: AZ-959 (3pt) + AZ-960 (2pt) + AZ-961 (1pt) all landed today as a coherent CSV-replay-input chain unblocking the relocated AZ-897 UI in `../ui`. Cycle-4 todo/ remainder for next batches: AZ-842 (3, docs) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1) = 6 SP product. OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) still in todo/ but sequenced after the Derkachi e2e green per user 2026-05-29 directive. Earlier same-day — **AZ-960 LANDED** (`gps-denied-render-map` CSV-truth dispatch): `load_ground_truth_track` now dispatches on `truth_path.suffix` (`.csv` → `load_csv_ground_truth`, else → `load_tlog_ground_truth`); `_maybe_render_map` short-circuit removed (AZ-959 workaround gone); CSV-path replay jobs now ship with `map_html_url` populated in the API response. 44/44 unit tests green across `test_az700_render_map.py` + `test_az701_replay_api.py` (17 pre-existing render-map tests pass per AC-2 + 3 new AZ-960 tests covering ACs 1+3+4 + AZ-959 happy-path test extended to assert `map_html_url`). AZ-960 moved todo/ → done/. Next batch: AZ-961 (ReportContext.tlog_path rename + label fix). Earlier same-day — **AZ-960 + AZ-961 filed as AZ-959 follow-ups** per user 2026-05-29 directive ("File AZ-960 (render-map CSV dispatch) + AZ-961 (ReportContext rename) and continue with one of them next"): the two deferred items surfaced during AZ-959 implementation are now tracked. **AZ-960** (2pt, todo/, `gps-denied-render-map`: dispatch `--truth` loader on extension to unblock CSV-path map render; deps AZ-700 + AZ-894 + AZ-959; no epic — UX-completing follow-up). **AZ-961** (1pt, todo/, `accuracy_report`: rename `ReportContext.tlog_path` → `ground_truth_path` + label fix in rendered report; deps AZ-699 + AZ-959; no epic — cosmetic cleanup). Next implement batch picks AZ-960 first (closes the UI map gap; AZ-961 sequenced after to avoid re-conflict on `_maybe_render_report`'s kwargs). Earlier same-day — **AZ-959 LANDED** (`replay_api` `POST /replay` CSV-path extension): handler now accepts `(video, tlog)` OR `(video, csv)` multipart with XOR validation; `validate_csv_kind` rejects malformed schemas at the API boundary referencing `csv_replay_format.md`; `SubprocessReplayRunner.run` dispatches `--imu` vs `--tlog`; `_maybe_render_report` dispatches GT loader via `load_csv_ground_truth` / `load_tlog_ground_truth`; `ReplayInputs` DTO carries `tlog_path: Path | None` + `csv_path: Path | None` with XOR `__post_init__`; `JobStorage` reserves both `tlog_path` and `csv_path`; new `GET /static/example-csv` endpoint serves the AZ-896 reference CSV via `REPLAY_API_EXAMPLE_CSV_PATH` env or source-checkout fallback. 27/27 unit tests green (18 pre-existing tlog tests pass unchanged per AC-7 + 9 new tests covering ACs 1-6 + 2 `validate_csv_kind` unit cases). **Deferred items (NOT silently fixed, surfaced to user as end-of-turn notes)**: (a) `gps-denied-render-map` only consumes binary tlog truth → CSV-path jobs return `map_html_url=None` (deferred to AZ-700 follow-up); (b) `ReportContext.tlog_path` field is now overloaded as "ground-truth source path"; rendered report's `Tlog: ` line is cosmetically misleading for CSV runs (deferred to AZ-699 follow-up). AZ-959 moved todo/ → done/. Cycle-4 active-scope SP delta: −3 SP (3 → 0). Earlier same-day — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) +**Date**: 2026-05-29 (cycle-4 Step 11 Validate — **Tier-2 Jetson e2e run NOT GREEN; AZ-962 + AZ-963 filed**: Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh` on Jetson AGX Orin (aarch64, JetPack 6.x). Result: **4 failed, 48 passed, 3 skipped, 1 xfailed, 1 xpassed in 90.59s**. Two distinct blockers surfaced. **Blocker 1**: the AZ-840 orchestrator test (`test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration`) — the test that should prove the full 7-step cycle-4 pipeline works — was SKIPPED, not PASSed. `docker-compose.test.jetson.yml` does not export `GPS_DENIED_OPERATOR_CONFIG_PATH` (despite the comment claiming the Jetson harness sets it) AND the `operator_replay.yaml` the README references does not exist anywhere in the repo. This means Epic AZ-835's "Done" status across AZ-836/AZ-838/AZ-839/AZ-840/AZ-842 was validated **by doc-content presence only**, not by end-to-end test execution — exactly the failure mode `meta-rule.mdc` warns against ("tests that pass by skipping the component they are supposed to exercise create false confidence"). Filed **AZ-962** (3 SP) to author the missing YAML + wire the env var into the compose so the orchestrator test can actually run. **Blocker 2**: 4 tests in `test_derkachi_1min.py` (60s smoke, AZ-265/AZ-404) regressed to FAIL with `EstimatorFatalError('eskf filter divergence on vio: mahalanobis²=212.311 > 100.0')` at frame 233. Root cause: AZ-895 made the CSV-driven path primary; the CSV path runs open-loop because the Derkachi fixture has no reference C6 tile cache (no satellite anchoring → C5 ESKF integrates open-loop → diverges in ~10s). Before AZ-895 the tlog path was primary and presumably exited cleanly even without anchoring; the deprecation didn't account for the runtime-semantic difference. Additionally `test_ac3_within_100m_80pct_of_ticks` XPASSed (was xfail, unexpectedly passed) — third silent-failure surface needing investigation. Filed **AZ-963** (3 SP) for triage + fix decision (options A–F documented in the spec). **AZ-842 caveat**: the AZ-840/AZ-842 "Done" tracker state I set earlier today (commits 10c2a1e / 2cc992d) was based on the prior misunderstanding that the cycle-4 pipeline was validated end-to-end; the user-skipped convention question would tilt toward "needs walk-back to In Testing" if convention (A) holds. Recorded as caveat in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. **Honest cycle-4 status**: NOT green. Cycle-4 todo/ now contains: AZ-835 Epic (stale Jira tracker; children done locally but orchestrator validation gap), AZ-943/AZ-951/AZ-952 OKVIS2 chain (still deferred per user 2026-05-29 directive until Derkachi e2e green — directive unchanged), AZ-962 + AZ-963 (newly filed). OKVIS2 remains deferred. Earlier same-day — **AZ-842 tracker drift fixed + wider Jira drift audit recorded as leftover**: AZ-842 was shipped 2026-05-29 in commit `42b1db6` (spec in `done/AZ-842_replay_protocol_and_orchestrator_docs.md`, Invariant 14 + cycle-4 redesign narrative landed in `replay_protocol.md` + `architecture.md` + `tests/e2e/replay/README.md`) but the Jira ticket was stuck in To Do. **Fixed**: To Do → In Progress → Done in Jira (read-back verified). **Wider audit triggered** when AZ-842 fix surfaced 10 more shipped tickets stuck in "In Testing" (AZ-836/838/839/840/894/895/896/899/900/901) and Epic AZ-835 stuck in "To Do" with all 5 children Done/deferred. Asked user A/B/C/D ("Done = shipped+tested" vs "Done = QA-accepted" convention question); user skipped — interpreted as "use judgment, don't block". Recorded in `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md` per scope-discipline rule (out-of-scope bulk modifications need explicit user direction). **Corrected cycle-4 todo/ remainder**: there is **NO product work** left in cycle-4 `todo/` — only Epic AZ-835 (stale tracker state, all children done) + the OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) which is deferred per user 2026-05-29 directive until after Derkachi e2e green. My earlier eighth + ninth bump narratives that listed "AZ-899 + AZ-900 + AZ-901 = 3 SP cycle-4 todo/ remainder" were fiction — those three specs have been in `done/` the whole time. Cycle-4 product work is effectively complete pending the Derkachi e2e flight test and the relocated AZ-897 UI in `../ui`. Earlier same-day — **AZ-961 LANDED** (`ReportContext.tlog_path` → `ground_truth_path` rename + label fix): `ReportContext` field renamed; rendered report line now reads `- Ground truth: ` for both tlog and CSV runs; AZ-959's inline comment documenting the field-overload removed. Two new symmetric tests in `test_az699_report_writer.py` assert the canonical label for both input formats. All 4 call sites updated: `helpers/accuracy_report.py` (field + docstring + render label), `replay_api/app.py` (kwarg), `tests/unit/test_az699_report_writer.py` (fixture + 2 new tests), `tests/e2e/replay/_e2e_orchestrator.py` (production e2e), `tests/e2e/replay/test_derkachi_real_tlog.py` (e2e test). 62/62 tests green across the three relevant unit-test modules. AZ-961 moved todo/ → done/. **Cycle-4 implement-batch sequence summary**: AZ-959 (3pt) + AZ-960 (2pt) + AZ-961 (1pt) all landed today as a coherent CSV-replay-input chain unblocking the relocated AZ-897 UI in `../ui`. Cycle-4 todo/ remainder for next batches: AZ-842 (3, docs) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1) = 6 SP product. OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) still in todo/ but sequenced after the Derkachi e2e green per user 2026-05-29 directive. Earlier same-day — **AZ-960 LANDED** (`gps-denied-render-map` CSV-truth dispatch): `load_ground_truth_track` now dispatches on `truth_path.suffix` (`.csv` → `load_csv_ground_truth`, else → `load_tlog_ground_truth`); `_maybe_render_map` short-circuit removed (AZ-959 workaround gone); CSV-path replay jobs now ship with `map_html_url` populated in the API response. 44/44 unit tests green across `test_az700_render_map.py` + `test_az701_replay_api.py` (17 pre-existing render-map tests pass per AC-2 + 3 new AZ-960 tests covering ACs 1+3+4 + AZ-959 happy-path test extended to assert `map_html_url`). AZ-960 moved todo/ → done/. Next batch: AZ-961 (ReportContext.tlog_path rename + label fix). Earlier same-day — **AZ-960 + AZ-961 filed as AZ-959 follow-ups** per user 2026-05-29 directive ("File AZ-960 (render-map CSV dispatch) + AZ-961 (ReportContext rename) and continue with one of them next"): the two deferred items surfaced during AZ-959 implementation are now tracked. **AZ-960** (2pt, todo/, `gps-denied-render-map`: dispatch `--truth` loader on extension to unblock CSV-path map render; deps AZ-700 + AZ-894 + AZ-959; no epic — UX-completing follow-up). **AZ-961** (1pt, todo/, `accuracy_report`: rename `ReportContext.tlog_path` → `ground_truth_path` + label fix in rendered report; deps AZ-699 + AZ-959; no epic — cosmetic cleanup). Next implement batch picks AZ-960 first (closes the UI map gap; AZ-961 sequenced after to avoid re-conflict on `_maybe_render_report`'s kwargs). Earlier same-day — **AZ-959 LANDED** (`replay_api` `POST /replay` CSV-path extension): handler now accepts `(video, tlog)` OR `(video, csv)` multipart with XOR validation; `validate_csv_kind` rejects malformed schemas at the API boundary referencing `csv_replay_format.md`; `SubprocessReplayRunner.run` dispatches `--imu` vs `--tlog`; `_maybe_render_report` dispatches GT loader via `load_csv_ground_truth` / `load_tlog_ground_truth`; `ReplayInputs` DTO carries `tlog_path: Path | None` + `csv_path: Path | None` with XOR `__post_init__`; `JobStorage` reserves both `tlog_path` and `csv_path`; new `GET /static/example-csv` endpoint serves the AZ-896 reference CSV via `REPLAY_API_EXAMPLE_CSV_PATH` env or source-checkout fallback. 27/27 unit tests green (18 pre-existing tlog tests pass unchanged per AC-7 + 9 new tests covering ACs 1-6 + 2 `validate_csv_kind` unit cases). **Deferred items (NOT silently fixed, surfaced to user as end-of-turn notes)**: (a) `gps-denied-render-map` only consumes binary tlog truth → CSV-path jobs return `map_html_url=None` (deferred to AZ-700 follow-up); (b) `ReportContext.tlog_path` field is now overloaded as "ground-truth source path"; rendered report's `Tlog: ` line is cosmetically misleading for CSV runs (deferred to AZ-699 follow-up). AZ-959 moved todo/ → done/. Cycle-4 active-scope SP delta: −3 SP (3 → 0). Earlier same-day — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) **Total Tasks**: 184 (143 product + 41 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +AZ-959 (1 product task, todo/, 3pt). AZ-897 was never in this table's row count (pre-existing gap — the cycle-4 AZ-89x specs land in todo/ but were not back-filled into the table rows; not fixing that here, out of scope). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 183 (142 product + 41 blackbox-test) — +AZ-951 + AZ-952 (2 product tasks, both backlog/, 3pt each). AZ-943 (5pt) moved todo/ → backlog/ (no count change). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 181 (140 product + 41 blackbox-test) → 182 (141 product) — +AZ-943 (1 product task, originally todo/, 5pt). AZ-944 + AZ-945 remain Jira-only at the time of this update (sibling tickets, local specs deferred to their own Implement turns); their Total-Tasks impact will be reconciled when their specs land. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 180 (139 product + 41 blackbox-test) — +AZ-908 (1 product task, backlog/, 3pt). Prior 2026-05-26 cycle-4 Step 9 bump: +AZ-899 + AZ-900 + AZ-901 (3 product tasks). AZ-841 moved todo/ → backlog/ (no count change; backlog tickets are still in the table). Prior 2026-05-23 refactor-run bump: 176 (135 product + 41 blackbox-test) — +AZ-844 (Epic, 0pt umbrella for refactor run 02) + AZ-845 + AZ-846 + AZ-847 (3 product tasks). Prior 2026-05-23 bump (Epic AZ-835 decomposition): 173 (132 product + 41 blackbox-test) = +AZ-835 (Epic) + AZ-836 (C1) + AZ-837 (test-stack hardening, not this Epic) + AZ-838 (C2) added 2026-05-22→2026-05-23 prior to that update; +AZ-839 (C3) + AZ-840 (C4) + AZ-841 (C5) + AZ-842 (C6) added in that update. AZ-777 stays in the table (now closed in Jira; spec at `done/AZ-777_derkachi_c6_reference_fixture.md` retains 8pt credit for Phases 1+2 shipped). Earlier counts: 165 (124 product + 41 blackbox-test) — AZ-317 retained in the table marked SUPERSEDED for audit; AZ-523 (C11 gate removal) + AZ-524 (C12 rename) added as 2 closed audit-trail tasks; AZ-526 = 2pt clock-helper hygiene; AZ-527 = 2pt c2 engine-dim helper hygiene; AZ-528 = 3pt c1_vio facade-spine hygiene; AZ-558 = 3pt MavlinkTransport routing follow-up; AZ-559 closed Won't Fix; AZ-589 + AZ-590 closed Won't Fix (kept in table as 0pt audit-trail rows); AZ-591 = 5pt cross-cutting compose_root bootstrap (todo/); AZ-592 = 5pt OKVIS2 Tier-2 placeholder (backlog/); AZ-593 = 5pt VINS-Mono Tier-2 placeholder (backlog/); AZ-618 = 0pt umbrella (split into AZ-619..AZ-624 on 2026-05-19); AZ-619..AZ-624 = 6 subtasks of AZ-618 covering Phase A..F of the airborne `pre_constructed` assembly, summing to 16pt actionable work; AZ-687 = 2pt replay-mode guard follow-up surfaced by AZ-618 Tier-2 run on 2026-05-19 **Total Complexity Points**: 584 (451 product + 133 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +3pt AZ-959. AZ-897 (5pt) was never table-counted here, so no decrement at this layer; the in-repo cycle-4 effort still drops by 5pt at the active-scope layer (AZ-897 work is now executed in `../ui`). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 581 (448 product + 133 blackbox-test) — +3pt AZ-951 + 3pt AZ-952 = +6 product pts. AZ-943 stays counted at 5pt (moved todo/ → backlog/, not deleted). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 580 (447 product + 133 blackbox-test) — +5pt AZ-943. AZ-944 (3pt) + AZ-945 (3pt) sibling tickets are filed Jira-side but not yet imported as local specs; their +6pt will land when AZ-944 / AZ-945 specs are authored. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 570 (437 product + 133 blackbox-test) — +3pt AZ-908. Prior 2026-05-26 cycle-4 Step 9 bump: +1pt AZ-899 + 1pt AZ-900 + 1pt AZ-901 + 1pt AZ-842 rescope (2→3) = +4 product pts. Prior 2026-05-23 refactor-run bump: 563 (430 product + 133 blackbox-test) — +2pt AZ-845 + 2pt AZ-846 + 2pt AZ-847 = +6 product pts on top of prior reconciled total (AZ-844 epic itself is 0pt umbrella). Prior 2026-05-23 reconciled total: 557 (424 product + 133 blackbox-test) — +5pt AZ-839 + 3pt AZ-840 + 1pt AZ-841 + 2pt AZ-842 = +11 product pts on top of prior reconciled total. AZ-836 (3pt) + AZ-838 (3pt) were added 2026-05-22→2026-05-23 prior to that update; AZ-837 (test-stack hardening, not this Epic) is unaccounted in that delta and should be folded in at the next preamble reconciliation. Earlier baseline: 546 (413 product + 133 blackbox-test) — +3pt AZ-776 + 8pt AZ-777 (5→8 override 2026-05-21 cycle-3 batch 104; see `_docs/_process_leftovers/2026-05-21_az777_complexity_override.md` for rationale + the spec refresh that pulled e2e-runner wiring + C11 contract adapt + Derkachi catalog seed + fixture replacement + un-xfail into one ticket) — AZ-523 = 3pt, AZ-524 = 2pt, AZ-526 = 2pt, AZ-527 = 2pt, AZ-528 = 3pt, AZ-558 = 3pt, AZ-589 + AZ-590 retained at 5pt each but closed Won't Fix (treated as 0 effective pts going forward), AZ-591 = 5pt, AZ-592 = 5pt placeholder, AZ-593 = 5pt placeholder, AZ-618 = 0pt umbrella post-split, AZ-619 = 2pt, AZ-620 = 3pt, AZ-621 = 3pt, AZ-622 = 3pt, AZ-623 = 3pt, AZ-624 = 2pt, AZ-687 = 2pt diff --git a/_docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md b/_docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md new file mode 100644 index 0000000..ed3ad92 --- /dev/null +++ b/_docs/02_tasks/todo/AZ-962_operator_config_jetson_wiring.md @@ -0,0 +1,100 @@ +# AZ-962 — Wire `GPS_DENIED_OPERATOR_CONFIG_PATH` + `operator_replay.yaml` into Tier-2 Jetson harness + +**Status**: To Do (Jira) / `todo/` (local) +**Issue type**: Task +**Complexity**: 3 SP +**Cycle**: cycle-4 e2e closure follow-up +**Jira**: https://denyspopov.atlassian.net/browse/AZ-962 +**Filed**: 2026-05-29 during cycle-4 Tier-2 validation run + +## Why + +Discovered 2026-05-29 during cycle-4 e2e validation run on Tier-2 Jetson AGX Orin. The AZ-840 orchestrator test (`tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration`) — the test that's supposed to prove the full 7-step pipeline works end-to-end — was SKIPPED with: + +``` +AZ-839 operator_pre_flight_setup requires GPS_DENIED_OPERATOR_CONFIG_PATH pointing at a YAML +that registers c6_tile_cache + c7_inference + c10_provisioning + c11_tile_manager blocks +(Jetson e2e harness sets this; dev macOS does not) +``` + +Two gaps: + +1. `docker-compose.test.jetson.yml` does NOT export `GPS_DENIED_OPERATOR_CONFIG_PATH` despite the comment claiming the Jetson harness sets it. Grep confirms the env var is absent from the compose file. +2. The YAML the README's Tier-2 invocation references (`/workspace/configs/operator_replay.yaml`) does NOT exist anywhere in the repo. No `configs/` directory, no `**/operator*.yaml` match. + +Net effect: the cycle-4 closure narrative (Epic AZ-835 + children AZ-836/AZ-838/AZ-839/AZ-840/AZ-842 all marked Done) was based on AC verification by **doc-content presence**, not by the orchestrator test actually running. The test has never been demonstrated to PASS end-to-end on the Jetson harness automatically. This is the exact failure mode `meta-rule.mdc` warns against ("Tests that pass by skipping the component they are supposed to exercise create false confidence"). + +## Goal + +Make the AZ-840 orchestrator test actually runnable on `bash scripts/run-tests-jetson.sh` (no out-of-band manual env-var setup). The test must either PASS, or fail with a NEW, real, attributable error that lands in a follow-up ticket — not skip silently. + +## Scope + +1. **Author `configs/operator_replay.yaml`** (final location TBD — `configs/` at repo root, or `tests/fixtures/operator_replay.yaml`, or another location consistent with the project's config conventions). + + * Must register at minimum: `c6_tile_cache`, `c7_inference`, `c10_provisioning`, `c11_tile_manager` (the four blocks `conftest.py:322-326` and `_build_operator_pre_flight_cache` consume). + * Schema must match what `load_config` parses (see `gps_denied_onboard/config/loader.py`). + * Component types must match what the runtime factories build (see `tests/e2e/replay/conftest.py:430-462` for the `c6_tile_cache.root_dir` override pattern). + * Imagery / FAISS settings sized for Derkachi fixture: route-driven seeding (AZ-836 / AZ-838), HNSW32 FAISS index, NetVLAD descriptors. + +2. **Wire the env var into `docker-compose.test.jetson.yml`**: + + * Add `GPS_DENIED_OPERATOR_CONFIG_PATH: /opt/configs/operator_replay.yaml` to the `e2e-runner.environment` block. + * Add a read-only bind mount for the configs dir: `./configs:/opt/configs:ro`. + * Verify the README's "Tier-2 invocation" example matches what the compose does automatically — no manual `export GPS_DENIED_OPERATOR_CONFIG_PATH=...` step required. + +3. **Re-run Tier-2 and capture the verdict**: + + * `JETSON_SSH_ALIAS= bash scripts/run-tests-jetson.sh` + * Confirm the AZ-840 test no longer skips with the env-var or config-file gate. + * Capture the verdict-report (`_docs/06_metrics/real_flight_validation_.md`) if PASS, or capture the new failure mode for follow-up ticket if FAIL. + +4. **Update README** if the wiring story now differs from the documented one. + +## Acceptance Criteria + +* **AC-1**: `docker-compose.test.jetson.yml` exports `GPS_DENIED_OPERATOR_CONFIG_PATH` pointing at a YAML that is bind-mounted into the e2e-runner container. +* **AC-2**: `configs/operator_replay.yaml` (or equivalent final path) exists in the repo, registers all 4 required component blocks (`c6_tile_cache` + `c7_inference` + `c10_provisioning` + `c11_tile_manager`), and is consumable by `load_config(os.environ, paths=[config_path])` without `KeyError`. +* **AC-3**: `bash scripts/run-tests-jetson.sh` no longer reports `SKIPPED [127]: AZ-839 operator_pre_flight_setup requires GPS_DENIED_OPERATOR_CONFIG_PATH ...` for `test_az840_e2e_real_flight_orchestration`. +* **AC-4**: The orchestrator test either PASSes (and the verdict report at `_docs/06_metrics/real_flight_validation_.md` is captured), or fails with a NEW error that is filed as a separate follow-up ticket (don't paper over the failure — failing test + new ticket is the honest outcome). +* **AC-5**: README's `### AZ-835 orchestrator test` section accurately describes what `scripts/run-tests-jetson.sh` does (no "set this env var manually" step required when running via the script). + +## Out of scope + +* The 4 regression failures in `test_derkachi_1min.py` (separate AZ-963 ticket). +* AZ-895 deprecation rollback. +* Adding a reference C6 tile cache for the Derkachi fixture (large separate work). +* Updating cycle-4 closure narrative / re-opening AZ-840/AZ-842 status decisions — those are tracker-state questions the user owns. + +## Dependencies + +* **AZ-835** (parent Epic, currently To Do in Jira but tracker-drift suspected) — this ticket closes a real validation gap in that Epic's deliverable. +* **AZ-839** (C3 fixture, Done locally / In Testing in Jira) — this ticket provides the missing input the fixture's skip-gate complains about. +* **AZ-840** (C4 orchestrator test, Done locally / In Testing in Jira) — this ticket makes that test actually run. + +## Estimate + +3 SP. Multi-step (YAML + compose wiring + verification re-run), moderate complexity (YAML schema must match runtime factories' expectations), moderate risk (might need iterative tuning on the first re-run). + +## Run-log evidence (2026-05-29 Tier-2) + +``` +JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh +... +e2e-runner-1 | collected 57 items +e2e-runner-1 | tests/e2e/replay/test_az835_e2e_real_flight.py::test_az840_e2e_real_flight_orchestration SKIPPED [ 1%] +... +e2e-runner-1 | = 4 failed, 48 passed, 3 skipped, 1 xfailed, 1 xpassed, 1 warning in 90.59s (0:01:30) = +e2e-runner-1 | SKIPPED [1] tests/e2e/replay/test_az835_e2e_real_flight.py:127: + AZ-839 operator_pre_flight_setup requires GPS_DENIED_OPERATOR_CONFIG_PATH pointing at a YAML + that registers c6_tile_cache + c7_inference + c10_provisioning + c11_tile_manager blocks + (Jetson e2e harness sets this; dev macOS does not) +``` + +## References + +* Compose: `docker-compose.test.jetson.yml` +* Test: `tests/e2e/replay/test_az835_e2e_real_flight.py:127` +* Skip-gate definition: `tests/e2e/replay/conftest.py:343-388` +* README: `tests/e2e/replay/README.md` § `AZ-835 orchestrator test` +* Sibling ticket (parallel work): AZ-963 — 60s smoke regression diff --git a/_docs/02_tasks/todo/AZ-963_eskf_divergence_60s_smoke_regression.md b/_docs/02_tasks/todo/AZ-963_eskf_divergence_60s_smoke_regression.md new file mode 100644 index 0000000..ed0510d --- /dev/null +++ b/_docs/02_tasks/todo/AZ-963_eskf_divergence_60s_smoke_regression.md @@ -0,0 +1,111 @@ +# AZ-963 — Fix Derkachi 60s smoke regressions: ESKF divergence on CSV-only path with no satellite anchoring (AZ-895 fallout) + +**Status**: To Do (Jira) / `todo/` (local) +**Issue type**: Task +**Complexity**: 3 SP (may bump to 5 SP after triage if option B is chosen) +**Cycle**: cycle-4 e2e closure follow-up +**Jira**: https://denyspopov.atlassian.net/browse/AZ-963 +**Filed**: 2026-05-29 during cycle-4 Tier-2 validation run + +## Why + +Discovered 2026-05-29 during cycle-4 e2e validation run on Tier-2 Jetson AGX Orin. Four tests in `tests/e2e/replay/test_derkachi_1min.py` regressed to FAIL after the AZ-895 deprecation made the CSV-driven replay path primary: + +* `test_ac1_exits_0_jsonl_count_match` — expects exit 0, got exit 1 +* `test_ac5_determinism_two_runs_diff` — expects two PASSing runs to diff cleanly, both exit 1 +* `test_ac6_pace_realtime_60s_within_5pct` — expects realtime pace within 5%, exits 1 before timing measurement is meaningful +* `test_ac6_pace_asap_under_30s` — expects asap under 30s, exits 1 in ~13s with fatal error + +All four fail with the same root cause: + +``` +ERROR c5.state.eskf_filter_divergence kv={"source":"vio","mahalanobis_sq":212.31,"threshold_sq":100.0} +ERROR replay_loop.state_add_vio_fatal frame=233 + EstimatorFatalError('eskf filter divergence on vio: mahalanobis²=212.311 > 100.0') +``` + +The CSV-driven path (now primary since AZ-895 deprecation) runs **open-loop** — the Derkachi fixture has no reference C6 tile cache so C2 VPR / C3 matcher / C4 pose-anchor stages are not wired: + +``` +WARN replay_loop.satellite_anchoring_not_wired: frame=0 — C2 VPR / C4 pose-anchor stages are not wired +in this run (Derkachi has no reference tile cache); estimator runs open-loop on VIO + IMU. Expect +monotonically growing position error. +``` + +After ~10s of open-loop integration, ESKF Mahalanobis distance exceeds the 100.0 threshold at frame 233 and the runner crashes with a non-zero exit code. The 4 tests don't care about accuracy but they require a clean exit — which they can't get on the CSV-only path. + +**Why this matters now**: before AZ-895, the tlog path was the primary replay surface and presumably exited cleanly (with some warning about divergence) without raising `EstimatorFatalError`. The AZ-895 deprecation didn't account for the runtime-semantic difference between the two paths in test fixtures that depended on "runner exits 0 even without satellite anchoring". + +## Related XPASS finding (in scope to investigate, may split into sub-ticket) + +`test_ac3_within_100m_80pct_of_ticks` showed up as XPASS in the same run. It was marked xfail because "AC-3 requires the C1+C2+C3+C4+C5 satellite-re-anchoring pipeline. Blocked by AZ-777...". XPASS means "marked xfail but unexpectedly passed" — which is impossible per the documented physics (open-loop ESKF can't meet ≤80% within 100m). Either the test is silently no-oping into a pass, or the xfail mark is stale, or the new semantics changed something that fixed it. Worth investigating because it could be a third silent-failure surface. + +## Goal + +The 4 currently-failing tests must either PASS, or have an explicit gating decision (xfail with a tracked reason, or skip with the right mark) that doesn't silently hide AC coverage. The AC matrix in the README must accurately reflect what's measured vs what's deferred. + +This ticket does NOT mandate a specific fix — the right answer requires triage. Options on the table: + +* **A**: Loosen the ESKF divergence threshold in the test harness path (changes production code; risky — the threshold exists for a real safety reason) +* **B**: Add a reference C6 tile cache for Derkachi so satellite anchoring works (AZ-777 follow-up scope; large; the fixture has no anchorable imagery yet) +* **C**: Gate the 4 tests behind a "satellite anchoring required" mark and skip them on the open-loop path (preserves the tests as documentation; doesn't restore AC coverage) +* **D**: Mark the divergence-driven failures as expected (xfail with rationale: "open-loop ESKF diverges on this fixture") +* **E**: Investigate why AC-3 XPASSes and whether that finding changes A–D +* **F**: Some combination after triage + +## Acceptance Criteria + +* **AC-1**: All 4 currently-failing tests (`test_ac1_exits_0_jsonl_count_match`, `test_ac5_determinism_two_runs_diff`, `test_ac6_pace_realtime_60s_within_5pct`, `test_ac6_pace_asap_under_30s`) are either PASSing or have an explicit gating decision with a tracked Jira reference — NOT silently disabled. +* **AC-2**: The `test_ac3_within_100m_80pct_of_ticks` XPASS is investigated and either becomes a real PASS (xfail mark removed with rationale) or stays xfail with an updated rationale (one of the two; not both, not silent). +* **AC-3**: No regression to the documented AC matrix in `tests/e2e/replay/README.md` § `AC matrix` — every AC row is still being measured in some form (PASS / honest xfail / honest skip with reason), and the README accurately reflects the current state. +* **AC-4**: The fix does not bring back the AZ-895-deprecated auto-sync surface (`--time-offset-ms`, `--skip-auto-sync-validation` CLI flags must remain deprecated). +* **AC-5**: A short triage memo lives at `_docs/03_implementation/batch_*_az963_triage.md` (or equivalent batch report) explaining which of options A–F was chosen and why, with the run-log evidence cited. + +## Out of scope + +* AZ-840 orchestrator test (separate AZ-962 ticket). +* Reverting AZ-895 to restore the tlog path as primary. +* Building a reference C6 tile cache for Derkachi (separate large work). +* Tracker-state cleanup for AZ-840 / AZ-842 (separate user decision). + +## Dependencies + +* **AZ-895** (Done locally / In Testing in Jira) — this ticket addresses fallout from that deprecation. +* **AZ-265 / AZ-404** (60s suite epic) — the regressed tests are deliverables of that epic. +* **AZ-777** (Phase 3 superseded) — referenced in the existing xfail rationale; understanding why it's superseded informs the triage. +* **AZ-962** (sibling) — the AZ-840 orchestrator test is blocked by a different gap; both are cycle-4 e2e closure work but they're independent and can be worked in parallel. + +## Estimate + +3 SP. Investigation + triage + implementation. May bump to 5 SP if option B (build reference tile cache) is chosen — in that case split into sub-tickets per the user's complexity-budget rule (≤5 SP per ticket). + +## Run-log evidence (2026-05-29 Tier-2) + +``` +e2e-runner-1 | = 4 failed, 48 passed, 3 skipped, 1 xfailed, 1 xpassed, 1 warning in 90.59s (0:01:30) = +e2e-runner-1 | FAILED tests/e2e/replay/test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match +e2e-runner-1 | FAILED tests/e2e/replay/test_derkachi_1min.py::test_ac5_determinism_two_runs_diff +e2e-runner-1 | FAILED tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_realtime_60s_within_5pct +e2e-runner-1 | FAILED tests/e2e/replay/test_derkachi_1min.py::test_ac6_pace_asap_under_30s +e2e-runner-1 | XPASS tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks +``` + +Excerpt from the stdout of the first failure (representative of all 4): + +``` +{"ts":"2026-05-29T10:34:50.397901Z","level":"ERROR","component":"c5_state.eskf_baseline", + "kind":"c5.state.eskf_filter_divergence", + "kv":{"source":"vio","mahalanobis_sq":212.31115250586484,"threshold_sq":100.0}} +{"ts":"2026-05-29T10:34:50.398356Z","level":"ERROR","component":"runtime_root.replay_loop", + "kind":"replay_loop.state_add_vio_fatal", + "msg":"replay_loop.state_add_vio_fatal: frame=233 EstimatorFatalError('eskf filter divergence on vio: mahalanobis²=212.311 > 100.0')"} +``` + +## References + +* Failing tests: `tests/e2e/replay/test_derkachi_1min.py:82, 387, 417, 433` +* XPASS: `tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks` +* ESKF threshold: `c5_state.eskf_baseline` (Mahalanobis² 100.0 threshold) +* Satellite-anchoring-not-wired warning: `runtime_root.replay_loop:replay_loop.satellite_anchoring_not_wired` +* README AC matrix: `tests/e2e/replay/README.md` § `AC matrix` +* Sibling ticket (parallel work): AZ-962 — orchestrator config wiring diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 1cca599..7e6c511 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -8,7 +8,7 @@ status: in_progress sub_step: phase: 6 name: implement-tasks - detail: "batch 8 = tracker-only fix for AZ-842 (To Do → Done, read-back verified) + wider Jira drift audit recorded as `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. 10 cycle-3/4 tickets (AZ-836/838/839/840/894/895/896/899/900/901) shipped to `done/` locally but stuck in 'In Testing' in Jira; Epic AZ-835 in `todo/` with all 5 children done. User skipped A/B/C/D convention question — leftover holds the bulk-transition payload for whichever convention they pick. **Corrected cycle-4 todo/ remainder**: nothing actionable. Earlier narratives that listed AZ-899/900/901 as 'cycle-4 todo/ remainder for next batches' were fiction — those specs have been in done/ the whole time. OKVIS2 chain (AZ-943/951/952) sits in todo/ but is deferred per user 2026-05-29 directive until after Derkachi e2e flight test passes. Cycle-4 product work is effectively complete pending Derkachi e2e green + AZ-897 UI in ../ui." + detail: "batch 9 = Tier-2 Jetson e2e validation run NOT GREEN. Ran `JETSON_SSH_ALIAS=jetson bash scripts/run-tests-jetson.sh`; result = 4 failed / 48 passed / 3 skipped / 1 xfailed / 1 xpassed in 90.59s. Two distinct blockers: (1) AZ-840 orchestrator test SKIPPED because `GPS_DENIED_OPERATOR_CONFIG_PATH` not exported by `docker-compose.test.jetson.yml` AND `operator_replay.yaml` missing from repo — Epic AZ-835's 'Done' status was validated by doc-content only, never by actual orchestrator test execution; (2) AZ-895 fallout — 4 tests in `test_derkachi_1min.py` regress with `EstimatorFatalError('eskf filter divergence: mahalanobis²=212.311 > 100.0')` at frame 233 because the CSV-driven path (now primary) runs open-loop on the Derkachi fixture (no reference C6 tile cache → no satellite anchoring). Filed AZ-962 (3 SP, operator config + compose wiring) and AZ-963 (3 SP, ESKF regression triage). OKVIS2 chain stays deferred per user 2026-05-29 directive ('after Derkachi e2e green' — directive unchanged; e2e not green). AZ-842 caveat: the AZ-840/AZ-842 'Done' tracker state set earlier today is contingent on whether convention (A) 'In Testing = shipped' or (B) 'Done = shipped+tested' applies; user-skipped convention question, leftover holds the walk-back payload if needed. Cycle-4 not green. Earlier same-day batch 8 = tracker-only fix for AZ-842 (To Do → Done, read-back verified) + wider Jira drift audit recorded as `_docs/_process_leftovers/2026-05-29_jira_status_drift_audit.md`. 10 cycle-3/4 tickets (AZ-836/838/839/840/894/895/896/899/900/901) shipped to `done/` locally but stuck in 'In Testing' in Jira; Epic AZ-835 in `todo/` with all 5 children done. User skipped A/B/C/D convention question — leftover holds the bulk-transition payload for whichever convention they pick. **Corrected cycle-4 todo/ remainder**: nothing actionable. Earlier narratives that listed AZ-899/900/901 as 'cycle-4 todo/ remainder for next batches' were fiction — those specs have been in done/ the whole time. OKVIS2 chain (AZ-943/951/952) sits in todo/ but is deferred per user 2026-05-29 directive until after Derkachi e2e flight test passes. Cycle-4 product work is effectively complete pending Derkachi e2e green + AZ-897 UI in ../ui." retry_count: 0 cycle: 4 tracker: jira