diff --git a/_docs/02_tasks/_dependencies_table.md b/_docs/02_tasks/_dependencies_table.md index 7382d77..1ffea0a 100644 --- a/_docs/02_tasks/_dependencies_table.md +++ b/_docs/02_tasks/_dependencies_table.md @@ -1,6 +1,6 @@ # Dependencies Table -**Date**: 2026-05-29 (cycle-4 Step 10 Implement — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) +**Date**: 2026-05-29 (cycle-4 Step 10 Implement — **AZ-959 LANDED** (`replay_api` `POST /replay` CSV-path extension): handler now accepts `(video, tlog)` OR `(video, csv)` multipart with XOR validation; `validate_csv_kind` rejects malformed schemas at the API boundary referencing `csv_replay_format.md`; `SubprocessReplayRunner.run` dispatches `--imu` vs `--tlog`; `_maybe_render_report` dispatches GT loader via `load_csv_ground_truth` / `load_tlog_ground_truth`; `ReplayInputs` DTO carries `tlog_path: Path | None` + `csv_path: Path | None` with XOR `__post_init__`; `JobStorage` reserves both `tlog_path` and `csv_path`; new `GET /static/example-csv` endpoint serves the AZ-896 reference CSV via `REPLAY_API_EXAMPLE_CSV_PATH` env or source-checkout fallback. 27/27 unit tests green (18 pre-existing tlog tests pass unchanged per AC-7 + 9 new tests covering ACs 1-6 + 2 `validate_csv_kind` unit cases). **Deferred items (NOT silently fixed, surfaced to user as end-of-turn notes)**: (a) `gps-denied-render-map` only consumes binary tlog truth → CSV-path jobs return `map_html_url=None` (deferred to AZ-700 follow-up); (b) `ReportContext.tlog_path` field is now overloaded as "ground-truth source path"; rendered report's `Tlog: ` line is cosmetically misleading for CSV runs (deferred to AZ-699 follow-up). AZ-959 moved todo/ → done/. Cycle-4 active-scope SP delta: −3 SP (3 → 0). Earlier same-day — OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) moved backlog/ → todo/ per user 2026-05-29 directive: "I have a feeling that it needed to be implemented after full e2e derkachi flight test would be finished successfully. So maybe put it back to todo?" Reasoning accepted: OKVIS2 is the planned NEXT phase after the cycle-4 Derkachi demo lands, not a cycle-5+ deferral. The 2026-05-27 production-default pivot directive remains in force; today's earlier "deferred to cycle-5+" framing was over-correction after the AZ-943 spec-reality gap. AZ-943 keeps its PAUSED preamble (still HARD-BLOCKED on AZ-951 + AZ-952; cannot be worked on until both blockers land). AZ-951 + AZ-952 are themselves NOT blocked — they ship the upstream patches that unblock AZ-943. Implementation sequence remains: finish the cycle-4 demo (AZ-959 backend extension + the existing CSV-replay path) → AZ-951 (covariance + ADR) → AZ-952 (tracking-stats) → AZ-943 (binding wiring) → AZ-944 (CI BUILD_OKVIS2=ON) → AZ-945 (Jetson Tier-2 `--vio-strategy okvis2`). Current implement-batch target stays AZ-959. Earlier same-day — AZ-897 relocated to `../ui` repo: original framing was wrong-shop. The Azaion suite already has a single React 19 SPA front-end at `../ui` per `ui/README.md`; spinning up a second React toolchain in `gps-denied-onboard` would have been parallel-pipeline duplication forbidden by coderule.mdc. Per user 2026-05-29 directive, AZ-897 description + summary rewritten to UI-only scope in `../ui` (adapted to take CSV + nadir-camera video uploads aligned with the AZ-894 CSV path); local AZ-897 spec deleted from `gps-denied-onboard/_docs/02_tasks/todo/` and re-authored into `../ui/_docs/02_tasks/todo/AZ-897_replay_ui_web_form.md` (no commit in `../ui` — left for that repo's autodev next cycle). Backend dependency filed as **AZ-959** (3pt, todo/, c1 replay_api extension to accept (video, csv) multipart + GET /static/example-csv endpoint; deps AZ-701 + AZ-894 + AZ-896; no epic) — extends the AZ-701 `POST /replay` to dispatch on `--imu` vs `--tlog` based on which upload field was present, with XOR validation. AZ-897 Jira linked `is blocked by` AZ-959. Cycle-4 in-repo effort: −5 SP (AZ-897) + 3 SP (AZ-959) = −2 SP net. Pivoting next implement batch to AZ-959. Earlier same-day — AZ-943 implementation attempt paused mid-batch on spec-reality gap: OKVIS2 v2 public API does NOT expose 6×6 pose covariance, feature counts, mean parallax, or MRE; the AZ-943 spec's "approach (a) in-binding subclass workaround" is structurally impossible because `ThreadedSlam::estimator_` is `private` and `ViSlamBackend` has no public telemetry accessor. The spec-documented "approach (b) upstream patch" fallback filed as **AZ-951** (3pt, backlog/, OKVIS2 v2 upstream patch: expose 6×6 pose covariance accessor + ADR for pin deviation; deps AZ-332 + AZ-592; epic AZ-254) + **AZ-952** (3pt, backlog/, OKVIS2 v2 upstream patch: expose tracking-stats accessor — feature counts + parallax + MRE; deps AZ-332 + AZ-592 + AZ-951 SOFT; epic AZ-254). Both linked Jira-side as `is blocked by` against AZ-943; AZ-943 transitioned In Progress → To Do with full audit comment. **AZ-943** moved todo/ → backlog/ with PAUSED preamble preserving original AC list for audit. Per user 2026-05-29 confirmation, cycle-4 Derkachi demo target stays KLT/RANSAC (per `tests/e2e/replay/conftest.py` line 159 `c1_vio: strategy: klt_ransac`); OKVIS2 chain (AZ-943 → AZ-944 → AZ-945 + AZ-951/952 blockers) deferred to cycle-5+ alongside AZ-945's Tier-2 `--vio-strategy okvis2` Jetson variant. Pivot to AZ-897 (replay UI web form). Earlier this session: OKVIS2 production-default pivot per user 2026-05-27 directive: AZ-592 placeholder split into 3 properly-sized sub-tickets per PBI rule, all three filed Jira-side then; local-spec import for AZ-943 happens this session before implement batch starts. **AZ-943** (5pt, **NOW backlog/** with PAUSED preamble, c1_vio, OKVIS2 binding wiring; replaces AZ-332 skeleton; deps AZ-332 + AZ-592 + **AZ-951 + AZ-952 (blockers)**; epic AZ-254). Sibling tickets remain Jira-only this session: **AZ-944** (3pt, Linux CI build env + DBoW2 small_voc + Tier-1 EuRoC smoke; Blocks chain AZ-943→AZ-944) and **AZ-945** (3pt, Jetson L4T + Tier-2 Derkachi `--vio-strategy okvis2` e2e; Blocks chain AZ-944→AZ-945). Local specs for AZ-944 + AZ-945 will be authored when their Implement turns come up. Earlier 2026-05-26 (cycle-4 Step 10 Implement — AZ-895 batch 3 user complexity decision: chose Option A "minimum deprecation" path. Filed **AZ-908** (3pt, backlog/, replay: hard removal of deprecated auto-sync surface — AZ-895 follow-up; deps AZ-895 HARD + AZ-842 HARD; no epic) to track the cycle-5+ physical removal that AZ-895's minimum-path explicitly defers. AZ-895 ships the no-op stubs + CLI deprecation warnings; AZ-908 will delete the stub files, drop the DTOs from `replay_input/interface.py`, remove the deprecated CLI flags, and drop the `auto_sync` config block. No SP change to cycle-4 totals (AZ-908 is cycle-5+ backlog, not cycle-4 active scope). Earlier same-day at Step 9 New Task — scope adjustments: (a) AZ-841 (1pt, un-xfail AZ-777 Tier-2 tests) moved from todo/ to backlog/ due to hard conflict with AZ-895 AC-4 (test_derkachi_real_tlog.py stays @xfail in cycle 4 because AZ-848 is backlogged) + partial overlap with AZ-894 AC-3 (CSV-path adapter covers the test_derkachi_1min.py un-xfail target); Jira comment added to AZ-841 documenting the deferral. (b) AZ-842 (2pt → **3pt**, +1 SP rescope) — dropped AZ-841 soft dependency, expanded replay_protocol.md scope to add new Invariant 13 covering single-canonical-clock model + cycle-4 CSV-driven replay narrative (AZ-894 + AZ-895 + AZ-896), plus architecture.md replay-input section updates. New deps: AZ-894 HARD + AZ-895 HARD + AZ-896 SOFT. (c) +**AZ-899** (1pt, product, todo/, land architecture_compliance_baseline.md — cycle-3 retro Top-3 #3 third try; deps None; no epic). (d) +**AZ-900** (1pt, product, todo/, autodev cycle-N+1 Step-9 retro-existence gate — cycle-3 retro Top-3 #2 + 2026-05-26 LESSONS process entry; deps None; no epic). (e) +**AZ-901** (1pt, product, todo/, fix EVIDENCE_OUT default path in e2e/runner/conftest.py:56 — closes 2026-05-26 leftover; deps None; no epic). Cycle-4 active scope: 6 product tickets in todo/ totaling **17 SP** = AZ-842 (3, docs) + AZ-894 (3, CSV adapter) + AZ-895 (2, auto-sync deprecation) + AZ-896 (1, format docs) + AZ-897 (5, replay UI) + AZ-899 (1) + AZ-900 (1) + AZ-901 (1). Dependency order: AZ-894 blocks AZ-895 + AZ-842 + AZ-897; AZ-896 blocks AZ-897 + AZ-842. AZ-899/AZ-900/AZ-901 standalone (no internal blockers). AZ-848 (5) + AZ-883 (2) + AZ-908 (3) remain in backlog/ (cycle-3 retro Top-3 #1 + AZ-895 follow-up deferred to cycle-5+; CSV-bypass strategy supersedes their fixes for the demo path). Earlier 2026-05-23 (cycle-3 Step 10 Implement, refactor run 02-az507-routespec-relocation — added AZ-844 (Epic, run dir `_docs/04_refactoring/02-az507-routespec-relocation/`) + AZ-845 (C01, 2pt relocate `RouteSpec` from `replay_input/tlog_route.py` to `_types/route.py`, deps None, epic AZ-844) + AZ-846 (C02, 2pt refresh `module-layout.md` cycle-3 entries — c11 + replay_input + `_types/route`, deps AZ-845, epic AZ-844) + AZ-847 (C03, 2pt widen `test_az270_compose_root` lint to enforce full rule-9 allow-list, deps AZ-845, epic AZ-844). Resolves cycle-3 cumulative review FAIL verdict (F1 High Architecture, F2 Medium Architecture, F3 Medium Maintainability) per `_docs/03_implementation/cumulative_review_batches_104-109_cycle3_report.md`. Jira "Blocks" links recorded: AZ-845 → AZ-846, AZ-845 → AZ-847. Earlier same-day at start of Step 10 Implement — Epic AZ-835 decomposed into 4 leaf tasks + AZ-777 closed: AZ-839 (C3, 5pt operator_pre_flight_setup real fixture, deps AZ-836+AZ-838+AZ-777Phase1+AZ-322+AZ-316+AZ-306, epic AZ-835), AZ-840 (C4, 3pt e2e orchestrator test (tlog,video,calibration), deps AZ-839+AZ-836+AZ-838+AZ-699+AZ-405+AZ-702+AZ-696, epic AZ-835), AZ-841 (C5, 1pt un-xfail AZ-777 AC-4+AC-5, deps AZ-839+AZ-840, epic AZ-835), AZ-842 (C6, 2pt docs — replay_protocol.md Invariant 12 + architecture.md + orchestrator README, soft dep AZ-841, epic AZ-835). AZ-777 transitioned to Done in Jira: Phases 1+2 shipped (batch 104 + between batches 104 and 106); Phases 3-5 superseded by Epic AZ-835 children per 2026-05-22 user directive. AZ-777 spec moved to done/. Earlier 2026-05-21 (cycle-3 Step 9 New Task — added AZ-776 (3pt open-loop ESKF composition profile via `c4_pose.enabled` flag, no deps, epic AZ-602) + AZ-777 (5pt Derkachi C6 reference tile cache + FAISS descriptor index from OSM/CARTO basemap, depends on AZ-776, epic AZ-602). Both unblock the 7 currently-`@xfail`-masked Derkachi e2e tests on Jetson; AZ-776 unblocks 5 (AC-1, AC-2, AC-5, AC-6 realtime, AC-6 asap), AZ-777 unblocks the remaining 2 (AC-3 + AZ-699 real-flight verdict). Earlier 2026-05-19 (refreshed late-morning after 11:27 Jetson Tier-2 e2e run for AZ-618 — surfaced a NEW gap: replay-mode `Config` lacks `c6_tile_cache` block, so `build_pre_constructed → _build_c6_descriptor_index → _c6_config` raises `KeyError` for AC-1/2/5/6. Follow-up filed as AZ-687 (2pt) under E-AZ-602 with guard at the bootstrap layer (NOT silent fallback in `_c6_config`). Earlier same-day mid-day after AZ-618 split: per the spec author's own Sizing-note recommendation + user-rule cap on PBI complexity, AZ-618 was split into 6 subtasks AZ-619..AZ-624 in Jira (subtasks of AZ-618; epic AZ-602 stays grandparent). AZ-618 retained at 0pt as the umbrella tracker; aggregate actionable work is 16pt across the subtasks (vs. AZ-618's original 5pt filing — author's "likely a true 8" caveat was understated due to c5_isam2_graph_handle ordering + GPU builder unknowns). Earlier same-day refresh at start of Step-7 rewind for AZ-618 — Step-11 Jetson tier-2 e2e gate identified missing internal product implementation: `runtime_root.main()` does not build the airborne `pre_constructed` infrastructure dict before `compose_root()`; AZ-618 = 5pt cross-cutting follow-up to AZ-591, lives under E-AZ-602; all 12 dep tasks are in `done/`. Earlier 2026-05-16 (cycle-1 completeness-gate post-mortem): AZ-589 + AZ-590 closed Won't Fix — were wrong abstraction (OKVIS v1 `ThreadedKFVio` API doesn't exist in OKVIS2 upstream; VINS-Mono `cpp/vins_mono/upstream/` submodule never existed; the actual production gap is the empty central `_STRATEGY_REGISTRY` affecting EVERY component with a strategy-selecting config field, not just c1_vio); replaced by AZ-591 (cross-cutting compose_root per-binary bootstrap, todo/, 5pt) + AZ-592 (AZ-332 Tier-2 validation bundle, backlog/, 5pt placeholder) + AZ-593 (AZ-333 Tier-2 validation bundle, backlog/, 5pt placeholder); AZ-332 + AZ-333 re-classified in gate report from FAIL to BLOCKED-on-Tier-2 per the original tasks' Implementation Notes deferral handles; earlier same-day after end of cycle-1 gate: AZ-589 + AZ-590 created (now closed); earlier same-day after end of Batch 64: AZ-558 implementation closed — `MavlinkTransport` seam now routes every C8 outbound MAVLink byte; AZ-401 AC-9 + AZ-404 AC-4b unskipped together; encoder helpers extracted to `_outbound_mavlink_payloads.py`; live-mode `compose_root` injection deferred to whichever future batch registers AP/iNav strategies in an airborne binary; earlier 2026-05-14: refreshed at start of Batch 63: AZ-559 closed Won't Fix — gap was illusory; `TileSource.ONBOARD_INGEST` + `TileMetadata.quality_metadata` + `write_tile`'s `FreshnessRejectionError` already cover the AZ-389 mid-flight ingest semantic without any new API; AZ-389 dep restored to AZ-303; earlier same-day after Batch 61: AZ-558 follow-up added — routes C8 outbound encoder bytes through `MavlinkTransport` seam; closes AZ-401 AC-9 deferred during batch 61 due to encoder-side routing not being in the AZ-401 task envelope; earlier same-day after cumulative review batches 52-54: AZ-528 hygiene PBI added for c1_vio strategy facade orchestration-spine 3-way duplication (Medium); earlier same-day after Batch 53: AZ-333 VINS-Mono landed — first c1_vio strategy after the AZ-332 OKVIS2 production-default; consolidation hygiene for the strategy-facade duplication deferred to a post-AZ-334 PBI; earlier same-day after Batch 51: AZ-527 hygiene PBI added from cumulative review batches 49-51 F1; 2026-05-13: AZ-526 hygiene PBI added from cumulative review batches 46-48 F1+F3; same-day refresh after Batch 44 SRP refactor: AZ-317 superseded; AZ-329 + AZ-330 specs rewritten; AZ-523 + AZ-524 audit-trail tickets added; E-C12 epic renamed `Operator Pre-flight Tooling` → `Operator Pre-flight Orchestrator`; earlier same-day refresh: AZ-507 + AZ-508 hygiene PBIs from cumulative review batches 31-33; 2026-05-11: AZ-489 + AZ-490 ADR-010 operator-origin path) **Total Tasks**: 184 (143 product + 41 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +AZ-959 (1 product task, todo/, 3pt). AZ-897 was never in this table's row count (pre-existing gap — the cycle-4 AZ-89x specs land in todo/ but were not back-filled into the table rows; not fixing that here, out of scope). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 183 (142 product + 41 blackbox-test) — +AZ-951 + AZ-952 (2 product tasks, both backlog/, 3pt each). AZ-943 (5pt) moved todo/ → backlog/ (no count change). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 181 (140 product + 41 blackbox-test) → 182 (141 product) — +AZ-943 (1 product task, originally todo/, 5pt). AZ-944 + AZ-945 remain Jira-only at the time of this update (sibling tickets, local specs deferred to their own Implement turns); their Total-Tasks impact will be reconciled when their specs land. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 180 (139 product + 41 blackbox-test) — +AZ-908 (1 product task, backlog/, 3pt). Prior 2026-05-26 cycle-4 Step 9 bump: +AZ-899 + AZ-900 + AZ-901 (3 product tasks). AZ-841 moved todo/ → backlog/ (no count change; backlog tickets are still in the table). Prior 2026-05-23 refactor-run bump: 176 (135 product + 41 blackbox-test) — +AZ-844 (Epic, 0pt umbrella for refactor run 02) + AZ-845 + AZ-846 + AZ-847 (3 product tasks). Prior 2026-05-23 bump (Epic AZ-835 decomposition): 173 (132 product + 41 blackbox-test) = +AZ-835 (Epic) + AZ-836 (C1) + AZ-837 (test-stack hardening, not this Epic) + AZ-838 (C2) added 2026-05-22→2026-05-23 prior to that update; +AZ-839 (C3) + AZ-840 (C4) + AZ-841 (C5) + AZ-842 (C6) added in that update. AZ-777 stays in the table (now closed in Jira; spec at `done/AZ-777_derkachi_c6_reference_fixture.md` retains 8pt credit for Phases 1+2 shipped). Earlier counts: 165 (124 product + 41 blackbox-test) — AZ-317 retained in the table marked SUPERSEDED for audit; AZ-523 (C11 gate removal) + AZ-524 (C12 rename) added as 2 closed audit-trail tasks; AZ-526 = 2pt clock-helper hygiene; AZ-527 = 2pt c2 engine-dim helper hygiene; AZ-528 = 3pt c1_vio facade-spine hygiene; AZ-558 = 3pt MavlinkTransport routing follow-up; AZ-559 closed Won't Fix; AZ-589 + AZ-590 closed Won't Fix (kept in table as 0pt audit-trail rows); AZ-591 = 5pt cross-cutting compose_root bootstrap (todo/); AZ-592 = 5pt OKVIS2 Tier-2 placeholder (backlog/); AZ-593 = 5pt VINS-Mono Tier-2 placeholder (backlog/); AZ-618 = 0pt umbrella (split into AZ-619..AZ-624 on 2026-05-19); AZ-619..AZ-624 = 6 subtasks of AZ-618 covering Phase A..F of the airborne `pre_constructed` assembly, summing to 16pt actionable work; AZ-687 = 2pt replay-mode guard follow-up surfaced by AZ-618 Tier-2 run on 2026-05-19 **Total Complexity Points**: 584 (451 product + 133 blackbox-test) — 2026-05-29 cycle-4 Step 10 third bump (AZ-897 relocation + AZ-959 filing): +3pt AZ-959. AZ-897 (5pt) was never table-counted here, so no decrement at this layer; the in-repo cycle-4 effort still drops by 5pt at the active-scope layer (AZ-897 work is now executed in `../ui`). Prior same-day 2026-05-29 second bump (AZ-943 paused, dependency PBIs filed): 581 (448 product + 133 blackbox-test) — +3pt AZ-951 + 3pt AZ-952 = +6 product pts. AZ-943 stays counted at 5pt (moved todo/ → backlog/, not deleted). Prior same-day 2026-05-29 bump (OKVIS2 binding session start): 580 (447 product + 133 blackbox-test) — +5pt AZ-943. AZ-944 (3pt) + AZ-945 (3pt) sibling tickets are filed Jira-side but not yet imported as local specs; their +6pt will land when AZ-944 / AZ-945 specs are authored. Prior 2026-05-26 cycle-4 Step 10 bump (AZ-895 batch 3 follow-up): 570 (437 product + 133 blackbox-test) — +3pt AZ-908. Prior 2026-05-26 cycle-4 Step 9 bump: +1pt AZ-899 + 1pt AZ-900 + 1pt AZ-901 + 1pt AZ-842 rescope (2→3) = +4 product pts. Prior 2026-05-23 refactor-run bump: 563 (430 product + 133 blackbox-test) — +2pt AZ-845 + 2pt AZ-846 + 2pt AZ-847 = +6 product pts on top of prior reconciled total (AZ-844 epic itself is 0pt umbrella). Prior 2026-05-23 reconciled total: 557 (424 product + 133 blackbox-test) — +5pt AZ-839 + 3pt AZ-840 + 1pt AZ-841 + 2pt AZ-842 = +11 product pts on top of prior reconciled total. AZ-836 (3pt) + AZ-838 (3pt) were added 2026-05-22→2026-05-23 prior to that update; AZ-837 (test-stack hardening, not this Epic) is unaccounted in that delta and should be folded in at the next preamble reconciliation. Earlier baseline: 546 (413 product + 133 blackbox-test) — +3pt AZ-776 + 8pt AZ-777 (5→8 override 2026-05-21 cycle-3 batch 104; see `_docs/_process_leftovers/2026-05-21_az777_complexity_override.md` for rationale + the spec refresh that pulled e2e-runner wiring + C11 contract adapt + Derkachi catalog seed + fixture replacement + un-xfail into one ticket) — AZ-523 = 3pt, AZ-524 = 2pt, AZ-526 = 2pt, AZ-527 = 2pt, AZ-528 = 3pt, AZ-558 = 3pt, AZ-589 + AZ-590 retained at 5pt each but closed Won't Fix (treated as 0 effective pts going forward), AZ-591 = 5pt, AZ-592 = 5pt placeholder, AZ-593 = 5pt placeholder, AZ-618 = 0pt umbrella post-split, AZ-619 = 2pt, AZ-620 = 3pt, AZ-621 = 3pt, AZ-622 = 3pt, AZ-623 = 3pt, AZ-624 = 2pt, AZ-687 = 2pt diff --git a/_docs/02_tasks/todo/AZ-959_replay_api_csv_path_endpoint.md b/_docs/02_tasks/done/AZ-959_replay_api_csv_path_endpoint.md similarity index 100% rename from _docs/02_tasks/todo/AZ-959_replay_api_csv_path_endpoint.md rename to _docs/02_tasks/done/AZ-959_replay_api_csv_path_endpoint.md diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 59603e9..75d2247 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -8,7 +8,7 @@ status: in_progress sub_step: phase: 6 name: implement-tasks - detail: "batch 5 of N: AZ-959 replay_api POST /replay CSV-path extension (pivoted from AZ-897 after relocation to ../ui repo per user 2026-05-29; AZ-959 is the backend slice that unblocks the relocated AZ-897 UI)" + detail: "batch 5 complete: AZ-959 replay_api POST /replay CSV-path extension landed (27/27 unit tests green, all 7 ACs covered: CSV happy path + XOR validation + malformed-CSV reject + /static/example-csv + CSV ground-truth dispatch in SubprocessReplayRunner._maybe_render_report + AZ-701 tlog tests unchanged). Map rendering for CSV path skipped (gps-denied-render-map only supports tlog truth today; deferred to AZ-700 follow-up). ReportContext.tlog_path field widened in-place to 'ground-truth source path' for the CSV case; the cosmetic 'Tlog:' label in the rendered report is now misleading for CSV runs (note for user — AZ-699 follow-up territory). Next batch: pick one of cycle-4 todo/ remainder (AZ-842 docs / AZ-899 / AZ-900 / AZ-901). OKVIS2 chain (AZ-943 + AZ-951 + AZ-952) sits in todo/ but is sequenced after the Derkachi e2e flight test passes per user 2026-05-29 directive." retry_count: 0 cycle: 4 tracker: jira diff --git a/src/gps_denied_onboard/replay_api/app.py b/src/gps_denied_onboard/replay_api/app.py index e235324..5af7a98 100644 --- a/src/gps_denied_onboard/replay_api/app.py +++ b/src/gps_denied_onboard/replay_api/app.py @@ -37,12 +37,14 @@ from gps_denied_onboard.replay_api.errors import ( UnsupportedFileKindError, ) from gps_denied_onboard.replay_api.handlers import ( + MIN_CSV_PROBE_BYTES, MIN_TLOG_PROBE_BYTES, MIN_VIDEO_PROBE_BYTES, auth_required, expected_bearer_token, extract_bearer_token, validate_calibration_kind, + validate_csv_kind, validate_tlog_kind, validate_upload_size, validate_video_kind, @@ -63,7 +65,9 @@ __all__ = ["SubprocessReplayRunner", "build_runner_from_env", "create_app"] _LOGGER = logging.getLogger("gps_denied_onboard.replay_api") -_PROBE_BYTES_MAX: int = max(MIN_TLOG_PROBE_BYTES, MIN_VIDEO_PROBE_BYTES, 64) +_PROBE_BYTES_MAX: int = max( + MIN_TLOG_PROBE_BYTES, MIN_VIDEO_PROBE_BYTES, MIN_CSV_PROBE_BYTES, 64 +) # --------------------------------------------------------------------- @@ -107,12 +111,16 @@ class SubprocessReplayRunner: signing_key_path.write_bytes(b"\x00" * 32) emissions_path = output_dir / "emissions.jsonl" + if inputs.csv_path is not None: + input_flag_pair = ["--imu", str(inputs.csv_path)] + else: + assert inputs.tlog_path is not None + input_flag_pair = ["--tlog", str(inputs.tlog_path)] argv = [ self._replay_binary, "--video", str(inputs.video_path), - "--tlog", - str(inputs.tlog_path), + *input_flag_pair, "--output", str(emissions_path), "--camera-calibration", @@ -175,6 +183,7 @@ class SubprocessReplayRunner: horizontal_error_distribution, ) from gps_denied_onboard.replay_input import ( + load_csv_ground_truth, load_tlog_ground_truth, ) except Exception as exc: @@ -191,7 +200,13 @@ class SubprocessReplayRunner: if not emissions: return None - gt_series = load_tlog_ground_truth(inputs.tlog_path).records + if inputs.csv_path is not None: + gt_series = load_csv_ground_truth(inputs.csv_path).records + gt_source_path = inputs.csv_path + else: + assert inputs.tlog_path is not None + gt_series = load_tlog_ground_truth(inputs.tlog_path).records + gt_source_path = inputs.tlog_path if not gt_series: return None @@ -222,7 +237,11 @@ class SubprocessReplayRunner: ) context = ReportContext( run_date_utc=datetime.utcnow().date().isoformat(), - tlog_path=inputs.tlog_path, + # tlog_path is widened to "ground-truth source" in cycle-4 + # (tlog or csv depending on which input drove the run); + # ReportContext field rename deferred to AZ-699 follow-up + # to keep AZ-959 scope minimal. + tlog_path=gt_source_path, video_path=inputs.video_path, calibration_acquisition_method=calibration_method, clip_duration_s=clip_duration_s, @@ -243,6 +262,16 @@ class SubprocessReplayRunner: output_dir: Path, report_path: Path | None, ) -> Path | None: + # gps-denied-render-map only understands binary tlog truth + # today; CSV-truth dispatch is an AZ-700 follow-up. For now, + # CSV-path runs ship without a map (report + emissions still + # render, see _maybe_render_report). + if inputs.tlog_path is None: + _LOGGER.info( + "skipping map render — CSV-path runs do not yet support " + "the gps-denied-render-map CLI (AZ-700 follow-up)" + ) + return None if not shutil.which(self._render_binary): venv_bin = Path(sys.executable).parent / self._render_binary if not venv_bin.exists(): @@ -433,8 +462,9 @@ def create_app( @app.post("/replay") async def post_replay( - tlog: Annotated[UploadFile, File()], video: Annotated[UploadFile, File()], + tlog: Annotated[UploadFile | None, File()] = None, + csv: Annotated[UploadFile | None, File()] = None, calibration: Annotated[UploadFile | None, File()] = None, pace: Annotated[str, Form()] = "asap", auto_trim: Annotated[bool, Form()] = True, @@ -442,9 +472,28 @@ def create_app( ) -> Response: _check_auth(authorization) - tlog_bytes = await tlog.read() - validate_upload_size(len(tlog_bytes), limit=max_upload_bytes) - validate_tlog_kind(tlog_bytes[:_PROBE_BYTES_MAX]) + # AC-2 / AC-3: exactly one of (tlog, csv) must be present. + if (tlog is None) == (csv is None): + raise MultipartMissingFieldError( + "POST /replay requires exactly one of `tlog` or `csv` " + "multipart fields (got " + f"tlog={'present' if tlog else 'absent'}, " + f"csv={'present' if csv else 'absent'}). See " + "_docs/02_document/contracts/replay/csv_replay_format.md " + "for the CSV schema." + ) + + tlog_bytes: bytes | None = None + csv_bytes: bytes | None = None + if tlog is not None: + tlog_bytes = await tlog.read() + validate_upload_size(len(tlog_bytes), limit=max_upload_bytes) + validate_tlog_kind(tlog_bytes[:_PROBE_BYTES_MAX]) + else: + assert csv is not None + csv_bytes = await csv.read() + validate_upload_size(len(csv_bytes), limit=max_upload_bytes) + validate_csv_kind(csv_bytes[:_PROBE_BYTES_MAX]) video_bytes = await video.read() validate_upload_size(len(video_bytes), limit=max_upload_bytes) @@ -461,7 +510,10 @@ def create_app( # Allocate per-job storage and write the uploads. job_id = _new_job_id() job_storage = storage.allocate_job(job_id) - job_storage.tlog_path.write_bytes(tlog_bytes) + if tlog_bytes is not None: + job_storage.tlog_path.write_bytes(tlog_bytes) + if csv_bytes is not None: + job_storage.csv_path.write_bytes(csv_bytes) job_storage.video_path.write_bytes(video_bytes) if calibration_bytes is not None: job_storage.calibration_path.write_bytes(calibration_bytes) @@ -477,7 +529,12 @@ def create_app( ) inputs = ReplayInputs( - tlog_path=job_storage.tlog_path, + tlog_path=( + job_storage.tlog_path if tlog_bytes is not None else None + ), + csv_path=( + job_storage.csv_path if csv_bytes is not None else None + ), video_path=job_storage.video_path, calibration_path=job_storage.calibration_path, pace=pace, @@ -572,6 +629,36 @@ def create_app( filename="map.html", ) + @app.get("/static/example-csv") + async def get_example_csv() -> Response: + """Serve the AZ-896 reference CSV for the AZ-897 UI workflow. + + No auth required — the example file is a public reference + document. Returns 503 when the file cannot be located, which + per the AZ-959 spec is treated as a deploy-misconfiguration + signal (file exists in the source tree). + """ + path = _example_csv_path() + if path is None: + return JSONResponse( + status_code=503, + content={ + "error_code": "example_csv_unavailable", + "message": ( + "example CSV not located — set " + "REPLAY_API_EXAMPLE_CSV_PATH or run from a " + "source checkout that contains " + "_docs/02_document/contracts/replay/" + "example_data_imu.csv" + ), + }, + ) + return FileResponse( + path=path, + media_type="text/csv; charset=utf-8", + filename="example_data_imu.csv", + ) + @app.get("/jobs/{job_id}/report") async def get_report( job_id: str, @@ -626,6 +713,32 @@ def _default_calibration_path() -> Path | None: return None +def _example_csv_path() -> Path | None: + """Locate the AZ-896 reference CSV. + + First honours ``REPLAY_API_EXAMPLE_CSV_PATH``. As a dev / source- + checkout fallback walks up from this module looking for the + canonical doc location. Returns ``None`` when neither path + yields a readable file — the handler then returns 503. + """ + raw = os.environ.get("REPLAY_API_EXAMPLE_CSV_PATH") + if raw: + configured = Path(raw) + return configured if configured.is_file() else None + for parent in Path(__file__).resolve().parents: + candidate = ( + parent + / "_docs" + / "02_document" + / "contracts" + / "replay" + / "example_data_imu.csv" + ) + if candidate.is_file(): + return candidate + return None + + def _await_terminal(registry: JobRegistry, job_id: str) -> JobSnapshot: """Block until ``job_id`` reaches a terminal state. diff --git a/src/gps_denied_onboard/replay_api/handlers.py b/src/gps_denied_onboard/replay_api/handlers.py index 31e10e3..7ef178d 100644 --- a/src/gps_denied_onboard/replay_api/handlers.py +++ b/src/gps_denied_onboard/replay_api/handlers.py @@ -18,12 +18,14 @@ from gps_denied_onboard.replay_api.errors import ( ) __all__ = [ + "MIN_CSV_PROBE_BYTES", "MIN_TLOG_PROBE_BYTES", "MIN_VIDEO_PROBE_BYTES", "auth_required", "expected_bearer_token", "extract_bearer_token", "validate_calibration_kind", + "validate_csv_kind", "validate_tlog_kind", "validate_upload_size", "validate_video_kind", @@ -48,6 +50,26 @@ _MP4_FTYP_MARKER: bytes = b"ftyp" MIN_VIDEO_PROBE_BYTES: int = 12 +# CSV header line for the AZ-896 replay format is ~410 chars; probe +# generously so we can read the full header regardless of OS line +# endings or operator whitespace. The validator only checks the +# headline column tokens; the parser in ``csv_ground_truth`` does +# the strict per-row validation downstream. +MIN_CSV_PROBE_BYTES: int = 512 + +_CSV_REQUIRED_HEADER_TOKENS: tuple[str, ...] = ( + "timestamp(ms)", + "Time", + "SCALED_IMU2.xacc", + "SCALED_IMU2.xgyro", + "GLOBAL_POSITION_INT.lat", + "GLOBAL_POSITION_INT.lon", +) +_CSV_FORMAT_DOC_PATH: str = ( + "_docs/02_document/contracts/replay/csv_replay_format.md" +) + + def validate_tlog_kind(probe_bytes: bytes) -> None: """Reject anything that doesn't open with a MAVLink magic byte. @@ -90,6 +112,34 @@ def validate_video_kind(probe_bytes: bytes) -> None: ) +def validate_csv_kind(probe_bytes: bytes) -> None: + """Reject anything that doesn't open with the AZ-896 CSV header. + + The strict per-row schema lives in ``csv_ground_truth.py``; this + boundary check just confirms the first line looks like the AZ-896 + header so we fail fast at the API before the subprocess hands the + error back through an opaque non-zero exit code. + """ + if len(probe_bytes) < 1: + raise UnsupportedFileKindError("csv upload is empty") + header_end = probe_bytes.find(b"\n") + header_bytes = probe_bytes if header_end < 0 else probe_bytes[:header_end] + try: + header = header_bytes.decode("utf-8").strip() + except UnicodeDecodeError as exc: + raise UnsupportedFileKindError( + "csv header is not valid UTF-8 (see " + f"{_CSV_FORMAT_DOC_PATH})" + ) from exc + columns = {col.strip() for col in header.split(",")} + missing = [token for token in _CSV_REQUIRED_HEADER_TOKENS if token not in columns] + if missing: + raise UnsupportedFileKindError( + "csv header is missing required columns " + f"{missing} (see {_CSV_FORMAT_DOC_PATH})" + ) + + def validate_calibration_kind(probe_bytes: bytes) -> None: """Light JSON-shape check; the renderer is the strict validator.""" if not probe_bytes: diff --git a/src/gps_denied_onboard/replay_api/interface.py b/src/gps_denied_onboard/replay_api/interface.py index f5fd5c6..03f4a83 100644 --- a/src/gps_denied_onboard/replay_api/interface.py +++ b/src/gps_denied_onboard/replay_api/interface.py @@ -38,21 +38,36 @@ class JobState(str, Enum): @dataclass(frozen=True, slots=True) class ReplayInputs: - """The (tlog + video + calibration) bundle a runner consumes. + """The (tlog|csv + video + calibration) bundle a runner consumes. Storage paths are absolute. The handler builds these from a per-job temp directory (see ``storage.py``). + Exactly one of ``tlog_path`` / ``csv_path`` must be set — the + handler validates this at the multipart boundary and the DTO + re-enforces it in ``__post_init__`` so any internal call site + that violates the contract fails fast. + ``pace`` and ``auto_trim`` mirror the ``gps-denied-replay`` CLI - flags; the runner is responsible for translating them into argv. + flags; the runner is responsible for translating them into argv + (``--imu`` for the csv path, ``--tlog`` for the tlog path). """ - tlog_path: Path video_path: Path calibration_path: Path + tlog_path: Path | None = None + csv_path: Path | None = None pace: str = "asap" auto_trim: bool = True + def __post_init__(self) -> None: + if (self.tlog_path is None) == (self.csv_path is None): + raise ValueError( + "ReplayInputs requires exactly one of tlog_path or " + "csv_path to be set (got " + f"tlog_path={self.tlog_path!r}, csv_path={self.csv_path!r})" + ) + @dataclass(frozen=True, slots=True) class ReplayJobResult: diff --git a/src/gps_denied_onboard/replay_api/storage.py b/src/gps_denied_onboard/replay_api/storage.py index d6611bb..05785fc 100644 --- a/src/gps_denied_onboard/replay_api/storage.py +++ b/src/gps_denied_onboard/replay_api/storage.py @@ -27,10 +27,17 @@ _LOGGER = logging.getLogger("gps_denied_onboard.replay_api.storage") @dataclass(frozen=True, slots=True) class JobStorage: - """The per-job paths the handler hands to the runner.""" + """The per-job paths the handler hands to the runner. + + Both ``tlog_path`` and ``csv_path`` are reserved on disk; the + handler writes to exactly one and leaves the other unused. The + ``ReplayInputs`` DTO carries ``None`` for the branch that wasn't + written so downstream consumers know which clock source applies. + """ root: Path tlog_path: Path + csv_path: Path video_path: Path calibration_path: Path output_dir: Path @@ -60,6 +67,7 @@ class StorageRoot: return JobStorage( root=job_root, tlog_path=job_root / "input.tlog", + csv_path=job_root / "input.csv", video_path=job_root / "input.mp4", calibration_path=job_root / "calibration.json", output_dir=output_dir, diff --git a/tests/unit/replay_api/test_az701_replay_api.py b/tests/unit/replay_api/test_az701_replay_api.py index 634b6ba..b82ab39 100644 --- a/tests/unit/replay_api/test_az701_replay_api.py +++ b/tests/unit/replay_api/test_az701_replay_api.py @@ -38,6 +38,7 @@ from gps_denied_onboard.replay_api import ( create_app, ) from gps_denied_onboard.replay_api.handlers import ( + validate_csv_kind, validate_tlog_kind, validate_video_kind, ) @@ -107,6 +108,30 @@ def _valid_calibration_bytes() -> bytes: return b'{"focal_length": 1, "acquisition_method": "factory-sheet"}' +def _valid_csv_bytes() -> bytes: + """Minimal AZ-896-schema CSV with 2 data rows. + + Header tokens match + ``_docs/02_document/contracts/replay/csv_replay_format.md``. + Values are minimal-but-valid; the API-boundary validator only + checks the header, the per-row checks live in + ``csv_ground_truth.py`` and aren't exercised by the multipart + handler. + """ + header = ( + "timestamp(ms),Time," + "SCALED_IMU2.xacc,SCALED_IMU2.yacc,SCALED_IMU2.zacc," + "SCALED_IMU2.xgyro,SCALED_IMU2.ygyro,SCALED_IMU2.zgyro," + "GLOBAL_POSITION_INT.lat,GLOBAL_POSITION_INT.lon," + "GLOBAL_POSITION_INT.alt,GLOBAL_POSITION_INT.vx," + "GLOBAL_POSITION_INT.vy,GLOBAL_POSITION_INT.vz," + "GLOBAL_POSITION_INT.hdg" + ) + row1 = "0,0.0,21,-3,-984,52,32,-5,50.0809634,36.1115442,141290,0,0,0,35041" + row2 = "100,0.1,-68,-9,-995,58,-17,1,50.0809634,36.1115441,141360,0,0,0,35042" + return f"{header}\n{row1}\n{row2}\n".encode("utf-8") + + @pytest.fixture(autouse=True) def _disable_auth_by_default(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: monkeypatch.setenv("REPLAY_API_AUTH_REQUIRED", "false") @@ -630,6 +655,254 @@ def test_post_replay_rejects_misnamed_zip_as_video( assert response.json()["error_code"] == "unsupported_file_kind" +# --------------------------------------------------------------------- +# AZ-959 — CSV-path multipart + XOR validation + /static/example-csv + + +def test_validate_csv_kind_accepts_az896_header() -> None: + # Act / Assert — must not raise on the canonical header + validate_csv_kind(_valid_csv_bytes()[:512]) + + +def test_validate_csv_kind_rejects_header_missing_time_column() -> None: + # Arrange — drop the Time column from an otherwise-valid header + bogus = ( + b"timestamp(ms)," + b"SCALED_IMU2.xacc,SCALED_IMU2.yacc,SCALED_IMU2.zacc," + b"SCALED_IMU2.xgyro,SCALED_IMU2.ygyro,SCALED_IMU2.zgyro," + b"GLOBAL_POSITION_INT.lat,GLOBAL_POSITION_INT.lon\n" + b"0,0,0,0,0,0,0,0,0\n" + ) + + # Act / Assert + with pytest.raises(Exception) as exc: + validate_csv_kind(bogus) + assert "Time" in str(exc.value) + assert "csv_replay_format.md" in str(exc.value) + + +def test_post_replay_csv_path_returns_200_and_dispatches_imu_flag( + fake_runner: _FakeRunner, + make_app: Any, +) -> None: + # Arrange — AC-1 + app = make_app(fake_runner) + client = TestClient(app) + + # Act + response = client.post( + "/replay", + files={ + "csv": ("data_imu.csv", _valid_csv_bytes(), "text/csv"), + "video": ("derkachi.mp4", _valid_mp4_bytes(), "video/mp4"), + "calibration": ( + "k.json", + _valid_calibration_bytes(), + "application/json", + ), + }, + data={"pace": "asap"}, + ) + + # Assert + assert response.status_code == 200, response.text + body = response.json() + assert body["state"] == JobState.DONE.value + assert body["sync"] is True + # Runner saw the csv_path branch (tlog_path is None for csv jobs) + assert len(fake_runner.calls) == 1 + inputs = fake_runner.calls[0] + assert inputs.tlog_path is None + assert inputs.csv_path is not None + assert inputs.csv_path.is_file() + assert inputs.csv_path.read_bytes() == _valid_csv_bytes() + + +def test_post_replay_rejects_both_tlog_and_csv( + fake_runner: _FakeRunner, + make_app: Any, +) -> None: + # Arrange — AC-2 + client = TestClient(make_app(fake_runner)) + + # Act + response = client.post( + "/replay", + files={ + "tlog": ("d.tlog", _valid_tlog_bytes(), "application/octet-stream"), + "csv": ("d.csv", _valid_csv_bytes(), "text/csv"), + "video": ("d.mp4", _valid_mp4_bytes(), "video/mp4"), + "calibration": ("k.json", _valid_calibration_bytes(), "application/json"), + }, + ) + + # Assert + assert response.status_code == 400 + body = response.json() + assert body["error_code"] == "multipart_missing_field" + assert "exactly one" in body["message"].lower() + + +def test_post_replay_rejects_neither_tlog_nor_csv( + fake_runner: _FakeRunner, + make_app: Any, +) -> None: + # Arrange — AC-3 + client = TestClient(make_app(fake_runner)) + + # Act + response = client.post( + "/replay", + files={ + "video": ("d.mp4", _valid_mp4_bytes(), "video/mp4"), + "calibration": ("k.json", _valid_calibration_bytes(), "application/json"), + }, + ) + + # Assert + assert response.status_code == 400 + body = response.json() + assert body["error_code"] == "multipart_missing_field" + assert "exactly one" in body["message"].lower() + + +def test_post_replay_rejects_malformed_csv_at_api_boundary( + fake_runner: _FakeRunner, + make_app: Any, +) -> None: + # Arrange — AC-4: CSV header missing the Time column + bogus_csv = ( + b"timestamp(ms)," + b"SCALED_IMU2.xacc,SCALED_IMU2.yacc,SCALED_IMU2.zacc," + b"SCALED_IMU2.xgyro,SCALED_IMU2.ygyro,SCALED_IMU2.zgyro," + b"GLOBAL_POSITION_INT.lat,GLOBAL_POSITION_INT.lon\n" + b"0,0,0,0,0,0,0,0,0\n" + ) + client = TestClient(make_app(fake_runner)) + + # Act + response = client.post( + "/replay", + files={ + "csv": ("bad.csv", bogus_csv, "text/csv"), + "video": ("d.mp4", _valid_mp4_bytes(), "video/mp4"), + "calibration": ("k.json", _valid_calibration_bytes(), "application/json"), + }, + ) + + # Assert + assert response.status_code == 400 + body = response.json() + assert body["error_code"] == "unsupported_file_kind" + assert "csv_replay_format.md" in body["message"] + + +def test_static_example_csv_serves_canonical_doc_file( + fake_runner: _FakeRunner, + make_app: Any, +) -> None: + # Arrange — AC-5: endpoint serves the source-tree CSV bytes + from gps_denied_onboard.replay_api.app import _example_csv_path + + canonical_path = _example_csv_path() + if canonical_path is None: + pytest.skip( + "example CSV not on disk — running outside a source checkout" + ) + client = TestClient(make_app(fake_runner)) + + # Act + response = client.get("/static/example-csv") + + # Assert + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/csv") + assert "charset=utf-8" in response.headers["content-type"] + assert response.content == canonical_path.read_bytes() + + +def test_static_example_csv_returns_503_when_path_misconfigured( + monkeypatch: pytest.MonkeyPatch, + fake_runner: _FakeRunner, + make_app: Any, + tmp_path: Path, +) -> None: + # Arrange — env var points at a path that does not exist; + # we want to also stop the source-checkout fallback from finding + # the canonical file. Easiest is to point the env var at a + # bogus path: the helper short-circuits to that branch and + # returns None without falling back. + monkeypatch.setenv( + "REPLAY_API_EXAMPLE_CSV_PATH", str(tmp_path / "nonexistent.csv") + ) + client = TestClient(make_app(fake_runner)) + + # Act + response = client.get("/static/example-csv") + + # Assert + assert response.status_code == 503 + body = response.json() + assert body["error_code"] == "example_csv_unavailable" + + +def test_subprocess_runner_renders_report_for_csv_ground_truth( + tmp_path: Path, +) -> None: + # Arrange — AC-6: ground-truth dispatch through the SubprocessReplayRunner. + # We call _maybe_render_report directly so the subprocess invocation + # itself doesn't have to run (the input branch under test is the GT + # loader, not the gps-denied-replay binary). + from gps_denied_onboard.replay_api.app import ( + SubprocessReplayRunner, + _example_csv_path, + ) + + csv_path = _example_csv_path() + if csv_path is None: + pytest.skip( + "example CSV not on disk — running outside a source checkout" + ) + runner = SubprocessReplayRunner() + output_dir = tmp_path / "output" + output_dir.mkdir() + calibration_path = tmp_path / "calib.json" + calibration_path.write_text(_valid_calibration_bytes().decode()) + video_path = tmp_path / "video.mp4" + video_path.write_bytes(_valid_mp4_bytes()) + emissions_path = output_dir / "emissions.jsonl" + emissions_path.write_text( + json.dumps( + { + "frame_id": 0, + "position_wgs84": { + "lat_deg": 50.0809634, + "lon_deg": 36.1115442, + "alt_m": 141.290, + }, + "emitted_at": 0, + } + ) + + "\n" + ) + inputs = ReplayInputs( + csv_path=csv_path, + video_path=video_path, + calibration_path=calibration_path, + ) + + # Act + report_path = runner._maybe_render_report( + inputs, emissions_path, output_dir + ) + + # Assert + assert report_path is not None + assert report_path.is_file() + text = report_path.read_text() + assert "Verdict" in text or "verdict" in text.lower() + + # --------------------------------------------------------------------- # Helpers