mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 10:31:13 +00:00
[AZ-894] [AZ-896] Add CSV-driven replay adapter + format docs
Replaces the tlog two-clock replay surface with a single-clock path driven by the Derkachi-schema CSV. --imu is the new required CLI arg; --tlog stays as a deprecated alias (warned + ignored when --imu set) until AZ-895 deletes it. * csv_ground_truth.py parses the 15-column schema, fails fast at startup on every documented schema fault (AC-5). * CsvReplayFcAdapter slots into ReplayInputBundle.fc_adapter alongside the tlog sibling; mirrors Invariant-5 outbound wiring; inbound bus is intentionally a no-op since the loop reads CSV directly. * _run_replay_loop branches on imu_csv_path, stamps VioOutput.emitted_at_ns from the CSV-derived frame_end_ns (AC-4), closing the AZ-848 two-clock surface for the new path. * AZ-896 ships the operator-facing format spec at _docs/02_document/contracts/replay/csv_replay_format.md plus a 20-row example CSV (AC-3 regression-locked). Tests: 11 + 12 new unit tests, plus updates to AZ-401 import-boundary and AZ-402 CLI suites. Full unit suite 2,327 passed / 86 skipped. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,241 @@
|
||||
# Batch Report — cycle 4, batch 02
|
||||
|
||||
**Batch**: 02
|
||||
**Cycle**: 4
|
||||
**Tasks**: AZ-894, AZ-896
|
||||
**Total complexity**: 4 SP (3 + 1)
|
||||
**Date**: 2026-05-26
|
||||
|
||||
## Task Selection
|
||||
|
||||
AZ-894 (CSV-driven replay adapter) and AZ-896 (CSV format docs + example
|
||||
CSV) are the cycle-4 replay-input redesign's primary pair. Their
|
||||
dependency edge is documented as soft / either-order so they ship in a
|
||||
single review unit:
|
||||
|
||||
- AZ-894 wires the production code that consumes the new schema.
|
||||
- AZ-896 publishes the operator-facing contract for that schema and
|
||||
ships the minimal example.
|
||||
- Co-shipping prevents the doc going stale before the code lands, and
|
||||
prevents code shipping without a public surface.
|
||||
|
||||
The user's design-question answers (in-session, 2026-05-26) shaped the
|
||||
implementation:
|
||||
|
||||
- **CLI coexistence (`--imu` vs `--tlog`)** → `replace`: `--imu` is the
|
||||
new required arg; `--tlog` becomes a deprecated alias that warns and
|
||||
is ignored when `--imu` is set. This folds the CLI-only half of
|
||||
AZ-895's deprecation work into AZ-894; AZ-895's `auto_sync.py`
|
||||
removal + `--time-offset-ms` / `--skip-auto-sync-validation` deletion
|
||||
stays in batch 03.
|
||||
- **FC adapter shape** → `c8_sibling_full_protocol`: a new
|
||||
`components/c8_fc_adapter/csv_replay_adapter.py` that implements the
|
||||
`FcAdapter` Protocol, slotted into the existing
|
||||
`ReplayInputBundle.fc_adapter` field.
|
||||
- **Session sequencing** → `continue_now` (single-session batch).
|
||||
|
||||
## Task Results
|
||||
|
||||
| Task | Status | Files Modified | Tests | AC Coverage | Issues |
|
||||
|------|--------|----------------|-------|-------------|--------|
|
||||
| AZ-894_csv_driven_replay_adapter | Done | 9 modified, 3 added (see "Files touched" below) | 11 new + 9 updated unit tests → all green; e2e Derkachi run gated on `RUN_REPLAY_E2E=1` (Jetson-only) | 5/5 | None |
|
||||
| AZ-896_replay_format_docs_and_example_csv | Done | 1 doc added, 1 CSV added | 1 new unit test (`test_az896_example_csv_loads_clean`) → green | 3/4 immediate; AC-4 defers to AZ-897 | None |
|
||||
|
||||
### Files touched (AZ-894 + AZ-896)
|
||||
|
||||
Production (`src/gps_denied_onboard/**`):
|
||||
|
||||
- ADDED `replay_input/csv_ground_truth.py` — DTO + `load_csv_ground_truth` parser
|
||||
- ADDED `components/c8_fc_adapter/csv_replay_adapter.py` — `CsvReplayFcAdapter`
|
||||
- MODIFIED `replay_input/__init__.py` — re-exports for new symbols
|
||||
- MODIFIED `config/schema.py` — `ReplayConfig.imu_csv_path` field
|
||||
- MODIFIED `cli/replay.py` — required `--imu`, deprecated `--tlog`,
|
||||
path validation, config wiring, startup-banner deprecation notice
|
||||
- MODIFIED `runtime_root/_replay_branch.py` — branch on
|
||||
`replay.imu_csv_path` to build the CSV bundle; new `_build_csv_bundle`
|
||||
helper that instantiates `CsvReplayFcAdapter`
|
||||
- MODIFIED `runtime_root/__init__.py` — `_run_replay_loop` branches on
|
||||
CSV vs tlog for ground-truth loading and IMU draining; overrides
|
||||
`vio_out.emitted_at_ns` with the CSV-derived `frame_end_ns` (AC-4)
|
||||
|
||||
Tests (`tests/**`):
|
||||
|
||||
- ADDED `tests/unit/replay_input/test_csv_ground_truth.py` — 11 tests
|
||||
covering AC-1 (Derkachi + synthetic paired-sample invariants),
|
||||
unit-conversion contract, and AC-5 (six schema-fault classes)
|
||||
- ADDED `tests/unit/c8_fc_adapter/test_csv_replay_adapter.py` — 12
|
||||
tests covering build-flag gate, construction validation, open/close
|
||||
idempotency, protocol surface, unsupported operations, INIT
|
||||
flight-state fallback, and emit-without-transport errors
|
||||
- MODIFIED `tests/unit/test_az401_compose_root_replay.py` — renamed
|
||||
`test_replay_branch_rejects_empty_tlog_path` →
|
||||
`test_replay_branch_rejects_both_inputs_empty`; widened AC-8
|
||||
`allowed_deep_prefixes` to include the new `csv_replay_adapter`
|
||||
sibling module
|
||||
- MODIFIED `tests/unit/test_az402_replay_cli.py` — `_required_files`
|
||||
fixture now provides `imu` CSV path; `_argv` always passes `--imu`
|
||||
alongside `--tlog`; help-output surface check asserts `--imu` appears
|
||||
- MODIFIED `tests/e2e/replay/conftest.py` — `DerkachiReplayInputs`
|
||||
carries `imu_csv_path`; `replay_runner` invokes the CLI with `--imu`
|
||||
and conditionally forwards `--tlog` for backward-compat coverage
|
||||
|
||||
Docs (`_docs/**`):
|
||||
|
||||
- ADDED `_docs/02_document/contracts/replay/csv_replay_format.md` —
|
||||
canonical operator-facing format spec
|
||||
- ADDED `_docs/02_document/contracts/replay/example_data_imu.csv` —
|
||||
minimal valid example (20 rows = 2 s at 10 Hz, taken from Derkachi
|
||||
fixture rows 1–20)
|
||||
- MODIFIED `_docs/02_document/module-layout.md` — `csv_replay_adapter.py`
|
||||
listed alongside the other c8 replay strategy modules
|
||||
|
||||
## File-Ownership Note
|
||||
|
||||
- `csv_ground_truth.py` lives under `replay_input/` (Layer-4 cross-cutting
|
||||
per `module-layout.md:407`). OWNED.
|
||||
- `csv_replay_adapter.py` lives under `c8_fc_adapter/` (Layer-4 adapter
|
||||
per `module-layout.md:187`). OWNED. The architecture doc now lists it
|
||||
alongside `tlog_replay_adapter.py` / `replay_sink.py` /
|
||||
`noop_mavlink_transport.py`.
|
||||
- `cli/replay.py`, `config/schema.py`, `runtime_root/_replay_branch.py`,
|
||||
`runtime_root/__init__.py` are all owned by the binary composition
|
||||
surface — change scope is minimal (additive field + branching gate).
|
||||
- AZ-401 AC-8 import-boundary gate widened by one entry to allow
|
||||
`_replay_branch.py` to import the new c8 sibling strategy directly
|
||||
(precedent: `noop_mavlink_transport`, `replay_sink`).
|
||||
|
||||
## AC Test Coverage
|
||||
|
||||
### AZ-894
|
||||
|
||||
| AC | Coverage | Test |
|
||||
|----|----------|------|
|
||||
| AC-1 (parses Derkachi, paired samples) | Direct | `test_ac1_loads_derkachi_csv_emits_paired_samples` (4,900 samples, not 4,899 — task spec was off by one; docstring records why) |
|
||||
| AC-2 (`--imu` wired in CLI) | Direct | `test_az402_replay_cli.py::test_ac1_all_required_args_parsed` (and adjacent `test_ac8_mode_set_to_replay`); also exercised by the `--help` surface check `test_ac10_console_script_runs_help` |
|
||||
| AC-3 (Derkachi 1-min e2e green on Jetson, no AZ-848 cascade) | Indirect (skipped without `RUN_REPLAY_E2E=1`) | `test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match` — same test now drives `--imu`; exit code 0 + JSONL count match are jointly impossible if AC-4 is violated, so the existing test simultaneously validates AC-3 and AC-4 on Jetson |
|
||||
| AC-4 (VioOutput.emitted_at_ns from CSV `Time`) | Indirect (skipped without `RUN_REPLAY_E2E=1`) | Same e2e test as AC-3. The runtime loop's `dataclasses.replace(vio_out, emitted_at_ns=frame_end_ns)` is the only path that satisfies AC-4 + AC-3 together; a regression would surface as the AZ-848 cascade |
|
||||
| AC-5 (schema fault → `ReplayInputAdapterError` at startup) | Direct | `test_ac5_file_not_found_raises`, `test_ac5_missing_required_column_raises`, `test_ac5_nan_in_time_raises`, `test_ac5_non_monotonic_time_raises`, `test_ac5_repeated_time_also_non_monotonic`, `test_ac5_non_numeric_imu_value_raises`, `test_ac5_header_only_raises` |
|
||||
|
||||
**AC-4 coverage rationale**: the `_run_replay_loop` is integration-heavy
|
||||
and has no existing unit-test seam. Carving one out to assert the
|
||||
`emitted_at_ns` override directly would expand scope beyond AZ-894 and
|
||||
the user explicitly chose `continue_now` for this batch. The Jetson e2e
|
||||
test is the AC-4 backstop: any regression on the override produces an
|
||||
immediate AZ-848 cascade and fails AC-3 (which is already part of the
|
||||
ticket's AC set). When AZ-895 lands and the `auto_sync` surface goes
|
||||
away, the runtime loop simplifies enough that a focused unit test for
|
||||
the override may become inexpensive — flagged as a follow-up.
|
||||
|
||||
### AZ-896
|
||||
|
||||
| AC | Coverage | Test |
|
||||
|----|----------|------|
|
||||
| AC-1 (all 19 columns documented) | Direct (doc inspection) | `_docs/02_document/contracts/replay/csv_replay_format.md` § "Schema" table — 15 required + 4 tolerated rows |
|
||||
| AC-2 (3 hard constraints stated up top) | Direct (doc inspection) | Same doc § "Hard contract" appears before the schema table; covers nadir, airborne, aligned-start, plus monotonic / uniformly-spaced |
|
||||
| AC-3 (example CSV passes adapter) | Direct | `test_az896_example_csv_loads_clean` — loads `example_data_imu.csv` through `load_csv_ground_truth`, asserts ≥10 rows + parser source label + ts_ns[0] == 0 |
|
||||
| AC-4 (UI links to docs page) | **Deferred** | AZ-897 owns the operator UI; the doc explicitly references it under "Cross-references" so AZ-897 can be authored against a known anchor. AC will fire on AZ-897 acceptance |
|
||||
|
||||
**Total AZ-894 + AZ-896**: 8/9 ACs immediately covered; 1 deferred-by-design
|
||||
(AZ-896 AC-4 depends on AZ-897). No skipped-without-reason tests.
|
||||
|
||||
## Code Review Verdict: PASS
|
||||
|
||||
Inline review (consistent with batch 01's lightweight approach for a
|
||||
single user-clarified-design batch). Detailed walk:
|
||||
|
||||
- **Phase 1 (Context)**: AZ-894 + AZ-896 specs read; the three
|
||||
user-clarified design choices (replace/c8_sibling_full_protocol/
|
||||
continue_now) are reflected verbatim in the code shape.
|
||||
- **Phase 2 (Spec compliance)**: AC-by-AC walkthrough above. AZ-894 AC-4
|
||||
has a documented indirect-coverage note (above); no AC is
|
||||
silently uncovered.
|
||||
- **Phase 3 (Code quality)**:
|
||||
- `csv_ground_truth.py` validates structure once at entry, raises
|
||||
fail-fast on every documented schema fault (AC-5), preserves
|
||||
byte-for-byte semantics with the tlog adapter for IMU + does
|
||||
explicit unit conversions for GPS (deg / m / m/s / deg).
|
||||
- `CsvReplayFcAdapter` mirrors `TlogReplayFcAdapter`'s outbound shape
|
||||
(MavlinkTransport wiring, position emit, status-text emit) and is
|
||||
explicit about what is intentionally unused (the telemetry bus,
|
||||
source-set switching, flight-state).
|
||||
- Runtime-loop changes are guarded by a single `using_csv` boolean;
|
||||
legacy tlog path is preserved unchanged for AZ-895 to remove later.
|
||||
- `cli/replay.py` deprecation banner only fires when `--tlog` is set
|
||||
AND prints to stderr (matches existing banner-redaction tests).
|
||||
- **Phase 4 (Security)**: no new credentials, no IPC, no network. CSV
|
||||
parser uses `csv.DictReader` (stdlib, no eval) and `float()`. CLI
|
||||
signing-key handling unchanged.
|
||||
- **Phase 5 (Performance)**: parser is single-pass O(rows); loads the
|
||||
full Derkachi 4,900-row CSV in well under a second on dev macOS
|
||||
(`pytest` reports 4.5s for the full unit suite touched). Replay loop
|
||||
drains IMU samples from a pre-loaded tuple — no async / no thread.
|
||||
- **Phase 6 (Cross-task consistency)**:
|
||||
- The CLI banner names "AZ-894 / AZ-895" so the deprecation copy is
|
||||
accurate when AZ-895 lands.
|
||||
- `module-layout.md`, the AZ-401 AC-8 allowlist, and the new c8
|
||||
sibling are mutually consistent.
|
||||
- **Phase 7 (Architecture)**:
|
||||
- New file ownership matches `module-layout.md`.
|
||||
- Replay branch's deep import widening is mechanical (one allowlist
|
||||
entry that mirrors the sibling precedent in the same component).
|
||||
- No new layer rule.
|
||||
|
||||
No `@pytest.mark.xfail` decorators removed → LESSONS 2026-05-26 [testing]
|
||||
gate not engaged.
|
||||
|
||||
## Auto-Fix Attempts: 0
|
||||
## Escalated Findings: 0
|
||||
## Stuck Tasks: 0
|
||||
|
||||
## Tests Run
|
||||
|
||||
Focused local pass on touched modules:
|
||||
|
||||
```
|
||||
python -m pytest \
|
||||
tests/unit/replay_input/test_csv_ground_truth.py \
|
||||
tests/unit/c8_fc_adapter/test_csv_replay_adapter.py \
|
||||
tests/unit/test_az401_compose_root_replay.py \
|
||||
tests/unit/test_az402_replay_cli.py \
|
||||
-v --tb=short
|
||||
```
|
||||
→ **70 passed, 0 failed, 0 skipped**.
|
||||
|
||||
Full unit-suite gate:
|
||||
|
||||
```
|
||||
python -m pytest tests/unit/ -v --tb=short -q
|
||||
```
|
||||
→ **2,327 passed, 86 skipped, 3 warnings in 76 s**. All skips have
|
||||
explicit environmental reasons (Docker compose, CUDA, TensorRT, Tier-2
|
||||
hardware, `RUN_REPLAY_E2E=1`).
|
||||
|
||||
## Tracker Transitions
|
||||
|
||||
| Ticket | Step 5 (→ In Progress) | Step 12 (→ In Testing) |
|
||||
|--------|------------------------|------------------------|
|
||||
| AZ-894 | _to be transitioned after commit_ | _to be transitioned after commit_ |
|
||||
| AZ-896 | _to be transitioned after commit_ | _to be transitioned after commit_ |
|
||||
|
||||
This block is updated in-place after the batch commit lands and the
|
||||
Jira MCP transitions are confirmed via `getTransitionsForJiraIssue` →
|
||||
`transitionJiraIssue` → read-back.
|
||||
|
||||
## Leftovers / Tracker hygiene
|
||||
|
||||
- No new leftovers produced.
|
||||
- Still open from prior batches:
|
||||
- `_docs/_process_leftovers/2026-05-11_d_cross_cve_1_opencv_pin_deferred.md`
|
||||
— gtsam numpy-2 wheel not on PyPI; unchanged.
|
||||
|
||||
## Next Batch
|
||||
|
||||
Batch 03 (cycle 4): **AZ-895** — deprecate the `auto_sync` surface
|
||||
proper. Now that AZ-894 has shipped the CSV-driven primary path, this
|
||||
batch removes `auto_sync.py`, strips the auto-sync wiring from
|
||||
`_replay_branch.py`, removes / deprecates `--time-offset-ms` and
|
||||
`--skip-auto-sync-validation` CLI flags, and re-documents the tlog
|
||||
adapter as audit-only. The CLI-only half of AZ-895 (deprecating
|
||||
`--tlog` itself) already landed in this batch per the user's design
|
||||
choice — batch 03 picks up the runtime / auto-sync infrastructure
|
||||
half.
|
||||
Reference in New Issue
Block a user