[AZ-894] [AZ-896] Add CSV-driven replay adapter + format docs

Replaces the tlog two-clock replay surface with a single-clock path
driven by the Derkachi-schema CSV. --imu is the new required CLI arg;
--tlog stays as a deprecated alias (warned + ignored when --imu set)
until AZ-895 deletes it.

* csv_ground_truth.py parses the 15-column schema, fails fast at
  startup on every documented schema fault (AC-5).
* CsvReplayFcAdapter slots into ReplayInputBundle.fc_adapter alongside
  the tlog sibling; mirrors Invariant-5 outbound wiring; inbound bus is
  intentionally a no-op since the loop reads CSV directly.
* _run_replay_loop branches on imu_csv_path, stamps
  VioOutput.emitted_at_ns from the CSV-derived frame_end_ns (AC-4),
  closing the AZ-848 two-clock surface for the new path.
* AZ-896 ships the operator-facing format spec at
  _docs/02_document/contracts/replay/csv_replay_format.md plus a
  20-row example CSV (AC-3 regression-locked).

Tests: 11 + 12 new unit tests, plus updates to AZ-401 import-boundary
and AZ-402 CLI suites. Full unit suite 2,327 passed / 86 skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-26 18:40:29 +03:00
parent 3020779404
commit 6be207cef3
19 changed files with 1833 additions and 93 deletions
@@ -0,0 +1,149 @@
# Replay-input CSV format (AZ-896)
**Status**: canonical operator-facing spec for the `--imu` argument of
`gps-denied-replay` (AZ-894).
**Audience**: operators preparing a (video, CSV) replay pair, plus engineers
implementing alternative replay backends.
**Companion artifacts**:
- `_docs/02_document/contracts/replay/example_data_imu.csv` — minimal valid
example (20 rows = 2 s at 10 Hz).
- `_docs/00_problem/input_data/flight_derkachi/data_imu.csv` — full Derkachi
fixture (4,900 rows = 489.9 s at 10 Hz).
- Parser implementation:
`src/gps_denied_onboard/replay_input/csv_ground_truth.py`.
## Hard contract (read before generating a file)
The replay pipeline trusts the CSV blindly inside the loop. Violations of any
of the following will produce silently wrong outputs (the parser only catches
schema-level faults, not semantic ones), so the operator owns these
invariants:
1. **Nadir camera.** The companion `.mp4` must be a nadir (straight-down)
recording. The C1 VIO and C2 VPR stages assume nadir framing; oblique
imagery breaks the satellite-anchor and VIO scale recovery.
2. **Airborne at row 0.** The UAV must already be airborne at the first CSV
row / first video frame. The replay pipeline does not implement a
take-off detector — feeding a ground-roll segment yields garbage IMU
integration.
3. **Aligned start.** Row 0's `Time = 0.0` must correspond to the first
video frame. The CLI does not perform sub-frame alignment; offset the
CSV/clip pair offline before invoking `gps-denied-replay`.
4. **Monotonic, uniformly-spaced `Time`.** Rows must be strictly increasing
on `Time` and uniformly spaced (the Derkachi fixture is 10 Hz). The
parser enforces monotonicity (AC-5); uniform spacing is the operator's
responsibility — non-uniform spacing skews the ESKF prediction step
without raising an error.
## Schema
The CSV must be header-first, comma-separated, UTF-8 encoded. Column order
does not matter — the parser uses `csv.DictReader` and looks up by name —
but the column **names** must match exactly (case-sensitive).
15 columns are required; up to 4 additional columns (mag fields,
`relative_alt`) are tolerated and ignored.
### Required columns
| # | Column | Unit | Type | Notes |
|---|--------|------|------|-------|
| 1 | `timestamp(ms)` | ms | float | Pixhawk wall clock at sample capture. **Ignored by the replay pipeline** — kept only for trace-back to the original tlog. |
| 2 | `Time` | s | float | **Canonical replay clock.** Must start at `0.0`, increase monotonically, and be uniformly spaced. The replay loop uses this column for every timestamp it emits. |
| 3 | `SCALED_IMU2.xacc` | mg | float | Body-frame X accelerometer, MAVLink `SCALED_IMU2` raw scaling. Forwarded unchanged into `ImuSample.accel_xyz[0]`. |
| 4 | `SCALED_IMU2.yacc` | mg | float | Body-frame Y accelerometer. |
| 5 | `SCALED_IMU2.zacc` | mg | float | Body-frame Z accelerometer. |
| 6 | `SCALED_IMU2.xgyro` | mrad/s | float | Body-frame X gyro, MAVLink `SCALED_IMU2` raw scaling. Forwarded unchanged into `ImuSample.gyro_xyz[0]`. |
| 7 | `SCALED_IMU2.ygyro` | mrad/s | float | Body-frame Y gyro. |
| 8 | `SCALED_IMU2.zgyro` | mrad/s | float | Body-frame Z gyro. |
| 9 | `GLOBAL_POSITION_INT.lat` | degrees | float | WGS84 latitude. **Already in decimal degrees** (Derkachi dump convention — pre-divided by 1e7 from MAVLink's int representation). |
| 10 | `GLOBAL_POSITION_INT.lon` | degrees | float | WGS84 longitude (same convention as `lat`). |
| 11 | `GLOBAL_POSITION_INT.alt` | mm | float | MSL altitude. Parser divides by 1000 to emit metres. |
| 12 | `GLOBAL_POSITION_INT.vx` | cm/s | float | NED north velocity. Parser divides by 100 to emit m/s. |
| 13 | `GLOBAL_POSITION_INT.vy` | cm/s | float | NED east velocity. |
| 14 | `GLOBAL_POSITION_INT.vz` | cm/s | float | NED down velocity. |
| 15 | `GLOBAL_POSITION_INT.hdg` | cdeg | float | Heading, 035999. Parser divides by 100 to emit degrees. |
### Tolerated extra columns
The following may be present but are not consumed:
| Column | Reason kept | Reason unused |
|--------|-------------|---------------|
| `SCALED_IMU2.xmag`, `.ymag`, `.zmag` | Symmetric with the accel/gyro triples in the Derkachi dump | The current ESKF does not integrate magnetometer; AZ-848 follow-up may add it |
| `GLOBAL_POSITION_INT.relative_alt` | Present in the MAVLink dump | The replay pipeline uses MSL `alt` only |
Additional columns beyond these are ignored without warning. Missing
required columns cause the load to raise
`ReplayInputAdapterError` before the replay loop starts (AC-5).
## Schema-level errors the parser catches
The parser raises `ReplayInputAdapterError` (CLI exit code 1) for any of:
- File does not exist or is not a regular file.
- File is empty (no header row).
- File has a header but no data rows.
- Any required column from the table above is missing from the header.
- The `Time` column at any row contains a non-numeric / NaN / Inf value.
- The `Time` column is non-monotonic (`Time[i] <= Time[i-1]`).
- Any required IMU or GPS column at any row contains a non-numeric / NaN /
Inf value.
The error message includes the row number (1-based, where row 1 is the
header — so the first data row is row 2). Operators should treat the first
parse failure as authoritative and fix the source CSV; the parser does not
continue after the first invalid row.
## Operator workflow
```bash
gps-denied-replay \
--video ./flight.mp4 \
--imu ./data_imu.csv \
--output ./estimator_output.jsonl \
--camera-calibration ./calib.json \
--config ./config.yaml \
--mavlink-signing-key ./signing_key.bin
```
`--tlog` is accepted as a deprecated alias and will be removed by AZ-895.
When both `--imu` and `--tlog` are supplied, `--imu` wins and a deprecation
warning is printed to stderr.
## Deriving a new CSV from an ArduPilot tlog
The Derkachi fixture was produced with `pymavlink`'s `mavlogdump.py`. The
short version:
```bash
mavlogdump.py --format csv \
--types SCALED_IMU2,GLOBAL_POSITION_INT \
./flight.tlog > ./raw_dump.csv
```
Then post-process to:
1. Rename / merge the per-message timestamp into a single `Time` column
relative to the first row.
2. Drop pre-takeoff rows (the UAV must be airborne at row 0 — see the hard
contract above).
3. Pre-divide `lat` / `lon` from the MAVLink `int * 1e7` representation
into decimal degrees.
4. Re-sample to a uniform 10 Hz cadence if the tlog dump produced
non-uniform spacing.
A reference post-processor script is **not** shipped — operators
historically write a one-off Python or Pandas pipeline per source aircraft.
## Cross-references
- AZ-894 — the CLI + adapter that consumes this format.
- AZ-895 — deletes the legacy `--tlog` argument once all callers migrate.
- AZ-897 — operator replay UI; links to this page and serves
`example_data_imu.csv`.
- `_docs/02_document/contracts/replay/replay_protocol.md` — the broader
replay orchestration contract.
- `_docs/00_problem/input_data/flight_derkachi/README.md` — fixture
provenance and license caveats.
@@ -0,0 +1,21 @@
timestamp(ms),Time,SCALED_IMU2.xacc,SCALED_IMU2.yacc,SCALED_IMU2.zacc,SCALED_IMU2.xgyro,SCALED_IMU2.ygyro,SCALED_IMU2.zgyro,SCALED_IMU2.xmag,SCALED_IMU2.ymag,SCALED_IMU2.zmag,GLOBAL_POSITION_INT.lat,GLOBAL_POSITION_INT.lon,GLOBAL_POSITION_INT.alt,GLOBAL_POSITION_INT.relative_alt,GLOBAL_POSITION_INT.vx,GLOBAL_POSITION_INT.vy,GLOBAL_POSITION_INT.vz,GLOBAL_POSITION_INT.hdg
4551116.348,0,21,-3,-984,52,32,-5,312,-1048,442,50.0809634,36.1115442,141290,23.182,-4,-6,-88,35041
4551216.348,0.1,-68,-9,-995,58,-17,1,309,-1016,441,50.0809634,36.1115441,141360,23.251,-5,-2,-89,35042
4551316.348,0.2,9,108,-988,69,-65,13,308,-964,436,50.0809633,36.1115441,141410,23.303,-1,-2,-86,35048
4551416.348,0.3,-20,27,-977,55,10,26,310,-988,438,50.0809633,36.1115441,141450,23.348,-5,-6,-84,35057
4551516.348,0.4,-40,40,-1026,0,65,10,306,-1076,440,50.0809633,36.111544,141510,23.402,-2,-2,-86,35065
4551616.348,0.5,30,126,-1050,-1,75,14,321,-1146,442,50.0809633,36.111544,141570,23.464,0,0,-88,35074
4551716.348,0.6,-64,67,-1031,-31,-6,21,314,-1066,438,50.0809632,36.1115439,141640,23.53,-5,1,-90,35080
4551816.348,0.7,-22,112,-1027,-61,-88,-5,302,-951,436,50.0809632,36.1115439,141710,23.601,-2,3,-90,35082
4551916.348,0.8,-123,-16,-998,-55,-104,-12,301,-942,440,50.0809631,36.1115439,141770,23.669,-10,0,-91,35079
4552016.348,0.9,-64,-13,-1003,13,-70,-30,301,-936,442,50.080963,36.1115439,141860,23.755,-2,0,-90,35073
4552116.348,1,-22,39,-995,73,20,-18,314,-988,436,50.080963,36.1115439,141930,23.826,-2,-2,-88,35070
4552216.348,1.1,-49,-69,-984,2,29,1,317,-992,433,50.080963,36.1115438,142010,23.9,-6,-2,-88,35068
4552316.348,1.2,-16,98,-991,-59,-28,-11,310,-970,435,50.080963,36.1115438,142080,23.975,-1,6,-86,35063
4552416.348,1.3,-6,169,-998,-29,2,-2,310,-983,435,50.0809629,36.1115438,142150,24.042,-3,5,-83,35059
4552516.348,1.4,-31,53,-1003,2,13,-10,317,-1042,438,50.0809629,36.1115438,142210,24.102,-3,3,-83,35051
4552616.348,1.5,-47,21,-1023,13,13,-14,320,-1069,439,50.0809629,36.1115438,142270,24.166,2,2,-83,35047
4552716.348,1.6,-30,-59,-1020,-18,24,0,315,-1083,438,50.0809629,36.1115439,142340,24.236,-5,1,-86,35049
4552816.348,1.7,-103,23,-1058,-59,26,-7,314,-1113,442,50.0809629,36.1115439,142430,24.321,-4,4,-90,35050
4552916.348,1.8,-17,51,-1037,-9,80,11,317,-1087,444,50.0809629,36.1115439,142510,24.404,-5,0,-93,35049
4553016.348,1.9,-87,72,-1022,-10,-45,0,309,-1004,439,50.0809628,36.111544,142600,24.494,-6,2,-97,35046
1 timestamp(ms) Time SCALED_IMU2.xacc SCALED_IMU2.yacc SCALED_IMU2.zacc SCALED_IMU2.xgyro SCALED_IMU2.ygyro SCALED_IMU2.zgyro SCALED_IMU2.xmag SCALED_IMU2.ymag SCALED_IMU2.zmag GLOBAL_POSITION_INT.lat GLOBAL_POSITION_INT.lon GLOBAL_POSITION_INT.alt GLOBAL_POSITION_INT.relative_alt GLOBAL_POSITION_INT.vx GLOBAL_POSITION_INT.vy GLOBAL_POSITION_INT.vz GLOBAL_POSITION_INT.hdg
2 4551116.348 0 21 -3 -984 52 32 -5 312 -1048 442 50.0809634 36.1115442 141290 23.182 -4 -6 -88 35041
3 4551216.348 0.1 -68 -9 -995 58 -17 1 309 -1016 441 50.0809634 36.1115441 141360 23.251 -5 -2 -89 35042
4 4551316.348 0.2 9 108 -988 69 -65 13 308 -964 436 50.0809633 36.1115441 141410 23.303 -1 -2 -86 35048
5 4551416.348 0.3 -20 27 -977 55 10 26 310 -988 438 50.0809633 36.1115441 141450 23.348 -5 -6 -84 35057
6 4551516.348 0.4 -40 40 -1026 0 65 10 306 -1076 440 50.0809633 36.111544 141510 23.402 -2 -2 -86 35065
7 4551616.348 0.5 30 126 -1050 -1 75 14 321 -1146 442 50.0809633 36.111544 141570 23.464 0 0 -88 35074
8 4551716.348 0.6 -64 67 -1031 -31 -6 21 314 -1066 438 50.0809632 36.1115439 141640 23.53 -5 1 -90 35080
9 4551816.348 0.7 -22 112 -1027 -61 -88 -5 302 -951 436 50.0809632 36.1115439 141710 23.601 -2 3 -90 35082
10 4551916.348 0.8 -123 -16 -998 -55 -104 -12 301 -942 440 50.0809631 36.1115439 141770 23.669 -10 0 -91 35079
11 4552016.348 0.9 -64 -13 -1003 13 -70 -30 301 -936 442 50.080963 36.1115439 141860 23.755 -2 0 -90 35073
12 4552116.348 1 -22 39 -995 73 20 -18 314 -988 436 50.080963 36.1115439 141930 23.826 -2 -2 -88 35070
13 4552216.348 1.1 -49 -69 -984 2 29 1 317 -992 433 50.080963 36.1115438 142010 23.9 -6 -2 -88 35068
14 4552316.348 1.2 -16 98 -991 -59 -28 -11 310 -970 435 50.080963 36.1115438 142080 23.975 -1 6 -86 35063
15 4552416.348 1.3 -6 169 -998 -29 2 -2 310 -983 435 50.0809629 36.1115438 142150 24.042 -3 5 -83 35059
16 4552516.348 1.4 -31 53 -1003 2 13 -10 317 -1042 438 50.0809629 36.1115438 142210 24.102 -3 3 -83 35051
17 4552616.348 1.5 -47 21 -1023 13 13 -14 320 -1069 439 50.0809629 36.1115438 142270 24.166 2 2 -83 35047
18 4552716.348 1.6 -30 -59 -1020 -18 24 0 315 -1083 438 50.0809629 36.1115439 142340 24.236 -5 1 -86 35049
19 4552816.348 1.7 -103 23 -1058 -59 26 -7 314 -1113 442 50.0809629 36.1115439 142430 24.321 -4 4 -90 35050
20 4552916.348 1.8 -17 51 -1037 -9 80 11 317 -1087 444 50.0809629 36.1115439 142510 24.404 -5 0 -93 35049
21 4553016.348 1.9 -87 72 -1022 -10 -45 0 309 -1004 439 50.0809628 36.111544 142600 24.494 -6 2 -97 35046
+1
View File
@@ -197,6 +197,7 @@ Bootstrap reference: `_docs/02_tasks/todo/AZ-263_initial_structure.md`. Architec
- `msp2_inav_adapter.py` (iNav via MSP2)
- `mavlink_gcs_adapter.py` (12 Hz downsampled summary to QGroundControl)
- `tlog_replay_adapter.py` (replay-mode `FcAdapter`; gated `BUILD_TLOG_REPLAY_ADAPTER`; ON in airborne per ADR-011; AZ-265)
- `csv_replay_adapter.py` (`CsvReplayFcAdapter` — outbound shim for the AZ-894 CSV-driven replay path; same `FcAdapter` Protocol parity as `tlog_replay_adapter`; gated `BUILD_TLOG_REPLAY_ADAPTER` for the airborne replay binary; AZ-894)
- `replay_sink.py` (`ReplaySink` interface + `JsonlReplaySink` impl; gated `BUILD_REPLAY_SINK_JSONL`; ON in airborne per ADR-011; AZ-265)
- `noop_mavlink_transport.py` (`NoopMavlinkTransport` for replay-mode outbound bytes; gated `BUILD_REPLAY_SINK_JSONL`; ON in airborne; AZ-265 / AZ-400)
- `serial_mavlink_transport.py` (`SerialMavlinkTransport` retrofit of the existing live-mode UART transport; AZ-265 / AZ-400 no-op restructure)
@@ -0,0 +1,241 @@
# Batch Report — cycle 4, batch 02
**Batch**: 02
**Cycle**: 4
**Tasks**: AZ-894, AZ-896
**Total complexity**: 4 SP (3 + 1)
**Date**: 2026-05-26
## Task Selection
AZ-894 (CSV-driven replay adapter) and AZ-896 (CSV format docs + example
CSV) are the cycle-4 replay-input redesign's primary pair. Their
dependency edge is documented as soft / either-order so they ship in a
single review unit:
- AZ-894 wires the production code that consumes the new schema.
- AZ-896 publishes the operator-facing contract for that schema and
ships the minimal example.
- Co-shipping prevents the doc going stale before the code lands, and
prevents code shipping without a public surface.
The user's design-question answers (in-session, 2026-05-26) shaped the
implementation:
- **CLI coexistence (`--imu` vs `--tlog`)** → `replace`: `--imu` is the
new required arg; `--tlog` becomes a deprecated alias that warns and
is ignored when `--imu` is set. This folds the CLI-only half of
AZ-895's deprecation work into AZ-894; AZ-895's `auto_sync.py`
removal + `--time-offset-ms` / `--skip-auto-sync-validation` deletion
stays in batch 03.
- **FC adapter shape** → `c8_sibling_full_protocol`: a new
`components/c8_fc_adapter/csv_replay_adapter.py` that implements the
`FcAdapter` Protocol, slotted into the existing
`ReplayInputBundle.fc_adapter` field.
- **Session sequencing** → `continue_now` (single-session batch).
## Task Results
| Task | Status | Files Modified | Tests | AC Coverage | Issues |
|------|--------|----------------|-------|-------------|--------|
| AZ-894_csv_driven_replay_adapter | Done | 9 modified, 3 added (see "Files touched" below) | 11 new + 9 updated unit tests → all green; e2e Derkachi run gated on `RUN_REPLAY_E2E=1` (Jetson-only) | 5/5 | None |
| AZ-896_replay_format_docs_and_example_csv | Done | 1 doc added, 1 CSV added | 1 new unit test (`test_az896_example_csv_loads_clean`) → green | 3/4 immediate; AC-4 defers to AZ-897 | None |
### Files touched (AZ-894 + AZ-896)
Production (`src/gps_denied_onboard/**`):
- ADDED `replay_input/csv_ground_truth.py` — DTO + `load_csv_ground_truth` parser
- ADDED `components/c8_fc_adapter/csv_replay_adapter.py``CsvReplayFcAdapter`
- MODIFIED `replay_input/__init__.py` — re-exports for new symbols
- MODIFIED `config/schema.py``ReplayConfig.imu_csv_path` field
- MODIFIED `cli/replay.py` — required `--imu`, deprecated `--tlog`,
path validation, config wiring, startup-banner deprecation notice
- MODIFIED `runtime_root/_replay_branch.py` — branch on
`replay.imu_csv_path` to build the CSV bundle; new `_build_csv_bundle`
helper that instantiates `CsvReplayFcAdapter`
- MODIFIED `runtime_root/__init__.py``_run_replay_loop` branches on
CSV vs tlog for ground-truth loading and IMU draining; overrides
`vio_out.emitted_at_ns` with the CSV-derived `frame_end_ns` (AC-4)
Tests (`tests/**`):
- ADDED `tests/unit/replay_input/test_csv_ground_truth.py` — 11 tests
covering AC-1 (Derkachi + synthetic paired-sample invariants),
unit-conversion contract, and AC-5 (six schema-fault classes)
- ADDED `tests/unit/c8_fc_adapter/test_csv_replay_adapter.py` — 12
tests covering build-flag gate, construction validation, open/close
idempotency, protocol surface, unsupported operations, INIT
flight-state fallback, and emit-without-transport errors
- MODIFIED `tests/unit/test_az401_compose_root_replay.py` — renamed
`test_replay_branch_rejects_empty_tlog_path`
`test_replay_branch_rejects_both_inputs_empty`; widened AC-8
`allowed_deep_prefixes` to include the new `csv_replay_adapter`
sibling module
- MODIFIED `tests/unit/test_az402_replay_cli.py``_required_files`
fixture now provides `imu` CSV path; `_argv` always passes `--imu`
alongside `--tlog`; help-output surface check asserts `--imu` appears
- MODIFIED `tests/e2e/replay/conftest.py``DerkachiReplayInputs`
carries `imu_csv_path`; `replay_runner` invokes the CLI with `--imu`
and conditionally forwards `--tlog` for backward-compat coverage
Docs (`_docs/**`):
- ADDED `_docs/02_document/contracts/replay/csv_replay_format.md`
canonical operator-facing format spec
- ADDED `_docs/02_document/contracts/replay/example_data_imu.csv`
minimal valid example (20 rows = 2 s at 10 Hz, taken from Derkachi
fixture rows 120)
- MODIFIED `_docs/02_document/module-layout.md``csv_replay_adapter.py`
listed alongside the other c8 replay strategy modules
## File-Ownership Note
- `csv_ground_truth.py` lives under `replay_input/` (Layer-4 cross-cutting
per `module-layout.md:407`). OWNED.
- `csv_replay_adapter.py` lives under `c8_fc_adapter/` (Layer-4 adapter
per `module-layout.md:187`). OWNED. The architecture doc now lists it
alongside `tlog_replay_adapter.py` / `replay_sink.py` /
`noop_mavlink_transport.py`.
- `cli/replay.py`, `config/schema.py`, `runtime_root/_replay_branch.py`,
`runtime_root/__init__.py` are all owned by the binary composition
surface — change scope is minimal (additive field + branching gate).
- AZ-401 AC-8 import-boundary gate widened by one entry to allow
`_replay_branch.py` to import the new c8 sibling strategy directly
(precedent: `noop_mavlink_transport`, `replay_sink`).
## AC Test Coverage
### AZ-894
| AC | Coverage | Test |
|----|----------|------|
| AC-1 (parses Derkachi, paired samples) | Direct | `test_ac1_loads_derkachi_csv_emits_paired_samples` (4,900 samples, not 4,899 — task spec was off by one; docstring records why) |
| AC-2 (`--imu` wired in CLI) | Direct | `test_az402_replay_cli.py::test_ac1_all_required_args_parsed` (and adjacent `test_ac8_mode_set_to_replay`); also exercised by the `--help` surface check `test_ac10_console_script_runs_help` |
| AC-3 (Derkachi 1-min e2e green on Jetson, no AZ-848 cascade) | Indirect (skipped without `RUN_REPLAY_E2E=1`) | `test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match` — same test now drives `--imu`; exit code 0 + JSONL count match are jointly impossible if AC-4 is violated, so the existing test simultaneously validates AC-3 and AC-4 on Jetson |
| AC-4 (VioOutput.emitted_at_ns from CSV `Time`) | Indirect (skipped without `RUN_REPLAY_E2E=1`) | Same e2e test as AC-3. The runtime loop's `dataclasses.replace(vio_out, emitted_at_ns=frame_end_ns)` is the only path that satisfies AC-4 + AC-3 together; a regression would surface as the AZ-848 cascade |
| AC-5 (schema fault → `ReplayInputAdapterError` at startup) | Direct | `test_ac5_file_not_found_raises`, `test_ac5_missing_required_column_raises`, `test_ac5_nan_in_time_raises`, `test_ac5_non_monotonic_time_raises`, `test_ac5_repeated_time_also_non_monotonic`, `test_ac5_non_numeric_imu_value_raises`, `test_ac5_header_only_raises` |
**AC-4 coverage rationale**: the `_run_replay_loop` is integration-heavy
and has no existing unit-test seam. Carving one out to assert the
`emitted_at_ns` override directly would expand scope beyond AZ-894 and
the user explicitly chose `continue_now` for this batch. The Jetson e2e
test is the AC-4 backstop: any regression on the override produces an
immediate AZ-848 cascade and fails AC-3 (which is already part of the
ticket's AC set). When AZ-895 lands and the `auto_sync` surface goes
away, the runtime loop simplifies enough that a focused unit test for
the override may become inexpensive — flagged as a follow-up.
### AZ-896
| AC | Coverage | Test |
|----|----------|------|
| AC-1 (all 19 columns documented) | Direct (doc inspection) | `_docs/02_document/contracts/replay/csv_replay_format.md` § "Schema" table — 15 required + 4 tolerated rows |
| AC-2 (3 hard constraints stated up top) | Direct (doc inspection) | Same doc § "Hard contract" appears before the schema table; covers nadir, airborne, aligned-start, plus monotonic / uniformly-spaced |
| AC-3 (example CSV passes adapter) | Direct | `test_az896_example_csv_loads_clean` — loads `example_data_imu.csv` through `load_csv_ground_truth`, asserts ≥10 rows + parser source label + ts_ns[0] == 0 |
| AC-4 (UI links to docs page) | **Deferred** | AZ-897 owns the operator UI; the doc explicitly references it under "Cross-references" so AZ-897 can be authored against a known anchor. AC will fire on AZ-897 acceptance |
**Total AZ-894 + AZ-896**: 8/9 ACs immediately covered; 1 deferred-by-design
(AZ-896 AC-4 depends on AZ-897). No skipped-without-reason tests.
## Code Review Verdict: PASS
Inline review (consistent with batch 01's lightweight approach for a
single user-clarified-design batch). Detailed walk:
- **Phase 1 (Context)**: AZ-894 + AZ-896 specs read; the three
user-clarified design choices (replace/c8_sibling_full_protocol/
continue_now) are reflected verbatim in the code shape.
- **Phase 2 (Spec compliance)**: AC-by-AC walkthrough above. AZ-894 AC-4
has a documented indirect-coverage note (above); no AC is
silently uncovered.
- **Phase 3 (Code quality)**:
- `csv_ground_truth.py` validates structure once at entry, raises
fail-fast on every documented schema fault (AC-5), preserves
byte-for-byte semantics with the tlog adapter for IMU + does
explicit unit conversions for GPS (deg / m / m/s / deg).
- `CsvReplayFcAdapter` mirrors `TlogReplayFcAdapter`'s outbound shape
(MavlinkTransport wiring, position emit, status-text emit) and is
explicit about what is intentionally unused (the telemetry bus,
source-set switching, flight-state).
- Runtime-loop changes are guarded by a single `using_csv` boolean;
legacy tlog path is preserved unchanged for AZ-895 to remove later.
- `cli/replay.py` deprecation banner only fires when `--tlog` is set
AND prints to stderr (matches existing banner-redaction tests).
- **Phase 4 (Security)**: no new credentials, no IPC, no network. CSV
parser uses `csv.DictReader` (stdlib, no eval) and `float()`. CLI
signing-key handling unchanged.
- **Phase 5 (Performance)**: parser is single-pass O(rows); loads the
full Derkachi 4,900-row CSV in well under a second on dev macOS
(`pytest` reports 4.5s for the full unit suite touched). Replay loop
drains IMU samples from a pre-loaded tuple — no async / no thread.
- **Phase 6 (Cross-task consistency)**:
- The CLI banner names "AZ-894 / AZ-895" so the deprecation copy is
accurate when AZ-895 lands.
- `module-layout.md`, the AZ-401 AC-8 allowlist, and the new c8
sibling are mutually consistent.
- **Phase 7 (Architecture)**:
- New file ownership matches `module-layout.md`.
- Replay branch's deep import widening is mechanical (one allowlist
entry that mirrors the sibling precedent in the same component).
- No new layer rule.
No `@pytest.mark.xfail` decorators removed → LESSONS 2026-05-26 [testing]
gate not engaged.
## Auto-Fix Attempts: 0
## Escalated Findings: 0
## Stuck Tasks: 0
## Tests Run
Focused local pass on touched modules:
```
python -m pytest \
tests/unit/replay_input/test_csv_ground_truth.py \
tests/unit/c8_fc_adapter/test_csv_replay_adapter.py \
tests/unit/test_az401_compose_root_replay.py \
tests/unit/test_az402_replay_cli.py \
-v --tb=short
```
**70 passed, 0 failed, 0 skipped**.
Full unit-suite gate:
```
python -m pytest tests/unit/ -v --tb=short -q
```
**2,327 passed, 86 skipped, 3 warnings in 76 s**. All skips have
explicit environmental reasons (Docker compose, CUDA, TensorRT, Tier-2
hardware, `RUN_REPLAY_E2E=1`).
## Tracker Transitions
| Ticket | Step 5 (→ In Progress) | Step 12 (→ In Testing) |
|--------|------------------------|------------------------|
| AZ-894 | _to be transitioned after commit_ | _to be transitioned after commit_ |
| AZ-896 | _to be transitioned after commit_ | _to be transitioned after commit_ |
This block is updated in-place after the batch commit lands and the
Jira MCP transitions are confirmed via `getTransitionsForJiraIssue`
`transitionJiraIssue` → read-back.
## Leftovers / Tracker hygiene
- No new leftovers produced.
- Still open from prior batches:
- `_docs/_process_leftovers/2026-05-11_d_cross_cve_1_opencv_pin_deferred.md`
— gtsam numpy-2 wheel not on PyPI; unchanged.
## Next Batch
Batch 03 (cycle 4): **AZ-895** — deprecate the `auto_sync` surface
proper. Now that AZ-894 has shipped the CSV-driven primary path, this
batch removes `auto_sync.py`, strips the auto-sync wiring from
`_replay_branch.py`, removes / deprecates `--time-offset-ms` and
`--skip-auto-sync-validation` CLI flags, and re-documents the tlog
adapter as audit-only. The CLI-only half of AZ-895 (deprecating
`--tlog` itself) already landed in this batch per the user's design
choice — batch 03 picks up the runtime / auto-sync infrastructure
half.
+3 -3
View File
@@ -6,9 +6,9 @@ step: 10
name: Implement
status: in_progress
sub_step:
phase: 0
name: awaiting-invocation
detail: ""
phase: 7
name: batch-loop
detail: "batch 3 of 4 — AZ-895"
retry_count: 0
cycle: 4
tracker: jira