azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 15:21:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-894] [AZ-896] Add CSV-driven replay adapter + format docs

Browse Source 
Replaces the tlog two-clock replay surface with a single-clock path
driven by the Derkachi-schema CSV. --imu is the new required CLI arg;
--tlog stays as a deprecated alias (warned + ignored when --imu set)
until AZ-895 deletes it.

* csv_ground_truth.py parses the 15-column schema, fails fast at
  startup on every documented schema fault (AC-5).
* CsvReplayFcAdapter slots into ReplayInputBundle.fc_adapter alongside
  the tlog sibling; mirrors Invariant-5 outbound wiring; inbound bus is
  intentionally a no-op since the loop reads CSV directly.
* _run_replay_loop branches on imu_csv_path, stamps
  VioOutput.emitted_at_ns from the CSV-derived frame_end_ns (AC-4),
  closing the AZ-848 two-clock surface for the new path.
* AZ-896 ships the operator-facing format spec at
  _docs/02_document/contracts/replay/csv_replay_format.md plus a
  20-row example CSV (AC-3 regression-locked).

Tests: 11 + 12 new unit tests, plus updates to AZ-401 import-boundary
and AZ-402 CLI suites. Full unit suite 2,327 passed / 86 skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-26 18:40:29 +03:00
parent 3020779404
commit 6be207cef3
19 changed files with 1833 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_tasks/todo/AZ-894_csv_driven_replay_adapter.md
-53
View File
@@ -1,53 +0,0 @@
# Replay: CSV-driven IMU+GPS adapter using single canonical clock
**Task**: AZ-894_csv_driven_replay_adapter
**Name**: Add a CSV-replay adapter that consumes the Derkachi-schema `data_imu.csv` (or any flight that ships with a paired CSV) and exposes IMU + GPS-ground-truth on a single canonical clock derived from the CSV's `Time` column
**Description**: Cycle 3 surfaced AZ-848 (eskf_out_of_order on frame 3) because the current replay pipeline imports two incompatible clocks: `VioOutput.emitted_at_ns` uses Jetson process-monotonic time, while `ImuWindow.ts_end_ns` uses FC-boot-relative time (parsed from MAVLink tlog messages). The single-clock model that production uses (single edge device, single clock at receipt) is not what replay does today. The Derkachi fixture's `data_imu.csv` already contains both IMU (`SCALED_IMU2.*`) and GPS ground truth (`GLOBAL_POSITION_INT.*`) on a single canonical clock (the `Time` column, 0..489.9 s at 10 Hz, aligned 3:1 with the 30 fps video). Using the CSV directly eliminates the clock-mismatch surface entirely for the test/demo path and matches the production single-clock model.
**Complexity**: 3 SP
**Dependencies**: AZ-896 (format docs land in the same cycle but can land in either order)
**Blocks**: AZ-895 (auto-sync deprecation), AZ-897 (replay UI)
**Component**: replay_input (new adapter), c8_fc_adapter (alternate ground-truth source), cli/replay
**Tracker**: AZ-894 (https://denyspopov.atlassian.net/browse/AZ-894)
**Parent Epic**: (none — cycle-4 replay-input redesign)
## Schema
The Derkachi CSV header (19 columns):
```
timestamp(ms), Time,
SCALED_IMU2.xacc, SCALED_IMU2.yacc, SCALED_IMU2.zacc,
SCALED_IMU2.xgyro, SCALED_IMU2.ygyro, SCALED_IMU2.zgyro,
SCALED_IMU2.xmag, SCALED_IMU2.ymag, SCALED_IMU2.zmag,
GLOBAL_POSITION_INT.lat, GLOBAL_POSITION_INT.lon, GLOBAL_POSITION_INT.alt,
GLOBAL_POSITION_INT.relative_alt,
GLOBAL_POSITION_INT.vx, GLOBAL_POSITION_INT.vy, GLOBAL_POSITION_INT.vz,
GLOBAL_POSITION_INT.hdg
```
- `timestamp(ms)`: FC-boot-relative milliseconds (kept for traceability; not used by C5)
- `Time`: flight-relative seconds (canonical clock — what C5 actually uses)
- `SCALED_IMU2.*`: 10 Hz IMU stream (accel mg, gyro mrad/s, mag mGauss per ArduPilot convention)
- `GLOBAL_POSITION_INT.*`: 10 Hz GPS ground truth (lat/lon in 1e-7 deg, alt in mm, vx/vy/vz in cm/s, hdg in cdeg)
## Acceptance Criteria
- **AC-1**: Adapter parses the Derkachi `data_imu.csv` end-to-end and emits 4,899 IMU samples + 4,899 GPS-ground-truth samples on a single monotonic clock anchored at row 0.
- **AC-2**: Wired into `cli/replay.py`; `gps-denied-replay --video flight_derkachi.mp4 --imu data_imu.csv` runs without invoking `tlog_replay_adapter.py`.
- **AC-3**: `test_derkachi_1min.py::test_ac1_exits_0_jsonl_count_match` passes on the Jetson e2e harness using the new path. AZ-848 cascade no longer triggers (no two-clock surface in the new path).
- **AC-4**: `VioOutput.emitted_at_ns` is populated from the CSV's `Time` column (or the frame-derived `t = N/fps`), not `time.monotonic_ns()`, when the new adapter is in use.
- **AC-5**: Schema mismatch (missing required column, NaN in `Time`, non-monotonic `Time`) raises a clear `ReplayInputAdapterError` at startup, not deep in the loop.
## Out of scope
- The structural AZ-848 / AZ-883 fix in the tlog adapter — those stay open as backlog.
- UI for picking the CSV — AZ-897.
- Other CSV schemas (PX4, generic MAVLink dumps) — future enhancement if needed.
## References
- Cycle-3 retro: `_docs/06_metrics/retro_2026-05-26.md`
- Bench-run evidence: `_docs/04_release/release_cycle3_jetson-bench_2026-05-26-1442.md`
- Companion tickets: AZ-895 (deprecate auto-sync), AZ-896 (format docs + example CSV), AZ-897 (replay UI)
- Supersedes (re bench-blocking): AZ-848 (VioOutput contract), AZ-883 (SCALED_IMU2 ts_ns=0)
_docs/02_tasks/todo/AZ-896_replay_format_docs_and_example_csv.md
-38
View File
@@ -1,38 +0,0 @@
# Docs: replay-input format spec + downloadable example CSV
**Task**: AZ-896_replay_format_docs_and_example_csv
**Name**: Author the operator-facing format spec for the (video, CSV) replay input pair, plus a minimal downloadable example CSV
**Description**: Operators using the replay/demo path need to know the exact CSV schema the system accepts, the hard contract (video t=0 ≡ CSV row 0; video must be nadir; UAV must already be airborne at t=0), and have a downloadable example to copy from. Operators today have no entry point that documents this.
**Complexity**: 1 SP
**Dependencies**: AZ-894 (the adapter that consumes the format — the doc describes what AZ-894 accepts)
**Blocks**: AZ-897 (UI links to the docs page and serves the example CSV)
**Component**: docs (_docs/04_release/)
**Tracker**: AZ-896 (https://denyspopov.atlassian.net/browse/AZ-896)
**Parent Epic**: (none — cycle-4 replay-input redesign)
## What
- Author a docs page at `_docs/04_release/replay_input_format.md` (or wherever the operator-facing docs land in cycle 4)
- Schema table: column names, units, types, expected rates, required vs optional
- Constraint statements up top, before the column table:
- Video: nadir camera; UAV already airborne at frame 0
- CSV: row 0 timestamp == video frame 0 timestamp; `Time` column starts at 0.0; rows monotonic and uniformly-spaced
- Ship `_docs/04_release/example_data_imu.csv` — a minimal valid example (e.g., 20 rows = 2 seconds at 10 Hz)
- Cross-link from the AZ-897 replay UI "Download example" button
## Acceptance Criteria
- **AC-1**: Schema page documents all 19 columns of the Derkachi CSV with units and types.
- **AC-2**: The three hard constraints (nadir / airborne / aligned-start) are stated up top, before the column table.
- **AC-3**: The example CSV (≥10 rows) passes through the AZ-894 CSV adapter without errors.
- **AC-4**: The page is reachable from the AZ-897 UI's "Download example" link.
## Out of scope
- Multi-schema support (PX4, generic MAVLink dumps).
## References
- Companion: AZ-894 (CSV adapter), AZ-897 (UI), AZ-895 (auto-sync deprecation)
- Source fixture: `_docs/00_problem/input_data/flight_derkachi/data_imu.csv`, README at `_docs/00_problem/input_data/flight_derkachi/README.md`