mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 11:21:13 +00:00
[AZ-894] [AZ-896] Add CSV-driven replay adapter + format docs
Replaces the tlog two-clock replay surface with a single-clock path driven by the Derkachi-schema CSV. --imu is the new required CLI arg; --tlog stays as a deprecated alias (warned + ignored when --imu set) until AZ-895 deletes it. * csv_ground_truth.py parses the 15-column schema, fails fast at startup on every documented schema fault (AC-5). * CsvReplayFcAdapter slots into ReplayInputBundle.fc_adapter alongside the tlog sibling; mirrors Invariant-5 outbound wiring; inbound bus is intentionally a no-op since the loop reads CSV directly. * _run_replay_loop branches on imu_csv_path, stamps VioOutput.emitted_at_ns from the CSV-derived frame_end_ns (AC-4), closing the AZ-848 two-clock surface for the new path. * AZ-896 ships the operator-facing format spec at _docs/02_document/contracts/replay/csv_replay_format.md plus a 20-row example CSV (AC-3 regression-locked). Tests: 11 + 12 new unit tests, plus updates to AZ-401 import-boundary and AZ-402 CLI suites. Full unit suite 2,327 passed / 86 skipped. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,149 @@
|
||||
# Replay-input CSV format (AZ-896)
|
||||
|
||||
**Status**: canonical operator-facing spec for the `--imu` argument of
|
||||
`gps-denied-replay` (AZ-894).
|
||||
**Audience**: operators preparing a (video, CSV) replay pair, plus engineers
|
||||
implementing alternative replay backends.
|
||||
**Companion artifacts**:
|
||||
|
||||
- `_docs/02_document/contracts/replay/example_data_imu.csv` — minimal valid
|
||||
example (20 rows = 2 s at 10 Hz).
|
||||
- `_docs/00_problem/input_data/flight_derkachi/data_imu.csv` — full Derkachi
|
||||
fixture (4,900 rows = 489.9 s at 10 Hz).
|
||||
- Parser implementation:
|
||||
`src/gps_denied_onboard/replay_input/csv_ground_truth.py`.
|
||||
|
||||
## Hard contract (read before generating a file)
|
||||
|
||||
The replay pipeline trusts the CSV blindly inside the loop. Violations of any
|
||||
of the following will produce silently wrong outputs (the parser only catches
|
||||
schema-level faults, not semantic ones), so the operator owns these
|
||||
invariants:
|
||||
|
||||
1. **Nadir camera.** The companion `.mp4` must be a nadir (straight-down)
|
||||
recording. The C1 VIO and C2 VPR stages assume nadir framing; oblique
|
||||
imagery breaks the satellite-anchor and VIO scale recovery.
|
||||
2. **Airborne at row 0.** The UAV must already be airborne at the first CSV
|
||||
row / first video frame. The replay pipeline does not implement a
|
||||
take-off detector — feeding a ground-roll segment yields garbage IMU
|
||||
integration.
|
||||
3. **Aligned start.** Row 0's `Time = 0.0` must correspond to the first
|
||||
video frame. The CLI does not perform sub-frame alignment; offset the
|
||||
CSV/clip pair offline before invoking `gps-denied-replay`.
|
||||
4. **Monotonic, uniformly-spaced `Time`.** Rows must be strictly increasing
|
||||
on `Time` and uniformly spaced (the Derkachi fixture is 10 Hz). The
|
||||
parser enforces monotonicity (AC-5); uniform spacing is the operator's
|
||||
responsibility — non-uniform spacing skews the ESKF prediction step
|
||||
without raising an error.
|
||||
|
||||
## Schema
|
||||
|
||||
The CSV must be header-first, comma-separated, UTF-8 encoded. Column order
|
||||
does not matter — the parser uses `csv.DictReader` and looks up by name —
|
||||
but the column **names** must match exactly (case-sensitive).
|
||||
|
||||
15 columns are required; up to 4 additional columns (mag fields,
|
||||
`relative_alt`) are tolerated and ignored.
|
||||
|
||||
### Required columns
|
||||
|
||||
| # | Column | Unit | Type | Notes |
|
||||
|---|--------|------|------|-------|
|
||||
| 1 | `timestamp(ms)` | ms | float | Pixhawk wall clock at sample capture. **Ignored by the replay pipeline** — kept only for trace-back to the original tlog. |
|
||||
| 2 | `Time` | s | float | **Canonical replay clock.** Must start at `0.0`, increase monotonically, and be uniformly spaced. The replay loop uses this column for every timestamp it emits. |
|
||||
| 3 | `SCALED_IMU2.xacc` | mg | float | Body-frame X accelerometer, MAVLink `SCALED_IMU2` raw scaling. Forwarded unchanged into `ImuSample.accel_xyz[0]`. |
|
||||
| 4 | `SCALED_IMU2.yacc` | mg | float | Body-frame Y accelerometer. |
|
||||
| 5 | `SCALED_IMU2.zacc` | mg | float | Body-frame Z accelerometer. |
|
||||
| 6 | `SCALED_IMU2.xgyro` | mrad/s | float | Body-frame X gyro, MAVLink `SCALED_IMU2` raw scaling. Forwarded unchanged into `ImuSample.gyro_xyz[0]`. |
|
||||
| 7 | `SCALED_IMU2.ygyro` | mrad/s | float | Body-frame Y gyro. |
|
||||
| 8 | `SCALED_IMU2.zgyro` | mrad/s | float | Body-frame Z gyro. |
|
||||
| 9 | `GLOBAL_POSITION_INT.lat` | degrees | float | WGS84 latitude. **Already in decimal degrees** (Derkachi dump convention — pre-divided by 1e7 from MAVLink's int representation). |
|
||||
| 10 | `GLOBAL_POSITION_INT.lon` | degrees | float | WGS84 longitude (same convention as `lat`). |
|
||||
| 11 | `GLOBAL_POSITION_INT.alt` | mm | float | MSL altitude. Parser divides by 1000 to emit metres. |
|
||||
| 12 | `GLOBAL_POSITION_INT.vx` | cm/s | float | NED north velocity. Parser divides by 100 to emit m/s. |
|
||||
| 13 | `GLOBAL_POSITION_INT.vy` | cm/s | float | NED east velocity. |
|
||||
| 14 | `GLOBAL_POSITION_INT.vz` | cm/s | float | NED down velocity. |
|
||||
| 15 | `GLOBAL_POSITION_INT.hdg` | cdeg | float | Heading, 0–35999. Parser divides by 100 to emit degrees. |
|
||||
|
||||
### Tolerated extra columns
|
||||
|
||||
The following may be present but are not consumed:
|
||||
|
||||
| Column | Reason kept | Reason unused |
|
||||
|--------|-------------|---------------|
|
||||
| `SCALED_IMU2.xmag`, `.ymag`, `.zmag` | Symmetric with the accel/gyro triples in the Derkachi dump | The current ESKF does not integrate magnetometer; AZ-848 follow-up may add it |
|
||||
| `GLOBAL_POSITION_INT.relative_alt` | Present in the MAVLink dump | The replay pipeline uses MSL `alt` only |
|
||||
|
||||
Additional columns beyond these are ignored without warning. Missing
|
||||
required columns cause the load to raise
|
||||
`ReplayInputAdapterError` before the replay loop starts (AC-5).
|
||||
|
||||
## Schema-level errors the parser catches
|
||||
|
||||
The parser raises `ReplayInputAdapterError` (CLI exit code 1) for any of:
|
||||
|
||||
- File does not exist or is not a regular file.
|
||||
- File is empty (no header row).
|
||||
- File has a header but no data rows.
|
||||
- Any required column from the table above is missing from the header.
|
||||
- The `Time` column at any row contains a non-numeric / NaN / Inf value.
|
||||
- The `Time` column is non-monotonic (`Time[i] <= Time[i-1]`).
|
||||
- Any required IMU or GPS column at any row contains a non-numeric / NaN /
|
||||
Inf value.
|
||||
|
||||
The error message includes the row number (1-based, where row 1 is the
|
||||
header — so the first data row is row 2). Operators should treat the first
|
||||
parse failure as authoritative and fix the source CSV; the parser does not
|
||||
continue after the first invalid row.
|
||||
|
||||
## Operator workflow
|
||||
|
||||
```bash
|
||||
gps-denied-replay \
|
||||
--video ./flight.mp4 \
|
||||
--imu ./data_imu.csv \
|
||||
--output ./estimator_output.jsonl \
|
||||
--camera-calibration ./calib.json \
|
||||
--config ./config.yaml \
|
||||
--mavlink-signing-key ./signing_key.bin
|
||||
```
|
||||
|
||||
`--tlog` is accepted as a deprecated alias and will be removed by AZ-895.
|
||||
When both `--imu` and `--tlog` are supplied, `--imu` wins and a deprecation
|
||||
warning is printed to stderr.
|
||||
|
||||
## Deriving a new CSV from an ArduPilot tlog
|
||||
|
||||
The Derkachi fixture was produced with `pymavlink`'s `mavlogdump.py`. The
|
||||
short version:
|
||||
|
||||
```bash
|
||||
mavlogdump.py --format csv \
|
||||
--types SCALED_IMU2,GLOBAL_POSITION_INT \
|
||||
./flight.tlog > ./raw_dump.csv
|
||||
```
|
||||
|
||||
Then post-process to:
|
||||
|
||||
1. Rename / merge the per-message timestamp into a single `Time` column
|
||||
relative to the first row.
|
||||
2. Drop pre-takeoff rows (the UAV must be airborne at row 0 — see the hard
|
||||
contract above).
|
||||
3. Pre-divide `lat` / `lon` from the MAVLink `int * 1e7` representation
|
||||
into decimal degrees.
|
||||
4. Re-sample to a uniform 10 Hz cadence if the tlog dump produced
|
||||
non-uniform spacing.
|
||||
|
||||
A reference post-processor script is **not** shipped — operators
|
||||
historically write a one-off Python or Pandas pipeline per source aircraft.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- AZ-894 — the CLI + adapter that consumes this format.
|
||||
- AZ-895 — deletes the legacy `--tlog` argument once all callers migrate.
|
||||
- AZ-897 — operator replay UI; links to this page and serves
|
||||
`example_data_imu.csv`.
|
||||
- `_docs/02_document/contracts/replay/replay_protocol.md` — the broader
|
||||
replay orchestration contract.
|
||||
- `_docs/00_problem/input_data/flight_derkachi/README.md` — fixture
|
||||
provenance and license caveats.
|
||||
@@ -0,0 +1,21 @@
|
||||
timestamp(ms),Time,SCALED_IMU2.xacc,SCALED_IMU2.yacc,SCALED_IMU2.zacc,SCALED_IMU2.xgyro,SCALED_IMU2.ygyro,SCALED_IMU2.zgyro,SCALED_IMU2.xmag,SCALED_IMU2.ymag,SCALED_IMU2.zmag,GLOBAL_POSITION_INT.lat,GLOBAL_POSITION_INT.lon,GLOBAL_POSITION_INT.alt,GLOBAL_POSITION_INT.relative_alt,GLOBAL_POSITION_INT.vx,GLOBAL_POSITION_INT.vy,GLOBAL_POSITION_INT.vz,GLOBAL_POSITION_INT.hdg
|
||||
4551116.348,0,21,-3,-984,52,32,-5,312,-1048,442,50.0809634,36.1115442,141290,23.182,-4,-6,-88,35041
|
||||
4551216.348,0.1,-68,-9,-995,58,-17,1,309,-1016,441,50.0809634,36.1115441,141360,23.251,-5,-2,-89,35042
|
||||
4551316.348,0.2,9,108,-988,69,-65,13,308,-964,436,50.0809633,36.1115441,141410,23.303,-1,-2,-86,35048
|
||||
4551416.348,0.3,-20,27,-977,55,10,26,310,-988,438,50.0809633,36.1115441,141450,23.348,-5,-6,-84,35057
|
||||
4551516.348,0.4,-40,40,-1026,0,65,10,306,-1076,440,50.0809633,36.111544,141510,23.402,-2,-2,-86,35065
|
||||
4551616.348,0.5,30,126,-1050,-1,75,14,321,-1146,442,50.0809633,36.111544,141570,23.464,0,0,-88,35074
|
||||
4551716.348,0.6,-64,67,-1031,-31,-6,21,314,-1066,438,50.0809632,36.1115439,141640,23.53,-5,1,-90,35080
|
||||
4551816.348,0.7,-22,112,-1027,-61,-88,-5,302,-951,436,50.0809632,36.1115439,141710,23.601,-2,3,-90,35082
|
||||
4551916.348,0.8,-123,-16,-998,-55,-104,-12,301,-942,440,50.0809631,36.1115439,141770,23.669,-10,0,-91,35079
|
||||
4552016.348,0.9,-64,-13,-1003,13,-70,-30,301,-936,442,50.080963,36.1115439,141860,23.755,-2,0,-90,35073
|
||||
4552116.348,1,-22,39,-995,73,20,-18,314,-988,436,50.080963,36.1115439,141930,23.826,-2,-2,-88,35070
|
||||
4552216.348,1.1,-49,-69,-984,2,29,1,317,-992,433,50.080963,36.1115438,142010,23.9,-6,-2,-88,35068
|
||||
4552316.348,1.2,-16,98,-991,-59,-28,-11,310,-970,435,50.080963,36.1115438,142080,23.975,-1,6,-86,35063
|
||||
4552416.348,1.3,-6,169,-998,-29,2,-2,310,-983,435,50.0809629,36.1115438,142150,24.042,-3,5,-83,35059
|
||||
4552516.348,1.4,-31,53,-1003,2,13,-10,317,-1042,438,50.0809629,36.1115438,142210,24.102,-3,3,-83,35051
|
||||
4552616.348,1.5,-47,21,-1023,13,13,-14,320,-1069,439,50.0809629,36.1115438,142270,24.166,2,2,-83,35047
|
||||
4552716.348,1.6,-30,-59,-1020,-18,24,0,315,-1083,438,50.0809629,36.1115439,142340,24.236,-5,1,-86,35049
|
||||
4552816.348,1.7,-103,23,-1058,-59,26,-7,314,-1113,442,50.0809629,36.1115439,142430,24.321,-4,4,-90,35050
|
||||
4552916.348,1.8,-17,51,-1037,-9,80,11,317,-1087,444,50.0809629,36.1115439,142510,24.404,-5,0,-93,35049
|
||||
4553016.348,1.9,-87,72,-1022,-10,-45,0,309,-1004,439,50.0809628,36.111544,142600,24.494,-6,2,-97,35046
|
||||
|
Reference in New Issue
Block a user