[AZ-599] Batch 79: FT-P-02 Derkachi builder + _common.py extraction

- Add build_p02_fixtures.py: IMU CSV → tlog conversion (RAW_IMU +
  ATTITUDE pairs, centidegrees→radians yaw) and orchestrator that
  runs gps-denied replay against Derkachi MP4 + generated tlog,
  verifying ≥1 record_type="estimate" in the FDR archive.
- Extract run_gps_denied_replay + FDR-parent-dir helpers into
  sitl_replay_builder/_common.py; refactor build_p01_fixtures.py
  to import from _common (b78 tests preserved).
- Add 20 unit tests under e2e/_unit_tests/fixtures/test_sitl_
  replay_builder_p02.py covering AC-1..AC-5; total unit suite
  686/686 passing (regression gate AC-6).
- README updated to document FT-P-01 + FT-P-02 builders.
- Advance autodev state: last_completed_batch=79, current_batch=80;
  prune verbose detail blob.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-17 13:40:07 +03:00
parent 2f1fb4d0d0
commit 4e0717e543
10 changed files with 1111 additions and 76 deletions
+74 -13
View File
@@ -1,16 +1,33 @@
# SITL Replay Fixture Builder (AZ-598)
# SITL Replay Fixture Builder (AZ-598, AZ-599)
Produces the `outbound_messages_<fc_kind>_<host>.json` +
`observer_<fc_kind>_<host>.json` fixtures consumed by the b75
`sitl_observer` module in offline FDR-replay mode (b75/b78).
Per-scenario fixture builders for the offline FDR-replay path used
by the b75 `sitl_observer` module + FT-* blackbox scenarios. Each
builder takes recorded flight inputs (still images / video / IMU
CSV / tlog) and produces the artifacts a specific scenario needs.
## Vertical-slice scope (this batch)
| Scenario | Builder | Inputs | Outputs |
|----------|---------|--------|---------|
| FT-P-01 (still-image accuracy) | `build_p01_fixtures.py` | 60 `AD0000NN.jpg` + coordinates CSV | `outbound_messages_<fc>_<host>.json` + `observer_<fc>_<host>.json` + `stills.mp4` + `stationary.tlog` + `fdr.jsonl` |
| FT-P-02 (Derkachi drift) | `build_p02_fixtures.py` | `flight_derkachi.mp4` + `data_imu.csv` | `derkachi.tlog` + `fdr/fdr.jsonl` (FDR archive) + `observer_<fc>_<host>.json` |
Only the FT-P-01 still-image accuracy scenario is supported. Other
scenarios (FT-P-02 Derkachi continuous flight, FT-N-04 blackout-spoof,
etc.) need their own capture flows and will land as follow-up tickets.
Other scenarios (FT-P-03 / 04 / 05 / 07 / 08 / 10 / 11, FT-N-01 / 02 / 03 / 04)
need their own capture flows and will land as follow-up tickets.
## Strategy
## Shared helpers (`_common.py`)
Both builders shell out to the production `gps-denied-replay` CLI and
write the same minimal `observer_<fc_kind>_<host>.json`. These two
operations live in `_common.py`:
* `run_gps_denied_replay(video, tlog, fdr_out, *, time_offset_ms=0, ...)`
* `write_observer_fixture(output_path)`
Future per-scenario builders should import from `_common.py` rather
than re-implementing.
## FT-P-01 (`build_p01_fixtures.py`)
### Strategy
Rather than spinning up a SITL container, this builder reuses the
production `gps-denied-replay` CLI + `ReplayInputAdapter`:
@@ -30,10 +47,10 @@ This avoids needing new SUT-side frame-ingestion code (HTTP endpoint,
file-watch source, etc.) which would otherwise be required to push
individual stills to a running SUT container.
## Usage
### Usage
```bash
gps-denied-build-p01-fixtures \
python -m e2e.fixtures.sitl_replay_builder.build_p01_fixtures \
--input-dir _docs/00_problem/input_data \
--output-dir e2e/fixtures/sitl_replay/p01 \
--fc-kind ardupilot \
@@ -55,7 +72,7 @@ E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01 \
pytest e2e/tests/positive/test_ft_p_01_still_image_accuracy.py
```
## Limitations
### Limitations
* The synthetic tlog encodes zero motion — auto-sync MUST be bypassed
via `--time-offset-ms 0` (the builder does this automatically).
@@ -67,9 +84,53 @@ E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01 \
* iNav adapter is NOT supported by this batch — only ArduPilot. iNav
will land as a follow-up once the AP path is validated end-to-end.
## FT-P-02 (`build_p02_fixtures.py`)
### Strategy
Same overall shape as FT-P-01 (drive `gps-denied-replay` against a
video + tlog pair), with two differences:
1. Video is already MP4 — skip the OpenCV still-image encoding step.
2. IMU is recorded telemetry (`data_imu.csv`, 10 Hz `SCALED_IMU2` +
`GLOBAL_POSITION_INT`). A CSV → tlog conversion packs each row as
a `RAW_IMU` + `ATTITUDE` MAVLink pair, with yaw synthesised from
`GLOBAL_POSITION_INT.hdg` (centidegrees → radians) and roll/pitch
= 0 (acceptable for the fixed-wing cruise data this represents).
Output is the SUT's natural FDR archive directory; the FT-P-02
scenario reads it via `runner.helpers.fdr_reader.iter_records`.
### Usage
```bash
python -m e2e.fixtures.sitl_replay_builder.build_p02_fixtures \
--derkachi-dir _docs/00_problem/input_data/flight_derkachi \
--output-dir e2e/fixtures/sitl_replay/p02 \
--fc-kind ardupilot \
--host sitl-host
```
Output:
* `derkachi.tlog` — generated from `data_imu.csv`.
* `fdr/fdr.jsonl` — the FDR archive from the replay run.
* `observer_ardupilot_sitl-host.json` — minimal observer config.
### Limitations
* The synthesised ATTITUDE has roll/pitch = 0 — acceptable for
fixed-wing cruise but unrealistic for aggressive manoeuvres.
* RAW_IMU is packed from `SCALED_IMU2` columns as pass-through (no
true scaled → raw unit conversion). If the SUT's tlog parser
strictly demands true raw counts the builder will need a units
conversion pass — surfaced as a follow-up after live-run.
* Auto-sync is bypassed via `--time-offset-ms 0` because the
Derkachi CSV is already aligned with the video.
## Testing
Unit tests under `e2e/_unit_tests/fixtures/test_sitl_replay_builder.py`
Unit tests under `e2e/_unit_tests/fixtures/test_sitl_replay_builder*.py`
mock all external dependencies (OpenCV, pymavlink, subprocess) so the
test suite runs without a real `gps-denied-replay` install. The actual
end-to-end run requires the SUT to be installed (`pip install -e .` at