gps-denied-onboard/e2e/fixtures/sitl_replay_builder/README.md

SITL Replay Fixture Builder (AZ-598, AZ-599)

Per-scenario fixture builders for the offline FDR-replay path used by the b75 sitl_observer module + FT-* blackbox scenarios. Each builder takes recorded flight inputs (still images / video / IMU CSV / tlog) and produces the artifacts a specific scenario needs.

Scenario	Builder	Inputs	Outputs
FT-P-01 (still-image accuracy)	build_p01_fixtures.py	60 AD0000NN.jpg + coordinates CSV	outbound_messages_<fc>_<host>.json + observer_<fc>_<host>.json + stills.mp4 + stationary.tlog + fdr.jsonl
FT-P-02 (Derkachi drift)	build_p02_fixtures.py	flight_derkachi.mp4 + data_imu.csv	derkachi.tlog + fdr/fdr.jsonl (FDR archive) + observer_<fc>_<host>.json

Other scenarios (FT-P-03 / 04 / 05 / 07 / 08 / 10 / 11, FT-N-01 / 02 / 03 / 04) need their own capture flows and will land as follow-up tickets.

Shared helpers (_common.py)

Both builders shell out to the production gps-denied-replay CLI and write the same minimal observer_<fc_kind>_<host>.json. These two operations live in _common.py:

• run_gps_denied_replay(video, tlog, fdr_out, *, time_offset_ms=0, ...)
• write_observer_fixture(output_path)

Future per-scenario builders should import from _common.py rather than re-implementing.

FT-P-01 (build_p01_fixtures.py)

Strategy

Rather than spinning up a SITL container, this builder reuses the production gps-denied-replay CLI + ReplayInputAdapter:

1. Encode the 60 AD0000NN.jpg still images into a 1 fps MP4.
2. Generate a synthetic stationary tlog (zero-motion RAW_IMU + ATTITUDE pairs at 200 Hz) — bypasses the AZ-405 take-off pre-validator without needing real flight data.
3. Run gps-denied-replay --video stills.mp4 --tlog stationary.tlog --time-offset-ms 0 --fdr-out fdr.jsonl (auto-sync bypassed because the synthetic tlog has no take-off signal).
4. Read fdr.jsonl, filter to kind == outbound_position_estimate, project each into the outbound_messages_* schema.
5. Write the two fixture JSON files into --output-dir.

This avoids needing new SUT-side frame-ingestion code (HTTP endpoint, file-watch source, etc.) which would otherwise be required to push individual stills to a running SUT container.

Usage

python -m e2e.fixtures.sitl_replay_builder.build_p01_fixtures \
  --input-dir _docs/00_problem/input_data \
  --output-dir e2e/fixtures/sitl_replay/p01 \
  --fc-kind ardupilot \
  --host sitl-host

The output directory will contain:

• stills.mp4 — the 60 images encoded at 1 fps.
• stationary.tlog — synthetic 120-s zero-motion tlog at 200 Hz.
• fdr.jsonl — the FDR JSONL stream from the replay run.
• outbound_messages_ardupilot_sitl-host.json — the consumed fixture.
• observer_ardupilot_sitl-host.json — the consumed fixture.

To activate the fixtures in a scenario run:

E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01 \
    pytest e2e/tests/positive/test_ft_p_01_still_image_accuracy.py

Limitations

• The synthetic tlog encodes zero motion — auto-sync MUST be bypassed via --time-offset-ms 0 (the builder does this automatically).
• The FDR record kind is assumed to be outbound_position_estimate — the --fdr-kind CLI flag overrides if the actual schema differs.
• Per-image timeout handling: if the SUT emits fewer outbound estimates than pushed frames, trailing image_ids are written as null entries (encoded as TimeoutError on scenario replay).
• iNav adapter is NOT supported by this batch — only ArduPilot. iNav will land as a follow-up once the AP path is validated end-to-end.

FT-P-02 (build_p02_fixtures.py)

Strategy

Same overall shape as FT-P-01 (drive gps-denied-replay against a video + tlog pair), with two differences:

1. Video is already MP4 — skip the OpenCV still-image encoding step.
2. IMU is recorded telemetry (data_imu.csv, 10 Hz SCALED_IMU2 + GLOBAL_POSITION_INT). A CSV → tlog conversion packs each row as a RAW_IMU + ATTITUDE MAVLink pair, with yaw synthesised from GLOBAL_POSITION_INT.hdg (centidegrees → radians) and roll/pitch = 0 (acceptable for the fixed-wing cruise data this represents).

Output is the SUT's natural FDR archive directory; the FT-P-02 scenario reads it via runner.helpers.fdr_reader.iter_records.

Usage

python -m e2e.fixtures.sitl_replay_builder.build_p02_fixtures \
  --derkachi-dir _docs/00_problem/input_data/flight_derkachi \
  --output-dir e2e/fixtures/sitl_replay/p02 \
  --fc-kind ardupilot \
  --host sitl-host

Output:

• derkachi.tlog — generated from data_imu.csv.
• fdr/fdr.jsonl — the FDR archive from the replay run.
• observer_ardupilot_sitl-host.json — minimal observer config.

Limitations

• The synthesised ATTITUDE has roll/pitch = 0 — acceptable for fixed-wing cruise but unrealistic for aggressive manoeuvres.
• RAW_IMU is packed from SCALED_IMU2 columns as pass-through (no true scaled → raw unit conversion). If the SUT's tlog parser strictly demands true raw counts the builder will need a units conversion pass — surfaced as a follow-up after live-run.
• Auto-sync is bypassed via --time-offset-ms 0 because the Derkachi CSV is already aligned with the video.

Testing

Unit tests under e2e/_unit_tests/fixtures/test_sitl_replay_builder*.py mock all external dependencies (OpenCV, pymavlink, subprocess) so the test suite runs without a real gps-denied-replay install. The actual end-to-end run requires the SUT to be installed (pip install -e . at repo root) and is documented as a manual step until CI infrastructure catches up.