gps-denied-onboard/e2e/fixtures/sitl_replay_builder/README.md

# SITL Replay Fixture Builder (AZ-598, AZ-599)

Per-scenario fixture builders for the offline FDR-replay path used
by the b75 `sitl_observer` module + FT-* blackbox scenarios. Each
builder takes recorded flight inputs (still images / video / IMU
CSV / tlog) and produces the artifacts a specific scenario needs.

| Scenario | Builder | Inputs | Outputs |
|----------|---------|--------|---------|
| FT-P-01 (still-image accuracy) | `build_p01_fixtures.py` | 60 `AD0000NN.jpg` + coordinates CSV | `outbound_messages_<fc>_<host>.json` + `observer_<fc>_<host>.json` + `stills.mp4` + `stationary.tlog` + `fdr.jsonl` |
| FT-P-02 (Derkachi drift) | `build_p02_fixtures.py` | `flight_derkachi.mp4` + `data_imu.csv` | `derkachi.tlog` + `fdr/fdr.jsonl` (FDR archive) + `observer_<fc>_<host>.json` |

Other scenarios (FT-P-03 / 04 / 05 / 07 / 08 / 10 / 11, FT-N-01 / 02 / 03 / 04)
need their own capture flows and will land as follow-up tickets.

## Shared helpers (`_common.py`)

Both builders shell out to the production `gps-denied-replay` CLI and
write the same minimal `observer_<fc_kind>_<host>.json`. These two
operations live in `_common.py`:

* `run_gps_denied_replay(video, tlog, fdr_out, *, time_offset_ms=0, ...)`
* `write_observer_fixture(output_path)`

Future per-scenario builders should import from `_common.py` rather
than re-implementing.

## FT-P-01 (`build_p01_fixtures.py`)

### Strategy

Rather than spinning up a SITL container, this builder reuses the
production `gps-denied-replay` CLI + `ReplayInputAdapter`:

1. Encode the 60 `AD0000NN.jpg` still images into a 1 fps MP4.
2. Generate a synthetic stationary tlog (zero-motion `RAW_IMU` +
   `ATTITUDE` pairs at 200 Hz) — bypasses the AZ-405 take-off
   pre-validator without needing real flight data.
3. Run `gps-denied-replay --video stills.mp4 --tlog stationary.tlog
   --time-offset-ms 0 --fdr-out fdr.jsonl` (auto-sync bypassed
   because the synthetic tlog has no take-off signal).
4. Read `fdr.jsonl`, filter to `kind == outbound_position_estimate`,
   project each into the `outbound_messages_*` schema.
5. Write the two fixture JSON files into `--output-dir`.

This avoids needing new SUT-side frame-ingestion code (HTTP endpoint,
file-watch source, etc.) which would otherwise be required to push
individual stills to a running SUT container.

### Usage

```bash
python -m e2e.fixtures.sitl_replay_builder.build_p01_fixtures \
  --input-dir _docs/00_problem/input_data \
  --output-dir e2e/fixtures/sitl_replay/p01 \
  --fc-kind ardupilot \
  --host sitl-host
```

The output directory will contain:

* `stills.mp4` — the 60 images encoded at 1 fps.
* `stationary.tlog` — synthetic 120-s zero-motion tlog at 200 Hz.
* `fdr.jsonl` — the FDR JSONL stream from the replay run.
* `outbound_messages_ardupilot_sitl-host.json` — the consumed fixture.
* `observer_ardupilot_sitl-host.json` — the consumed fixture.

To activate the fixtures in a scenario run:

```bash
E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01 \
    pytest e2e/tests/positive/test_ft_p_01_still_image_accuracy.py
```

### Limitations

* The synthetic tlog encodes zero motion — auto-sync MUST be bypassed
  via `--time-offset-ms 0` (the builder does this automatically).
* The FDR record `kind` is assumed to be `outbound_position_estimate`
  — the `--fdr-kind` CLI flag overrides if the actual schema differs.
* Per-image timeout handling: if the SUT emits fewer outbound estimates
  than pushed frames, trailing image_ids are written as `null` entries
  (encoded as TimeoutError on scenario replay).
* iNav adapter is NOT supported by this batch — only ArduPilot. iNav
  will land as a follow-up once the AP path is validated end-to-end.

## FT-P-02 (`build_p02_fixtures.py`)

### Strategy

Same overall shape as FT-P-01 (drive `gps-denied-replay` against a
video + tlog pair), with two differences:

1. Video is already MP4 — skip the OpenCV still-image encoding step.
2. IMU is recorded telemetry (`data_imu.csv`, 10 Hz `SCALED_IMU2` +
   `GLOBAL_POSITION_INT`). A CSV → tlog conversion packs each row as
   a `RAW_IMU` + `ATTITUDE` MAVLink pair, with yaw synthesised from
   `GLOBAL_POSITION_INT.hdg` (centidegrees → radians) and roll/pitch
   = 0 (acceptable for the fixed-wing cruise data this represents).

Output is the SUT's natural FDR archive directory; the FT-P-02
scenario reads it via `runner.helpers.fdr_reader.iter_records`.

### Usage

```bash
python -m e2e.fixtures.sitl_replay_builder.build_p02_fixtures \
  --derkachi-dir _docs/00_problem/input_data/flight_derkachi \
  --output-dir e2e/fixtures/sitl_replay/p02 \
  --fc-kind ardupilot \
  --host sitl-host
```

Output:

* `derkachi.tlog` — generated from `data_imu.csv`.
* `fdr/fdr.jsonl` — the FDR archive from the replay run.
* `observer_ardupilot_sitl-host.json` — minimal observer config.

### Limitations

* The synthesised ATTITUDE has roll/pitch = 0 — acceptable for
  fixed-wing cruise but unrealistic for aggressive manoeuvres.
* RAW_IMU is packed from `SCALED_IMU2` columns as pass-through (no
  true scaled → raw unit conversion). If the SUT's tlog parser
  strictly demands true raw counts the builder will need a units
  conversion pass — surfaced as a follow-up after live-run.
* Auto-sync is bypassed via `--time-offset-ms 0` because the
  Derkachi CSV is already aligned with the video.

## Testing

Unit tests under `e2e/_unit_tests/fixtures/test_sitl_replay_builder*.py`
mock all external dependencies (OpenCV, pymavlink, subprocess) so the
test suite runs without a real `gps-denied-replay` install. The actual
end-to-end run requires the SUT to be installed (`pip install -e .` at
repo root) and is documented as a manual step until CI infrastructure
catches up.