gps-denied-onboard/e2e/fixtures/sitl_replay_builder/README.md

# SITL Replay Fixture Builder (AZ-598, AZ-599, AZ-600)

Parameterized fixture-builder framework for the offline FDR-replay path
used by the b75 `sitl_observer` module + FT-* blackbox scenarios. A new
scenario typically only writes a ~60-line config factory + CLI on top of
the framework — no new strategy code required.

| Scenario | Builder | Inputs | Outputs |
|----------|---------|--------|---------|
| FT-P-01 (still-image accuracy) | `build_p01_fixtures.py` | 60 `AD0000NN.jpg` | `outbound_messages_<fc>_<host>.json` + `observer_<fc>_<host>.json` + `stills.mp4` + `stationary.tlog` + `fdr.jsonl` |
| FT-P-02 (Derkachi drift) | `build_p02_fixtures.py` | `flight_derkachi.mp4` + `data_imu.csv` | `derkachi.tlog` + `fdr/fdr.jsonl` (FDR archive) + `observer_<fc>_<host>.json` |

Other scenarios (FT-P-03 / 04 / 05 / 07 / 08 / 10 / 11, FT-N-01..04) will
land as follow-ups; each will reuse the strategies below.

## Framework (`builder.py`)

Three strategy ABCs decompose the per-scenario variance:

| Strategy | Concrete impls | Used by |
|----------|----------------|---------|
| `VideoSource` — materialize the MP4 the replay CLI consumes | `StillImagesSource(image_paths, fps)`, `Mp4PassthroughSource(mp4_path)` | b78 / b79 |
| `TlogSource` — materialize the tlog the replay CLI consumes | `SyntheticStationaryTlog(duration_s, hz)`, `ImuCsvTlog(csv_path, schema=DEFAULT_DERKACHI_IMU_SCHEMA)` | b78 / b79 |
| `FdrProjection` — translate the FDR JSONL into scenario fixture shape | `RawFdrPassthrough(verify_estimates=True)`, `OutboundMessagesProjection(image_ids, fdr_kind="outbound_position_estimate")` | b79 / b78 |

The `build_fixtures(cfg: FixtureBuilderConfig)` orchestrator composes the
three strategies plus the shared `run_gps_denied_replay` subprocess driver
and `write_observer_fixture` helper.

Shared helpers (in `builder.py`):

* `run_gps_denied_replay(video, tlog, fdr_out, *, time_offset_ms=0, ...)` — shells out to the production CLI.
* `write_observer_fixture(output_path)` — writes the minimal `observer_*.json` `sitl_observer.get_observer` requires.
* `pack_raw_imu(time_usec, *, xacc=0, yacc=0, zacc=0, xgyro=0, ygyro=0, zgyro=0)` — parameterized RAW_IMU packer. Stationary callers pass `zacc=STATIONARY_Z_ACCEL_MG` (gravity).
* `pack_attitude(time_boot_ms, *, roll=0.0, pitch=0.0, yaw=0.0)` — parameterized ATTITUDE packer.
* `parse_fdr_for_outbound_estimates(fdr_path, *, fdr_kind, lat_key, lon_key)` — read FDR JSONL into per-image dicts.
* `verify_fdr_has_estimates(fdr_path)` — assert ≥1 `record_type=="estimate"` record.
* `hdg_centideg_to_rad(hdg_cdeg)` — utility for ATTITUDE yaw synthesis.

## Adding a new scenario (worked example: FT-P-04)

FT-P-04 (Derkachi frame-to-frame registration) reuses the same Derkachi MP4
+ IMU CSV as FT-P-02 but consumes the FDR archive differently. With the
framework in place, the new builder is purely a config factory:

```python
# e2e/fixtures/sitl_replay_builder/build_p04_fixtures.py  (sketch)
from dataclasses import dataclass
from pathlib import Path
from e2e.fixtures.sitl_replay_builder.builder import (
    DEFAULT_CLI_BIN,
    FixtureBuilderConfig,
    ImuCsvTlog,
    Mp4PassthroughSource,
    RawFdrPassthrough,
    build_fixtures,
)


@dataclass(frozen=True)
class P04BuilderConfig:
    derkachi_dir: Path
    output_dir: Path
    fc_kind: str = "ardupilot"
    host: str = "sitl-host"


def build_p04_fixtures(cfg, **deps):
    mp4 = cfg.derkachi_dir / "flight_derkachi.mp4"
    csv_path = cfg.derkachi_dir / "data_imu.csv"
    builder_cfg = FixtureBuilderConfig(
        video_source=Mp4PassthroughSource(mp4_path=mp4),
        tlog_source=ImuCsvTlog(csv_path=csv_path),
        fdr_projection=RawFdrPassthrough(verify_estimates=True),
        output_dir=cfg.output_dir,
        fc_kind=cfg.fc_kind, host=cfg.host,
        tlog_filename="derkachi.tlog", fdr_subdir="fdr",
    )
    return build_fixtures(builder_cfg, **deps)
```

Total new code: ~30 lines + argparse CLI. No new strategy class is needed
because every Derkachi-based scenario consumes the same `Mp4PassthroughSource +
ImuCsvTlog + RawFdrPassthrough` triple. A scenario that emits a *new* fixture
shape (e.g. FT-P-13's "anchor-search-region" record extraction) writes a new
`FdrProjection` subclass alongside.

## Per-scenario usage

### FT-P-01

```bash
python -m e2e.fixtures.sitl_replay_builder.build_p01_fixtures \
  --input-dir _docs/00_problem/input_data \
  --output-dir e2e/fixtures/sitl_replay/p01 \
  --fc-kind ardupilot \
  --host sitl-host
```

Activation:

```bash
E2E_SITL_REPLAY_DIR=e2e/fixtures/sitl_replay/p01 \
    pytest e2e/tests/positive/test_ft_p_01_still_image_accuracy.py
```

### FT-P-02

```bash
python -m e2e.fixtures.sitl_replay_builder.build_p02_fixtures \
  --derkachi-dir _docs/00_problem/input_data/flight_derkachi \
  --output-dir e2e/fixtures/sitl_replay/p02 \
  --fc-kind ardupilot \
  --host sitl-host
```

## Limitations

* The synthesised ATTITUDE has roll/pitch = 0 — acceptable for fixed-wing
  cruise but unrealistic for aggressive manoeuvres. Override the packer call
  inside a custom `TlogSource` when needed.
* RAW_IMU is packed from `SCALED_IMU2` columns as pass-through (no true
  scaled → raw unit conversion). If the SUT's tlog parser strictly demands
  true raw counts the builder will need a units conversion pass — surfaced
  as a follow-up after the first live run.
* Auto-sync (`time_offset_ms != 0`) is bypassed by every scenario currently;
  operators running this against truly independent tlog+video pairs should
  override `FixtureBuilderConfig.time_offset_ms`.
* iNav adapter is NOT supported by the existing builders — ArduPilot only.
* The FDR record `kind`/`record_type` schemas are assumed to match the
  production contract; overrides live on each projection class.

## Testing

Unit tests under `e2e/_unit_tests/fixtures/`:

* `test_sitl_replay_builder_builder.py` — strategy-level tests
  (`VideoSource`, `TlogSource`, `FdrProjection` impls + shared helpers +
  `build_fixtures` orchestrator).
* `test_sitl_replay_builder.py` — FT-P-01 scenario integration.
* `test_sitl_replay_builder_p02.py` — FT-P-02 scenario integration.

All external dependencies (OpenCV, pymavlink, subprocess) are mocked via
the underscore-prefixed `_runner` / `_video_writer_factory` / `_imread` /
`_mavlink_writer_factory` injection points so the suite runs without a
real `gps-denied-replay` install. The actual end-to-end run requires the
SUT to be installed (`pip install -e .` at repo root) and is documented as
a manual step until CI infrastructure catches up.