[AZ-265] Replay as configuration of airborne binary (ADR-011)

Re-design replay mode per user direction: replay is no longer a fourth
Docker image with a reduced component set, but a `config.mode = "replay"`
branch of the single airborne binary. The pre-flight workflow (route in
suite UI -> C12 tile download via real satellite-provider -> C10
manifest+engines build) is identical between live and replay; only three
strategies swap at compose time:

  FrameSource:      Live <-> Video
  FcAdapter:        Pymavlink/MSP2 <-> TlogReplay
  MavlinkTransport: Serial <-> Noop

The C8 outbound MAVLink encoders run unchanged in both modes; their
bytes hit `NoopMavlinkTransport` in replay and disappear. A new
`JsonlReplaySink` taps C5's `EstimatorOutput` stream so the parent-suite
UI sees per-tick coordinates by tailing `results.jsonl`. MAVLink 2.0
signing key remains mandatory (operator supplies a dummy file).

A new `replay_input/` Layer-4 cross-cutting coordinator owns
`(video, tlog) -> (FrameSource, FcAdapter, Clock)` convergence; the
composition root sees only standard interfaces past `.open()`.

Docs:
- architecture.md: new ADR-011 with full rationale; ADR-002 binary
  narrative updated.
- contracts/replay/replay_protocol.md: bumped to v2.0.0; 12 invariants
  (notably mode-agnosticism + encoder byte-equality + signing key
  mandatory + real C6 cache in replay).
- module-layout.md: Build-Time Exclusion Map dropped from 4 to 3 binary
  columns; replay-mode `BUILD_*` flags default ON in airborne;
  `shared/replay_input` cross-cutting entry added.
- epics.md: E-DEMO-REPLAY scope reframed; story points 27-32 -> 19-24.

Task respecs:
- AZ-401: shrunk 3 -> 2 pts; `compose_root` mode branch + JSONL sink +
  NoopMavlinkTransport wiring; legacy `compose_replay` export deleted.
- AZ-402: console-script wrapper that mutates `config.mode = "replay"`
  and dispatches into the shared airborne main; `--mavlink-signing-key`
  mandatory.
- AZ-403: CANCELLED. Moved to done/ with banner; Jira transition deferred
  via `_docs/_process_leftovers/2026-05-14_az_403_cancellation_pending_tracker.md`.
- AZ-404: AC-4 reworded as mode-agnosticism AST scan + encoder
  byte-equality test; new AC-8 operator-workflow rehearsal.
- AZ-405: also owns the `replay_input/` module + `ReplayInputAdapter`.

_dependencies_table.md updated: AZ-401 gains AZ-405 dep; AZ-404 drops
AZ-403 dep; AZ-403 row marked CANCELLED.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-14 09:01:04 +03:00
parent fa3742d582
commit 5adf3dd04f
13 changed files with 765 additions and 418 deletions
+85 -31
View File
@@ -1,54 +1,99 @@
# Replay — Auto-sync video↔tlog via IMU take-off detection (AC-7 / AC-8)
# Replay — `replay_input/` coordinator + auto-sync video↔tlog via IMU take-off detection
**Task**: AZ-405_replay_auto_sync
**Name**: Auto-sync of videotlog via IMU take-off detection (AC-7 / AC-8; `--time-offset-ms` remains the manual override)
**Description**: Implement auto-detection of the video↔tlog timestamp offset for the replay CLI, mitigating R-DEMO-1 (recordings are often started independently — camera and FC may be minutes apart). Algorithm: (1) parse the tlog for the IMU take-off pattern — sustained vertical accel > 0.5 g for ≥ 0.5 s + change in attitude rate > 1 rad/s in the same window (typical quadcopter take-off signature); compute `tlog_takeoff_ns`. (2) Analyse the video for motion-onset — pyramidal optical flow magnitude crossing a configurable threshold sustained for ≥ 0.5 s; compute `video_motion_onset_ns`. (3) Offset = `tlog_takeoff_ns - video_motion_onset_ns` (positive offset = video starts before take-off recorded in tlog). Confidence-scoring: confidence is high (≥ 80 %) when both signals are well-defined; low when ambiguous (e.g., fixed-wing hand-launch — no clear vertical-accel-above-0.5g pulse). If confidence < 80 %, log WARN + use the best-guess offset and proceed. `--time-offset-ms` always overrides auto-detect (manual override per AC-7). AC-8 hard-fail (exit code 2): if the resulting offset produces ≤ 95 % of frames matching at least one IMU window within ± 100 ms, the CLI exits with code 2 and prints both the auto-detected offset (if any) and the per-frame match percentage so the operator can debug.
**Complexity**: 5 points
**Dependencies**: AZ-402 (CLI hosts the auto-sync logic at startup); AZ-399 (tlog parser); AZ-398 (VideoFileFrameSource for video-side analysis); AZ-263, AZ-269, AZ-266, AZ-272 (FDR for confidence + decision logging)
**Component**: replay-auto-sync (epic AZ-265 / E-DEMO-REPLAY) — auto-sync helper at `src/gps_denied_onboard/cli/replay_auto_sync.py`
**Name**: `replay_input/` Layer-4 cross-cutting coordinator (`ReplayInputAdapter`) + auto-sync of videotlog timestamp offset via IMU take-off detection (AC-7 / AC-8; `--time-offset-ms` is the manual override)
**Description**: Per ADR-011, replay is a configuration of the airborne binary; the architectural integration point is the new `replay_input/` Layer-4 cross-cutting module that converges `(video, tlog)` inputs into the standard `FrameSource` + `FcAdapter` + `Clock` surfaces the composition root already consumes. This task creates the `replay_input/` module and owns the time-alignment concern inside it (auto-sync + manual offset application).
The module:
1. Hosts the `ReplayInputAdapter` class in `src/gps_denied_onboard/replay_input/tlog_video_adapter.py` (public re-export in `__init__.py`). Constructor takes `(video_path, tlog_path, camera_calibration, target_fc_dialect, wgs_converter, pace, manual_time_offset_ms, auto_sync_config)`. `.open()` resolves the time-offset (auto-sync OR manual override), instantiates `VideoFileFrameSource` + `TlogReplayFcAdapter` + chosen `Clock` (`TlogDerivedClock` for pace=ASAP; `WallClock` for pace=REALTIME), and returns a `ReplayInputBundle(frame_source, fc_adapter, clock, resolved_time_offset_ms, auto_sync_result)` for the composition root to wire.
2. Hosts the auto-sync logic in `src/gps_denied_onboard/replay_input/auto_sync.py`:
- `detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult` — parses the tlog for the IMU take-off pattern (sustained vertical accel > 0.5 g for ≥ 0.5 s + change in attitude rate > 1 rad/s in the same window — typical quadcopter take-off signature); returns `(tlog_takeoff_ns, confidence)`.
- `detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult` — analyses the video for motion-onset via pyramidal optical flow magnitude crossing a configurable threshold sustained for ≥ 0.5 s; returns `(video_motion_onset_ns, confidence)`.
- `compute_offset(tlog_result, video_result) -> AutoSyncOffset` — combines the two; offset = `tlog_takeoff_ns - video_motion_onset_ns` (positive offset = video starts before take-off recorded in tlog); confidence = combined.
- `validate_offset_or_fail(offset, tlog_path, video_path, frame_rate_hz, threshold_pct) -> int` — runs the AC-8 frame-window match-percentage check: for each video frame, find the nearest IMU window within ± 100 ms after applying the offset; return 0 if ≥ 95 % of frames have a match, 2 otherwise.
3. Confidence-scoring: confidence is high (≥ 80 %) when both signals are well-defined; low when ambiguous (e.g., fixed-wing hand-launch — no clear vertical-accel-above-0.5g pulse). If combined confidence < 80 %, `ReplayInputAdapter.open()` logs WARN + uses the best-guess offset and proceeds. `manual_time_offset_ms is not None` always overrides auto-detect.
4. AC-8 hard-fail: if `validate_offset_or_fail` returns 2 (either after auto-sync OR after manual override), `ReplayInputAdapter.open()` raises `ReplayInputAdapterError("auto-sync hard-fail: …")` which the shared main maps to CLI exit code 2.
The composition root's replay-mode branch (AZ-401) instantiates `ReplayInputAdapter`, calls `.open()`, and consumes the returned bundle. No replay-aware code lives outside this module + AZ-400's transport seam + AZ-401's composition-root branch.
**Complexity**: 5 points (unchanged from v1.0.0 — same algorithmic work; the coordinator class is a small addition since it just instantiates strategies the algorithm already needs).
**Dependencies**: AZ-402 (CLI provides the args that feed `ReplayInputAdapter`); AZ-399 (`TlogReplayFcAdapter` is instantiated by `ReplayInputAdapter.open()`); AZ-398 (`VideoFileFrameSource` + `Clock` strategies are instantiated by `ReplayInputAdapter.open()`); AZ-279 (`WgsConverter` constructor-injected); AZ-263 (`runtime_root` bootstrap); AZ-269 / AZ-270 (`Config.replay.auto_sync` sub-config); AZ-266 (logging); AZ-272 (FDR record schema for confidence + decision logging).
**Component**: replay-input (epic AZ-265 / E-DEMO-REPLAY) — module at `src/gps_denied_onboard/replay_input/`.
**Tracker**: AZ-405
**Epic**: AZ-265 (E-DEMO-REPLAY)
### Document Dependencies
- `_docs/02_document/contracts/replay/replay_protocol.md` `time_offset_ms` semantics (Invariant 8).
- `_docs/02_document/architecture.md` — R-DEMO-1 mitigation.
- Epic AZ-265 description in `_docs/02_document/epics.md` — AC-7 / AC-8.
- `_docs/02_document/contracts/replay/replay_protocol.md` (v2.0.0) — `ReplayInputAdapter` API; `time_offset_ms` semantics (Invariant 8).
- `_docs/02_document/architecture.md` **ADR-011** (replay-as-configuration; ReplayInputAdapter is the architectural seam between (video, tlog) and the rest of the system) + R-DEMO-1 mitigation.
- `_docs/02_document/module-layout.md``shared/replay_input` cross-cutting entry.
- Epic AZ-265 description in `_docs/02_document/epics.md` — AC-7 / AC-8 / AC-9 / AC-10.
## Problem
Without this task, the replay CLI relies on the operator passing `--time-offset-ms N` manually, which is error-prone (operators often don't have a stopwatch on the moment of take-off; the camera and FC are routinely started at different times). R-DEMO-1 is a recurring real-world concern. AC-7 / AC-8 codify the auto-sync expectation.
Two problems:
1. **Without `replay_input/`** there is no module-level home for the `(video, tlog)``(FrameSource, FcAdapter, Clock)` convergence; the composition root would need to instantiate each strategy individually + know about auto-sync + apply the manual override — all replay-specific code leaking into `compose_root`. Per ADR-011 the composition root should see only standard `FrameSource` + `FcAdapter` + `Clock` instances after the coordinator is opened; this task creates the coordinator.
2. **Without auto-sync** the replay CLI relies on the operator passing `--time-offset-ms N` manually, which is error-prone (operators often don't have a stopwatch on the moment of take-off; the camera and FC are routinely started at different times). R-DEMO-1 is a recurring real-world concern. AC-7 / AC-8 codify the auto-sync expectation.
## Outcome
- `src/gps_denied_onboard/cli/replay_auto_sync.py`:
- `detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult` — returns `(tlog_takeoff_ns, confidence)`.
- `detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult` — returns `(video_motion_onset_ns, confidence)`.
- `compute_offset(tlog_result, video_result) -> AutoSyncOffset` — combines the two; emits final confidence + offset.
- `validate_offset_or_fail(offset, tlog_path, video_path, ...) -> int` — runs the AC-8 frame-window match-percentage check; returns 0 if ≥ 95 %, 2 otherwise (caller maps to CLI exit code).
- CLI wiring (in `cli/replay.py`): when `--time-offset-ms` is NOT provided, the CLI invokes `detect_*` + `compute_offset` + `validate_offset_or_fail`; if validation returns 2, the CLI exits 2 with the diagnostic message per AC-8.
- `src/gps_denied_onboard/replay_input/__init__.py`:
- Re-exports `ReplayInputAdapter`, `ReplayInputBundle`, `AutoSyncDecision`, `AutoSyncConfig`, `ReplayInputAdapterError`.
- `src/gps_denied_onboard/replay_input/interface.py`:
- `ReplayInputBundle` frozen+slots dataclass.
- `AutoSyncDecision` frozen+slots dataclass.
- `AutoSyncConfig` frozen+slots dataclass (defaults + thresholds).
- `src/gps_denied_onboard/replay_input/tlog_video_adapter.py`:
- `ReplayInputAdapter` class with `open()` + `close()` (idempotent close).
- Inside `open()`: resolve time-offset (auto-sync OR manual) → instantiate strategies → return bundle.
- Fails fast if required tlog message types absent (R-DEMO-3); raises `ReplayInputAdapterError("tlog missing required message types: ...")`.
- `src/gps_denied_onboard/replay_input/auto_sync.py`:
- `detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult` — pymavlink stream-parse; sustained vertical-accel + attitude-rate detector.
- `detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult` — OpenCV pyramidal optical flow.
- `compute_offset(tlog_result, video_result) -> AutoSyncOffset` — combination + confidence.
- `validate_offset_or_fail(offset, tlog_path, video_path, frame_rate_hz, threshold_pct) -> int` — AC-8 validator.
- `src/gps_denied_onboard/replay_input/tests/` — unit tests:
- `test_tlog_takeoff_detector_positive` (AC-1).
- `test_tlog_takeoff_detector_ambiguous` (AC-2).
- `test_tlog_takeoff_detector_hand_launch` (AC-3).
- `test_video_motion_onset_positive` (AC-4).
- `test_combined_offset_within_200ms` (AC-5).
- `test_combined_offset_low_confidence_warn_and_proceed` (AC-6).
- `test_ac8_validator_hard_fail` (AC-7).
- `test_manual_override_bypasses_auto_detect` (AC-8).
- `test_frame_window_match_validator_threshold` (AC-9).
- `test_confidence_score_deterministic` (AC-10).
- `test_replay_input_adapter_open_returns_bundle` (covers the coordinator wiring; AC-11 below).
- `test_replay_input_adapter_clock_strategy_pace_asap` (TlogDerivedClock).
- `test_replay_input_adapter_clock_strategy_pace_realtime` (WallClock).
- `test_replay_input_adapter_close_idempotent`.
- `test_replay_input_adapter_missing_tlog_messages_fails_fast` (R-DEMO-3).
- INFO log on auto-detect success: `kind="replay.auto_sync.detected"` with `{tlog_takeoff_ns, video_motion_onset_ns, offset_ms, tlog_confidence, video_confidence, combined_confidence}`.
- WARN log on low confidence: `kind="replay.auto_sync.low_confidence"` with the same fields + `proceeding_with_best_guess: true`.
- ERROR log on AC-8 fail: `kind="replay.auto_sync.ac8_validation_failed"` with `{frame_window_match_pct, threshold_pct: 95.0}`.
- FDR records mirror all three log kinds.
- Unit tests: tlog-takeoff detector against synthetic IMU traces (positive case + ambiguous case + hand-launch case); video-motion detector against synthetic video frames; combined offset within tolerance for synchronised inputs; AC-8 validation hard-fails on degenerate offsets.
## Scope
### Included
- `replay_input/` module structure (`__init__.py`, `interface.py`, `tlog_video_adapter.py`, `auto_sync.py`, `tests/`).
- `ReplayInputAdapter` class with `open()` + `close()`.
- Tlog-takeoff detector (sustained vertical accel + attitude rate).
- Video-motion-onset detector (pyramidal optical flow).
- Combined offset computation + confidence.
- AC-8 frame-window match-percentage validator.
- CLI wiring at startup.
- Manual override (`--time-offset-ms`) bypass path.
- Manual override (`manual_time_offset_ms is not None`) bypass path.
- Structured logging + FDR.
- Unit tests covering positive / ambiguous / hand-launch / hard-fail cases.
- All unit tests listed above.
### Excluded
- E2E test against the Derkachi fixture — owned by E2E task (this task ships unit tests; E2E task adds an integration assertion AC-7 / AC-8).
- The CLI argparse + entrypoint — owned by CLI task.
- Modifications to `TlogReplayFcAdapter` — this task consumes the adapter's tlog stream and the FrameSource's video frames; no API changes.
- E2E test against the Derkachi fixture — owned by AZ-404 (this task ships unit tests; AZ-404 adds the integration assertion AC-7 / AC-8 / AC-9).
- The CLI argparse + entrypoint — owned by AZ-402.
- The composition root branch on `config.mode` — owned by AZ-401.
- `VideoFileFrameSource` + `Clock` strategies themselves — owned by AZ-398.
- `TlogReplayFcAdapter` itself — owned by AZ-399.
## Acceptance Criteria
@@ -62,16 +107,22 @@ Without this task, the replay CLI relies on the operator passing `--time-offset-
**AC-5: Combined offset within ± 200 ms (epic AC-7)** — for a fixture with KNOWN ground-truth offset (e.g., constructed test case offset = 5000 ms), `compute_offset` returns within ± 200 ms of ground truth.
**AC-6: Low combined confidence WARN-and-proceed** — when `combined_confidence < 0.80`, `compute_offset` returns the best-guess offset + WARN log; the CLI proceeds (does NOT exit) — verified via the unit test of the CLI wiring.
**AC-6: Low combined confidence WARN-and-proceed** — when `combined_confidence < 0.80`, `ReplayInputAdapter.open()` returns the bundle with the best-guess offset + WARN log; does NOT raise — verified via the unit test of the coordinator.
**AC-7: AC-8 hard-fail exit 2** — wire a `validate_offset_or_fail` against a deliberately-bad offset (e.g., 60 s offset on a 60 s clip — every frame would be off the tlog window); function returns 2; CLI exit code 2; ERROR log + FDR fired.
**AC-7: AC-8 hard-fail raises** — wire a `validate_offset_or_fail` against a deliberately-bad offset (e.g., 60 s offset on a 60 s clip — every frame would be off the tlog window); `ReplayInputAdapter.open()` raises `ReplayInputAdapterError("auto-sync hard-fail: …")` so the shared main maps to CLI exit code 2; ERROR log + FDR fired.
**AC-8: Manual override bypasses auto-detect**`--time-offset-ms 5000` passed → auto-detect functions are NOT invoked (verified via call-count assertion); the manual offset flows directly into `TlogReplayFcAdapter`.
**AC-8: Manual override bypasses auto-detect**`ReplayInputAdapter(manual_time_offset_ms=5000, …).open()``detect_*` and `compute_offset` are NOT invoked (verified via call-count assertion); the manual offset flows directly into `TlogReplayFcAdapter`. AC-8 validator still runs (so a wildly wrong manual offset still fails fast).
**AC-9: Frame-window match-percentage validator** — for a known-good offset, validator computes ≥ 95 % match (returns 0); for a known-bad offset, computes ≤ 95 % (returns 2). Threshold is configurable via `config.replay.auto_sync_match_threshold_pct` (default 95.0).
**AC-10: Confidence-score determinism** — re-run the auto-sync against the same input twice; assert confidence values match within 1e-9 (algorithmic determinism).
**AC-11: ReplayInputAdapter.open() returns a complete bundle**`bundle = adapter.open()` returns a `ReplayInputBundle` with `isinstance(bundle.frame_source, VideoFileFrameSource)`, `isinstance(bundle.fc_adapter, TlogReplayFcAdapter)`, and `bundle.clock` matching the pace (`TlogDerivedClock` for ASAP, `WallClock` for REALTIME). The `resolved_time_offset_ms` field equals either the manual override or the auto-sync result.
**AC-12: Close is idempotent**`adapter.open(); adapter.close(); adapter.close()` does not raise; the second close is a no-op.
**AC-13: Missing tlog messages fail fast** — open against a tlog missing `RAW_IMU` (AP) or `MSP2_RAW_IMU` (iNav); assert `ReplayInputAdapterError("tlog missing required message types: ['RAW_IMU']")` is raised inside `open()` BEFORE any video read (R-DEMO-3).
## Non-Functional Requirements
- Auto-sync startup overhead p99 ≤ 3 s (within the epic's cold-start ≤ 5 s budget combined with composition).
@@ -85,21 +136,24 @@ Without this task, the replay CLI relies on the operator passing `--time-offset-
- The take-off pattern thresholds (0.5 g, 1 rad/s, 0.5 s sustained) are in `config.replay.auto_sync.takeoff_*` with documented defaults.
- The video-motion threshold is similarly configurable.
- AC-8's 95 % match threshold is configurable per `config.replay.auto_sync_match_threshold_pct`.
- `ReplayInputAdapter` is a Layer-4 module (per `module-layout.md`); it imports from Layer 1 (`frame_source` interface, `clock` interface, `_types`, `config`, `logging`, `fdr_client`, `helpers.wgs_converter`) and instantiates Layer-4 strategies (`c8_fc_adapter.tlog_replay_adapter`, `frame_source.video_file_frame_source`); it does NOT import from Layer 3 (no component-level dependencies).
## Risks & Mitigation
- **R-DEMO-1 (drift / unsynchronised recordings)** — *Mitigation*: this task IS the mitigation; AC-1..AC-5 cover the positive cases; AC-6 covers the WARN-and-proceed regime; AC-8 covers the hard-fail regime.
- **R-DEMO-1 (drift / unsynchronised recordings)** — *Mitigation*: this task IS the mitigation; AC-1..AC-5 cover the positive cases; AC-6 covers the WARN-and-proceed regime; AC-7 covers the hard-fail regime.
- **R-DEMO-3 (demo footage missing required FC messages)** — *Mitigation*: AC-13 fails fast at startup with a clear message naming the missing types.
- **Risk: optical-flow false-positives on jitter-only video** — *Mitigation*: configurable threshold; sustained-for-0.5 s requirement matches the take-off semantics; AC-2 covers the ambiguous case.
- **Risk: fixed-wing hand-launch hits the WARN regime even on legitimate footage** — *Mitigation*: documented; operator can pass `--time-offset-ms` manually; AC-3 documents the expected confidence drop.
- **Risk: AC-8 95 % threshold too strict for short clips with sparse IMU** — *Mitigation*: threshold is configurable; default 95 % is calibrated for typical tlog rates (50200 Hz IMU).
- **Risk (new): the coordinator class adds a new architectural seam that might leak `if mode == replay` plumbing into `compose_root`** — *Mitigation*: AZ-401's AC-7 (AST scan) catches this; the coordinator's API surface (open() → bundle) is designed so the composition root sees only standard interfaces past `.open()`.
## Runtime Completeness
- **Named capability**: videotlog auto-sync via IMU take-off detection.
- **Production code**: real OpenCV optical flow, real pymavlink tlog scan, real confidence-scored combined offset, real AC-8 validator.
- **Named capability**: `replay_input/` Layer-4 coordinator that converges `(video, tlog)` into the standard `FrameSource` + `FcAdapter` + `Clock` surfaces, owning time-alignment between them.
- **Production code**: real OpenCV optical flow, real pymavlink tlog scan, real confidence-scored combined offset, real AC-8 validator, real strategy instantiation, real Clock-pace selection.
- **Allowed external stubs**: test fakes only.
- **Unacceptable substitutes**: a hardcoded `time_offset_ms = 0` default (defeats R-DEMO-1 mitigation).
- **Unacceptable substitutes**: a hardcoded `time_offset_ms = 0` default (defeats R-DEMO-1 mitigation); placing the coordinator inside `cli/replay.py` (defeats the Layer-4 separation and forces the CLI to know about strategy instantiation — that belongs in the composition root branch, which itself delegates to `replay_input/`).
## Contract
Implements epic AZ-265 ACs 7 + 8; mitigates R-DEMO-1.
Implements epic AZ-265 ACs 7 + 8; mitigates R-DEMO-1 + R-DEMO-3. Implements the `ReplayInputAdapter` surface specified in `_docs/02_document/contracts/replay/replay_protocol.md` (v2.0.0). Operationalises the `replay_input/` cross-cutting module from ADR-011.