gps-denied-onboard/_docs/02_tasks/todo/AZ-405_replay_auto_sync.md

Replay — replay_input/ coordinator + auto-sync video↔tlog via IMU take-off detection

Task: AZ-405_replay_auto_sync Name: replay_input/ Layer-4 cross-cutting coordinator (ReplayInputAdapter) + auto-sync of video↔tlog timestamp offset via IMU take-off detection (AC-7 / AC-8; --time-offset-ms is the manual override) Description: Per ADR-011, replay is a configuration of the airborne binary; the architectural integration point is the new replay_input/ Layer-4 cross-cutting module that converges (video, tlog) inputs into the standard FrameSource + FcAdapter + Clock surfaces the composition root already consumes. This task creates the replay_input/ module and owns the time-alignment concern inside it (auto-sync + manual offset application).

The module:

1. Hosts the ReplayInputAdapter class in src/gps_denied_onboard/replay_input/tlog_video_adapter.py (public re-export in __init__.py). Constructor takes (video_path, tlog_path, camera_calibration, target_fc_dialect, wgs_converter, pace, manual_time_offset_ms, auto_sync_config). .open() resolves the time-offset (auto-sync OR manual override), instantiates VideoFileFrameSource + TlogReplayFcAdapter + chosen Clock (TlogDerivedClock for pace=ASAP; WallClock for pace=REALTIME), and returns a ReplayInputBundle(frame_source, fc_adapter, clock, resolved_time_offset_ms, auto_sync_result) for the composition root to wire.
2. Hosts the auto-sync logic in src/gps_denied_onboard/replay_input/auto_sync.py:
   • detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult — parses the tlog for the IMU take-off pattern (sustained vertical accel > 0.5 g for ≥ 0.5 s + change in attitude rate > 1 rad/s in the same window — typical quadcopter take-off signature); returns (tlog_takeoff_ns, confidence).
   • detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult — analyses the video for motion-onset via pyramidal optical flow magnitude crossing a configurable threshold sustained for ≥ 0.5 s; returns (video_motion_onset_ns, confidence).
   • compute_offset(tlog_result, video_result) -> AutoSyncOffset — combines the two; offset = tlog_takeoff_ns - video_motion_onset_ns (positive offset = video starts before take-off recorded in tlog); confidence = combined.
   • validate_offset_or_fail(offset, tlog_path, video_path, frame_rate_hz, threshold_pct) -> int — runs the AC-8 frame-window match-percentage check: for each video frame, find the nearest IMU window within ± 100 ms after applying the offset; return 0 if ≥ 95 % of frames have a match, 2 otherwise.
3. Confidence-scoring: confidence is high (≥ 80 %) when both signals are well-defined; low when ambiguous (e.g., fixed-wing hand-launch — no clear vertical-accel-above-0.5g pulse). If combined confidence < 80 %, ReplayInputAdapter.open() logs WARN + uses the best-guess offset and proceeds. manual_time_offset_ms is not None always overrides auto-detect.
4. AC-8 hard-fail: if validate_offset_or_fail returns 2 (either after auto-sync OR after manual override), ReplayInputAdapter.open() raises ReplayInputAdapterError("auto-sync hard-fail: …") which the shared main maps to CLI exit code 2.

The composition root's replay-mode branch (AZ-401) instantiates ReplayInputAdapter, calls .open(), and consumes the returned bundle. No replay-aware code lives outside this module + AZ-400's transport seam + AZ-401's composition-root branch.

Complexity: 5 points (unchanged from v1.0.0 — same algorithmic work; the coordinator class is a small addition since it just instantiates strategies the algorithm already needs). Dependencies: AZ-402 (CLI provides the args that feed ReplayInputAdapter); AZ-399 (TlogReplayFcAdapter is instantiated by ReplayInputAdapter.open()); AZ-398 (VideoFileFrameSource + Clock strategies are instantiated by ReplayInputAdapter.open()); AZ-279 (WgsConverter constructor-injected); AZ-263 (runtime_root bootstrap); AZ-269 / AZ-270 (Config.replay.auto_sync sub-config); AZ-266 (logging); AZ-272 (FDR record schema for confidence + decision logging). Component: replay-input (epic AZ-265 / E-DEMO-REPLAY) — module at src/gps_denied_onboard/replay_input/. Tracker: AZ-405 Epic: AZ-265 (E-DEMO-REPLAY)

Document Dependencies

• _docs/02_document/contracts/replay/replay_protocol.md (v2.0.0) — ReplayInputAdapter API; time_offset_ms semantics (Invariant 8).
• _docs/02_document/architecture.md — ADR-011 (replay-as-configuration; ReplayInputAdapter is the architectural seam between (video, tlog) and the rest of the system) + R-DEMO-1 mitigation.
• _docs/02_document/module-layout.md — shared/replay_input cross-cutting entry.
• Epic AZ-265 description in _docs/02_document/epics.md — AC-7 / AC-8 / AC-9 / AC-10.

Problem

Two problems:

1. Without replay_input/ there is no module-level home for the (video, tlog) → (FrameSource, FcAdapter, Clock) convergence; the composition root would need to instantiate each strategy individually + know about auto-sync + apply the manual override — all replay-specific code leaking into compose_root. Per ADR-011 the composition root should see only standard FrameSource + FcAdapter + Clock instances after the coordinator is opened; this task creates the coordinator.
2. Without auto-sync the replay CLI relies on the operator passing --time-offset-ms N manually, which is error-prone (operators often don't have a stopwatch on the moment of take-off; the camera and FC are routinely started at different times). R-DEMO-1 is a recurring real-world concern. AC-7 / AC-8 codify the auto-sync expectation.

Outcome

• src/gps_denied_onboard/replay_input/__init__.py:
  • Re-exports ReplayInputAdapter, ReplayInputBundle, AutoSyncDecision, AutoSyncConfig, ReplayInputAdapterError.
• src/gps_denied_onboard/replay_input/interface.py:
  • ReplayInputBundle frozen+slots dataclass.
  • AutoSyncDecision frozen+slots dataclass.
  • AutoSyncConfig frozen+slots dataclass (defaults + thresholds).
• src/gps_denied_onboard/replay_input/tlog_video_adapter.py:
  • ReplayInputAdapter class with open() + close() (idempotent close).
  • Inside open(): resolve time-offset (auto-sync OR manual) → instantiate strategies → return bundle.
  • Fails fast if required tlog message types absent (R-DEMO-3); raises ReplayInputAdapterError("tlog missing required message types: ...").
• src/gps_denied_onboard/replay_input/auto_sync.py:
  • detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult — pymavlink stream-parse; sustained vertical-accel + attitude-rate detector.
  • detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult — OpenCV pyramidal optical flow.
  • compute_offset(tlog_result, video_result) -> AutoSyncOffset — combination + confidence.
  • validate_offset_or_fail(offset, tlog_path, video_path, frame_rate_hz, threshold_pct) -> int — AC-8 validator.
• src/gps_denied_onboard/replay_input/tests/ — unit tests:
  • test_tlog_takeoff_detector_positive (AC-1).
  • test_tlog_takeoff_detector_ambiguous (AC-2).
  • test_tlog_takeoff_detector_hand_launch (AC-3).
  • test_video_motion_onset_positive (AC-4).
  • test_combined_offset_within_200ms (AC-5).
  • test_combined_offset_low_confidence_warn_and_proceed (AC-6).
  • test_ac8_validator_hard_fail (AC-7).
  • test_manual_override_bypasses_auto_detect (AC-8).
  • test_frame_window_match_validator_threshold (AC-9).
  • test_confidence_score_deterministic (AC-10).
  • test_replay_input_adapter_open_returns_bundle (covers the coordinator wiring; AC-11 below).
  • test_replay_input_adapter_clock_strategy_pace_asap (TlogDerivedClock).
  • test_replay_input_adapter_clock_strategy_pace_realtime (WallClock).
  • test_replay_input_adapter_close_idempotent.
  • test_replay_input_adapter_missing_tlog_messages_fails_fast (R-DEMO-3).
• INFO log on auto-detect success: kind="replay.auto_sync.detected" with {tlog_takeoff_ns, video_motion_onset_ns, offset_ms, tlog_confidence, video_confidence, combined_confidence}.
• WARN log on low confidence: kind="replay.auto_sync.low_confidence" with the same fields + proceeding_with_best_guess: true.
• ERROR log on AC-8 fail: kind="replay.auto_sync.ac8_validation_failed" with {frame_window_match_pct, threshold_pct: 95.0}.
• FDR records mirror all three log kinds.

Scope

Included

• replay_input/ module structure (__init__.py, interface.py, tlog_video_adapter.py, auto_sync.py, tests/).
• ReplayInputAdapter class with open() + close().
• Tlog-takeoff detector (sustained vertical accel + attitude rate).
• Video-motion-onset detector (pyramidal optical flow).
• Combined offset computation + confidence.
• AC-8 frame-window match-percentage validator.
• Manual override (manual_time_offset_ms is not None) bypass path.
• Structured logging + FDR.
• All unit tests listed above.

Excluded

• E2E test against the Derkachi fixture — owned by AZ-404 (this task ships unit tests; AZ-404 adds the integration assertion AC-7 / AC-8 / AC-9).
• The CLI argparse + entrypoint — owned by AZ-402.
• The composition root branch on config.mode — owned by AZ-401.
• VideoFileFrameSource + Clock strategies themselves — owned by AZ-398.
• TlogReplayFcAdapter itself — owned by AZ-399.

Acceptance Criteria

AC-1: Tlog take-off detector positive — synthetic AP IMU trace with a clear take-off (sustained 1.2 g vertical for 1 s + 1.5 rad/s attitude rate) → tlog_takeoff_ns matches the synthetic onset within ± 50 ms; confidence ≥ 0.85.

AC-2: Tlog take-off detector ambiguous — synthetic IMU with low-amplitude vibration (0.3 g) but no take-off → confidence < 0.50.

AC-3: Tlog take-off detector hand-launch — synthetic IMU with abrupt 0.8 g impulse but no sustained climb → confidence < 0.80 (in the WARN-and-proceed regime per AC-7).

AC-4: Video motion-onset positive — synthetic 60-frame video with first 10 frames stationary and frames 11+ moving → video_motion_onset_ns matches the onset of frame 11 within ± 1 frame.

AC-5: Combined offset within ± 200 ms (epic AC-7) — for a fixture with KNOWN ground-truth offset (e.g., constructed test case offset = 5000 ms), compute_offset returns within ± 200 ms of ground truth.

AC-6: Low combined confidence WARN-and-proceed — when combined_confidence < 0.80, ReplayInputAdapter.open() returns the bundle with the best-guess offset + WARN log; does NOT raise — verified via the unit test of the coordinator.

AC-7: AC-8 hard-fail raises — wire a validate_offset_or_fail against a deliberately-bad offset (e.g., 60 s offset on a 60 s clip — every frame would be off the tlog window); ReplayInputAdapter.open() raises ReplayInputAdapterError("auto-sync hard-fail: …") so the shared main maps to CLI exit code 2; ERROR log + FDR fired.

AC-8: Manual override bypasses auto-detect — ReplayInputAdapter(manual_time_offset_ms=5000, …).open() → detect_* and compute_offset are NOT invoked (verified via call-count assertion); the manual offset flows directly into TlogReplayFcAdapter. AC-8 validator still runs (so a wildly wrong manual offset still fails fast).

AC-9: Frame-window match-percentage validator — for a known-good offset, validator computes ≥ 95 % match (returns 0); for a known-bad offset, computes ≤ 95 % (returns 2). Threshold is configurable via config.replay.auto_sync_match_threshold_pct (default 95.0).

AC-10: Confidence-score determinism — re-run the auto-sync against the same input twice; assert confidence values match within 1e-9 (algorithmic determinism).

AC-11: ReplayInputAdapter.open() returns a complete bundle — bundle = adapter.open() returns a ReplayInputBundle with isinstance(bundle.frame_source, VideoFileFrameSource), isinstance(bundle.fc_adapter, TlogReplayFcAdapter), and bundle.clock matching the pace (TlogDerivedClock for ASAP, WallClock for REALTIME). The resolved_time_offset_ms field equals either the manual override or the auto-sync result.

AC-12: Close is idempotent — adapter.open(); adapter.close(); adapter.close() does not raise; the second close is a no-op.

AC-13: Missing tlog messages fail fast — open against a tlog missing RAW_IMU (AP) or MSP2_RAW_IMU (iNav); assert ReplayInputAdapterError("tlog missing required message types: ['RAW_IMU']") is raised inside open() BEFORE any video read (R-DEMO-3).

Non-Functional Requirements

• Auto-sync startup overhead p99 ≤ 3 s (within the epic's cold-start ≤ 5 s budget combined with composition).
• Tlog-takeoff detection: full tlog scan ≤ 1 s for tlogs up to 100 MB (typical 1–2 min clip is ~10 MB).
• Video-motion-onset detection: scan the first 10 s of the video; ≤ 1 s on Tier-1 hardware.

Constraints

• OpenCV (already in deps for video) is the optical flow library.
• pymavlink (already bundled per D-C8-3) is the tlog reader.
• The take-off pattern thresholds (0.5 g, 1 rad/s, 0.5 s sustained) are in config.replay.auto_sync.takeoff_* with documented defaults.
• The video-motion threshold is similarly configurable.
• AC-8's 95 % match threshold is configurable per config.replay.auto_sync_match_threshold_pct.
• ReplayInputAdapter is a Layer-4 module (per module-layout.md); it imports from Layer 1 (frame_source interface, clock interface, _types, config, logging, fdr_client, helpers.wgs_converter) and instantiates Layer-4 strategies (c8_fc_adapter.tlog_replay_adapter, frame_source.video_file_frame_source); it does NOT import from Layer 3 (no component-level dependencies).

Risks & Mitigation

• R-DEMO-1 (drift / unsynchronised recordings) — Mitigation: this task IS the mitigation; AC-1..AC-5 cover the positive cases; AC-6 covers the WARN-and-proceed regime; AC-7 covers the hard-fail regime.
• R-DEMO-3 (demo footage missing required FC messages) — Mitigation: AC-13 fails fast at startup with a clear message naming the missing types.
• Risk: optical-flow false-positives on jitter-only video — Mitigation: configurable threshold; sustained-for-0.5 s requirement matches the take-off semantics; AC-2 covers the ambiguous case.
• Risk: fixed-wing hand-launch hits the WARN regime even on legitimate footage — Mitigation: documented; operator can pass --time-offset-ms manually; AC-3 documents the expected confidence drop.
• Risk: AC-8 95 % threshold too strict for short clips with sparse IMU — Mitigation: threshold is configurable; default 95 % is calibrated for typical tlog rates (50–200 Hz IMU).
• Risk (new): the coordinator class adds a new architectural seam that might leak if mode == replay plumbing into compose_root — Mitigation: AZ-401's AC-7 (AST scan) catches this; the coordinator's API surface (open() → bundle) is designed so the composition root sees only standard interfaces past .open().

Runtime Completeness

• Named capability: replay_input/ Layer-4 coordinator that converges (video, tlog) into the standard FrameSource + FcAdapter + Clock surfaces, owning time-alignment between them.
• Production code: real OpenCV optical flow, real pymavlink tlog scan, real confidence-scored combined offset, real AC-8 validator, real strategy instantiation, real Clock-pace selection.
• Allowed external stubs: test fakes only.
• Unacceptable substitutes: a hardcoded time_offset_ms = 0 default (defeats R-DEMO-1 mitigation); placing the coordinator inside cli/replay.py (defeats the Layer-4 separation and forces the CLI to know about strategy instantiation — that belongs in the composition root branch, which itself delegates to replay_input/).

Contract

Implements epic AZ-265 ACs 7 + 8; mitigates R-DEMO-1 + R-DEMO-3. Implements the ReplayInputAdapter surface specified in _docs/02_document/contracts/replay/replay_protocol.md (v2.0.0). Operationalises the replay_input/ cross-cutting module from ADR-011.