gps-denied-onboard/_docs/02_tasks/todo/AZ-405_replay_auto_sync.md

Replay — Auto-sync video↔tlog via IMU take-off detection (AC-7 / AC-8)

Task: AZ-405_replay_auto_sync Name: Auto-sync of video ↔ tlog via IMU take-off detection (AC-7 / AC-8; --time-offset-ms remains the manual override) Description: Implement auto-detection of the video↔tlog timestamp offset for the replay CLI, mitigating R-DEMO-1 (recordings are often started independently — camera and FC may be minutes apart). Algorithm: (1) parse the tlog for the IMU take-off pattern — sustained vertical accel > 0.5 g for ≥ 0.5 s + change in attitude rate > 1 rad/s in the same window (typical quadcopter take-off signature); compute tlog_takeoff_ns. (2) Analyse the video for motion-onset — pyramidal optical flow magnitude crossing a configurable threshold sustained for ≥ 0.5 s; compute video_motion_onset_ns. (3) Offset = tlog_takeoff_ns - video_motion_onset_ns (positive offset = video starts before take-off recorded in tlog). Confidence-scoring: confidence is high (≥ 80 %) when both signals are well-defined; low when ambiguous (e.g., fixed-wing hand-launch — no clear vertical-accel-above-0.5g pulse). If confidence < 80 %, log WARN + use the best-guess offset and proceed. --time-offset-ms always overrides auto-detect (manual override per AC-7). AC-8 hard-fail (exit code 2): if the resulting offset produces ≤ 95 % of frames matching at least one IMU window within ± 100 ms, the CLI exits with code 2 and prints both the auto-detected offset (if any) and the per-frame match percentage so the operator can debug. Complexity: 5 points Dependencies: AZ-402 (CLI hosts the auto-sync logic at startup); AZ-399 (tlog parser); AZ-398 (VideoFileFrameSource for video-side analysis); AZ-263, AZ-269, AZ-266, AZ-272 (FDR for confidence + decision logging) Component: replay-auto-sync (epic AZ-265 / E-DEMO-REPLAY) — auto-sync helper at src/gps_denied_onboard/cli/replay_auto_sync.py Tracker: AZ-405 Epic: AZ-265 (E-DEMO-REPLAY)

Document Dependencies

• _docs/02_document/contracts/replay/replay_protocol.md — time_offset_ms semantics (Invariant 8).
• _docs/02_document/architecture.md — R-DEMO-1 mitigation.
• Epic AZ-265 description in _docs/02_document/epics.md — AC-7 / AC-8.

Problem

Without this task, the replay CLI relies on the operator passing --time-offset-ms N manually, which is error-prone (operators often don't have a stopwatch on the moment of take-off; the camera and FC are routinely started at different times). R-DEMO-1 is a recurring real-world concern. AC-7 / AC-8 codify the auto-sync expectation.

Outcome

• src/gps_denied_onboard/cli/replay_auto_sync.py:
  • detect_tlog_takeoff(tlog_path, target_fc_dialect) -> AutoSyncResult — returns (tlog_takeoff_ns, confidence).
  • detect_video_motion_onset(video_path, frame_rate_hz) -> AutoSyncResult — returns (video_motion_onset_ns, confidence).
  • compute_offset(tlog_result, video_result) -> AutoSyncOffset — combines the two; emits final confidence + offset.
  • validate_offset_or_fail(offset, tlog_path, video_path, ...) -> int — runs the AC-8 frame-window match-percentage check; returns 0 if ≥ 95 %, 2 otherwise (caller maps to CLI exit code).
• CLI wiring (in cli/replay.py): when --time-offset-ms is NOT provided, the CLI invokes detect_* + compute_offset + validate_offset_or_fail; if validation returns 2, the CLI exits 2 with the diagnostic message per AC-8.
• INFO log on auto-detect success: kind="replay.auto_sync.detected" with {tlog_takeoff_ns, video_motion_onset_ns, offset_ms, tlog_confidence, video_confidence, combined_confidence}.
• WARN log on low confidence: kind="replay.auto_sync.low_confidence" with the same fields + proceeding_with_best_guess: true.
• ERROR log on AC-8 fail: kind="replay.auto_sync.ac8_validation_failed" with {frame_window_match_pct, threshold_pct: 95.0}.
• FDR records mirror all three log kinds.
• Unit tests: tlog-takeoff detector against synthetic IMU traces (positive case + ambiguous case + hand-launch case); video-motion detector against synthetic video frames; combined offset within tolerance for synchronised inputs; AC-8 validation hard-fails on degenerate offsets.

Scope

Included

• Tlog-takeoff detector (sustained vertical accel + attitude rate).
• Video-motion-onset detector (pyramidal optical flow).
• Combined offset computation + confidence.
• AC-8 frame-window match-percentage validator.
• CLI wiring at startup.
• Manual override (--time-offset-ms) bypass path.
• Structured logging + FDR.
• Unit tests covering positive / ambiguous / hand-launch / hard-fail cases.

Excluded

• E2E test against the Derkachi fixture — owned by E2E task (this task ships unit tests; E2E task adds an integration assertion AC-7 / AC-8).
• The CLI argparse + entrypoint — owned by CLI task.
• Modifications to TlogReplayFcAdapter — this task consumes the adapter's tlog stream and the FrameSource's video frames; no API changes.

Acceptance Criteria

AC-1: Tlog take-off detector positive — synthetic AP IMU trace with a clear take-off (sustained 1.2 g vertical for 1 s + 1.5 rad/s attitude rate) → tlog_takeoff_ns matches the synthetic onset within ± 50 ms; confidence ≥ 0.85.

AC-2: Tlog take-off detector ambiguous — synthetic IMU with low-amplitude vibration (0.3 g) but no take-off → confidence < 0.50.

AC-3: Tlog take-off detector hand-launch — synthetic IMU with abrupt 0.8 g impulse but no sustained climb → confidence < 0.80 (in the WARN-and-proceed regime per AC-7).

AC-4: Video motion-onset positive — synthetic 60-frame video with first 10 frames stationary and frames 11+ moving → video_motion_onset_ns matches the onset of frame 11 within ± 1 frame.

AC-5: Combined offset within ± 200 ms (epic AC-7) — for a fixture with KNOWN ground-truth offset (e.g., constructed test case offset = 5000 ms), compute_offset returns within ± 200 ms of ground truth.

AC-6: Low combined confidence WARN-and-proceed — when combined_confidence < 0.80, compute_offset returns the best-guess offset + WARN log; the CLI proceeds (does NOT exit) — verified via the unit test of the CLI wiring.

AC-7: AC-8 hard-fail exit 2 — wire a validate_offset_or_fail against a deliberately-bad offset (e.g., 60 s offset on a 60 s clip — every frame would be off the tlog window); function returns 2; CLI exit code 2; ERROR log + FDR fired.

AC-8: Manual override bypasses auto-detect — --time-offset-ms 5000 passed → auto-detect functions are NOT invoked (verified via call-count assertion); the manual offset flows directly into TlogReplayFcAdapter.

AC-9: Frame-window match-percentage validator — for a known-good offset, validator computes ≥ 95 % match (returns 0); for a known-bad offset, computes ≤ 95 % (returns 2). Threshold is configurable via config.replay.auto_sync_match_threshold_pct (default 95.0).

AC-10: Confidence-score determinism — re-run the auto-sync against the same input twice; assert confidence values match within 1e-9 (algorithmic determinism).

Non-Functional Requirements

• Auto-sync startup overhead p99 ≤ 3 s (within the epic's cold-start ≤ 5 s budget combined with composition).
• Tlog-takeoff detection: full tlog scan ≤ 1 s for tlogs up to 100 MB (typical 1–2 min clip is ~10 MB).
• Video-motion-onset detection: scan the first 10 s of the video; ≤ 1 s on Tier-1 hardware.

Constraints

• OpenCV (already in deps for video) is the optical flow library.
• pymavlink (already bundled per D-C8-3) is the tlog reader.
• The take-off pattern thresholds (0.5 g, 1 rad/s, 0.5 s sustained) are in config.replay.auto_sync.takeoff_* with documented defaults.
• The video-motion threshold is similarly configurable.
• AC-8's 95 % match threshold is configurable per config.replay.auto_sync_match_threshold_pct.

Risks & Mitigation

• R-DEMO-1 (drift / unsynchronised recordings) — Mitigation: this task IS the mitigation; AC-1..AC-5 cover the positive cases; AC-6 covers the WARN-and-proceed regime; AC-8 covers the hard-fail regime.
• Risk: optical-flow false-positives on jitter-only video — Mitigation: configurable threshold; sustained-for-0.5 s requirement matches the take-off semantics; AC-2 covers the ambiguous case.
• Risk: fixed-wing hand-launch hits the WARN regime even on legitimate footage — Mitigation: documented; operator can pass --time-offset-ms manually; AC-3 documents the expected confidence drop.
• Risk: AC-8 95 % threshold too strict for short clips with sparse IMU — Mitigation: threshold is configurable; default 95 % is calibrated for typical tlog rates (50–200 Hz IMU).

Runtime Completeness

• Named capability: video↔tlog auto-sync via IMU take-off detection.
• Production code: real OpenCV optical flow, real pymavlink tlog scan, real confidence-scored combined offset, real AC-8 validator.
• Allowed external stubs: test fakes only.
• Unacceptable substitutes: a hardcoded time_offset_ms = 0 default (defeats R-DEMO-1 mitigation).

Contract

Implements epic AZ-265 ACs 7 + 8; mitigates R-DEMO-1.