From e054a55804a6796603dbd97d193225d85100910c Mon Sep 17 00:00:00 2001 From: Oleksandr Bezdieniezhnykh Date: Mon, 18 May 2026 09:50:11 +0300 Subject: [PATCH] [AZ-611] [AZ-614] [AZ-618] Step-11 Cycle-3 report + autodev state Cycle-3 addendum captures the layered Jetson rerun progression: synth time-base fix (AZ-614) drops offset_ms from 1.7e12 to -4334; AZ-611 skip-auto-sync then crosses the AC-9 validator; AZ-602 build-flag completeness opens VideoFileFrameSource and TlogReplayFcAdapter; composition root logs 'replay.compose_root.ready: auto_sync_used=false', then crashes inside runtime_root.airborne_bootstrap because production main() never builds c13_fdr / c6_* / c7_inference / c3_lightglue_runtime / c3_feature_extractor / c2_82_ransac_filter into pre_constructed. The bootstrap gap is filed as AZ-618 (Story under AZ-602). It affects both live and replay binaries -- every prior Reality-Gate run died at auto-sync before the composition graph was walked, so the gap was hidden. The 38 compose_root unit tests pass only via the replay_components_factory stub kwarg, which bypasses the bootstrap entirely. Autodev sub_step advances to phase 8 'az614-az611-landed-bootstrap-gap-discovered' pending the user's decision on whether to start AZ-618 immediately or close out Step 11 with the current Reality-Gate signal. Co-authored-by: Cursor --- .../run_tests_step11_report.md | 92 +++++++++++++++++++ _docs/_autodev_state.md | 6 +- 2 files changed, 95 insertions(+), 3 deletions(-) diff --git a/_docs/03_implementation/run_tests_step11_report.md b/_docs/03_implementation/run_tests_step11_report.md index 2ad76b6..6140a8e 100644 --- a/_docs/03_implementation/run_tests_step11_report.md +++ b/_docs/03_implementation/run_tests_step11_report.md @@ -386,3 +386,95 @@ stages they entail). * git LFS-tracked fixtures (the 269 MB Derkachi mp4) must be pre-smudged on the Mac BEFORE the rsync step; otherwise the Jetson receives the 134 B pointer and tests fail at fixture-load. + +## Cycle-3 addendum: AZ-614 + AZ-611 landed; next blocker is airborne bootstrap (AZ-618) + +Date: 2026-05-18. Track-2 Reality-Gate work continued on Jetson with +two SUT-side fixes layered on top of the AZ-615 harness. + +### Commits landed this cycle + +| Commit | Ticket | What | +|--------|--------|------| +| `e114bfd` | AZ-614 | `_TLOG_BASE_TIMESTAMP_US = 0` so the synth tlog shares the video's t=0 anchor | +| `bd41956` | AZ-611 | `--skip-auto-sync` CLI flag + `ReplayConfig.skip_auto_sync_validation` field + 5 unit tests | +| `324bbd6` | AZ-602 | `docker-compose.test.yml` + `docker-compose.test.jetson.yml`: set all three replay `BUILD_*` flags | +| `b7012d2` | AZ-615 | `scripts/run-tests-jetson.sh`: resolve `~` against remote `$HOME` before the heredoc cd | + +### Jetson rerun progression + +Each rerun isolated one fix to keep the diagnostic signal clean. + +| Rerun | Run scope | `tlog_takeoff_ns` | resolved `offset_ms` | Failure layer | +|-------|-----------|-------------------|----------------------|---------------| +| Pre-fix (`915484.txt`) | AZ-614 unverified | `1.7e18` (53 yr anchor) | `1.7e12` (~53 yr) | auto-sync AC-9 | +| Rerun 1 (`527631.txt`) | AZ-614 only | `0` | `-4334` (~4.3 s) | auto-sync AC-9 (false-positive motion) | +| Rerun 2 (`224191.txt`) | AZ-614 + AZ-611 | `0` | `0` (manual, validation skipped) | runtime_root build flag (`BUILD_VIDEO_FILE_FRAME_SOURCE`) | +| Rerun 3 (`110515.txt`) | + AZ-602 build flags | `0` | `0` | **runtime_root airborne_bootstrap** ← NEW | + +### Reality-Gate verdict (cycle 3) + +The Jetson run now successfully: + +* Reads the synth tlog (`message_counts: SCALED_IMU2/ATTITUDE/GPS_RAW_INT/HEARTBEAT`) +* Opens `VideoFileFrameSource` against the 269 MB Derkachi mp4 +* Opens `TlogReplayFcAdapter` and `JsonlReplaySink` +* Logs `replay.compose_root.ready: pace=asap resolved_offset_ms=0 auto_sync_used=false` + +…then immediately crashes inside `runtime_root.airborne_bootstrap`: + +``` +runtime_root: airborne_bootstrap: component 'c4_pose' requires +pre_constructed['c282_ransac_filter'] to be populated before +compose_root() runs; available keys in constructed: +['clock', 'fc_adapter', 'frame_source', 'mavlink_transport', 'replay_sink']. +Production main() must build infrastructure (c13_fdr, c6_*, c7_inference, etc.) +into pre_constructed and pass it to compose_root(config, pre_constructed=...). +``` + +This affects **both live and replay binaries**. Every prior "green" Reality-Gate +run died at auto-sync (AZ-614 root cause) BEFORE the composition graph was +walked, so the gap stayed hidden through Track 1 + AZ-615. AZ-591 registered the +strategy *wrappers*; `runtime_root.main()` still does not *construct* the +infrastructure dependencies those wrappers consume. The 38 unit tests for +`compose_root` pass only because they inject a stub factory via the +`replay_components_factory` kwarg, bypassing the bootstrap entirely. + +Filed as **AZ-618** (Story under AZ-602, 5 pts capped per local rules, with a +6-subtask split recommended during refinement: c13_fdr+clock, c6_*, c7_inference, +c3_lightglue+feature_extractor, c2_82_ransac_filter, integration wiring). + +### Tier-2 e2e count breakdown (Cycle 3) + +Same 5 failures, three layers deeper into the SUT than Cycle 2. + +| Outcome | Count | Tests | +|---------|------:|-------| +| PASSED | 17 | AC-4 AST scan, AC-7 skip-gate, 14× AC-9 helpers | +| FAILED | 5 | AC-1, AC-2, AC-5, AC-6 pace-realtime, AC-6 pace-asap | +| SKIPPED | 1 | AC-8 (unchanged: D-PROJ-2 mock-sat stub) | +| XFAIL | 1 | AC-3 (unchanged: calibration intrinsics unknown) | +| **Wall clock** | **20s** | (vs ~10 min Cycle 2: now fails fast at composition root instead of timing out at auto-sync) | + +### Jira state at end of cycle 3 + +| Issue | Title | Status | +|-------|-------|--------| +| AZ-602 | E2E Tier-1 harness rehabilitation (Epic) | TO DO (Track-2 still in progress) | +| AZ-611 | Auto-sync escape hatch (`--skip-auto-sync`) | **DONE this cycle** | +| AZ-614 | Derkachi tlog synth time-base mismatch | **DONE this cycle** | +| AZ-615 | Jetson Tier-2 harness | DONE (+ tilde-fix this cycle) | +| AZ-618 | **Airborne main() must build pre_constructed infrastructure for compose_root** | **NEW — TO DO (next Reality-Gate blocker)** | + +### Discovered followup (no commits, just a note) + +`scripts/run-tests-jetson.sh` still does `docker compose up` against the +`docker-compose.test.jetson.yml` runner stack, which tries to pull +`operator-orchestrator:dev` etc. from Docker Hub (they only exist as local +build tags). Rerun 3 worked around this by skipping compose entirely and +invoking `docker run --rm --runtime=nvidia --gpus all gps-denied-onboard/e2e-runner:jetson …` +directly. Compose isn't needed until the test reaches into the DB / mock-sat / +companion services — which currently never happens because the run dies at +`airborne_bootstrap`. Recommend revisiting the script after AZ-618 lands so the +compose dependency graph is meaningful. + diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 9d2578f..160e8bc 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -6,9 +6,9 @@ step: 11 name: Run Tests status: passed_with_followups sub_step: - phase: 7 - name: jetson-harness-online - detail: "Track 1 done + AZ-615 Jetson Tier-2 harness wired. First Jetson run: identical to Colima (17 pass / 5 fail / 1 skip / 1 xfail, 10m09s). Same 5 failures hit AZ-614 (tlog synth time-base, arch-independent) BEFORE reaching the GPU. Image builds, GPU exposed, SUT runs — infrastructure proven. Next: fix AZ-614 to actually exercise the GPU. AZ-616 (real ../satellite-provider) + AZ-617 (tier2 marks done) queued." + phase: 8 + name: az614-az611-landed-bootstrap-gap-discovered + detail: "AZ-614 + AZ-611 + AZ-602 build-flags + AZ-615 tilde-fix all landed (commits e114bfd, bd41956, 324bbd6, b7012d2). Jetson Cycle-3 rerun (terminal 110515.txt): replay path now reaches `replay.compose_root.ready: auto_sync_used=false`, then crashes in `runtime_root.airborne_bootstrap` with `pre_constructed['c282_ransac_filter']` missing. Same 5 heavy ACs still fail but 3 layers deeper — `runtime_root.main()` calls `register_airborne_strategies()` but does NOT build c13_fdr/c6_*/c7_inference/c3_*/c2_82_ransac_filter into pre_constructed. Filed AZ-618 (Story under AZ-602, 5 pts capped). Pending user decision on whether to start AZ-618 immediately or close out Step 11 with the current Reality-Gate signal." retry_count: 0 cycle: 1 tracker: jira