[AZ-611] [AZ-614] [AZ-618] Step-11 Cycle-3 report + autodev state

Cycle-3 addendum captures the layered Jetson rerun progression:
synth time-base fix (AZ-614) drops offset_ms from 1.7e12 to -4334;
AZ-611 skip-auto-sync then crosses the AC-9 validator; AZ-602
build-flag completeness opens VideoFileFrameSource and
TlogReplayFcAdapter; composition root logs
'replay.compose_root.ready: auto_sync_used=false', then crashes
inside runtime_root.airborne_bootstrap because production main()
never builds c13_fdr / c6_* / c7_inference / c3_lightglue_runtime /
c3_feature_extractor / c2_82_ransac_filter into pre_constructed.

The bootstrap gap is filed as AZ-618 (Story under AZ-602). It
affects both live and replay binaries -- every prior Reality-Gate
run died at auto-sync before the composition graph was walked, so
the gap was hidden. The 38 compose_root unit tests pass only via
the replay_components_factory stub kwarg, which bypasses the
bootstrap entirely.

Autodev sub_step advances to phase 8
'az614-az611-landed-bootstrap-gap-discovered' pending the user's
decision on whether to start AZ-618 immediately or close out
Step 11 with the current Reality-Gate signal.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-18 09:50:11 +03:00
parent b7012d2787
commit e054a55804
2 changed files with 95 additions and 3 deletions
@@ -386,3 +386,95 @@ stages they entail).
* git LFS-tracked fixtures (the 269 MB Derkachi mp4) must be
pre-smudged on the Mac BEFORE the rsync step; otherwise the Jetson
receives the 134 B pointer and tests fail at fixture-load.
## Cycle-3 addendum: AZ-614 + AZ-611 landed; next blocker is airborne bootstrap (AZ-618)
Date: 2026-05-18. Track-2 Reality-Gate work continued on Jetson with
two SUT-side fixes layered on top of the AZ-615 harness.
### Commits landed this cycle
| Commit | Ticket | What |
|--------|--------|------|
| `e114bfd` | AZ-614 | `_TLOG_BASE_TIMESTAMP_US = 0` so the synth tlog shares the video's t=0 anchor |
| `bd41956` | AZ-611 | `--skip-auto-sync` CLI flag + `ReplayConfig.skip_auto_sync_validation` field + 5 unit tests |
| `324bbd6` | AZ-602 | `docker-compose.test.yml` + `docker-compose.test.jetson.yml`: set all three replay `BUILD_*` flags |
| `b7012d2` | AZ-615 | `scripts/run-tests-jetson.sh`: resolve `~` against remote `$HOME` before the heredoc cd |
### Jetson rerun progression
Each rerun isolated one fix to keep the diagnostic signal clean.
| Rerun | Run scope | `tlog_takeoff_ns` | resolved `offset_ms` | Failure layer |
|-------|-----------|-------------------|----------------------|---------------|
| Pre-fix (`915484.txt`) | AZ-614 unverified | `1.7e18` (53 yr anchor) | `1.7e12` (~53 yr) | auto-sync AC-9 |
| Rerun 1 (`527631.txt`) | AZ-614 only | `0` | `-4334` (~4.3 s) | auto-sync AC-9 (false-positive motion) |
| Rerun 2 (`224191.txt`) | AZ-614 + AZ-611 | `0` | `0` (manual, validation skipped) | runtime_root build flag (`BUILD_VIDEO_FILE_FRAME_SOURCE`) |
| Rerun 3 (`110515.txt`) | + AZ-602 build flags | `0` | `0` | **runtime_root airborne_bootstrap** ← NEW |
### Reality-Gate verdict (cycle 3)
The Jetson run now successfully:
* Reads the synth tlog (`message_counts: SCALED_IMU2/ATTITUDE/GPS_RAW_INT/HEARTBEAT`)
* Opens `VideoFileFrameSource` against the 269 MB Derkachi mp4
* Opens `TlogReplayFcAdapter` and `JsonlReplaySink`
* Logs `replay.compose_root.ready: pace=asap resolved_offset_ms=0 auto_sync_used=false`
…then immediately crashes inside `runtime_root.airborne_bootstrap`:
```
runtime_root: airborne_bootstrap: component 'c4_pose' requires
pre_constructed['c282_ransac_filter'] to be populated before
compose_root() runs; available keys in constructed:
['clock', 'fc_adapter', 'frame_source', 'mavlink_transport', 'replay_sink'].
Production main() must build infrastructure (c13_fdr, c6_*, c7_inference, etc.)
into pre_constructed and pass it to compose_root(config, pre_constructed=...).
```
This affects **both live and replay binaries**. Every prior "green" Reality-Gate
run died at auto-sync (AZ-614 root cause) BEFORE the composition graph was
walked, so the gap stayed hidden through Track 1 + AZ-615. AZ-591 registered the
strategy *wrappers*; `runtime_root.main()` still does not *construct* the
infrastructure dependencies those wrappers consume. The 38 unit tests for
`compose_root` pass only because they inject a stub factory via the
`replay_components_factory` kwarg, bypassing the bootstrap entirely.
Filed as **AZ-618** (Story under AZ-602, 5 pts capped per local rules, with a
6-subtask split recommended during refinement: c13_fdr+clock, c6_*, c7_inference,
c3_lightglue+feature_extractor, c2_82_ransac_filter, integration wiring).
### Tier-2 e2e count breakdown (Cycle 3)
Same 5 failures, three layers deeper into the SUT than Cycle 2.
| Outcome | Count | Tests |
|---------|------:|-------|
| PASSED | 17 | AC-4 AST scan, AC-7 skip-gate, 14× AC-9 helpers |
| FAILED | 5 | AC-1, AC-2, AC-5, AC-6 pace-realtime, AC-6 pace-asap |
| SKIPPED | 1 | AC-8 (unchanged: D-PROJ-2 mock-sat stub) |
| XFAIL | 1 | AC-3 (unchanged: calibration intrinsics unknown) |
| **Wall clock** | **20s** | (vs ~10 min Cycle 2: now fails fast at composition root instead of timing out at auto-sync) |
### Jira state at end of cycle 3
| Issue | Title | Status |
|-------|-------|--------|
| AZ-602 | E2E Tier-1 harness rehabilitation (Epic) | TO DO (Track-2 still in progress) |
| AZ-611 | Auto-sync escape hatch (`--skip-auto-sync`) | **DONE this cycle** |
| AZ-614 | Derkachi tlog synth time-base mismatch | **DONE this cycle** |
| AZ-615 | Jetson Tier-2 harness | DONE (+ tilde-fix this cycle) |
| AZ-618 | **Airborne main() must build pre_constructed infrastructure for compose_root** | **NEW — TO DO (next Reality-Gate blocker)** |
### Discovered followup (no commits, just a note)
`scripts/run-tests-jetson.sh` still does `docker compose up` against the
`docker-compose.test.jetson.yml` runner stack, which tries to pull
`operator-orchestrator:dev` etc. from Docker Hub (they only exist as local
build tags). Rerun 3 worked around this by skipping compose entirely and
invoking `docker run --rm --runtime=nvidia --gpus all gps-denied-onboard/e2e-runner:jetson …`
directly. Compose isn't needed until the test reaches into the DB / mock-sat /
companion services — which currently never happens because the run dies at
`airborne_bootstrap`. Recommend revisiting the script after AZ-618 lands so the
compose dependency graph is meaningful.
+3 -3
View File
@@ -6,9 +6,9 @@ step: 11
name: Run Tests
status: passed_with_followups
sub_step:
phase: 7
name: jetson-harness-online
detail: "Track 1 done + AZ-615 Jetson Tier-2 harness wired. First Jetson run: identical to Colima (17 pass / 5 fail / 1 skip / 1 xfail, 10m09s). Same 5 failures hit AZ-614 (tlog synth time-base, arch-independent) BEFORE reaching the GPU. Image builds, GPU exposed, SUT runs — infrastructure proven. Next: fix AZ-614 to actually exercise the GPU. AZ-616 (real ../satellite-provider) + AZ-617 (tier2 marks done) queued."
phase: 8
name: az614-az611-landed-bootstrap-gap-discovered
detail: "AZ-614 + AZ-611 + AZ-602 build-flags + AZ-615 tilde-fix all landed (commits e114bfd, bd41956, 324bbd6, b7012d2). Jetson Cycle-3 rerun (terminal 110515.txt): replay path now reaches `replay.compose_root.ready: auto_sync_used=false`, then crashes in `runtime_root.airborne_bootstrap` with `pre_constructed['c282_ransac_filter']` missing. Same 5 heavy ACs still fail but 3 layers deeper — `runtime_root.main()` calls `register_airborne_strategies()` but does NOT build c13_fdr/c6_*/c7_inference/c3_*/c2_82_ransac_filter into pre_constructed. Filed AZ-618 (Story under AZ-602, 5 pts capped). Pending user decision on whether to start AZ-618 immediately or close out Step 11 with the current Reality-Gate signal."
retry_count: 0
cycle: 1
tracker: jira