mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 08:31:13 +00:00
[AZ-611] [AZ-614] [AZ-618] Step-11 Cycle-3 report + autodev state
Cycle-3 addendum captures the layered Jetson rerun progression: synth time-base fix (AZ-614) drops offset_ms from 1.7e12 to -4334; AZ-611 skip-auto-sync then crosses the AC-9 validator; AZ-602 build-flag completeness opens VideoFileFrameSource and TlogReplayFcAdapter; composition root logs 'replay.compose_root.ready: auto_sync_used=false', then crashes inside runtime_root.airborne_bootstrap because production main() never builds c13_fdr / c6_* / c7_inference / c3_lightglue_runtime / c3_feature_extractor / c2_82_ransac_filter into pre_constructed. The bootstrap gap is filed as AZ-618 (Story under AZ-602). It affects both live and replay binaries -- every prior Reality-Gate run died at auto-sync before the composition graph was walked, so the gap was hidden. The 38 compose_root unit tests pass only via the replay_components_factory stub kwarg, which bypasses the bootstrap entirely. Autodev sub_step advances to phase 8 'az614-az611-landed-bootstrap-gap-discovered' pending the user's decision on whether to start AZ-618 immediately or close out Step 11 with the current Reality-Gate signal. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -386,3 +386,95 @@ stages they entail).
|
||||
* git LFS-tracked fixtures (the 269 MB Derkachi mp4) must be
|
||||
pre-smudged on the Mac BEFORE the rsync step; otherwise the Jetson
|
||||
receives the 134 B pointer and tests fail at fixture-load.
|
||||
|
||||
## Cycle-3 addendum: AZ-614 + AZ-611 landed; next blocker is airborne bootstrap (AZ-618)
|
||||
|
||||
Date: 2026-05-18. Track-2 Reality-Gate work continued on Jetson with
|
||||
two SUT-side fixes layered on top of the AZ-615 harness.
|
||||
|
||||
### Commits landed this cycle
|
||||
|
||||
| Commit | Ticket | What |
|
||||
|--------|--------|------|
|
||||
| `e114bfd` | AZ-614 | `_TLOG_BASE_TIMESTAMP_US = 0` so the synth tlog shares the video's t=0 anchor |
|
||||
| `bd41956` | AZ-611 | `--skip-auto-sync` CLI flag + `ReplayConfig.skip_auto_sync_validation` field + 5 unit tests |
|
||||
| `324bbd6` | AZ-602 | `docker-compose.test.yml` + `docker-compose.test.jetson.yml`: set all three replay `BUILD_*` flags |
|
||||
| `b7012d2` | AZ-615 | `scripts/run-tests-jetson.sh`: resolve `~` against remote `$HOME` before the heredoc cd |
|
||||
|
||||
### Jetson rerun progression
|
||||
|
||||
Each rerun isolated one fix to keep the diagnostic signal clean.
|
||||
|
||||
| Rerun | Run scope | `tlog_takeoff_ns` | resolved `offset_ms` | Failure layer |
|
||||
|-------|-----------|-------------------|----------------------|---------------|
|
||||
| Pre-fix (`915484.txt`) | AZ-614 unverified | `1.7e18` (53 yr anchor) | `1.7e12` (~53 yr) | auto-sync AC-9 |
|
||||
| Rerun 1 (`527631.txt`) | AZ-614 only | `0` | `-4334` (~4.3 s) | auto-sync AC-9 (false-positive motion) |
|
||||
| Rerun 2 (`224191.txt`) | AZ-614 + AZ-611 | `0` | `0` (manual, validation skipped) | runtime_root build flag (`BUILD_VIDEO_FILE_FRAME_SOURCE`) |
|
||||
| Rerun 3 (`110515.txt`) | + AZ-602 build flags | `0` | `0` | **runtime_root airborne_bootstrap** ← NEW |
|
||||
|
||||
### Reality-Gate verdict (cycle 3)
|
||||
|
||||
The Jetson run now successfully:
|
||||
|
||||
* Reads the synth tlog (`message_counts: SCALED_IMU2/ATTITUDE/GPS_RAW_INT/HEARTBEAT`)
|
||||
* Opens `VideoFileFrameSource` against the 269 MB Derkachi mp4
|
||||
* Opens `TlogReplayFcAdapter` and `JsonlReplaySink`
|
||||
* Logs `replay.compose_root.ready: pace=asap resolved_offset_ms=0 auto_sync_used=false`
|
||||
|
||||
…then immediately crashes inside `runtime_root.airborne_bootstrap`:
|
||||
|
||||
```
|
||||
runtime_root: airborne_bootstrap: component 'c4_pose' requires
|
||||
pre_constructed['c282_ransac_filter'] to be populated before
|
||||
compose_root() runs; available keys in constructed:
|
||||
['clock', 'fc_adapter', 'frame_source', 'mavlink_transport', 'replay_sink'].
|
||||
Production main() must build infrastructure (c13_fdr, c6_*, c7_inference, etc.)
|
||||
into pre_constructed and pass it to compose_root(config, pre_constructed=...).
|
||||
```
|
||||
|
||||
This affects **both live and replay binaries**. Every prior "green" Reality-Gate
|
||||
run died at auto-sync (AZ-614 root cause) BEFORE the composition graph was
|
||||
walked, so the gap stayed hidden through Track 1 + AZ-615. AZ-591 registered the
|
||||
strategy *wrappers*; `runtime_root.main()` still does not *construct* the
|
||||
infrastructure dependencies those wrappers consume. The 38 unit tests for
|
||||
`compose_root` pass only because they inject a stub factory via the
|
||||
`replay_components_factory` kwarg, bypassing the bootstrap entirely.
|
||||
|
||||
Filed as **AZ-618** (Story under AZ-602, 5 pts capped per local rules, with a
|
||||
6-subtask split recommended during refinement: c13_fdr+clock, c6_*, c7_inference,
|
||||
c3_lightglue+feature_extractor, c2_82_ransac_filter, integration wiring).
|
||||
|
||||
### Tier-2 e2e count breakdown (Cycle 3)
|
||||
|
||||
Same 5 failures, three layers deeper into the SUT than Cycle 2.
|
||||
|
||||
| Outcome | Count | Tests |
|
||||
|---------|------:|-------|
|
||||
| PASSED | 17 | AC-4 AST scan, AC-7 skip-gate, 14× AC-9 helpers |
|
||||
| FAILED | 5 | AC-1, AC-2, AC-5, AC-6 pace-realtime, AC-6 pace-asap |
|
||||
| SKIPPED | 1 | AC-8 (unchanged: D-PROJ-2 mock-sat stub) |
|
||||
| XFAIL | 1 | AC-3 (unchanged: calibration intrinsics unknown) |
|
||||
| **Wall clock** | **20s** | (vs ~10 min Cycle 2: now fails fast at composition root instead of timing out at auto-sync) |
|
||||
|
||||
### Jira state at end of cycle 3
|
||||
|
||||
| Issue | Title | Status |
|
||||
|-------|-------|--------|
|
||||
| AZ-602 | E2E Tier-1 harness rehabilitation (Epic) | TO DO (Track-2 still in progress) |
|
||||
| AZ-611 | Auto-sync escape hatch (`--skip-auto-sync`) | **DONE this cycle** |
|
||||
| AZ-614 | Derkachi tlog synth time-base mismatch | **DONE this cycle** |
|
||||
| AZ-615 | Jetson Tier-2 harness | DONE (+ tilde-fix this cycle) |
|
||||
| AZ-618 | **Airborne main() must build pre_constructed infrastructure for compose_root** | **NEW — TO DO (next Reality-Gate blocker)** |
|
||||
|
||||
### Discovered followup (no commits, just a note)
|
||||
|
||||
`scripts/run-tests-jetson.sh` still does `docker compose up` against the
|
||||
`docker-compose.test.jetson.yml` runner stack, which tries to pull
|
||||
`operator-orchestrator:dev` etc. from Docker Hub (they only exist as local
|
||||
build tags). Rerun 3 worked around this by skipping compose entirely and
|
||||
invoking `docker run --rm --runtime=nvidia --gpus all gps-denied-onboard/e2e-runner:jetson …`
|
||||
directly. Compose isn't needed until the test reaches into the DB / mock-sat /
|
||||
companion services — which currently never happens because the run dies at
|
||||
`airborne_bootstrap`. Recommend revisiting the script after AZ-618 lands so the
|
||||
compose dependency graph is meaningful.
|
||||
|
||||
|
||||
@@ -6,9 +6,9 @@ step: 11
|
||||
name: Run Tests
|
||||
status: passed_with_followups
|
||||
sub_step:
|
||||
phase: 7
|
||||
name: jetson-harness-online
|
||||
detail: "Track 1 done + AZ-615 Jetson Tier-2 harness wired. First Jetson run: identical to Colima (17 pass / 5 fail / 1 skip / 1 xfail, 10m09s). Same 5 failures hit AZ-614 (tlog synth time-base, arch-independent) BEFORE reaching the GPU. Image builds, GPU exposed, SUT runs — infrastructure proven. Next: fix AZ-614 to actually exercise the GPU. AZ-616 (real ../satellite-provider) + AZ-617 (tier2 marks done) queued."
|
||||
phase: 8
|
||||
name: az614-az611-landed-bootstrap-gap-discovered
|
||||
detail: "AZ-614 + AZ-611 + AZ-602 build-flags + AZ-615 tilde-fix all landed (commits e114bfd, bd41956, 324bbd6, b7012d2). Jetson Cycle-3 rerun (terminal 110515.txt): replay path now reaches `replay.compose_root.ready: auto_sync_used=false`, then crashes in `runtime_root.airborne_bootstrap` with `pre_constructed['c282_ransac_filter']` missing. Same 5 heavy ACs still fail but 3 layers deeper — `runtime_root.main()` calls `register_airborne_strategies()` but does NOT build c13_fdr/c6_*/c7_inference/c3_*/c2_82_ransac_filter into pre_constructed. Filed AZ-618 (Story under AZ-602, 5 pts capped). Pending user decision on whether to start AZ-618 immediately or close out Step 11 with the current Reality-Gate signal."
|
||||
retry_count: 0
|
||||
cycle: 1
|
||||
tracker: jira
|
||||
|
||||
Reference in New Issue
Block a user