[AZ-321] C10 EngineCompiler: hardware-tied TRT compile + cache reuse

Land the C10 per-model engine compile + cache-reuse orchestrator.
`EngineCompiler.compile_engines_for_corpus(request)` walks the
corpus, computes the canonical engine filename via AZ-281
`EngineFilenameSchema.build`, and either reuses the cached binary
(cache hit, AZ-280 `Sha256Sidecar.verify` returns True) or delegates
to the AZ-297 `compile_engine` on the injected runtime (cache miss;
the runtime owns the write path). Returns one `EngineCompileResult`
per backbone carrying the canonical `EngineCacheEntry`, outcome
(BUILT / REUSED), and `compile_duration_s` (None on reuse).
Hardware-tied reuse (D-C10-6 / D-C10-7) falls out of the filename
schema — a host change rebuilds at the new path and leaves the old
files untouched (AC-4).

Design corrections vs. the task spec body:
- The spec proposed a c10-local `EngineCacheEntry` carrying outcome
  and duration; that name is already taken by the AZ-297 canonical
  DTO. The wrapper is renamed `EngineCompileResult`; the canonical
  shape wins.
- The spec called `InferenceRuntime.host_info()`, which is not in
  the AZ-297 Protocol. `HostCapabilities` is threaded through
  `EngineCompileRequest` instead so the composition root owns host
  probing and the compiler stays decoupled.
- The c10 layer cannot import `components.c7_inference` (arch rule
  `test_az270_compose_root.test_ac6`). `engine_compiler.py` defines
  `CompileEngineCallable` — a structural Protocol cut of
  `InferenceRuntime` exposing only `compile_engine` — and catches
  broad `Exception` (re-raising preserves the original type;
  `error_class` is recorded in the ERROR log payload).

Production
- engine_compiler.py: `CompileOutcome` enum, `BackboneSpec`,
  `EngineCompileRequest`, `EngineCompileResult`,
  `EngineCompileSummary` DTOs; `CompileEngineCallable` Protocol;
  `EngineCompiler` with the single public method.
- config.py: `BackboneConfig` + `C10ProvisioningConfig`
  (`workspace_mb` default 4 GiB to match C7 NFT-LIM-01); validate
  positive shape dims and duplicate model_name detection in
  `__post_init__`.
- runtime_root/c10_factory.py: `build_engine_compiler(config)` wires
  the existing `build_inference_runtime` factory through;
  `build_backbone_specs(config)` materialises the `BackboneSpec`
  tuple from the config block.
- components/c10_provisioning/__init__.py: re-exports the AZ-321
  surface and registers the new config block.

Tests
- test_engine_compiler.py: covers AC-1..AC-10 + missing-sidecar
  sibling case for AC-5. Tier-1 via fake runtime that writes through
  the REAL `Sha256Sidecar.write_atomic_and_sidecar`. Tier-2
  placeholders for the cache-hit p99 NFR (200 MB engine sweep) and
  kill-during-compile atomic-write NFR.

Docs
- module-layout.md: c10_provisioning Per-Component Mapping lists the
  new internal modules (engine_compiler.py, config.py), the
  composition-root c10_factory.py, the AZ-321 public re-export
  surface, and the registered config block.
- batch_33_cycle1_report.md + reviews/batch_33_review.md:
  PASS_WITH_WARNINGS (4 Low findings accepted).

Tests run: c10_provisioning 13 passing + 2 Tier-2 skips; combined
unit suite (excluding pending components) 543 passing, 21
env-skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-13 00:09:53 +03:00
parent 0ad3278b12
commit 0dfe7c5301
10 changed files with 1538 additions and 7 deletions
@@ -0,0 +1,218 @@
# Batch 33 / Cycle 1 — Implementation Report
**Date**: 2026-05-12
**Tasks**: AZ-321 (C10 EngineCompiler — per-model TRT compile +
hardware-tied cache reuse + AZ-281 filename schema + AZ-280 sidecar
gate)
**Story points landed**: 5
**Status**: complete (AZ-321 → In Testing)
## Scope summary
Single-task batch landing the C10 per-model engine compile + cache-
reuse orchestrator. `EngineCompiler.compile_engines_for_corpus(req)`
walks the corpus, computes the canonical engine filename via AZ-281
`EngineFilenameSchema.build(...)`, and either reuses the cached
binary (cache hit; AZ-280 `Sha256Sidecar.verify` returns True) or
delegates to the AZ-297 `compile_engine` method on the injected
runtime (cache miss; the runtime owns the write path and the sidecar
emission). The orchestrator returns one `EngineCompileResult` per
backbone carrying the canonical `EngineCacheEntry`, the
`CompileOutcome.{BUILT,REUSED}` label, and the `compile_duration_s`
(None on reuse). Hardware-tied cache reuse (D-C10-6 / D-C10-7) falls
out naturally from the filename schema: an engine compiled on
`(sm=87, jp=6.2, trt=10.3, fp16)` lives at a different path than one
compiled on `(sm=89, jp=6.3, trt=10.5, fp16)`, so a hardware change
produces cache misses for the new device and leaves the old files
untouched (AC-4).
Two design corrections vs. the task spec body:
- **`EngineCacheEntry` shape** — the task spec proposed a c10-local
`EngineCacheEntry` with `outcome` and `compile_duration_s` fields.
That clashes with the canonical AZ-297
`_types.inference.EngineCacheEntry` already re-exported from
`components.c10_provisioning`. The canonical shape wins; the AZ-321
wrapper is renamed `EngineCompileResult` and carries
`{entry, outcome, compile_duration_s}` cleanly.
- **`InferenceRuntime.host_info()`** — the task spec calls a
hypothetical `host_info()` method on the runtime to retrieve
`(sm, jp, trt)`. The AZ-297 Protocol does NOT expose host info.
Rather than expand the frozen Protocol mid-cycle, we accept a
`HostCapabilities` field on `EngineCompileRequest` so the
composition root threads the host from its own probe (Tier-2
device introspection or Tier-1 test fixture). The compiler stays
decoupled from any runtime-side introspection surface.
The C10 layer is also forbidden by `test_az270_compose_root.test_ac6`
from importing from `components.c7_inference` directly — that rule
applies across all `components/*/*.py` files regardless of what the
prose-level `module-layout.md` declares the "Imports from" list to be.
The lint test wins. To respect it, `engine_compiler.py` defines
`CompileEngineCallable` — a structural Protocol cut of
`InferenceRuntime` exposing only the single `compile_engine` method
the compiler actually uses — and catches the broader `Exception`
class (the AZ-297 C7 error family stays the runtime's contract; the
compiler dispatches on `type(exc).__name__` in its ERROR log payload
and re-raises so the original exception type propagates to the
caller intact).
## Files added / modified
### New (production)
- `src/gps_denied_onboard/components/c10_provisioning/engine_compiler.py`
`CompileOutcome` enum (`BUILT` / `REUSED`), `BackboneSpec` DTO,
`EngineCompileRequest` DTO, `EngineCompileResult` DTO,
`EngineCompileSummary` DTO, `CompileEngineCallable` structural
Protocol, and the `EngineCompiler` class with the single public
`compile_engines_for_corpus` method. Helpers:
`_build_config_for_backbone` (synthesises one
`OptimizationProfile` with `min == opt == max ==
expected_input_shape` from the backbone spec; richer dynamic-shape
ranges are out of scope for AZ-321), `_summarise` (aggregate counts
for the `c10.engine.compile.summary` log).
- `src/gps_denied_onboard/components/c10_provisioning/config.py`
`BackboneConfig` DTO (`model_name`, `onnx_path`,
`expected_input_shape`, `input_name` with `"input"` default) +
`C10ProvisioningConfig` (`backbones` tuple, `workspace_mb` default
4096 to match C7 NFT-LIM-01). Both validate in `__post_init__`
(non-empty strings, positive shape dims, duplicate model_name
detection).
- `src/gps_denied_onboard/runtime_root/c10_factory.py`
`build_engine_compiler(config)` wires the existing
`build_inference_runtime` factory through to a new `EngineCompiler`
instance with a c10-scoped structured logger; `build_backbone_specs
(config)` materialises the `BackboneSpec` tuple from
`config.components['c10_provisioning'].backbones`.
### Modified (production)
- `src/gps_denied_onboard/components/c10_provisioning/__init__.py`
re-exports the AZ-321 public surface (`EngineCompiler`,
`BackboneSpec`, `EngineCompileRequest`, `EngineCompileResult`,
`CompileOutcome`, `EngineCompileSummary`, `CompileEngineCallable`,
`BackboneConfig`, `C10ProvisioningConfig`) and registers the new
config block via `register_component_block("c10_provisioning",
C10ProvisioningConfig)`. `CacheProvisioner` / `Manifest` /
`EngineCacheEntry` re-exports unchanged.
### New (tests)
- `tests/unit/c10_provisioning/test_engine_compiler.py`**NEW**
Tier-1 suite covering every AC + the 2 Tier-2 NFR placeholders:
- **AC-1** cold cache + 3 backbones → all `BUILT`; 3 `.engine` +
3 `.sha256` files on disk; 3 `c10.engine.cache.miss` WARN logs;
1 `c10.engine.compile.summary` INFO log with `engines_built=3`.
- **AC-2** warm cache + identical request → all `REUSED`;
`compile_duration_s is None` for every result; ZERO calls to the
fake runtime; 3 `c10.engine.cache.hit` INFO logs.
- **AC-3** mixed (1 hit + 2 miss) — DINOv2 reused, LightGlue +
ALIKED built; 2 calls to the fake runtime.
- **AC-4** hardware change (sm 87→89, jp 6.2→6.3, trt 10.3→10.5):
every backbone rebuilt at the new filename; old files at the old
filename untouched on disk.
- **AC-5** tampered sidecar (overwrite LightGlue's `.sha256` with
`0`×64): LightGlue rebuilt; DINOv2 + ALIKED still reused; 1
`c10.engine.sidecar.mismatch` WARN log with `model_name=
lightglue` and `reason=digest_mismatch`. Plus a sibling case
where the sidecar file is deleted entirely (`Sha256Sidecar.verify`
raises) — same WARN-then-rebuild outcome.
- **AC-6** `EngineBuildError` mid-corpus (backbone 2 of 3 fails):
error propagates; backbone 1 (pre-cached, reused) untouched on
disk; backbone 2's would-be engine NOT on disk (atomic-write
guarantee from the fake mirrors AZ-298's real behaviour);
backbone 3 never attempted (single call recorded for backbone 2).
- **AC-7** `CalibrationCacheError` propagates with the
`c10.engine.compile.error` ERROR log carrying `model_name`,
`calibration_path`, `error_class=CalibrationCacheError`.
- **AC-8** filename is exactly
`dinov2_vpr__sm87_jp6.2_trt10.3_fp16.engine` (per AZ-281
canonical schema with the `__` separator between model and
`sm`); sidecar at `*.engine.sha256` with 64-hex digest;
`EngineFilenameSchema.parse` round-trip + `Sha256Sidecar.verify`
both pass.
- **AC-9** `compile_duration_s` is a positive float for every
`BUILT` result, `None` for every `REUSED` result.
- **AC-10** empty `backbones` tuple → empty result; ZERO runtime
calls; ZERO files written; 1 summary log with all-zero counts.
- **NFR-perf-cache-hit** Tier-2 placeholder skip (200 MB engine
sweep belongs in the AZ-321 microbench harness on Jetson).
- **NFR-reliability-atomic-write** Tier-2 placeholder skip (kill-
during-compile scenario lives in the microbench harness; the
atomicity contract itself is owned by AZ-280's tests).
The Tier-1 tests use a `_FakeRuntime` that satisfies
`CompileEngineCallable` and writes deterministic engine bytes via
the REAL `Sha256Sidecar.write_atomic_and_sidecar` — so the cache-hit
/ cache-miss / tampered-sidecar paths run against the same helper
the production wiring uses. Only the C7-runtime-specific compile
internals (TRT engine bytes, calibration cache, GPU memory) are
mocked.
### Modified (docs)
- `_docs/02_document/module-layout.md` — c10_provisioning Per-
Component Mapping now lists the new internal modules
(`engine_compiler.py`, `config.py`) and the composition-root
`c10_factory.py`; the Public API re-export list is extended with
the AZ-321 surface; the `Config block` line is added (registered
on import). `default_provisioner.py` row marked `pending` until
the AZ-325 task lands.
## Acceptance criteria coverage
| AC | Test | Status |
|----|------|--------|
| AC-1 Cold cache → all built | `test_ac1_cold_cache_compiles_every_backbone` | passing |
| AC-2 Warm cache → all reused, zero compile calls | `test_ac2_warm_cache_reuses_every_backbone` | passing |
| AC-3 Mixed cache | `test_ac3_mixed_cache_hits_and_misses` | passing |
| AC-4 Hardware change invalidates filename | `test_ac4_hardware_change_invalidates_cache` | passing |
| AC-5 Tampered sidecar + missing sidecar paths | `test_ac5_tampered_sidecar_invalidates_that_engine` + `test_missing_sidecar_treated_as_cache_miss` | passing |
| AC-6 `EngineBuildError` propagates, partial state consistent | `test_ac6_engine_build_error_propagates_and_third_backbone_untouched` | passing |
| AC-7 `CalibrationCacheError` propagates with diagnostic log | `test_ac7_calibration_cache_error_propagates` | passing |
| AC-8 Filename + sidecar layout matches AZ-281 schema | `test_ac8_filename_and_sidecar_layout` | passing |
| AC-9 `compile_duration_s` recorded for built only | `test_ac9_compile_duration_recorded_for_built_only` | passing |
| AC-10 Empty backbones → empty result, no side effects | `test_ac10_empty_backbones_returns_empty` | passing |
| NFR-perf-cache-hit p99 ≤ 1.5 s for 200 MB engine | `test_nfr_perf_cache_hit_p99_under_1500ms_for_200mb_engine` (Tier-2) | Tier-2 skipped |
| NFR-reliability atomic-write no half-engine after kill | `test_nfr_reliability_atomic_write_no_half_engine_after_kill` (Tier-2) | Tier-2 skipped |
## AC Test Coverage: 10 of 10 covered (+ 2 NFRs)
## Code Review Verdict: PASS_WITH_WARNINGS (4 Low accepted; see Findings)
## Auto-Fix Attempts: 0
## Stuck Agents: None
## Findings (self-review)
| # | Severity | Category | Location | Note | Resolution |
|---|----------|----------|----------|------|------------|
| 1 | Low | Architecture | `engine_compiler.py::_compile_one` | Catches the broad `Exception` (not the specific AZ-297 `RuntimeError` family) because the c10 layer cannot import `components.c7_inference` (architecture rule `test_az270_compose_root.test_ac6`). The C7 contract scopes its runtime exceptions to its own family; ANY exception bubbling out of `compile_engine` is treated as a compile failure here. Re-raise preserves the original type. Inline comment documents the rule. | Open (Low) — accepted; architecture rule wins. |
| 2 | Low | Maintainability | `engine_compiler.py::CompileEngineCallable` | Duplicates the `compile_engine` method shape from the C7 `InferenceRuntime` Protocol. Mirrors the LightGlue dual-Protocol pattern already in `_types/manifests.py` (consumer-side structural cut vs. producer-side opaque marker). | Open (Low) — accepted; matches established pattern. |
| 3 | Low | Architecture | `engine_compiler.py` ↔ C7InferenceConfig | `EngineCompileRequest.cache_root` MUST equal the directory the C7 runtime writes to (`C7InferenceConfig.engine_cache_dir`). The composition root (`build_engine_compiler` + the C10 corpus driver T5 in AZ-325) is responsible for keeping the two in sync; the compiler itself trusts the request. A divergence would cause cache hits to always miss. | Open (Low) — flagged for AZ-325 to enforce. |
| 4 | Low | Scope | `engine_compiler.py::_build_config_for_backbone` | Synthesises exactly one `OptimizationProfile` with `min == opt == max == expected_input_shape`. Backbones requiring dynamic input ranges would need a richer `BackboneSpec` carrying explicit `OptimizationProfile` tuples. None of the AZ-321 corpus backbones (DINOv2-VPR, LightGlue, ALIKED) need dynamic shapes today, but the limitation is real. | Open (Low) — accepted; future extension. |
## Tracker
- AZ-321 transitioned to **In Progress** at session start; will move
to **In Testing** post-commit per `protocols.md`.
## Test suite
- `tests/unit/c10_provisioning/` — 13 passing, 2 Tier-2 skips
(cache-hit p99 NFR + atomic-write kill scenario).
- Combined unit suite excluding pending components (c1, c2, c2.5,
c3, c3.5, c4, c5, c8, c11, c12) and the c6 collection blocker on
this host (missing `psycopg_pool` is a known dev-machine env issue,
pre-existing) — 543 passing, 21 environment-skipped, 1 warning
(pre-existing `pynvml` FutureWarning unrelated to AZ-321).
## Next batch
Cycle 1 advances per the greenfield queue — autodev re-detects the
next AZ ticket in the Step 7 batch loop. AZ-321 unblocks AZ-322
(C10 Descriptor Batcher), AZ-337 (C2 UltraVPR), AZ-345 / AZ-346 /
AZ-347 (C3 matchers), and AZ-349 (C3.5 refiner) at the topological
level; the next ready batch is computed by `compute-next-batch`.
A cumulative review (batches 3133) will fire at the next sub-skill
phase boundary per Step 14.5's K=3 trigger.
@@ -0,0 +1,54 @@
# Code Review Report — Batch 33 / Cycle 1
**Batch**: 33
**Tasks**: AZ-321 (C10 EngineCompiler)
**Date**: 2026-05-12
**Verdict**: PASS_WITH_WARNINGS
## Findings
| # | Severity | Category | File:Line | Title |
|---|----------|----------|-----------|-------|
| 1 | Low | Architecture | `engine_compiler.py::_compile_one` | Broad `except Exception` rather than the AZ-297 `RuntimeError` family |
| 2 | Low | Maintainability | `engine_compiler.py::CompileEngineCallable` | Duplicates the `compile_engine` signature from the C7 `InferenceRuntime` Protocol |
| 3 | Low | Architecture | `engine_compiler.py` ↔ C7InferenceConfig | `EngineCompileRequest.cache_root` MUST mirror `C7InferenceConfig.engine_cache_dir` — invariant enforced by the composition root, not the compiler |
| 4 | Low | Scope | `engine_compiler.py::_build_config_for_backbone` | Single static `OptimizationProfile` synthesised per backbone; dynamic shape ranges out of scope |
### Finding Details
**F1: Broad `except Exception` on `compile_engine`** (Low / Architecture)
- Location: `src/gps_denied_onboard/components/c10_provisioning/engine_compiler.py::EngineCompiler._compile_one`
- Description: The C7 contract scopes `InferenceRuntime.compile_engine` exceptions to the C7-local `RuntimeError` family (`EngineBuildError`, `CalibrationCacheError`, ...). The c10 layer is forbidden from importing `components.c7_inference` (architecture rule `test_az270_compose_root.test_ac6` walks all `components/*/*.py` files and flags any cross-component import — including TYPE_CHECKING-guarded ones). Catching the broader `Exception` and dispatching by `type(exc).__name__` in the log payload is the cheapest fix that respects the rule. Re-raise preserves the original exception type for the caller.
- Suggestion: A longer-term cleanup would either (a) hoist the C7 error envelope to `_types/inference.py` (parallels the `EngineCacheEntry` move) or (b) extend the architecture-lint to allow Public-API-only imports from sibling components. Both are bigger scope than AZ-321.
- Task: AZ-321
- Resolution: Open (Low) — accepted as documented; inline comment in source.
**F2: `CompileEngineCallable` shadow Protocol** (Low / Maintainability)
- Location: `engine_compiler.py::CompileEngineCallable`
- Description: Defines a structural Protocol carrying only the single `compile_engine` method shape — the c10 compiler's narrow consumer cut of the AZ-297 `InferenceRuntime` Protocol. Mirrors the LightGlue dual-Protocol pattern already documented in `_types/manifests.py` (`EngineHandle` consumer-cut Protocol vs `c7_inference.EngineHandle` opaque marker class).
- Suggestion: None — same pattern already accepted across the codebase. A future "Public-API cross-component import allowlist" lint update could collapse this dual.
- Task: AZ-321
- Resolution: Open (Low) — accepted; matches existing pattern.
**F3: `cache_root` / `engine_cache_dir` invariant** (Low / Architecture)
- Location: `engine_compiler.py::EngineCompileRequest.cache_root` vs `C7InferenceConfig.engine_cache_dir`
- Description: The compiler's cache-hit detection writes nothing — it just checks `target_path.exists()` + `Sha256Sidecar.verify`. The C7 runtime owns the `.engine` write path and writes to `C7InferenceConfig.engine_cache_dir`. If the composition root passes a different `cache_root` to the compiler than the C7 runtime is configured for, cache hits will never fire (the compiler will look at the wrong directory). The compiler itself trusts the request; the wiring invariant lives in `build_engine_compiler` and (later) the AZ-325 `CacheProvisioner` driver.
- Suggestion: AZ-325 (C10 Cache Provisioner — the orchestrator T5 that drives the compiler) should pull both paths from the same config field or assert their equality at construction time. The compiler stays scope-bound to one model at a time.
- Task: AZ-321 (flag for AZ-325 follow-up)
- Resolution: Open (Low) — accepted; flagged for AZ-325.
**F4: Single static `OptimizationProfile` per backbone** (Low / Scope)
- Location: `engine_compiler.py::_build_config_for_backbone`
- Description: The synthesised `BuildConfig` carries exactly one `OptimizationProfile` with `min == opt == max == expected_input_shape`. Backbones requiring dynamic input ranges (variable batch size, variable image resolution) would need a richer `BackboneSpec` carrying explicit `OptimizationProfile` tuples. AZ-321's named corpus (DINOv2-VPR, LightGlue, ALIKED) uses fixed shapes; this is OK today but a real limitation.
- Suggestion: When the first dynamic-shape backbone arrives, extend `BackboneSpec` with a `dynamic_profiles: tuple[OptimizationProfile, ...]` field and prefer it over the synthetic single-profile fallback inside `_build_config_for_backbone`.
- Task: AZ-321
- Resolution: Open (Low) — accepted; future extension.
## Verdict Logic
- 0 Critical
- 0 High
- 0 Medium
- 4 Low
**PASS_WITH_WARNINGS**: only Low findings; all accepted as documented.