gps-denied-onboard/_docs/02_tasks/todo/AZ-321_c10_engine_compiler.md

C10 Engine Compiler — Per-Model TRT Compile + Hardware-Tied Cache Reuse

Task: AZ-321_c10_engine_compiler Name: C10 Engine Compiler Description: Implement EngineCompiler, the C10-internal phase that compiles or re-uses TensorRT engines for every backbone the corpus needs (DINOv2 reduced for VPR, LightGlue, ALIKED descriptor head, plus any C7-runtime-required model). For each backbone, computes the AZ-281 self-describing filename {model}_{sm}_{jp}_{trt}_{precision}.engine, looks for an existing engine + sidecar at that path, and either re-uses it (cache hit, D-C10-6) or invokes AZ-298's TensorRT runtime to compile from the ONNX source + calibration cache. Writes each new engine via AZ-280's Sha256Sidecar for the takeoff content-hash gate. Returns a list[EngineCacheEntry] recording the per-backbone outcome (built / reused) plus the cache hit ratio. The compile is hardware-tied: SM, Jetpack, TRT version, and precision flags are baked into the filename so re-running on a different device produces a cache miss (correct behaviour, not a bug). Complexity: 5 points Dependencies: AZ-263_initial_structure, AZ-269_config_loader, AZ-266_log_module, AZ-280_sha256_sidecar, AZ-281_engine_filename_schema, AZ-298_c7_tensorrt_runtime Component: c10_provisioning (epic AZ-252 / E-C10) Tracker: AZ-321 Epic: AZ-252 (E-C10)

Document Dependencies

• _docs/02_document/contracts/shared_helpers/engine_filename_schema.md — filename shape + parser (AZ-281).
• _docs/02_document/contracts/shared_helpers/sha256_sidecar.md — atomic write + sidecar pattern (AZ-280).
• _docs/02_document/contracts/c7_inference/inference_runtime_protocol.md — engine compile API (AZ-298).
• _docs/02_document/components/11_c10_provisioning/description.md — § 5 error handling, § 7 caveats (D-C10-6 hardware-tied).

Problem

Without a real engine compiler:

• AC-NEW-1 (no engine deserialization at takeoff before manifest verify) collapses on the build side — F1 cannot produce the .engine artifacts the airborne C7 deserialise step expects.
• D-C10-6 (calibration cache reuse on identical hardware) is unobservable — every build re-compiles from scratch, blowing the C10-PT-01 ≤ 12 min cold target on warm runs.
• D-C10-7 (self-describing engine filename) has no producer — without {model}_{sm}_{jp}_{trt}_{precision}.engine, hardware mismatches between operator workstation and Jetson airborne would silently load wrong-arch engines.
• The C10-PT-01 warm idempotent re-run target (≤ 1 min) cannot be hit; engines dominate build time.
• C10-IT-05 (Tier-2 build produces SM 87 / JP 6.2 / TRT 10.3 / FP16 engines) has no implementation.
• Operators have no way to inspect which engines came from cache vs. were rebuilt — a critical signal for diagnosing GPU-OOM or calibration regressions.

This task delivers the per-model compile + cache-reuse logic. It does NOT own the orchestration (T5 owns build_cache_artifacts), the descriptor batching (T2), or the manifest writing (T3).

Outcome

• An EngineCompiler class at src/gps_denied_onboard/components/c10_provisioning/engine_compiler.py:
  • Constructor: __init__(self, *, inference_runtime: InferenceRuntime, sidecar: Sha256Sidecar, filename_schema: EngineFilenameSchema, logger: Logger).
  • Public method: compile_engines_for_corpus(request: EngineCompileRequest) -> list[EngineCacheEntry].
  • EngineCompileRequest (@dataclass(frozen=True)): backbones: tuple[BackboneSpec, ...], calibration_path: Path, cache_root: Path, precision: enum {fp16, int8}.
  • BackboneSpec (@dataclass(frozen=True)): model_name: str, onnx_path: Path, expected_input_shape: tuple[int, ...].
  • EngineCacheEntry (@dataclass(frozen=True)): model_name: str, engine_path: Path, sidecar_path: Path, outcome: enum {built, reused}, compile_duration_s: float | None, engine_sha256_hex: str.
• Method flow:
  1. For each BackboneSpec: a. Detect runtime hardware (SM, JP, TRT version) via inference_runtime.host_info(). b. Compute the target filename via filename_schema.format(...): {model}_{sm}_{jp}_{trt}_{precision}.engine. c. Compute the target path: {cache_root}/engines/{filename}. d. If target_path.exists() AND sidecar.verify(target_path) returns True:
     • Outcome = reused; emit INFO log kind="c10.engine.cache.hit"; append EngineCacheEntry; continue. e. Else (cache miss):
     • Emit WARN log kind="c10.engine.cache.miss" with {model_name, target_filename}.
     • Call inference_runtime.compile_engine(onnx_path, calibration_path, precision, expected_input_shape) -> bytes (raises EngineBuildError or CalibrationCacheError on failure — propagate).
     • Write the engine bytes via sidecar.write_with_sidecar(target_path, engine_bytes) (atomic write + SHA-256 sidecar at {target_path}.sha256).
     • Outcome = built; record compile_duration_s from time.monotonic() deltas; append EngineCacheEntry.
  2. Return the list. Aggregate count: engines_built, engines_reused, total cache hit ratio. INFO log kind="c10.engine.compile.summary" with the totals.
• The composition root constructs EngineCompiler and injects it into the T5 CacheProvisioner. Factory: build_engine_compiler(config) -> EngineCompiler.
• A BackboneSpec registry at src/gps_denied_onboard/runtime_root/c10_factory.py enumerates the project's backbones (initially DINOv2-VPR + LightGlue + ALIKED — cross-referenced against E-C2/E-C2.5/E-C3 component descriptions). The list is config-driven via config.c10.backbones: list[BackboneSpec] so a future model addition does not require code change.
• INFO log on every cache hit; WARN on every cache miss; ERROR on EngineBuildError / CalibrationCacheError with the offending model name.

Scope

Included

• EngineCompiler class with the single public method.
• The 3 DTOs (EngineCompileRequest, BackboneSpec, EngineCacheEntry) plus their enum types.
• Hardware-tied filename construction via AZ-281's schema.
• Cache-hit detection via sidecar.verify (sha256 sidecar matches).
• Cache-miss compile via AZ-298's InferenceRuntime.compile_engine.
• Atomic engine write + sidecar via AZ-280.
• Composition-root factory.
• Conformance test: a fake InferenceRuntime returns scripted engine bytes; the test asserts cache hit / miss outcomes for the documented matrix.
• Per-cache-entry timing instrumentation.
• config.c10.backbones schema extension on AZ-269's loader.

Excluded

• The orchestration of when to compile (T5 owns build_cache_artifacts).
• Descriptor generation (T2 owns).
• Manifest writing (T3 owns).
• TensorRT internals — owned by AZ-298 (the compile_engine impl); this task only consumes the protocol.
• Engine deserialization at takeoff — owned by AZ-298 (load side) + the C7 component runtime self-check.
• Engine version compatibility checks across deployments — out of scope; the filename schema (AZ-281) carries enough signal that mismatches surface as cache miss.
• Multi-GPU compile — operator workstation is single-GPU per RESTRICT-OPS-2.
• A re-build-now CLI flag — operator workflow goes through T5; force-rebuild is achieved by deleting the engine cache directory.

Acceptance Criteria

AC-1: Cold cache compiles every backbone Given an empty cache_root/engines/ and 3 backbones in BackboneSpec[] When compile_engines_for_corpus(request) is called Then 3 EngineCacheEntry are returned, all with outcome = built; 3 .engine files + 3 .sha256 sidecars are present at cache_root/engines/; ONE WARN log per backbone (c10.engine.cache.miss); ONE INFO log summary with engines_built=3, engines_reused=0

AC-2: Warm cache reuses every backbone Given the same cache_root/engines/ populated by a prior cold run When compile_engines_for_corpus(request) is called with identical request Then 3 EngineCacheEntry are returned, all outcome = reused; ZERO calls to inference_runtime.compile_engine (verifiable via spy); ONE INFO log per backbone (c10.engine.cache.hit); summary log shows engines_reused=3

AC-3: Mixed cache (1 hit + 2 miss) Given the cache contains only the DINOv2 engine; LightGlue and ALIKED are missing When compile_engines_for_corpus(request) is called Then DINOv2 → reused, LightGlue + ALIKED → built; the report shows engines_built=2, engines_reused=1

AC-4: Hardware change invalidates cache Given a cache populated for (sm=87, jp=6.2, trt=10.3, fp16) and the runtime now reports (sm=89, jp=6.3, trt=10.5, fp16) When compile_engines_for_corpus(request) is called Then ALL backbones have outcome = built (the filename differs, so the existing engines are not even consulted); the existing engines remain on disk (this task does NOT delete stale engines — that's the orchestrator's call)

AC-5: Tampered sidecar invalidates that one engine Given a .engine file matches its sidecar but a malicious actor flipped a bit in the sidecar (or the engine bytes drifted) When compile_engines_for_corpus(request) is called Then sidecar.verify returns False for that entry; that backbone is recompiled (outcome = built); ONE WARN log kind="c10.engine.sidecar.mismatch" with the offending path

AC-6: EngineBuildError propagates without partial state Given inference_runtime.compile_engine raises EngineBuildError("CUDA OOM") on the second of 3 backbones When compile_engines_for_corpus(request) is called Then EngineBuildError is raised; the first backbone's engine + sidecar ARE present (already-written cache reuse from prior runs); the second backbone's engine is NOT half-written (atomic write); the third backbone is NOT attempted; ONE ERROR log with the model name

AC-7: CalibrationCacheError propagates with diagnostic Given inference_runtime.compile_engine raises CalibrationCacheError("calibration table missing for INT8") When the compiler hits the failing backbone Then the error propagates; ONE ERROR log with {model_name, calibration_path}; partial state is consistent (atomic writes guarantee no half-engine on disk)

AC-8: Filename schema + sidecar layout matches spec Given a freshly-built DINOv2 engine on Tier-2 hardware (SM 87, JP 6.2, TRT 10.3, FP16) When inspecting cache_root/engines/ Then the file is named dinov2_vpr_sm87_jp62_trt103_fp16.engine; the sidecar at dinov2_vpr_sm87_jp62_trt103_fp16.engine.sha256 contains the 64-char hex digest; both match EngineFilenameSchema.parse and Sha256Sidecar.verify

AC-9: compile_duration_s recorded for built; None for reused Given a mix of hits and misses When inspecting EngineCacheEntry Then compile_duration_s is not None for every built entry; compile_duration_s is None for every reused entry; built durations are positive floats

AC-10: Empty backbones list returns empty result Given request.backbones == () When compile_engines_for_corpus(request) is called Then [] is returned; ZERO calls to inference_runtime.compile_engine; ZERO files written; ONE INFO log summary with all-zero counts

Non-Functional Requirements

Performance

• Cache-hit path per backbone ≤ 100 ms (one filename construction + one Path.exists + one sidecar verify dominated by SHA-256 of the engine file, which is bounded by disk read bandwidth). For a 200 MB engine, this is ~1 s on NVMe — measure and document.
• Cold compile is dominated by AZ-298's TensorRT runtime; this task imposes no additional time budget beyond AZ-298's.

Compatibility

• AZ-281 (EngineFilenameSchema) and AZ-280 (Sha256Sidecar) are the schema and atomic-write helpers; this task introduces NO new third-party dependencies.

Reliability

• Atomic writes via AZ-280 guarantee no half-engine on disk after a process kill.
• Cache-miss recompile is idempotent — running the same compile twice produces identical bytes (TRT engine determinism is owned by AZ-298; this task assumes it).

Unit Tests

AC Ref	What to Test	Required Outcome
AC-1	Empty cache_root + 3 backbones	All built; sidecars present
AC-2	Warm cache + identical request	All reused; zero compile_engine calls
AC-3	Cache populated for 1 of 3 backbones	1 reused + 2 built
AC-4	Hardware change (different SM in fake runtime)	All built; old engines untouched
AC-5	Tampered sidecar (flip 1 byte)	That engine rebuilds; WARN log
AC-6	Fake runtime raises EngineBuildError mid-run	Error propagates; partial state consistent
AC-7	Fake runtime raises CalibrationCacheError	Error propagates with diagnostic
AC-8	Inspect filename + sidecar layout	Matches schema; both verify
AC-9	Compile_duration recorded	Set on built, None on reused
AC-10	Empty backbones	Empty result; zero side effects
NFR-perf-cache-hit	Microbench cache-hit path × 100 with 200 MB engine	p99 ≤ 1.5 s (mostly SHA-256 read)
NFR-reliability-atomic-write	Kill process mid-compile_engine	No half-engine on disk after restart

Constraints

• The filename schema is canonical via AZ-281; this task does NOT invent its own (per coderule.mdc "follow established project patterns").
• The atomic-write + sidecar pattern is canonical via AZ-280; this task does NOT use open(...).write() or naked pathlib.Path.write_bytes().
• Cache hit is decided by sidecar.verify (file SHA-256 matches sidecar value); filename match alone is NOT sufficient (defends against bit-rot or bit-flip).
• The BackboneSpec registry is config-driven; adding a new model is a config change, not a code change.
• This task does NOT clean up stale engines (the orchestrator T5 may emit ManifestCoverageError on orphan files; cleanup is the operator's call).
• This task introduces no new third-party dependencies.

Risks & Mitigation

Risk 1: SHA-256 verification of large engines is slow on warm path

• Risk: 200 MB engine × 5 backbones = 1 GB SHA-256 per warm idempotent run; on slow disks, this exceeds C10-PT-01's 1 min budget alone.
• Mitigation: AZ-280's Sha256Sidecar.verify uses sendfile / mmap paths where available; benchmark documented in AZ-280. If still too slow, a future task adds an mtime + size quick-check fallback (out of scope this cycle).

Risk 2: Partial cache after EngineBuildError on backbone N

• Risk: Backbones 1..N-1 are built and on disk; the N-th fails; backbones N+1..M are never attempted. The cache is "partially valid" — the orchestrator (T5) sees inconsistent state.
• Mitigation: T5's coverage check + ManifestCoverageError surface this. The compiler does NOT delete the partial state; T5 decides whether to retry, fail, or roll back per the operator's request mode.

Risk 3: TensorRT engine determinism not guaranteed across builds

• Risk: Two compiles of the same ONNX + calibration produce different bytes; cache-hit detection via SHA-256 fails post-rebuild.
• Mitigation: TRT engine determinism is AZ-298's contract obligation; if it fails, this task's cache-hit ratio drops to 0 and operators see WARN logs. AZ-298's tests assert determinism; this task assumes it.

Risk 4: Operator manually edits engine file but not sidecar

• Risk: Hand-debugging or manual tuning leaves an engine file whose bytes don't match its sidecar; AC-5 covers detection.
• Mitigation: AC-5 + WARN log c10.engine.sidecar.mismatch surface the case immediately on next compile run; operators should re-generate via the build command.

Runtime Completeness

• Named capability: TRT engine compile + hardware-tied cache reuse per D-C10-6 + D-C10-7 (description.md § 5; epic § Acceptance C10-IT-05; AC-NEW-1).
• Production code that must exist: real EngineCompiler orchestrating real AZ-298 compile_engine + real AZ-280 atomic write/verify + real AZ-281 filename construction; real config-driven BackboneSpec registry.
• Allowed external stubs: tests MAY use a fake InferenceRuntime that returns scripted bytes + a fake host_info() for hardware variation; production wiring uses the real AZ-298 runtime + real Sha256Sidecar.
• Unacceptable substitutes: a Python-level pickle of a "fake engine" object (TRT engines are opaque CUDA blobs; faking them in production breaks AC-NEW-1's takeoff verify); skipping the sidecar (loses bit-rot detection); inventing a new filename scheme inside this task (defeats D-C10-7); Path.write_bytes() instead of AZ-280 (no atomicity guarantee).