# C7 TensorrtRuntime — Engine Compile + Deserialize + Infer + GPU Memory

**Task**: AZ-298_c7_tensorrt_runtime
**Name**: C7 TensorrtRuntime
**Description**: Implement `TensorrtRuntime`, the production-default `InferenceRuntime` strategy on JetPack 6.2 + TensorRT 10.3 (per D-C7-9). Owns the full TRT lifecycle: engine compilation via the Polygraphy / trtexec / IBuilderConfig hybrid (FP16 + INT8 + Mixed precision; INT8 calibration cache trust enforcement); engine deserialization at F2 takeoff load (delegating manifest content-hash + filename schema validation to AZ-301 EngineGate); per-flight resident `EngineHandle` GPU memory management; sync per-call `infer` on the F3 hot path with per-model latency budgets from C7-PT-01; release on flight end; CUDA stream ownership.
**Complexity**: 5 points
**Dependencies**: AZ-297_c7_runtime_protocol, AZ-301_c7_engine_gate, AZ-280_sha256_sidecar, AZ-281_engine_filename_schema, AZ-263_initial_structure, AZ-269_config_loader, AZ-266_log_module
**Component**: c7_inference (epic AZ-249 / E-C7)
**Tracker**: AZ-298
**Epic**: AZ-249 (E-C7)

### Document Dependencies

- `_docs/02_document/contracts/c7_inference/inference_runtime_protocol.md` — the Protocol this task implements; produced by AZ-297.
- `_docs/02_document/contracts/shared_helpers/engine_filename_schema.md` — the schema parser used at deserialise time (delegated to EngineGate).
- `_docs/02_document/contracts/shared_helpers/sha256_sidecar.md` — engine sidecar trust check (delegated to EngineGate).
- `_docs/02_document/contracts/shared_config/composition_root_protocol.md` — Config object that provides engine cache dir, calibration dataset path, precision selection.

## Problem

Without a real TensorRT 10.3 strategy, the on-Jetson hot path cannot meet the C7-PT-01 per-model latency targets (UltraVPR ≤ 60 ms, LightGlue ≤ 30 ms, AdHoP ≤ 90 ms, DISK ≤ 50 ms p95) and the AC-4.1 system E2E budget < 400 ms p95 collapses. The Protocol from AZ-297 is just types; this task is what actually runs ML on the Jetson Orin Nano Super.

Concretely, without this task:

- F1 pre-flight engine compilation has no production producer; C10 CacheProvisioner cannot build the engine cache.
- F2 takeoff load has no engine deserialiser; flights cannot start.
- F3 hot path has no `infer`; every consumer (C2 / C2.5 / C3 / C3.5) has nothing to call.
- INT8 calibration cache trust (D-C10-6) has no production gate at compile time.
- Per-flight GPU memory budget (NFT-LIM-01: ≤ 4 GB resident across all engines, C7-PT-02) has no enforcement point.

This task delivers the canonical production runtime — every other strategy (AZ-299 ONNX-RT, AZ-300 PyTorch) is a fallback against this one's numbers.

## Outcome

- A `TensorrtRuntime` class at `src/gps_denied_onboard/components/c7_inference/tensorrt_runtime.py` conforming to the `InferenceRuntime` Protocol from AZ-297; `current_runtime_label() == "tensorrt"`.
- `compile_engine(model_path, build_config) -> EngineCacheEntry` produces an FP16 / INT8 / Mixed engine using the Polygraphy + trtexec + IBuilderConfig hybrid: Polygraphy for orchestration + ONNX import; trtexec for build profiling and binary outputs where appropriate; IBuilderConfig for fine-grained TRT controls (workspace size, optimization profiles, INT8 calibrator). The output `.engine` file lands at the cache path with the D-C10-7 filename schema and a sha256 sidecar from AZ-280.
- INT8 calibration cache is trusted iff its content hash matches the calibration-dataset hash recorded at compile time. A mismatch raises `CalibrationCacheError` and forces a full rebuild — never a silent fallback.
- `deserialize_engine(EngineCacheEntry) -> EngineHandle` calls AZ-301 `EngineGate` first (raises `EngineHashMismatchError` / `EngineSchemaMismatchError` / `EngineSidecarMissingError` on refusal); on success, builds an `IExecutionContext`, allocates GPU buffers per the optimization profile's `opt_shape`, allocates one CUDA stream, and returns a `TrtEngineHandle` (`EngineHandle` subclass) that holds the runtime-resident state.
- `infer(handle, inputs) -> outputs` does sync GPU stream execution: H2D copy of every named input → `enqueueV3` on the engine's exec context → D2H copy of every named output → stream sync → return. No pinned-memory pool reuse beyond what TRT itself allocates (keeps the simple-baseline path viable).
- `release_engine(handle)` frees GPU buffers, destroys the exec context, releases the CUDA stream. Called once per `EngineHandle` at flight end (or on `OutOfMemoryError`-driven cleanup).
- Concurrent `infer` calls against different engines are serialised on a single CUDA stream per Runtime instance (matches the description.md "typically one stream because the F3 hot path is single-threaded").
- `EngineBuildError` on compile failure (e.g., ONNX op unsupported by TRT 10.3 plugins); `EngineDeserializeError` on engine corruption; `InferenceError` on transient CUDA fault; `OutOfMemoryError` on GPU OOM with the offending engine name in the message.
- The `current_runtime_label()` returns `"tensorrt"` exactly; the FDR-stamped runtime label is consistent across logs, FDR records, and operator post-flight inspection.

## Scope

### Included

- `TensorrtRuntime` class implementation conforming to the AZ-297 Protocol.
- `compile_engine`: Polygraphy network creation + ONNX parser + IBuilderConfig setup (workspace MB from `BuildConfig`, optimization profiles, FP16 / INT8 / Mixed flags) + INT8 calibrator wiring with calibration-dataset path from `BuildConfig.calibration_dataset` + trtexec invocation for build (subprocess) when `BuildConfig.use_trtexec=True` (faster on FP16; IBuilderConfig direct path is the default for INT8). Output: `.engine` file written via `helpers.sha256_sidecar.atomic_write_with_sidecar`.
- INT8 calibration cache trust: at compile time, write a `.calib_cache` file alongside the `.engine` and a sidecar `.calib_cache.sha256`. The calibration dataset's content hash is stamped into the cache header. On reuse, the calibrator reads the existing cache iff the dataset hash matches; otherwise rebuilds and overwrites — never silently uses a stale cache.
- `deserialize_engine`: invokes AZ-301 `EngineGate.validate(EngineCacheEntry)` first (raising the gate's documented errors); then loads the engine via `IRuntime.deserialize_cuda_engine`, builds `IExecutionContext`, allocates GPU buffers from `opt_shape`, returns `TrtEngineHandle`.
- `infer`: sync GPU stream execution per the description.md § 5; H2D and D2H via `cudaMemcpyAsync` on the owned stream; `enqueueV3`; stream sync; per-frame DEBUG log with backbone name and elapsed milliseconds.
- `release_engine`: destroys the exec context, frees GPU buffers, releases the stream; idempotent on a handle that was already released (returns silently).
- `thermal_state()` is delegated to AZ-302's publisher via a constructor-injected `ThermalStatePublisher` reference; this task wires the delegation, AZ-302 owns the polling loop.
- Per-engine GPU memory budget enforcement: at deserialize time, sum the buffer allocations against `config.inference.gpu_memory_budget_bytes` (default 4 GB per C7-PT-02). Refuse to deserialize if the budget would be exceeded — raises `OutOfMemoryError` BEFORE allocating, with a message identifying which engine pushed it over.
- Error envelope per the AZ-297 Protocol: `EngineBuildError`, `EngineDeserializeError`, `CalibrationCacheError`, `InferenceError`, `OutOfMemoryError`.
- Diagnostic INFO log on `compile_engine` start/end with elapsed seconds and output path; INFO log on `deserialize_engine` with engine identity + warm-up confirmation; per-frame DEBUG log on `infer` (off by default; enabled by config).
- A standalone CLI entry point `python -m c7_inference.tensorrt_runtime compile <onnx_path> <build_config_path>` is exposed for C10 CacheProvisioner to invoke pre-flight without holding a runtime instance. The CLI is a thin wrapper around `compile_engine`; it is not a separate compile path.

### Excluded

- AZ-301 EngineGate validation logic (filename-schema parse, manifest content-hash check) — this task only INVOKES the gate.
- AZ-302 ThermalState polling thread — this task only delegates to a publisher reference.
- AZ-299 OnnxTrtEpRuntime fallback — separate task.
- AZ-300 PytorchFp16Runtime baseline — separate task.
- C10 CacheProvisioner orchestration of compile_engine — owned by E-C10. This task exposes the API; C10 calls it.
- Engine warm-up beyond the deserialize-side buffer allocation (the warm-up that AC-NEW-1 measures end-to-end is owned by C10's pre-flight orchestration; the per-engine warm-up cost lives here).
- ONNX op unsupported by TRT 10.3 — out of scope to "fix" via plugins; raises `EngineBuildError` and the operator chooses ONNX-RT fallback (AZ-299).
- Multi-stream concurrent execution — out of scope this cycle (description.md notes the F3 hot path is single-threaded; future task if needed).

## Acceptance Criteria

**AC-1: compile_engine produces an FP16 .engine + sidecar at the canonical path**
Given a valid ONNX model and `BuildConfig(precision=Fp16, ...)`
When `compile_engine(model_path, build_config)` runs to completion
Then a `.engine` file exists at the D-C10-7 filename schema path with a matching `.sha256` sidecar from AZ-280; the returned `EngineCacheEntry` carries that path, the sha256, and the `(SM, JP, TRT, precision)` tuple

**AC-2: compile_engine produces an INT8 .engine with a calibration cache + sidecar**
Given a valid ONNX model, a calibration dataset directory with at least 100 images, and `BuildConfig(precision=Int8, calibration_dataset=...)`
When `compile_engine` runs
Then a `.engine` file is produced AND a `.calib_cache` file is produced with a `.calib_cache.sha256` sidecar; the cache header records the calibration-dataset content hash; a second `compile_engine` call with the same dataset reuses the cache (verifiable: the second call is < 30 s, the first is minutes)

**AC-3: stale calibration cache forces rebuild**
Given an existing `.calib_cache` with dataset hash A, and the dataset on disk now hashes to B
When `compile_engine` runs with the same calibration_dataset path
Then the calibrator detects the mismatch, rebuilds the cache from scratch (cache header now records hash B), and writes a new sidecar; the prior cache is overwritten — no silent reuse, no `CalibrationCacheError`

**AC-4: corrupted calibration cache raises CalibrationCacheError**
Given an existing `.calib_cache` whose sidecar `.calib_cache.sha256` does not match the cache file's actual sha256
When `compile_engine` runs
Then `CalibrationCacheError` is raised; no engine is built; the cache file is NOT silently overwritten (operator must explicitly delete or replace)

**AC-5: deserialize_engine invokes EngineGate before any GPU allocation**
Given an `EngineCacheEntry` whose engine file's filename schema does not match the running Jetson tuple
When `deserialize_engine(entry)` is called
Then `EngineGate.validate(entry)` is invoked first, raises `EngineSchemaMismatchError`; no `IRuntime.deserialize_cuda_engine` call is made; no GPU memory is allocated (verifiable via NVML before/after delta)

**AC-6: deserialize_engine refuses when GPU memory budget would be exceeded**
Given a deserialize that would allocate 1.2 GB and a runtime instance currently holding 3.0 GB resident with budget = 4 GB
When `deserialize_engine` is called
Then `OutOfMemoryError` is raised with the engine name in the message; no partial allocation occurs; the prior 3.0 GB resident state is unchanged

**AC-7: infer round-trips H2D + enqueueV3 + D2H on the owned stream**
Given a deserialised `TrtEngineHandle` and a fixed input dict
When `infer(handle, inputs)` is called and a profiling tool counts CUDA API events
Then the call sequence is: `cudaMemcpyAsync(H2D)` × N inputs, `enqueueV3`, `cudaMemcpyAsync(D2H)` × M outputs, `cudaStreamSynchronize`; output dict has the M expected named tensors

**AC-8: per-model latency p95 budget**
Given the production-default UltraVPR / LightGlue / AdHoP / DISK FP16 engines deserialised on Tier-2 (Jetson Orin Nano Super)
When the C7-PT-01 latency benchmark runs (scripted load matching production)
Then per-model p95 latencies are: UltraVPR ≤ 60 ms, LightGlue ≤ 30 ms, AdHoP ≤ 90 ms, DISK ≤ 50 ms; failure thresholds (100 / 60 / 150 / 90 ms respectively) are NOT crossed

**AC-9: GPU memory budget compliance**
Given all production-default engines resident concurrently
When the C7-PT-02 memory benchmark runs
Then GPU resident memory across engines ≤ 4 GB (failure threshold 5 GB) and process RAM ≤ 1.5 GB (failure threshold 2 GB)

**AC-10: release_engine fully frees GPU state and is idempotent**
Given a deserialised handle holding 1 GB GPU memory
When `release_engine(handle)` is called once and then again
Then after the first call NVML reports 1 GB freed; after the second call no error is raised and NVML state is unchanged

## Non-Functional Requirements

**Performance**
- Per-model p95 latencies as in AC-8.
- GPU memory ≤ 4 GB resident, RAM ≤ 1.5 GB process — AC-9 / NFT-LIM-01 / NFT-LIM-04.
- `compile_engine` for FP16 sub-minute typical; INT8 minutes (calibration dominates) — bounded by hardware, not budget.
- `deserialize_engine` p95 ≤ 5 s per engine on Tier-2 (engineering sanity bound; AC-NEW-1 cold-start budget is end-to-end and is asserted by C7-IT-01 in the test phase).

**Compatibility**
- TensorRT 10.3 only this cycle (per D-C7-9 JetPack 6.2 lock). No 9.x / 8.x compatibility shims.
- Built on Polygraphy + trtexec + IBuilderConfig — three TRT-supported entry points; no custom TRT builder calls outside that surface.

**Reliability**
- All errors are caught and rewrapped into the AZ-297 family. No raw `RuntimeError` or TRT-specific exception leaks to consumers.
- `infer` NEVER blocks on the producer side; sync stream execution is on the consumer's calling thread.
- INT8 calibration cache trust is the lurking foot-gun (per description.md § 7); the AC-2 / AC-3 / AC-4 flow is the only protection. A new strategy or refactor MUST preserve these three.

**Concurrency**
- One CUDA stream per `TensorrtRuntime` instance. The F3 hot path is single-threaded (per description.md); concurrent `infer` calls from different threads are NOT supported this cycle (would corrupt the stream). Documented as a constraint.
- Compile path (`compile_engine`) is allowed to run while a runtime instance is also serving `infer` calls only if the compile is in a separate process (the C10 CLI entry point); same-process concurrent compile + infer is NOT supported.

## Unit Tests

| AC Ref | What to Test | Required Outcome |
|--------|-------------|-----------------|
| AC-1 | compile FP16 with a tiny ONNX (sanity) | Engine + sidecar produced at canonical path; EngineCacheEntry shape correct |
| AC-2 | compile INT8 with a 100-image calibration dataset; rerun | Cache + sidecar produced; rerun reuses cache (under 30 s); cache header has dataset hash |
| AC-3 | mutate calibration dataset; rerun compile | Cache rebuilt; new dataset hash in header |
| AC-4 | corrupt calibration sidecar; rerun compile | `CalibrationCacheError`; engine NOT built |
| AC-5 | deserialise an engine with a mismatched filename SM | `EngineSchemaMismatchError` from gate; no GPU allocation (NVML diff = 0) |
| AC-6 | deserialise that would push past budget | `OutOfMemoryError`; prior resident state unchanged |
| AC-7 | infer with a profiler attached | Exact CUDA call sequence: N H2D, enqueueV3, M D2H, stream sync |
| AC-8 | C7-PT-01 microbench against the four production engines | p95 within budgets per the table |
| AC-9 | C7-PT-02 memory test with all engines resident | ≤ 4 GB GPU, ≤ 1.5 GB RAM |
| AC-10 | release_engine called twice | First call frees memory; second is a no-op |
| NFR-perf-deserialize | Microbench deserialise per engine | p95 ≤ 5 s on Tier-2 |
| NFR-reliability-error-rewrap | Inject a TRT C++ exception via mock | Rewrapped into the AZ-297 family; original message preserved |

## Constraints

- TensorRT 10.3 + JetPack 6.2 lock per D-C7-9.
- Polygraphy + trtexec + IBuilderConfig only — no custom TRT builder API outside this surface.
- `helpers.sha256_sidecar` (AZ-280) for atomic write + sidecar pattern.
- `helpers.engine_filename_schema` (AZ-281) for the D-C10-7 filename schema (consumed at deserialise via the gate).
- The `EngineHandle` subclass `TrtEngineHandle` is opaque to consumers; consumers MUST NOT introspect its fields. Implementation may add diagnostic fields (e.g., `last_infer_elapsed_ms`) but they are NOT part of the AZ-297 Protocol.
- The CLI entry point `python -m c7_inference.tensorrt_runtime compile ...` is for C10 to invoke; not part of any consumer's public API.
- This task introduces no new third-party dependencies beyond TensorRT 10.3, Polygraphy, and the trtexec binary that ships with TRT.
- Per-frame DEBUG logging defaults to OFF (would flood at 39 Hz aggregate per description.md); enabled only via `config.inference.per_frame_debug_log = true`.

## Risks & Mitigation

**Risk 1: ONNX op unsupported by TRT 10.3**
- *Risk*: A backbone author exports an op TRT cannot lower; `compile_engine` raises `EngineBuildError` and the operator has nothing to fall back to.
- *Mitigation*: `EngineBuildError` is surfaced to the operator pre-flight (per description.md error-handling spec); the operator switches the runtime config to `onnx_trt_ep` (AZ-299) which has wider op coverage. Documented in the operator playbook (out of scope here).

**Risk 2: INT8 calibration silently uses a stale cache**
- *Risk*: AC-3 / AC-4 fails in a corner case (dataset mutated atomically, hash check races with compile start).
- *Mitigation*: The calibration-dataset hash is computed at the START of `compile_engine` and compared to the cache header's hash; the dataset is treated as immutable for the duration of the call. AC-3 + AC-4 cover the obvious cases; the corner case requires a separate test where the dataset is replaced mid-compile, which is operator error and out of scope.

**Risk 3: GPU memory budget is exceeded under sustained use due to TRT internal scratch**
- *Risk*: `deserialize_engine` reports buffer size based on `opt_shape`, but TRT may allocate additional internal scratch beyond what is reported.
- *Mitigation*: AC-9 measures actual NVML resident memory, not reported buffer size; a regression here fails the test. The budget includes a safety margin (4 GB target with 5 GB hard fail) that absorbs typical TRT scratch.

**Risk 4: trtexec subprocess hangs on a malformed ONNX**
- *Risk*: trtexec can hang silently on certain malformed inputs.
- *Mitigation*: The trtexec invocation has a config-driven timeout (default 600 s = 10 minutes); on timeout, the subprocess is killed and `EngineBuildError("trtexec timeout after 600s")` is raised. The IBuilderConfig direct path is preferred for INT8 anyway; trtexec is mainly for FP16 build profiling.

## Runtime Completeness

- **Named capability**: TensorRT 10.3 production runtime + Polygraphy / trtexec / IBuilderConfig hybrid + INT8 calibration cache trust + GPU memory budget enforcement (architecture / E-C7 / D-C7-9 / NFT-PERF-01 / NFT-LIM-01).
- **Production code that must exist**: real `TensorrtRuntime` class implementing the AZ-297 Protocol; real Polygraphy + trtexec + IBuilderConfig compile path; real `IRuntime.deserialize_cuda_engine` + `IExecutionContext` + GPU buffer allocations; real `enqueueV3` + sync-stream execution; real INT8 calibrator with dataset-hash cache trust; real GPU memory budget check at deserialise.
- **Allowed external stubs**: tests MAY substitute a `FakeCudaProfiler` to count CUDA events (AC-7); production wiring uses real TRT + real CUDA. C10 CacheProvisioner is the production caller of `compile_engine`; tests MAY drive the CLI directly.
- **Unacceptable substitutes**: a Python-level fake "engine" that bypasses TRT (would defeat the whole point), a calibration cache "always trusted" path (would break D-C10-6), an `infer` that uses `torch.compile` instead of `enqueueV3` (that's AZ-300 PyTorch baseline territory), or an in-memory engine that skips the file-on-disk path (would break the engine cache lifecycle that C10 + F2 takeoff load depend on).
