# Source Registry — C7: On-Jetson inference runtime

> Mode A Phase 2 — engine Step 2 (Source Tiering & Exhaustive Web Investigation). Sources backing the C7 cross-cutting integration row ([`../06_component_fit_matrix/C7_inference_runtime.md`](../06_component_fit_matrix/C7_inference_runtime.md)) and C7 fact cards ([`../02_fact_cards/C7_inference_runtime.md`](../02_fact_cards/C7_inference_runtime.md)).
>
> Index: [`00_summary.md`](00_summary.md). Sibling component sources: [C1 VIO](C1_vio.md), [C2 VPR](C2_vpr.md), [C3 Matchers](C3_matchers.md), [C4 Pose](C4_pose_estimation.md), [C5 State estimator](C5_state_estimator.md), [C6 Tile cache](C6_tile_cache_spatial_index.md). Sub-question sources: [SQ6 external positioning](SQ6_external_positioning.md), [SQ1 existing systems](SQ1_existing_systems.md), [SQ2 canonical pipeline](SQ2_canonical_pipeline.md).

---

## Scope summary

C7 is a **cross-cutting integration row** rather than a per-component candidate row: it pins how the C1 VIO learned-frontend (if any), C2 VPR backbone, and C3 matcher actually run on the Jetson Orin Nano Super under JetPack 6 — TensorRT vs ONNX Runtime+TRT EP vs pure PyTorch FP16. Per the user-pinned scope (locked via `/autodev` AskQuestion 2026-05-08 — see `_docs/_autodev_state.md` `c7_breadth=B`, `c7_quantization=A`, `c7_overkill_options=A`), three documentary candidate rows are evaluated: **TensorRT native primary** + **ONNX Runtime + TensorRT EP interop alternate** + **pure PyTorch FP16 mandatory simple-baseline**. INT8 primary + FP16 fallback per candidate; INT8-only candidates Experimental until calibration data exists. Triton / DeepStream / CUDA-Python custom kernels noted-and-rejected in one sentence (server/video-pipeline class or out-of-budget for embedded 8 h mission). Cand-row candidates inherit and propagate Plan-phase gates already opened by C2 (D-C2-5 DINOv2 ViT-export to TensorRT FP16/INT8) and C3 (D-C3-2 LightGlue inference runtime path).

---

## Sources

### Source #99 — NVIDIA TensorRT 10.x official documentation portal (context7-indexed)

**Title**: NVIDIA TensorRT — SDK for optimizing and accelerating deep learning inference on NVIDIA GPUs (mixed precision, dynamic shapes, transformer optimizations)
**Tier**: L1 — official authoritative SDK documentation (NVIDIA primary)
**URL**: <https://docs.nvidia.com/deeplearning/tensorrt/latest/> + context7 indexed at `/websites/nvidia_deeplearning_tensorrt`
**Access date**: 2026-05-08
**Direct verification**: ✅ context7 query "INT8 calibration EntropyCalibrator2 ICudaEngine deserialize Jetson Orin Nano FP16 mixed precision deployment workflow Python builder" returned 9371 code snippets at Source Reputation High + Benchmark Score 75.25.

**Key APIs verified**:
- **INT8 calibrator hierarchy**: `nvinfer1::IInt8Calibrator` (abstract base) + `nvinfer1::IInt8EntropyCalibrator` (deprecated) + `nvinfer1::IInt8EntropyCalibrator2` (current canonical) + `nvinfer1::IInt8MinMaxCalibrator`. Each defines `getBatchSize()` + `getBatch(void* bindings[], const char* names[], int32_t nbBindings)` + `readCalibrationCache(size_t& length)` + `writeCalibrationCache(const void* ptr, size_t length)` + `getAlgorithm()` returning `kENTROPY_CALIBRATION_2` for the canonical path.
- **Python builder INT8 enable pattern** (canonical TensorRT 10.x):
  ```python
  config.set_flag(trt.BuilderFlag.INT8)
  config.int8_calibrator = Int8_calibrator
  Int8_calibrator = EntropyCalibrator(["input_node_name"], batchstream)
  ```
- **Mixed-precision flag pattern**: `config.set_flag(trt.BuilderFlag.FP16)` + `config.set_flag(trt.BuilderFlag.INT8)` for combined FP16+INT8 mixed precision (TensorRT auto-selects per-layer precision based on calibration data).

**Use**: backs Fact #94 (TensorRT native primary candidate) per-mode API verification block + Plan-phase D-C7-1 calibration-dataset-strategy + D-C7-2 mixed-precision flag matrix.

---

### Source #100 — Microsoft ONNX Runtime official documentation (context7-indexed) + Jetson AI Lab community wheel index

**Title**: Microsoft ONNX Runtime — cross-platform ML inference and training accelerator with TensorRT execution provider; Jetson-specific install path via Jetson AI Lab community PyPI index
**Tier**: L1 — official authoritative SDK documentation (Microsoft primary) + L2 community-maintained Jetson wheel index
**URL**: <https://onnxruntime.ai/> + context7 indexed at `/microsoft/onnxruntime` (v1.25.0) + <https://pypi.jetson-ai-lab.io/jp6/cu126/> + <https://github.com/dusty-nv/jetson-containers/issues/1283> + <https://github.com/microsoft/onnxruntime/issues/20503> + <https://github.com/microsoft/onnxruntime/issues/27562>
**Access date**: 2026-05-08
**Direct verification**: ✅ context7 query "TensorRT execution provider TrtFp16Enable TrtInt8Enable TrtCachePath onnxruntime-gpu Jetson ARM64 inference session options" returned 1462 code snippets at Source Reputation High + Benchmark Score 82.23 (highest of the 3 C7 candidate context7 lookups).

**Key APIs verified**:
- **Provider enumeration + config pattern** (canonical Python API):
  ```python
  import onnxruntime as ort
  print(ort.get_available_providers())
  tensorrt_options = {'device_id': 0, 'trt_max_workspace_size': 2147483648, 'trt_fp16_enable': True}
  cuda_options = {'device_id': 0, 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_mem_limit': 2 * 1024 * 1024 * 1024}
  session_trt = ort.InferenceSession(
      "model.onnx",
      providers=[('TensorrtExecutionProvider', tensorrt_options), ('CUDAExecutionProvider', cuda_options), 'CPUExecutionProvider']
  )
  ```
- **Provider-cascade behavior**: ORT TRT EP attempts to optimize each subgraph via TensorRT; falls back to CUDA EP for unsupported ops; falls back to CPU EP if neither GPU EP applies. Subgraph fallback is automatic and per-op transparent.

**Jetson install constraints (CRITICAL)**:
- **Standard `pip install onnxruntime-gpu` does NOT work on Jetson Tegra** — Microsoft does not publish prebuilt aarch64 wheels with CUDA/TensorRT EPs (per Issue #20503: "NVIDIA does not have CI infrastructure to publish them").
- **Canonical install path (JetPack 6 + CUDA 12.6 + Ubuntu 22.04)**: `pip3 install onnxruntime-gpu --index-url https://pypi.jetson-ai-lab.io/jp6/cu126`.
- **Alternate index (CUDA 12.9 + Ubuntu 24.04)**: `pip3 install onnxruntime-gpu --index-url https://pypi.jetson-ai-lab.io/jp6/cu129`.
- **Known incompatibility**: onnxruntime-gpu v1.23.0 wheels for JetPack 6 were built against `numpy<2.0.0`; importing under `numpy>=2.0.0` raises a compatibility error per Issue #27562. Pin numpy<2 in project requirements until upstream rebuild is published.
- **Standard pip install `onnxruntime` (CPU-only) succeeds but exposes only `CPUExecutionProvider` and `AzureExecutionProvider`** — does NOT include CUDA EP or TensorRT EP.

**Use**: backs Fact #95 (ONNX Runtime + TensorRT EP interop alternate candidate) per-mode API verification block + Plan-phase D-C7-3 ORT-Jetson-wheel-pin + D-C7-4 numpy-version-pin.

---

### Source #101 — PyTorch official documentation (context7-indexed) + Jetson AI Lab PyTorch wheel availability for JetPack 6

**Title**: PyTorch — open-source machine learning framework (tensor computation with strong GPU acceleration; tape-based autograd); Jetson-specific wheels available via Jetson AI Lab + NVIDIA forums
**Tier**: L1 — official authoritative SDK documentation (PyTorch Foundation primary) + L1 NVIDIA Developer Forums (canonical Jetson PyTorch distribution channel)
**URL**: <https://pytorch.org/docs/stable/amp.html> + context7 indexed at `/pytorch/pytorch` (v2.5.1, v2.8.0, v2.9.1, v2.11.0) + <https://forums.developer.nvidia.com/t/installing-pytorch-for-jetpack-6-2/349519> + <https://forums.developer.nvidia.com/t/jetpack-6-2-and-pytorch-2-6-0-on-jetson-nano-orin/331972>
**Access date**: 2026-05-08
**Direct verification**: ✅ context7 query "torch.cuda.amp.autocast half precision FP16 inference mode no_grad CUDA Jetson Orin ARM64 model.half() torch.compile inference deployment" returned 4866 code snippets at Source Reputation High + Benchmark Score 76.69.

**Key APIs verified**:
- **`torch.amp.autocast(device_type, dtype, enabled, cache_enabled)`** — canonical AMP context manager (since PyTorch 1.10). Replaces deprecated `torch.cuda.amp.autocast`. Inference pattern:
  ```python
  with torch.no_grad():
      with torch.autocast(device_type='cuda', dtype=torch.float16, enabled=True):
          output = model(input)
  ```
- **`torch.compile(model, backend='inductor')`** — graph-mode optimization for further speedup; tradeoff is cold-start compile cost (~10-60 sec depending on model complexity).
- **`model.half()`** — eager-mode FP16 weight conversion (full-precision FP16 throughout, vs autocast's per-op precision selection).

**Jetson install constraints**:
- **Standard `pip install torch` does NOT include CUDA support on Jetson** — must use NVIDIA-published or Jetson AI Lab community wheels.
- **JetPack 6.2 + CUDA 12.6 + Ubuntu 22.04 + Python 3.10 canonical wheel**: `torch-2.9.0-cp310-cp310-linux_aarch64.whl` from Jetson AI Lab (per NVIDIA forum recommendation). Earlier stable combination: PyTorch 2.5 + torchvision 0.20.
- **Known dependency issues**: missing `libcudss.so.0` and `libnvdla_runtime.so` on PyTorch 2.9 cu129 wheel under JetPack 6.2 (CUDA 12.6) — version mismatch between wheel build target and installed JetPack CUDA. Mitigation: prefer the cu126 variant for JetPack 6.2.
- **CUDA capability**: Jetson Orin Nano Super GPU = compute capability **SM 87** (Ampere class).

**Use**: backs Fact #96 (pure PyTorch FP16 mandatory simple-baseline candidate) per-mode API verification block + D-C7-5 PyTorch-Jetson-wheel-pin.

---

### Source #102 — Ultralytics YOLO26 benchmark suite on Jetson Orin Nano Super (April 2026)

**Title**: Update NVIDIA Jetson Orin Nano Super benchmarks with YOLO26 (Ultralytics 8.4.33; commit 8d4e6e8 April 2026)
**Tier**: L1 — official authoritative benchmark suite (Ultralytics is the canonical YOLO maintainer)
**URL**: <https://github.com/ultralytics/ultralytics/pull/24097> + <https://github.com/ultralytics/ultralytics/commit/8d4e6e841c89f6598b322695cb2bc816eeba8b93>
**Access date**: 2026-05-08
**Direct verification**: ✅ Web search results explicitly cite the per-export-format inference times measured on Jetson Orin Nano Super.

**Key data extracted (YOLO26n on Jetson Orin Nano Super, April 2026 measurement)**:

| Export format | Inference time (ms) | mAP50-95 | Speedup vs FP32 | Accuracy delta vs FP16 |
|---|---|---|---|---|
| TensorRT FP32 | 7.53 | 0.4770 | 1.00× | — |
| TensorRT FP16 | 4.57 | 0.4800 | 1.65× | baseline (slightly higher than FP32 due to noise) |
| TensorRT INT8 | 3.80 | 0.4490 | 1.98× | **-6.5% mAP50-95** |

**Key data extracted (YOLOv8s on Jetson Orin Nano, NVIDIA forum)**:
- **INT8**: ~157 QPS (~6.4 ms/inference)
- **FP16**: ~103 QPS (~9.7 ms/inference)
- **INT8 vs FP16 speedup**: ~1.5× (vs ~1.20× on YOLO26n — model architecture and memory bandwidth dependent)

**Use**: backs Fact #94 (TensorRT) latency claims for object-detection-class CNN backbones on Jetson Orin Nano Super; provides empirical anchor for the engine's "INT8 primary + FP16 fallback" precision strategy. Caveat: YOLO is a detection network; feature-matching networks (LightGlue / DISK / XFeat) are known to be more quantization-sensitive (see Source #103).

---

### Source #103 — LightGlue ONNX Runtime + TensorRT acceleration (canonical reference) + FP8 ModelOpt quantization findings (Fabio Sim's Journal)

**Title**: Accelerating LightGlue Inference with ONNX Runtime and TensorRT (Fabio Sim's Journal, canonical author of `fabio-sim/LightGlue-ONNX`) + FP8 Quantized LightGlue in TensorRT with NVIDIA Model Optimizer (subsequent post)
**Tier**: L1 — canonical author of the canonical LightGlue ONNX/TensorRT export pathway (already cited as Source #73 in C3 row)
**URL**: <https://fabio-sim.github.io/blog/accelerating-lightglue-inference-onnx-runtime-tensorrt/> + <https://fabio-sim.github.io/blog/fp8-quantized-lightglue-tensorrt-nvidia-model-optimizer/> + <https://github.com/qdLMF/LightGlue-with-FlashAttentionV2-TensorRT> (community Jetson Orin NX TensorRT 8.5.2 + FlashAttentionV2 plugin reference implementation)
**Access date**: 2026-05-08
**Direct verification**: ✅ Web search results explicitly cite the 2-4× ONNX Runtime + TensorRT speedup over compiled PyTorch and the FP8 5.97× / 0.32× engine-size results.

**Key data extracted**:
- **LightGlue (transformer-based feature matcher) — ONNX Runtime + TensorRT inference**: 2-4× speedup over compiled PyTorch across various batch sizes and sequence lengths.
- **FP8 quantized LightGlue (NVIDIA ModelOpt) on Hopper/Ada/Blackwell**:
  - Engine size ~0.32× of FP32 (~68% smaller).
  - Up to 5.97× speedup vs FP32.
  - **Material accuracy degradation**: "match counts dropped. Sometimes they dropped hard." This is qualitatively different from YOLO-class detection networks where INT8 is well-tolerated.
  - **FP8 hardware support**: requires Hopper / Ada / Blackwell architecture. **Jetson Orin Nano Super is Ampere (SM 87) — NOT FP8-native**. FP8 ModelOpt path applies only via INT8 emulation fallback on Ampere.
- **Two FP8 formats**: E4M3 (4 exponent bits + 3 mantissa bits, better precision for activations) + E5M2 (5 exponent bits + 2 mantissa bits, better dynamic range for gradients).
- **Community Jetson reference implementation**: `qdLMF/LightGlue-with-FlashAttentionV2-TensorRT` deploys on Jetson Orin NX 8 GB with TensorRT 8.5.2 + custom FlashAttentionV2 plugin.

**Use**: backs Fact #94 (TensorRT) feature-matching-network INT8 caveat; backs the "INT8-only candidates Experimental until calibration data exists" engine ruling per user-pinned `c7_quantization=A` scope; raises Plan-phase gate D-C7-6 INT8-vs-FP16-per-model-family-precision-policy.

---

### Source #104 — JetPack SDK release notes (NVIDIA official) — JetPack 6.0 / 6.1 / 6.2 version matrix

**Title**: NVIDIA JetPack 6.x SDK Release Notes — TensorRT/CUDA/cuDNN versions per release; Super Mode introduction in JetPack 6.2 (January 2025)
**Tier**: L1 — official authoritative release notes (NVIDIA Developer)
**URL**: <https://developer.nvidia.com/embedded/jetpack-sdk-60> + <https://developer.nvidia.com/embedded/jetpack-sdk-61> + <https://developer.nvidia.com/embedded/jetpack-sdk-62> + <https://developer.nvidia.com/blog/nvidia-jetpack-6-2-brings-super-mode-to-nvidia-jetson-orin-nano-and-jetson-orin-nx-modules/>
**Access date**: 2026-05-08
**Direct verification**: ✅ Web search results explicitly enumerate TensorRT / CUDA / cuDNN per JetPack release.

**Key data extracted**:

| JetPack | CUDA | TensorRT | cuDNN | Super Mode | Released |
|---|---|---|---|---|---|
| 6.0 | 12.2 | 8.6 | 8.9 | No | early 2024 |
| 6.1 | 12.6 | 10.3 | 9.3 | MAXN mode (dev kit only) | mid-2024 |
| **6.2** | **12.6** | **10.3** | **9.3** | **YES — Orin Nano Super + Orin NX production modules** | **2025-01-16** |

- **Super Mode performance gains** (vs base Orin Nano): up to 2× higher generative AI inference performance, 70% AI TOPS increase, 50% memory bandwidth boost.
- **TensorRT 10.3** is the canonical inference runtime version for JetPack 6.1 / 6.2 deployments. Major API upgrade from TensorRT 8.x → 10.x — `IInt8EntropyCalibrator2` API surface is preserved; `INetworkDefinition` and `IBuilderConfig` semantics unchanged.

**Use**: pins the project's target software stack to **JetPack 6.2 + CUDA 12.6 + TensorRT 10.3 + cuDNN 9.3 + Super Mode enabled** for the Jetson Orin Nano Super target hardware. Backs Facts #94, #95, #96 deployability claims.

---

### Source #105 — TensorRT-on-Jetson canonical install constraints (Ultralytics issue reports + NVIDIA forum)

**Title**: TensorRT 10.x on Jetson Orin Nano — install path, hardware-specificity, memory-pressure-during-build constraints
**Tier**: L2 — community-reported issues with NVIDIA-acknowledged root causes (high signal-to-noise on canonical constraints)
**URL**: <https://github.com/ultralytics/ultralytics/issues/18882> ("TensorRT does not currently build wheels for Tegra systems") + <https://forums.developer.nvidia.com/t/tensorrt-10-7-0-on-orin-nano/364236> (SM 87 compute-capability mismatch) + <https://github.com/ultralytics/ultralytics/issues/18730> (laptop-GPU-built engine cannot load on Jetson) + <https://github.com/ultralytics/ultralytics/issues/21281> (TensorRT export memory pressure on Orin AGX)
**Access date**: 2026-05-08
**Direct verification**: ✅ Web search returned direct issue links with NVIDIA-confirmed root causes.

**Key constraints extracted** (CRITICAL for C7 deployment design):

1. **TensorRT Python wheels are NOT installed via pip on Jetson Tegra**. Standard `pip install tensorrt` raises: `RuntimeError: TensorRT does not currently build wheels for Tegra systems`. The canonical install path is the JetPack-bundled TensorRT (already present after `apt install nvidia-jetpack`), accessed via the system Python at `/usr/lib/python3.10/dist-packages/tensorrt`.
2. **TensorRT engines are hardware-specific** — engines built against a laptop / dev-machine GPU CANNOT be loaded on the Jetson at runtime. **Engines must be built directly on the Jetson target**.
3. **GPU compute capability mismatch is silent at build-time, fatal at load-time**: laptop GPUs (e.g., RTX 4090 = SM 89) and Jetson Orin Nano Super (SM 87) produce incompatible engines; the build emits no error, the load logs `Target GPU SM 87 is not supported by this TensorRT release` — version-and-SM-compatibility matrix must be respected.
4. **TensorRT engine builds on Jetson under memory pressure can segfault during tactic profiling** (8 GB shared CPU+GPU is tight; a rich layer-fusion search consumes peak RAM during `tactic.profile` phase). Mitigation: limit `config.max_workspace_size` to a fraction of the budget (e.g., 1-2 GB) and avoid concurrent inference / Postgres / FAISS during builds.
5. **JetPack 6.x ships the canonical TensorRT version** (TensorRT 10.3 for JP 6.1/6.2 per Source #104); upgrading TensorRT independently of JetPack is not officially supported.

**Use**: drives D-C7-7 build-on-Jetson-vs-prebuilt-engine-shipping-strategy + D-C7-8 max-workspace-size-cap-for-build-stability + D-C7-9 SM-compatibility-version-pin.

---

(Subsequent sources #106+ added during fact extraction below as candidate-specific evidence is gathered. Closure target: 3 candidate rows + 1 cross-cutting integration matrix.)
