# TensorRT Engine Deployment on NVIDIA Jetson Orin Nano Super — Research Summary

**Context**: GPS-denied UAV visual navigation system with models: SuperPoint, LightGlue, DINOv2 ViT-S/14, LiteSAM, EfficientLoFTR. Target: migrate all to TensorRT Engine format on Jetson Orin Nano Super.

---

## 1. Jetson Orin Nano Super Specifications

| Spec | Value |
|------|-------|
| **GPU Architecture** | NVIDIA Ampere |
| **CUDA Cores** | 1,024 |
| **Tensor Cores** | 32 |
| **GPU Clock** | 1,020 MHz (Super Mode) vs 635 MHz (original) |
| **AI Performance** | 67 TOPS (Sparse) / 33 TOPS (Dense) / 17 FP16 TFLOPs |
| **Memory** | 8 GB LPDDR5, 128-bit |
| **Memory Bandwidth** | 102 GB/s (Super Mode) vs 68 GB/s (original) |
| **CPU** | 6-core Arm Cortex-A78AE v8.2 64-bit @ 1.7 GHz |
| **Power Modes** | 15W, 25W, MAXN SUPER (uncapped) |
| **Compute Capability** | 8.7 (Ampere) |

**Sources**: [NVIDIA Jetson Orin Nano Super Blog](https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-developer-kit-gets-a-super-boost/), [JetPack 6.2 Release Notes](https://docs.nvidia.com/jetson/archives/jetpack-archived/jetpack-62/release-notes/index.html)

---

## 2. TensorRT vs ONNX Runtime on Jetson — Inference Speed Comparison

| Aspect | TensorRT (Native) | ONNX Runtime (TensorRT EP) |
|-------|-------------------|----------------------------|
| **Throughput** | Higher (typically 2–4× vs PyTorch) | Lower than native TRT; ~3× gap in some benchmarks |
| **Inception benchmark** | ~3× higher throughput than ONNX RT TRT EP | [GitHub #11356](https://github.com/microsoft/onnxruntime/issues/11356) |
| **SuperPoint+LightGlue** | Up to 4× speedup vs PyTorch (RTX4080) | TensorRT EP fastest among ORT providers |
| **Ease of use** | Manual GPU memory, bindings, session setup | Few lines of Python, simpler API |
| **Recommendation** | Maximum throughput on Jetson | Cross-platform portability, simpler dev |

**Typical speedup ONNX Runtime → TensorRT**: Single-digit to high double-digit %; model-dependent. One Jetson Xavier AGX case: ~3× (128.97 vs 41.11 queries/sec).

**Sources**: [PUT Vision Lab comparison](https://putvision.github.io/article/2021/12/20/jetson-onnxruntime-tensorrt.html), [LightGlue ONNX/TRT blog](https://fabio-sim.github.io/blog/accelerating-lightglue-inference-onnx-runtime-tensorrt/), [ONNX Runtime #11356](https://github.com/microsoft/onnxruntime/issues/11356)

---

## 3. TensorRT Conversion Pipeline

### Option A: PyTorch → ONNX → TensorRT Engine

1. Export PyTorch to ONNX (`torch.onnx.export`)
2. Simplify ONNX (`onnx-simplifier`)
3. Convert ONNX to TRT engine (`trtexec` or TensorRT Python API)

**Typical requirements**: CUDA 11.6+, PyTorch 1.9+, ONNX 1.11+, TensorRT 8.4+

### Option B: Torch-TensorRT (Direct)

**JIT with torch.compile**:
```python
optimized_model = torch.compile(model, backend="tensorrt")
```

**Ahead-of-time (torch.export)**:
```python
torch_tensorrt.compile(model, ir="dynamo", inputs)
```

**Compilation phases**: Lowering → Partitioning → Conversion (PyTorch ops → TRT ops) → Optimization (engine build).

**Sources**: [Torch-TensorRT docs](https://docs.pytorch.org/TensorRT/index.html), [SPSG-TRT](https://github.com/lacie-life/SPSG-TRT)

---

## 4. TensorRT Precision Options on Orin Nano Super

| Precision | Support | Notes |
|-----------|---------|-------|
| **FP32** | Yes | Baseline |
| **FP16** | Yes | ~17 TFLOPs; recommended for most models |
| **INT8** | Yes | ~33 TOPS; needs calibration; **ViT-S+DPT can regress ~2.7×** |
| **FP8** | Limited | TensorRT 9.0+; Ampere support improved in 10.x; Ada/Hopper preferred |

**INT8 caveats**: Calibration critical; ViT-style models can show regressions. Enable `nvpmodel -m 2` and `jetson_clocks` for benchmarking.

**Sources**: [NVIDIA Forums FP16/INT8](https://forums.developer.nvidia.com/t/jetson-orin-nano-fp16-int8-performance/326723), [ViT-S DPT INT8 regression](https://forums.developer.nvidia.com/t/tensorrt-model-optimizer-int8-quantization-causes-2-7x-performance-regression-on-jetson-orin-nano-4gb-vit-s-dpt-architecture/357835)

---

## 5. Jetson Orin Nano Super vs Desktop GPU TRT Features

| Feature | Orin Nano Super | Desktop (e.g. RTX 40xx) |
|---------|-----------------|--------------------------|
| FP32/FP16/INT8 | Yes | Yes |
| FP8 (native) | Limited (Ampere) | Full (Ada/Hopper) |
| DLA | No (Orin Nano) | N/A |
| Unified memory | Yes (8 GB shared) | Separate VRAM |
| TensorRT API | Same | Same |

Orin Nano Super uses the same TensorRT API; differences are hardware limits (memory, FP8, DLA).

---

## 6. Typical Speedup: ONNX Runtime → TensorRT

- **Range**: ~10–200%+ depending on model
- **Reported**: ~3× in some Jetson Xavier AGX benchmarks
- **SuperPoint+LightGlue**: Up to 4× vs PyTorch with TensorRT (RTX4080)
- **DINOv2-base**: 1.68× on Orin Nano Super vs original (FP16 TRT)

---

## 7. Maximum Model Size / VRAM on Orin Nano Super

| Constraint | Value |
|------------|-------|
| **Physical RAM** | 8 GB LPDDR5 (unified CPU+GPU) |
| **Practical limit** | ~6–7 GB for models + OS + app |
| **VLMs** | 16 GB often recommended; 8 GB marginal |
| **Swap** | NVMe as CPU swap only, not GPU memory |
| **Recommendation** | Quantization (INT8/FP16) and smaller models for 8 GB |

**Model footprint (approx)**:
- DINOv2 ViT-S/14: ~22M params, ~90 MB FP16
- LiteSAM: 6.31M params, ~25 MB FP16
- SuperPoint + LightGlue: ~15–20 MB combined

All five models should fit in 8 GB with FP16; INT8 reduces footprint further.

**Sources**: [Jetson Orin Nano memory](https://nvidia-jetson.piveral.com/jetson-orin-nano/gpu-memory-limitations-and-nvme-usage-on-jetson-orin-nano/), [Insufficient GPU memory forum](https://forums.developer.nvidia.com/t/jetson-orin-nano-super-insufficient-gpu-memory/330777)

---

## 8. Multiple TRT Engines on Orin Nano Super

| Approach | Result |
|----------|--------|
| **Multiple engines, multiple threads** | Degradation: 10 engines × 10 threads → ~50 ms → ~300 ms per thread |
| **Single engine + batch** | Better throughput and lower memory |
| **Async CUDA streams** | Prefer single engine + streams over multiple engines |
| **Multi-threading vs multi-process** | Prefer multi-threading; multi-process causes time-slicing and higher latency |
| **Concurrent models (e.g. detection + SR)** | Possible tearing/artifacts; sync/memory issues |

**Recommendation**: One engine per model, sequential or carefully pipelined execution; avoid many concurrent engines.

**Sources**: [TensorRT #3716](https://github.com/NVIDIA/TensorRT/issues/3716), [TensorRT #4358](https://github.com/NVIDIA/TensorRT/issues/4358), [Isaac ROS forum](https://forums.developer.nvidia.com/t/multiple-model-inference-and-runtime-model-switching/292394)

---

## 9. JetPack and TensorRT Versions for Orin Nano Super

| Component | Version |
|-----------|---------|
| **JetPack** | 6.1 (Super Mode), 6.2 (full support) |
| **TensorRT** | 10.3.0 |
| **CUDA** | 12.6.10 |
| **cuDNN** | 9.3.0 |
| **L4T** | 36.4.3 |
| **Kernel** | 5.15+ |
| **OS** | Ubuntu 22.04 LTS |

**Super Mode config**: `jetson-orin-nano-devkit-super.conf`; power mode `nvpmodel -m 2` (MAXN).

**Sources**: [JetPack 6.2 Release Notes](https://docs.nvidia.com/jetson/archives/jetpack-archived/jetpack-62/release-notes/index.html), [NVIDIA JetPack SDK](https://developer.nvidia.com/embedded/jetpack-sdk-62)

---

## 10. Vision Transformer (ViT) / DINOv2 on TensorRT

### Support and Performance

- **DINOv2-base-patch14**: 75 → 126 FPS on Orin Nano Super (FP16 TRT), ~1.68×
- **ViT optimization**: Kernel fusion, dynamic tensor memory, compiler tuning
- **NanoOWL (OWL-ViT)**: 95 FPS on Jetson AGX Orin with ViT-B/32 via TRT

### Known Issues

1. **INT8 on ViT-S+DPT**: ~2.7× slowdown on Orin Nano 4GB; avoid INT8 for this class
2. **TRT-ViT**: Research shows TRT-oriented ViTs can be ~2.7× faster than standard ViTs
3. **Quantization + pruning**: Combined can give ~4× speedup with &lt;1% accuracy drop on AGX Orin

### Conversion Path

- Export to ONNX, then build TRT engine
- FP16 generally safe; INT8 needs validation per model
- DINOv2 has no dedicated TRT conversion guide; use standard PyTorch→ONNX→TRT

**Sources**: [Embedl ViT optimization](https://www.embedl.com/optimizing-vision-transformers-for-peak-performance-on-nvidia-jetson-agx-orinvidia-jetson-agx-orin), [TRT-ViT paper](https://arxiv.org/abs/2205.09579), [NanoOWL](https://github.com/NVIDIA-AI-IOT/nanoowl), [NVIDIA ViT blog](https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-developer-kit-gets-a-super-boost/)

---

## Model-Specific Notes for Your Stack

| Model | TRT Conversion | Notes |
|-------|----------------|-------|
| **SuperPoint** | PyTorch→ONNX→TRT; existing repos | [superpoint_lightglue_tensorrt](https://github.com/fettahyildizz/superpoint_lightglue_tensorrt), [SPSG-TRT](https://github.com/lacie-life/SPSG-TRT) |
| **LightGlue** | ONNX→TRT; FP8 via Model Optimizer | Up to ~6× vs FP32; 3840 keypoint limit in TRT TopK |
| **DINOv2 ViT-S/14** | PyTorch→ONNX→TRT | FP16 recommended; avoid INT8 |
| **LiteSAM** | PyTorch→ONNX→TRT | Small (6.31M params); standard ViT-style path |
| **EfficientLoFTR** | PyTorch→ONNX→TRT | Transformer-based; similar to LightGlue |

---

## Source URLs

1. https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-developer-kit-gets-a-super-boost/
2. https://docs.nvidia.com/jetson/archives/jetpack-archived/jetpack-62/release-notes/index.html
3. https://developer.nvidia.com/blog/nvidia-jetpack-6-2-brings-super-mode-to-nvidia-jetson-orin-nano-and-jetson-orin-nx-modules/
4. https://putvision.github.io/article/2021/12/20/jetson-onnxruntime-tensorrt.html
5. https://fabio-sim.github.io/blog/accelerating-lightglue-inference-onnx-runtime-tensorrt/
6. https://fabio-sim.github.io/blog/fp8-quantized-lightglue-tensorrt-nvidia-model-optimizer/
7. https://github.com/microsoft/onnxruntime/issues/11356
8. https://github.com/NVIDIA/TensorRT/issues/3716
9. https://github.com/NVIDIA/TensorRT/issues/4358
10. https://forums.developer.nvidia.com/t/jetson-orin-nano-fp16-int8-performance/326723
11. https://forums.developer.nvidia.com/t/tensorrt-model-optimizer-int8-quantization-causes-2-7x-performance-regression-on-jetson-orin-nano-4gb-vit-s-dpt-architecture/357835
12. https://www.embedl.com/optimizing-vision-transformers-for-peak-performance-on-nvidia-jetson-agx-orinvidia-jetson-agx-orin
13. https://arxiv.org/abs/2205.09579
14. https://github.com/NVIDIA-AI-IOT/nanoowl
15. https://github.com/fettahyildizz/superpoint_lightglue_tensorrt
16. https://github.com/lacie-life/SPSG-TRT
17. https://docs.pytorch.org/TensorRT/index.html
18. https://nvidia-jetson.piveral.com/jetson-orin-nano/gpu-memory-limitations-and-nvme-usage-on-jetson-orin-nano/
19. https://forums.developer.nvidia.com/t/jetson-orin-nano-super-insufficient-gpu-memory/330777
