# Fact Cards

## Fact #1
- **Statement**: ONNX Runtime CUDA Execution Provider on Jetson Orin Nano (JetPack 6.1) is 7-8x slower than TensorRT standalone due to tensor cores not being utilized with default settings.
- **Source**: Source #1 (https://github.com/microsoft/onnxruntime/issues/24085)
- **Phase**: Assessment
- **Target Audience**: Jetson Orin Nano developers using ONNX Runtime
- **Confidence**: ✅ High (confirmed by issue author with NSight profiling, MSFT acknowledged)
- **Related Dimension**: Performance

## Fact #2
- **Statement**: The workaround for Fact #1 is to remove the `cudnn_conv_algo_search` option (which defaults to EXHAUSTIVE) and use FP16 models. This restores tensor core usage.
- **Source**: Source #1
- **Phase**: Assessment
- **Target Audience**: Jetson Orin Nano developers
- **Confidence**: ✅ High (confirmed fix by issue author)
- **Related Dimension**: Performance

## Fact #3
- **Statement**: ONNX Runtime TRT-EP keeps serialized TRT engine in memory (~420-440MB) throughout execution. Native TRT via trtexec drops to 130-140MB after engine deserialization by calling releaseBlob().
- **Source**: Source #2 (https://github.com/microsoft/onnxruntime/issues/20457)
- **Phase**: Assessment
- **Target Audience**: All ONNX RT TRT-EP users, especially memory-constrained devices
- **Confidence**: ✅ High (confirmed by MSFT developer @chilo-ms with detailed explanation)
- **Related Dimension**: Memory consumption

## Fact #4
- **Statement**: The ~280-300MB extra memory from ONNX RT TRT-EP (Fact #3) is because the serialized engine is retained across compute function calls for dynamic shape models. Native TRT releases it after deserialization.
- **Source**: Source #2
- **Phase**: Assessment
- **Target Audience**: Memory-constrained Jetson deployments
- **Confidence**: ✅ High (MSFT developer explanation)
- **Related Dimension**: Memory consumption

## Fact #5
- **Statement**: MSFT engineer states "TensorRT EP can achieve performance parity with native TensorRT" — both use the same TRT kernels internally. Benefit of TRT-EP is automatic fallback for unsupported ops.
- **Source**: Source #3 (https://github.com/microsoft/onnxruntime/issues/12083)
- **Phase**: Assessment
- **Target Audience**: General
- **Confidence**: ⚠️ Medium (official statement but contradicted by real benchmarks in some cases)
- **Related Dimension**: Performance

## Fact #6
- **Statement**: Real benchmark of InceptionV3/4 showed ONNX RT TRT-EP achieving ~41 inferences/sec vs native TRT at ~129 inferences/sec — approximately 3x performance gap.
- **Source**: Source #4 (https://github.com/microsoft/onnxruntime/issues/11356)
- **Phase**: Assessment
- **Target Audience**: CNN model deployers
- **Confidence**: ⚠️ Medium (community report, older ONNX RT version, model-specific)
- **Related Dimension**: Performance

## Fact #7
- **Statement**: Jetson Orin Nano Super specs: 67 TOPS sparse INT8 / 33 TOPS dense, GPU at 1020 MHz, 8GB LPDDR5 shared, 102 GB/s bandwidth, 25W TDP. NO DLA cores.
- **Source**: Source #6, Source #7
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (official NVIDIA specs)
- **Related Dimension**: Hardware constraints

## Fact #8
- **Statement**: Jetson Orin Nano has ZERO DLA (Deep Learning Accelerator) cores. DLA is only available on Orin NX (1-2 cores) and AGX Orin (2 cores).
- **Source**: Source #7, Source #14
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (official hardware specifications)
- **Related Dimension**: Hardware constraints

## Fact #9
- **Statement**: TensorRT engines are tied to specific GPU models, not just architectures. Must be built on the target device. Cannot cross-compile from x86 to aarch64.
- **Source**: Source #8 (https://github.com/NVIDIA/TensorRT/issues/1920)
- **Phase**: Assessment
- **Target Audience**: TRT deployers
- **Confidence**: ✅ High (NVIDIA confirmed)
- **Related Dimension**: Deployment workflow

## Fact #10
- **Statement**: Standard conversion workflow: PyTorch → ONNX (torch.onnx.export) → trtexec --onnx=model.onnx --saveEngine=model.engine --fp16. Use --memPoolSize instead of deprecated --workspace flag.
- **Source**: Source #9, Source #13
- **Phase**: Assessment
- **Target Audience**: Model deployers on Jetson
- **Confidence**: ✅ High (official NVIDIA workflow)
- **Related Dimension**: Deployment workflow

## Fact #11
- **Statement**: TensorRT 10.x Python API: load engine via runtime.deserialize_cuda_engine(data). Async inference via context.enqueue_v3(stream_handle). Uses tensor-name-based API (not binding indices).
- **Source**: Source #10
- **Phase**: Assessment
- **Target Audience**: Python TRT developers
- **Confidence**: ✅ High (official NVIDIA docs)
- **Related Dimension**: API/integration

## Fact #12
- **Statement**: Torch-TensorRT supports Jetson aarch64 with JetPack 6.2. Supports ahead-of-time (AOT) compilation, FP16/INT8, dynamic and static shapes. Alternative path to ONNX→trtexec.
- **Source**: Source #11
- **Phase**: Assessment
- **Target Audience**: PyTorch developers on Jetson
- **Confidence**: ✅ High (official documentation)
- **Related Dimension**: Deployment workflow

## Fact #13
- **Statement**: XFeatTensorRT repo exists — C++ TensorRT implementation of XFeat feature extraction. Confirms XFeat is TRT-convertible.
- **Source**: Source #12
- **Phase**: Assessment
- **Target Audience**: Our project (XFeat users)
- **Confidence**: ✅ High (working open-source implementation)
- **Related Dimension**: Model-specific conversion

## Fact #14
- **Statement**: cuVSLAM is a closed-source NVIDIA CUDA library (PyCuVSLAM). It is NOT an ONNX or PyTorch model. It cannot and does not need to be converted to TRT — it's already native CUDA-optimized for Jetson.
- **Source**: cuVSLAM documentation (https://github.com/NVlabs/PyCuVSLAM)
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (verified from PyCuVSLAM docs)
- **Related Dimension**: Model applicability

## Fact #15
- **Statement**: JetPack 6.2 ships with TensorRT 10.3, CUDA 12.6, cuDNN 9.3. The tensorrt Python module is pre-installed and accessible on Jetson.
- **Source**: Source #5
- **Phase**: Assessment
- **Target Audience**: Jetson developers
- **Confidence**: ✅ High (official release notes)
- **Related Dimension**: Software stack

## Fact #16
- **Statement**: TRT engine build on Jetson Orin Nano Super (8GB) can cause OOM for large models during the build phase, even if inference fits in memory. Workaround: build on a more powerful machine with same GPU architecture, or use Torch-TensorRT PyTorch workflow.
- **Source**: Source #5 (https://github.com/NVIDIA/TensorRT-LLM/issues/3149)
- **Phase**: Assessment
- **Target Audience**: Jetson Orin Nano developers building large TRT engines
- **Confidence**: ✅ High (confirmed in NVIDIA TRT-LLM issue)
- **Related Dimension**: Deployment workflow

## Fact #17
- **Statement**: LiteSAM uses MobileOne backbone which is reparameterizable — multi-branch training structure collapses to a single feed-forward path. This is critical for TRT optimization: fewer layers, better fusion, faster inference.
- **Source**: Solution draft03, LiteSAM paper
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (published paper)
- **Related Dimension**: Model-specific conversion

## Fact #18
- **Statement**: INT8 quantization is safe for CNN layers (MobileOne backbone) but NOT for transformer components (TAIFormer in LiteSAM). FP16 is safe for both CNN and transformer layers.
- **Source**: Solution draft02/03 analysis
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ⚠️ Medium (general best practice, not verified on LiteSAM specifically)
- **Related Dimension**: Quantization strategy

## Fact #19
- **Statement**: On 8GB shared memory Jetson: OS+runtime ~1.5GB, cuVSLAM ~200-500MB, tiles ~200MB. Remaining budget: ~5.8-6.1GB. ONNX RT TRT-EP overhead of ~280-300MB per model is significant. Native TRT saves this memory.
- **Source**: Solution draft03 memory budget + Source #2
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (computed from verified facts)
- **Related Dimension**: Memory consumption

## Fact #20
- **Statement**: LiteSAM's MinGRU subpixel refinement (Eqs 12-16) uses: z_t = σ(Linear(C_f)), h̃_t = Linear(C_f), h_t = (1-z_t)⊙h_{t-1} + z_t⊙h̃_t. Gates depend ONLY on input C_f (not h_{t-1}). Operates on 3×3 window (9 candidates), 4 stacked layers. All ops are standard: Linear, Sigmoid, Mul, Add, ReLU, Tanh.
- **Source**: LiteSAM paper (MDPI Remote Sensing, 2025, Eqs 12-16, Section 3.4.2)
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (from the published paper)
- **Related Dimension**: LiteSAM TRT compatibility

## Fact #21
- **Statement**: MinGRU's parallel implementation can use logcumsumexp (log-space parallel scan), which is NOT a standard ONNX operator. However, for seq_len=9 (LiteSAM's 3×3 window), a simple unrolled loop is equivalent and uses only standard ops.
- **Source**: minGRU paper + lucidrains/minGRU-pytorch implementation
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ⚠️ Medium (logcumsumexp risk depends on implementation; seq_len=9 makes rewrite trivial)
- **Related Dimension**: LiteSAM TRT compatibility

## Fact #22
- **Statement**: EfficientLoFTR has a proven TRT conversion path via Coarse_LoFTR_TRT (138 stars). The paper documents specific code changes needed: replace einsum with elementary ops (view, bmm, reshape, sum), adapt for TRT-compatible functions. Tested on Jetson Nano 2GB (~5 FPS with distilled model).
- **Source**: Coarse_LoFTR_TRT paper (arXiv:2202.00770) + GitHub repo
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (published paper + working open-source implementation)
- **Related Dimension**: Fallback satellite matcher

## Fact #23
- **Statement**: EfficientLoFTR has 15.05M params (2.4x more than LiteSAM's 6.31M). On AGX Orin with PyTorch: ~620ms (LiteSAM is 19.8% faster). Semi-dense matching. CVPR 2024. Available on HuggingFace under Apache 2.0.
- **Source**: LiteSAM paper comparison + EfficientLoFTR docs
- **Phase**: Assessment
- **Target Audience**: Our project
- **Confidence**: ✅ High (published benchmarks)
- **Related Dimension**: Fallback satellite matcher

## Fact #24
- **Statement**: SAM2's MemoryAttention showed performance DEGRADATION with TRT: 30ms PyTorch → 100ms TRT FP16. RoPEAttention identified as bottleneck. This warns that transformer attention modules may not always benefit from TRT conversion.
- **Source**: https://github.com/facebookresearch/sam2/issues/639
- **Phase**: Assessment
- **Target Audience**: Transformer model deployers
- **Confidence**: ⚠️ Medium (different model, but relevant warning for attention layers)
- **Related Dimension**: TRT conversion risks
