# TensorRT Conversion Research: DINOv2, LiteSAM, EfficientLoFTR

**Research date**: 2025-03-15  
**Target platform**: Jetson Orin Nano Super (17 FP16 TFLOPs, 1024 CUDA cores, 8GB LPDDR5)  
**Context**: Visual navigation system — DINOv2 for image retrieval, LiteSAM/EfficientLoFTR for satellite-aerial feature matching

---

## Executive Summary

| Model | TRT Conversion Feasibility | Blocking Issues | Expected Jetson Orin Nano Super | Recommended Path |
|-------|---------------------------|-----------------|---------------------------------|------------------|
| **DINOv2 ViT-S/14** | ⚠️ Feasible with caveats | FMHA fusion failure; embedding accuracy degradation (FP32/FP16); no official ONNX | ~15–35 ms (estimate) | PyTorch→ONNX (wrapper) → TRT FP16; validate embeddings vs PyTorch |
| **LiteSAM** | ❌ High effort | No ONNX; MinGRU custom ops; immature repo | N/A (no TRT path) | Use EfficientLoFTR TRT as fallback; custom ONNX export if LiteSAM required |
| **EfficientLoFTR** | ⚠️ Partial | LoFTR_TRT targets original LoFTR; EfficientLoFTR-specific ONNX not documented | ~80–150 ms (estimate from LoFTR) | Adapt LoFTR_TRT export scripts; or use original LoFTR TRT |

---

## 1. DINOv2 ViT-S/14

### 1.1 Has DINOv2 been successfully converted to TensorRT?

**Yes, with significant caveats.**

- **GitHub**: facebookresearch/dinov2 PR #129 (closed Feb 2025) — draft ONNX export script; HuggingFace model `RoundtTble/dinov2_vits14_onnx` exists.
- **NVIDIA TAO Deploy**: DINO with TAO Deploy targets **object detection DINO** (Deformable DETR), not DINOv2 (self-supervised embeddings). Different models.
- **Community**: dinov2_onnx (sefaburakokcu), Medium article (Thuan Bui Huy Nov 2025) — PyTorch→ONNX→TRT workflow described; 2–3× speedup claimed.
- **NVIDIA Forums**: User on Jetson Orin Nano (JetPack 6.0) converted dinov2_vits14 to ONNX then TRT FP32; **embedding distances degraded** (different vs same product separation collapsed).

**Sources**:
- [facebookresearch/dinov2 PR #129](https://github.com/facebookresearch/dinov2/pull/129)
- [NVIDIA Forums: Dinov2 TensorRT model performance issue](https://forums.developer.nvidia.com/t/dinov2-tensorrt-model-performance-issue/312251)
- [Medium: Accelerating Vision AI with TensorRT (YOLOv8, DINOv2)](https://medium.com/@testth02/accelerating-vision-ai-inference-with-tensorrt-yolov8-and-dinov2-optimization-in-practice-287acd4c73e1)
- [dinov2_onnx](https://github.com/sefaburakokcu/dinov2_onnx)

### 1.2 ViT TensorRT Challenges

| Challenge | Details |
|-----------|---------|
| **Attention ops** | FMHA (Fused Multi-Head Attention) fusion often fails. TensorRT 10.8: DINOv2 ONNX does not produce `mha`/`fused_mha_v2` layers; attention stays as separate MatMul/Softmax. |
| **Layer norm** | Standard LayerNorm generally supported; no major blockers. |
| **Dynamic shapes** | ViT patch count varies with input size. Use `dynamic_axes` for batch and spatial dims; min/opt/max shapes must align with ONNX export. |
| **ONNX export** | Mask inputs must be wrapped out; `opset_version` 17+ recommended. |

**Sources**:
- [NVIDIA TensorRT Issue #4537: DINOv2 FMHA Fusion Failure](https://github.com/NVIDIA/tensorrt/issues/4537)
- [NVIDIA TensorRT Issue #4404: DINO ONNX vs TRT output inconsistency](https://github.com/NVIDIA/tensorrt/issues/4404) (object detection DINO; different model but relevant for TRT accuracy)

### 1.3 torch-tensorrt ViT Support

No explicit documentation for ViT/DINOv2. torch-tensorrt focuses on common CNN patterns. ViT models typically go via **ONNX → trtexec** or **ONNX Runtime with TensorRT EP**, not torch-tensorrt direct.

### 1.4 Precision (FP16/INT8) for DINOv2

| Precision | Status | Notes |
|----------|--------|-------|
| **FP32** | Works | NVIDIA forum user: embedding quality still degraded vs PyTorch/ONNX. |
| **FP16** | Works | Medium article: 2–3× speedup; best trade-off. Validate embedding distances. |
| **INT8** | Possible | TAO Deploy supports INT8 for DINO (object detection); no public reports for DINOv2 embeddings. Calibration required. |

**Critical**: Embedding space compression observed — same vs different product distances too close. Validate retrieval metrics (e.g., mAP, recall@k) before deployment.

### 1.5 Expected DINOv2 ViT-S/14 on Jetson Orin Nano Super

- **Jetson Orin Nano Super**: 17 FP16 TFLOPs, 1024 CUDA cores.
- **Baseline**: DINOv2 ViT-S ~10 ms (ViT-S14-Mix), ~25–30 ms (ViT-B) on desktop GPUs.
- **Estimate**: ViT-S/14 TRT FP16 on Orin Nano Super: **~15–35 ms** (conservative; no published Jetson benchmarks for DINOv2).

### 1.6 NVIDIA Alternatives for Image Retrieval on Jetson

- **Jetson AI Lab**: NanoSAM, Efficient ViT, NanoOWL — optimized for segmentation/detection, not retrieval embeddings.
- **TAO**: DINO in TAO is object detection, not DINOv2.
- **Conclusion**: No direct NVIDIA-optimized DINOv2 alternative for retrieval. DINOv2 ONNX→TRT remains the main path; validate accuracy.

---

## 2. LiteSAM

### 2.1 ONNX Export and MinGRU

- **LiteSAM**: No ONNX export script; no TensorRT support. Repo: 5 stars, 4 commits, last update Oct 2025.
- **Architecture**: MobileOne-S3 backbone, TAIFormer (attention), **MinGRU** for subpixel refinement (replaces heatmap-based refinement in EfficientLoFTR).
- **MinGRU**: Recurrent layer. PyTorch `nn.GRU` exports to ONNX via standard `torch.onnx.export`; ONNX supports `GRU` op. **No known custom ops** in MinGRU that would block ONNX.
- **Blocking factors**: (1) No export script; (2) TAIFormer (custom attention) may need graph fixes; (3) Semi-dense output — variable match count (see Section 4).

### 2.2 Conversion Path (Theoretical)

1. Implement ONNX export (wrap model, handle optional inputs).
2. Verify GRU and TAIFormer export cleanly.
3. Use fixed max matches or dynamic axes for output (see Section 4).
4. ONNX → TensorRT via trtexec.

**Feasibility**: Theoretically possible; requires custom development. No prior art for LiteSAM.

### 2.3 Expected Performance

- **Paper**: LiteSAM 83.79 ms on RTX 3090 @ 1184×1184; Jetson AGX Orin (50W): 497.49 ms (opt.).
- **Jetson Orin Nano Super**: ~1/3 of AGX Orin GPU; estimate **~500–800 ms** PyTorch. TRT: unknown until conversion exists.

---

## 3. EfficientLoFTR

### 3.1 ONNX/TensorRT Support

- **Official repo**: zju3dv/EfficientLoFTR — no ONNX or TensorRT export in main repo.
- **LoFTR_TRT**: [Kolkir/LoFTR_TRT](https://github.com/Kolkir/LoFTR_TRT) — TensorRT adaptation of **original LoFTR**, not EfficientLoFTR. Includes `export_onnx.py`.
- **Coarse_LoFTR_TRT**: Coarse-only variant for low-end devices.
- **HuggingFace**: EfficientLoFTR in Transformers; no ONNX export documented.
- **EfficientLoFTR differences**: Aggregated attention, two-stage correlation (vs heatmap refinement). Architecture differs from LoFTR; LoFTR_TRT scripts need adaptation.

### 3.2 Conversion Path

1. Adapt LoFTR_TRT `export_onnx.py` for EfficientLoFTR (different modules).
2. Or use original LoFTR with TRT — lower accuracy but proven path.
3. EfficientLoFTR paper: ~27 ms @ 640×480 mixed precision on optimized hardware.

### 3.3 Expected Jetson Orin Nano Super

- LoFTR family: coarse-to-fine, transformer-heavy. Jetson AGX Orin: LiteSAM 497 ms; EfficientLoFTR likely similar or slower.
- **Estimate**: EfficientLoFTR TRT on Orin Nano Super: **~80–150 ms** at 640×480 if conversion succeeds; **~300–500 ms** at 1184×1184.

---

## 4. Dynamic Output Shapes (Semi-Dense Matchers)

Semi-dense matchers (LiteSAM, EfficientLoFTR, LoFTR) output **variable numbers of matches** per image pair.

### 4.1 TensorRT Handling

- **Fixed output**: Export with fixed max matches (e.g., top 2048). Pad unused slots; mask in post-processing.
- **Dynamic axes**: ONNX `dynamic_axes` for output dimension (e.g., `{0: "batch", 1: "num_matches"}`). TensorRT supports dynamic shapes with min/opt/max profiles.
- **TopK / filtering**: If model outputs scores, use TopK or threshold in ONNX/TRT to bound output size.
- **Practical approach**: Fixed max matches is most robust; variable shapes add complexity and can hurt TRT optimization.

---

## 5. GRU / Recurrent Ops in TensorRT

- **ONNX**: `GRU` op is standard; PyTorch `nn.GRU` exports to it.
- **TensorRT**: Supports GRU/LSTM. No custom plugin needed for standard GRU.
- **MinGRU**: If it is a standard GRU variant, export should work. Custom implementations (e.g., minimal hidden size, custom activations) may need verification.

---

## 6. Summary Table

| Model | TRT Feasible | Blockers | Jetson Orin Nano Super (est.) | Conversion Path |
|-------|--------------|----------|-------------------------------|-----------------|
| **DINOv2 ViT-S/14** | ⚠️ Yes | Embedding accuracy; FMHA not fusing | ~15–35 ms | ONNX (wrapper) → TRT FP16; validate retrieval |
| **LiteSAM** | ❌ No path | No ONNX; custom work required | N/A | Use EfficientLoFTR TRT; or custom ONNX→TRT |
| **EfficientLoFTR** | ⚠️ Partial | Adapt LoFTR_TRT for EfficientLoFTR | ~80–150 ms (640×480) | Adapt export scripts; or use LoFTR TRT |

---

## 7. Recommended Actions

1. **DINOv2**: Export via [dinov2_onnx](https://github.com/sefaburakokcu/dinov2_onnx) or PR #129 approach; convert to TRT FP16; **validate embedding distances and retrieval metrics** on representative data.
2. **LiteSAM**: Treat as PyTorch-only for now. Use EfficientLoFTR (or LoFTR) TRT as fallback for satellite matching.
3. **EfficientLoFTR**: Fork LoFTR_TRT; adapt `export_onnx.py` for EfficientLoFTR modules; test ONNX→TRT; benchmark on Orin Nano Super.
4. **Dynamic outputs**: Use fixed max matches for TRT; pad and mask in post-processing.

---

## 8. Source URLs

| Source | URL |
|--------|-----|
| DINOv2 ONNX PR | https://github.com/facebookresearch/dinov2/pull/129 |
| DINOv2 TensorRT performance (NVIDIA Forums) | https://forums.developer.nvidia.com/t/dinov2-tensorrt-model-performance-issue/312251 |
| DINOv2 FMHA fusion issue | https://github.com/NVIDIA/tensorrt/issues/4537 |
| DINO ONNX vs TRT inconsistency | https://github.com/NVIDIA/tensorrt/issues/4404 |
| Medium: TensorRT YOLOv8 DINOv2 | https://medium.com/@testth02/accelerating-vision-ai-inference-with-tensorrt-yolov8-and-dinov2-optimization-in-practice-287acd4c73e1 |
| dinov2_onnx | https://github.com/sefaburakokcu/dinov2_onnx |
| TAO Deploy DINO | https://docs.nvidia.com/tao/tao-toolkit/text/tao_deploy/dino.html |
| LoFTR_TRT | https://github.com/Kolkir/LoFTR_TRT |
| EfficientLoFTR | https://github.com/zju3dv/EfficientLoFTR |
| LiteSAM | https://github.com/boyagesmile/LiteSAM |
| Jetson Orin Nano Super | https://developer.nvidia.com/blog/nvidia-jetson-orin-nano-developer-kit-gets-a-super-boost/ |
| ViT-TensorRT (PyPI) | https://pypi.org/project/ViT-TensorRT/ |
| TRT-ViT paper | https://arxiv.org/abs/2205.09579 |
| Jetson AI Lab ViT | https://jetson-ai-lab.com/vit/index.html |
