gps-denied-onboard/_docs/02_document/tests/performance-tests.md

# Performance Tests

> Deployment-binding numbers require Tier T4 (real Jetson Orin Nano Super @ 25 W). T1 runs are functional plausibility checks only — same caveat as `test-data.md` D2.

---

### NFT-PERF-01: End-to-end latency p95 ≤400 ms (AC-4.1)

**Summary**: From camera-frame capture to GPS_INPUT emission, p95 latency ≤ 400 ms on Orin Nano Super @ 25 W.
**Traces to**: AC-4.1. Tier: T4 (`deferred-hil`) for binding result; T1 functional smoke.
**Metric**: end-to-end latency in ms, sampled per-frame, aggregated to p50 / p95 / p99.

**Preconditions**:
- Tier T4: real Jetson Orin Nano Super, 25 W power mode (`nvpmodel -m 0` + 25 W profile), thermals stabilized at +25 °C ambient.
- TRT engines warmed (≥1 min steady-state replay before measurement).
- 30-min sustained replay of `synthetic_8h_load` slice (or AerialVL S03 mid-segment).
- Frame timestamping uses the camera-shim `time_usec` and matches against the GPS_INPUT `time_usec`.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Stream nav-cam frames at 3 fps for 30 min after warm-up | per-frame `(t_emit_gps_input - t_capture)` |
| 2 | Drop the first 60 s as warm-up | aggregate the rest |
| 3 | Compute p50, p95, p99, max | report |
| 4 | Verify drop rate | `dropped_frames / total_frames ≤ 10%` |

**Pass criteria**: p95 ≤ 400 ms; drop rate ≤ 10 % (per AC-4.1's "skip-allowed" clause).
**Duration**: 30 min + 60 s warm-up.

---

### NFT-PERF-02: cuVSLAM single-frame latency ≤20 ms

**Summary**: cuVSLAM inference completes within 20 ms per frame.
**Traces to**: results_report row 37, F-T1b. Tier: T4 binding; T1 functional.
**Metric**: cuVSLAM per-frame inference duration, p95.

**Preconditions**: cuVSLAM warmed; mono+IMU mode.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Replay 5 min of nav-cam frames at 3 fps | per-frame `cuvslam_inference_ms` (publicly exposed metric) |
| 2 | p95 over the run | report |

**Pass criteria**: p95 ≤ 20 ms.
**Duration**: 5 min.

---

### NFT-PERF-03: Cross-view matcher latency

**Summary**: Inline matcher (SP+LG TRT FP16/INT8) ≤ 200 ms / pair; LiteSAM re-loc fallback ≤ 2000 ms / pair.
**Traces to**: AC-4.1 (sub-budget), results_report row 38. Tier: T4 binding.
**Metric**: per-pair matcher inference time, p95.

**Preconditions**: matcher warmed; representative resolution (1024×768 SP+LG / GIM-LG).

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Replay 1000 cross-view pairs through inline path | `inline_matcher_ms` per pair |
| 2 | Replay 100 cross-view pairs through re-loc path | `reloc_matcher_ms` per pair |

**Pass criteria**: inline p95 ≤ 200 ms; re-loc p95 ≤ 2000 ms.
**Duration**: ≤30 min.

---

### NFT-PERF-04: Orthority per-frame latency ≤50 ms

**Summary**: Orthority's per-frame ortho call on Orin Nano Super stays within budget.
**Traces to**: F-T14, M-27. Tier: T4 binding. If exceeded, fall back to `cv2.warpPerspective + bilinear DEM` per Component 1b documented fall-back.
**Metric**: ortho per-frame duration, p95.

**Preconditions**: Orthority loaded; SRTM-30 m DEM mmap warm; sector classified `flat` or `moderate`.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Replay 1000 frames | per-frame `ortho_ms` |

**Pass criteria**: p95 ≤ 50 ms. If FAIL: open task to switch to fall-back path (not a blocking gate at this test, but a flow trigger).
**Duration**: ≤10 min.

---

### NFT-PERF-05: Spoofing-promotion latency ≤3 s p95 (AC-NEW-2)

**Summary**: Time from spoof onset to SUT promotion as primary GPS source.
**Traces to**: AC-NEW-2. Tier: T3 (`deferred-sitl`).
**Metric**: t_promote = `t_promotion_event - t_spoof_onset`, p95 over 50 trials.

**Preconditions**: SITL + `gps-spoof-injector`; FC EKF3 lane-switch event observable via `EKF_STATUS_REPORT`.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | At t=0 inject spoof signal | observe SUT GPS_INPUT promotion (raised `fix_type` to 3D-fix-with-priority + STATUSTEXT `PROMOTE`) |
| 2 | Repeat 50 trials with randomised spoof magnitudes | distribution |

**Pass criteria**: p95 ≤ 3 s.
**Duration**: ≤30 min.

---

### NFT-PERF-06: Frame-by-frame output cadence (AC-4.4)

**Summary**: GPS_INPUT is streamed per-frame, not batched.
**Traces to**: AC-4.4. Tier: T1 + T4.
**Metric**: inter-frame interval distribution.

**Preconditions**: 30 min steady-state replay.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Replay at 3 fps | sniff GPS_INPUT timestamps |
| 2 | Compute inter-arrival deltas | distribution |
| 3 | Verify no frame is delayed >1 inter-frame interval | — |

**Pass criteria**: |Δt - 1/3 s| ≤ 50 ms for ≥99 % of frames; no batches (no clusters of frames within the same 50 ms window).
**Duration**: 30 min.

---

### NFT-PERF-07: GPS_INPUT message rate (results_report row 9)

**Summary**: GPS_INPUT emitted at 5–10 Hz continuous (matches per-frame at 3 fps + duplicates for FC stability when configured).
**Traces to**: AC-4.3, results_report row 9. Tier: T1.
**Metric**: rate over 60 s windows.

**Preconditions**: steady-state tracking.

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Sniff GPS_INPUT for 5 min | per-second rate |

**Pass criteria**: rate ∈ [5, 10] Hz throughout.
**Duration**: 5 min.

---

### NFT-PERF-08: VPR latency under conditional invocation

**Summary**: VPR's DINOv2 forward only fires on re-loc triggers; in cruise it stays near zero CPU/GPU.
**Traces to**: AC-8.6, restrictions §Satellite (VPR retrieval unit). Tier: T4.
**Metric**: VPR invocations / second; cruise idle vs re-loc burst.

**Preconditions**: 60-min replay with scripted re-loc triggers (cold start, sharp turn, σ_xy > 50 m, VO failure ≥2 frames).

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Run replay | per-second `vpr_invocations` counter |
| 2 | Compute average across cruise window vs re-loc window | — |

**Pass criteria**:
- Cruise window (no triggers): VPR invocations / 100 frames ≤ 1 (i.e., not invoked per-frame).
- Re-loc window: VPR invokes within 1 frame of trigger; latency ≤ 200 ms p95 for the DINOv2 forward.
**Duration**: 60 min.

---

### NFT-PERF-09: Top-K dynamic sizing matches sector / σ_xy

**Summary**: VPR top-K honours AC-8.6 dynamic-K rules.
**Traces to**: AC-8.6. Tier: T1 + T4.
**Metric**: K value selected per VPR call vs sector class + σ_xy.

**Preconditions**: scripted scenarios with (sector ∈ {stable, active}) × (σ_xy ∈ {10, 30, 60}).

**Steps**:

| Step | Consumer Action | Measurement |
|------|----------------|-------------|
| 1 | Trigger VPR in each combination | observe `vpr_top_k` metric |

**Pass criteria**:
- stable + σ_xy ≤ 20 m → K=5.
- active-conflict → K=20.
- expanding-window fallback (σ_xy > 50 m or fail-N) → K=50.
**Duration**: 5 min.

---

### NFT-PERF-10: Failsafe latency ≤3 s no-fix → FC fallback (AC-5.2)

**Summary**: When SUT cannot produce any estimate for >3 s, FC observably falls back to IMU-only DR.
**Traces to**: AC-5.2. Tier: T3.
**Metric**: time from last-fix-emission to FC fallback signal in `EKF_STATUS_REPORT`.

**Preconditions**: scripted blackout in SITL.

**Steps**: blackout pipeline; observe FC.

**Pass criteria**: FC fallback observable within 4 s of blackout (3 s budget + 1 s observation latency).
**Duration**: 5 min.

---

### NFT-PERF-11: Bench-off candidates — accuracy-vs-latency frontier

**Summary**: Score inline matcher candidates on the documented bench-off corpora.
**Traces to**: AC-1.1 / AC-1.2 / AC-2.2 / R2 / R3, F-T15. Tier: T2.
**Metric**: per-candidate (recall@30 m, p95 latency, peak GPU mem, sustained 30-min thermal stability, seasonal-robustness score).

**Preconditions**: AerialVL, UAV-VisLoc, AerialExtreMatch, 2chADCNN, TartanAir V2, internal Mavic.

**Steps**: run each candidate (SP+LG, GIM-LG, XFeat sparse, XFeat semi-dense) and each ceiling reference (RoMa v2, MASt3R-SLAM, MapGlue, MATCHA — offline only) over the corpora.

**Pass criteria**:
- Inline candidates must fit in 200 ms / pair on Orin Nano Super @ 25 W.
- Re-loc candidates (LiteSAM) must fit in 2 s / pair.
- Selected inline matcher's recall@30 m on AerialVL S03 must support AC-1.1 / AC-1.2.
**Duration**: 4 h Monte Carlo.

---

### NFT-PERF-12: Latency under adversarial input — no infinite stall

**Summary**: Pathological inputs (uniform-grey frame, all-black frame, very low contrast) do not cause unbounded latency.
**Traces to**: AC-3.x (resilience), AC-4.1 (negative). Tier: T1.
**Metric**: per-frame latency capped.

**Preconditions**: replay with 5 % of frames replaced by uniform-grey or all-black.

**Steps**: replay 30 min; observe latency CDF.

**Pass criteria**: each frame's latency ≤ 600 ms (1.5× p95 budget); pipeline never stalls beyond a single frame interval.
**Duration**: 30 min.

---

## Test execution caveats

- **T1 runs**: produced numbers are NOT deployment-binding. AC-4.1 / NFT-PERF-01 specifically requires Orin Nano Super 25 W (T4) for binding pass.
- **T4 runs**: bench scheduler enforces single-tenant access; thermal warm-up ≥1 min before measurement window starts.
- **Frame-rate floor**: AC-4.1 allows ~10 % drop under sustained load. Drop rate IS measured and reported in NFT-PERF-01.