gps-denied-onboard/_docs/02_document/tests/performance-tests.md

Performance Tests

All performance tests honor the per-tier execution profile from environment.md. Latency and memory tests bound to Jetson Orin Nano Super hardware run on Tier-2 only; metrics that don't depend on hardware (e.g. inter-emit interval correctness, GCS rate) run on both tiers.

NFT-PERF-01: End-to-end latency p95 budget

Summary: Validates the AC-4.1 end-to-end latency budget (camera capture → GPS to FC) on the pinned hardware. Traces to: AC-4.1, D-CROSS-LATENCY-1 Metric: Wall-clock latency from frame-capture timestamp to outbound GPS_INPUT (AP) / MSP2_SENSOR_GPS (iNav) reception at the SITL container.

Preconditions:

• Tier-2 only — Jetson Orin Nano Super, JetPack 6.2, TensorRT 10.3 per D-C7-9.
• tile-cache-fixture pre-loaded.
• SUT cold-started THEN warmed up for 30 s of replay before measurement window starts.
• Two configurations measured: (a) K=3 baseline at +25 °C, (b) K=2 + Jacobian-cov hybrid auto-degrade at +50 °C ambient (NFT-9 in the solution draft).

Steps:

Step	Consumer Action	Measurement
1	Run 30 s warm-up replay (excluded from measurement)	none
2	Run 5 min Derkachi replay at 3 Hz target cadence	per-frame latency: t_emit_at_sitl − t_capture
3	Record per-frame latency to CSV; compute p50, p95, p99	distribution
4	Repeat at +50 °C ambient (chamber if available, else flagged)	distribution under thermal-throttle hybrid

Pass criteria:

• (a) K=3 baseline: p95 ≤ 400 ms (AC-4.1 hard bound).
• (b) K=2 + Jacobian-cov hybrid: p95 ≤ 400 ms still satisfied after auto-degrade (proves D-CROSS-LATENCY-1 effective).
• ≤10% frame drops under sustained load (AC-4.1 allowance).
• Per-stage latency partitioning (D-CROSS-LATENCY-1 table) recorded for all stages: C1 OKVIS2 / C2 UltraVPR / C2.5 / C3 / C3.5 / C4 / C4 cov / C5 / serialization / OS jitter — used in NFT-PERF-01 evidence bundle for budget-margin tracking.

Duration: 2 × 5.5 min replays (warm-up + measurement) per configuration; ~25 min total per FC adapter.

---

NFT-PERF-02: Frame-by-frame streaming (no batching)

Summary: Validates AC-4.4 — estimates streamed frame-by-frame with no batching/delay. Traces to: AC-4.4 Metric: Inter-emit interval at SITL.

Preconditions:

• Tier-1 OR Tier-2.
• SUT warmed up for 30 s.

Steps:

Step	Consumer Action	Measurement
1	Replay Derkachi 5 min at 3 Hz	per-frame inter-emit interval at SITL
2	Compute distribution	p95 of inter-emit interval

Pass criteria: p95 inter-emit interval ≤ inter-frame-interval × 1.05 (i.e. ≤ ~350 ms at 3 Hz target). No window of ≥3 missed-emit gaps.

Duration: 6 min.

---

NFT-PERF-03: Cold-start TTFF

Summary: Validates AC-NEW-1 cold-start time-to-first-fix from companion boot. Traces to: AC-NEW-1 Metric: Wall-clock from SUT container-ready event (or systemctl start on Tier-2) to first valid outbound GPS_INPUT / MSP2_SENSOR_GPS arrival at SITL.

Preconditions:

• Tier-2 (Jetson) for the canonical run; Tier-1 acceptable for trend-tracking.
• cold-boot-fixture provides the FC EKF snapshot (loaded into SITL before the SUT cold boot).
• tile-cache-fixture already mounted (cache-load is part of the TTFF budget per AC-NEW-1 wording "from boot").
• 50 cold boots executed back-to-back to populate distribution.

Steps:

Step	Consumer Action	Measurement
1	Stop SUT; clear in-memory state	container down
2	Start SUT (record t_start)	timestamp
3	First outbound message arrives at SITL (record t_first_emit)	TTFF = t_first_emit − t_start
4	Repeat 50 times	distribution

Pass criteria: p95 TTFF < 30 s.

Duration: ~30 min (50 × ~30 s + restart overhead).

---

NFT-PERF-04: Spoofing-promotion latency

Summary: Validates AC-NEW-2 — when FC signals GPS denial/spoof, promote onboard estimate to FC's primary position source within < 3 s p95. Traces to: AC-NEW-2 Metric: Latency from spoof-onset signal to FC-side EKF source-set switch (AP: EK3_SRC1_POSXY flips to companion-source value; iNav: GPS provider state reflects companion as primary).

Preconditions:

• Tier-1 acceptable (mostly software loops + SITL).
• derkachi-fixture running with SUT in satellite_anchored steady state.
• Spoof injector primed.

Steps:

Step	Consumer Action	Measurement
1	Inject false GPS into FC SITL (record t_spoof_onset)	timestamp
2	Observe FC EKF source-set state via parameter read polling at 100 Hz (record t_promotion)	promotion latency = t_promotion − t_spoof_onset
3	Repeat 50 trials per FC (parameterized on ardupilot + inav)	distribution per FC

Pass criteria: p95 < 3 s on both FCs.

Duration: ~25 min per FC (50 trials × ~30 s including pre-trial reset).

---

Per-stage latency partition record (informational, not pass/fail)

NFT-PERF-01 captures per-stage latencies matching the D-CROSS-LATENCY-1 partition table from solution.md. The recorded targets are tracked for budget-margin trend (regression detector), not as independent pass/fail thresholds — only AC-4.1 p95 ≤ 400 ms is the hard gate.

Stage	K=3 target p95	K=2 hybrid target p95
C1 OKVIS2 VIO	≤ 60 ms	≤ 60 ms
C2 UltraVPR query	≤ 15 ms	≤ 15 ms
C2.5 Top-N re-rank	≤ 80 ms	≤ 80 ms
C3 DISK+LightGlue × N	≤ 200 ms (steady)	≤ 140 ms (thermal)
C3.5 AdHoP (conditional, p99)	≤ 100 ms when triggered	≤ 60 ms when triggered
C4 solvePnPRansac	≤ 25 ms	≤ 25 ms
C4 covariance recovery	≤ 100 ms (steady)	≤ 25 ms (thermal)
C5 iSAM2 update	≤ 15 ms	≤ 15 ms
MAVLink/MSP2 + UART/USB	≤ 30 ms	≤ 30 ms
OS scheduling jitter (p99)	≤ 50 ms	≤ 50 ms