mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-04-22 22:36:37 +00:00
Oleksandr Bezdieniezhnykh
139 lines
4.8 KiB
Markdown
139 lines
4.8 KiB
Markdown
# Performance Tests
|
|
|
|
### NFT-PERF-01: End-to-End Per-Frame Latency
|
|
|
|
**Summary**: Validate total pipeline latency from camera capture to GPS_INPUT output is <400ms.
|
|
**Traces to**: AC-07 (< 400ms end-to-end per frame)
|
|
**Metric**: End-to-end latency (camera frame timestamp → GPS_INPUT message timestamp)
|
|
|
|
**Preconditions**:
|
|
- System running on Jetson Orin Nano Super (GPU-mode)
|
|
- Camera-replay serving frames at 0.7fps
|
|
- System in steady state (warm-up: ≥10 frames processed)
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Camera-replay sends frame with known timestamp | Record t_capture |
|
|
| 2 | Monitor GPS_INPUT messages at mavlink-inspector | Record t_gps_input for first GPS_INPUT update after t_capture |
|
|
| 3 | Compute latency = t_gps_input - t_capture | Per-frame latency |
|
|
| 4 | Repeat for 30 consecutive frames | Array of 30 latency values |
|
|
|
|
**Pass criteria**: p95 latency < 400ms; max latency < 500ms
|
|
**Duration**: 50s (~30 frames at 0.7fps + warm-up)
|
|
|
|
---
|
|
|
|
### NFT-PERF-02: GPS_INPUT Output Rate Consistency
|
|
|
|
**Summary**: Validate GPS_INPUT messages are delivered at a sustained 5-10Hz with no gaps.
|
|
**Traces to**: AC-08 (GPS_INPUT via MAVLink at 5-10Hz)
|
|
**Metric**: Message rate (Hz), maximum inter-message gap (ms)
|
|
|
|
**Preconditions**:
|
|
- System in steady state
|
|
- Camera-replay active
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Capture GPS_INPUT messages for 60 seconds | Count messages, record timestamps |
|
|
| 2 | Compute rate: count / 60 | 5 ≤ rate ≤ 10 Hz |
|
|
| 3 | Compute max gap between consecutive messages | max_gap ≤ 250ms |
|
|
| 4 | Compute jitter: std_dev of inter-message intervals | jitter < 50ms |
|
|
|
|
**Pass criteria**: Rate 5-10Hz; max gap ≤ 250ms; jitter < 50ms
|
|
**Duration**: 60s
|
|
|
|
---
|
|
|
|
### NFT-PERF-03: cuVSLAM Visual Odometry Processing Time
|
|
|
|
**Summary**: Validate cuVSLAM processes each frame within 20ms.
|
|
**Traces to**: AC-07 (real-time processing budget)
|
|
**Metric**: Per-frame cuVSLAM inference time (ms)
|
|
|
|
**Preconditions**:
|
|
- System running on Jetson Orin Nano Super
|
|
- Steady state (≥10 frames processed)
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Replay 30 frames, read processing time from SSE events or health endpoint metrics | Per-frame VO time |
|
|
| 2 | Compute p95 of VO time | p95 ≤ 20ms |
|
|
|
|
**Pass criteria**: p95 cuVSLAM inference time ≤ 20ms
|
|
**Duration**: 50s
|
|
|
|
---
|
|
|
|
### NFT-PERF-04: Satellite Matching Latency (Async)
|
|
|
|
**Summary**: Validate satellite matching completes within 330ms per keyframe (async, does not block VO).
|
|
**Traces to**: AC-07 (within frame budget), solution processing time budget
|
|
**Metric**: Per-keyframe satellite matching latency (ms)
|
|
|
|
**Preconditions**:
|
|
- System running on Jetson Orin Nano Super
|
|
- Satellite tiles loaded
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Monitor satellite match events over 60s (expect ~4-8 matches at 0.07-0.14Hz) | Per-match latency from health/metrics endpoint |
|
|
| 2 | Verify no VO frame was blocked during satellite matching | VO timestamps maintain 0.7fps cadence |
|
|
|
|
**Pass criteria**: p95 satellite matching ≤ 330ms; VO cadence unaffected
|
|
**Duration**: 60s
|
|
|
|
---
|
|
|
|
### NFT-PERF-05: TRT Engine Load Time
|
|
|
|
**Summary**: Validate all TensorRT engines load within 10 seconds total.
|
|
**Traces to**: AC-11 (startup), solution startup sequence
|
|
**Metric**: Engine load time (seconds)
|
|
|
|
**Preconditions**:
|
|
- Cold start on Jetson Orin Nano Super
|
|
- Engines pre-built and available on storage
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Start system, monitor startup log for engine load timestamps | t_start_load, t_end_load per engine |
|
|
| 2 | Compute total: sum of all engine load times | ≤ 10s total |
|
|
|
|
**Pass criteria**: Total TRT engine load time ≤ 10s
|
|
**Duration**: 30s (includes boot time)
|
|
|
|
---
|
|
|
|
### NFT-PERF-06: Sustained 30-Minute Processing
|
|
|
|
**Summary**: Validate the system maintains consistent performance over a 30-minute continuous session without degradation.
|
|
**Traces to**: AC-07 (real-time), AC-08 (memory < 8GB)
|
|
**Metric**: Per-frame latency, GPS_INPUT rate, position accuracy over time
|
|
|
|
**Preconditions**:
|
|
- System running on Jetson Orin Nano Super
|
|
- Camera-replay looping flight-sequence-60 (re-starts after frame 60)
|
|
- Satellite tiles available
|
|
|
|
**Steps**:
|
|
|
|
| Step | Consumer Action | Measurement |
|
|
|------|----------------|-------------|
|
|
| 1 | Run for 30 minutes, collect per-minute stats | Latency, rate, accuracy |
|
|
| 2 | Compare first-5-min stats vs last-5-min stats | No degradation >10% |
|
|
| 3 | Monitor for any position output gaps > 1s | Count gaps |
|
|
|
|
**Pass criteria**: No latency degradation >10% over 30 min; GPS_INPUT rate remains 5-10Hz; no output gaps >1s
|
|
**Duration**: 30 minutes
|