Oleksandr Bezdieniezhnykh
4.8 KiB
Performance Tests
NFT-PERF-01: End-to-End Per-Frame Latency
Summary: Validate total pipeline latency from camera capture to GPS_INPUT output is <400ms. Traces to: AC-07 (< 400ms end-to-end per frame) Metric: End-to-end latency (camera frame timestamp → GPS_INPUT message timestamp)
Preconditions:
- System running on Jetson Orin Nano Super (GPU-mode)
- Camera-replay serving frames at 0.7fps
- System in steady state (warm-up: ≥10 frames processed)
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Camera-replay sends frame with known timestamp | Record t_capture |
| 2 | Monitor GPS_INPUT messages at mavlink-inspector | Record t_gps_input for first GPS_INPUT update after t_capture |
| 3 | Compute latency = t_gps_input - t_capture | Per-frame latency |
| 4 | Repeat for 30 consecutive frames | Array of 30 latency values |
Pass criteria: p95 latency < 400ms; max latency < 500ms Duration: 50s (~30 frames at 0.7fps + warm-up)
NFT-PERF-02: GPS_INPUT Output Rate Consistency
Summary: Validate GPS_INPUT messages are delivered at a sustained 5-10Hz with no gaps. Traces to: AC-08 (GPS_INPUT via MAVLink at 5-10Hz) Metric: Message rate (Hz), maximum inter-message gap (ms)
Preconditions:
- System in steady state
- Camera-replay active
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Capture GPS_INPUT messages for 60 seconds | Count messages, record timestamps |
| 2 | Compute rate: count / 60 | 5 ≤ rate ≤ 10 Hz |
| 3 | Compute max gap between consecutive messages | max_gap ≤ 250ms |
| 4 | Compute jitter: std_dev of inter-message intervals | jitter < 50ms |
Pass criteria: Rate 5-10Hz; max gap ≤ 250ms; jitter < 50ms Duration: 60s
NFT-PERF-03: cuVSLAM Visual Odometry Processing Time
Summary: Validate cuVSLAM processes each frame within 20ms. Traces to: AC-07 (real-time processing budget) Metric: Per-frame cuVSLAM inference time (ms)
Preconditions:
- System running on Jetson Orin Nano Super
- Steady state (≥10 frames processed)
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Replay 30 frames, read processing time from SSE events or health endpoint metrics | Per-frame VO time |
| 2 | Compute p95 of VO time | p95 ≤ 20ms |
Pass criteria: p95 cuVSLAM inference time ≤ 20ms Duration: 50s
NFT-PERF-04: Satellite Matching Latency (Async)
Summary: Validate satellite matching completes within 330ms per keyframe (async, does not block VO). Traces to: AC-07 (within frame budget), solution processing time budget Metric: Per-keyframe satellite matching latency (ms)
Preconditions:
- System running on Jetson Orin Nano Super
- Satellite tiles loaded
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Monitor satellite match events over 60s (expect ~4-8 matches at 0.07-0.14Hz) | Per-match latency from health/metrics endpoint |
| 2 | Verify no VO frame was blocked during satellite matching | VO timestamps maintain 0.7fps cadence |
Pass criteria: p95 satellite matching ≤ 330ms; VO cadence unaffected Duration: 60s
NFT-PERF-05: TRT Engine Load Time
Summary: Validate all TensorRT engines load within 10 seconds total. Traces to: AC-11 (startup), solution startup sequence Metric: Engine load time (seconds)
Preconditions:
- Cold start on Jetson Orin Nano Super
- Engines pre-built and available on storage
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Start system, monitor startup log for engine load timestamps | t_start_load, t_end_load per engine |
| 2 | Compute total: sum of all engine load times | ≤ 10s total |
Pass criteria: Total TRT engine load time ≤ 10s Duration: 30s (includes boot time)
NFT-PERF-06: Sustained 30-Minute Processing
Summary: Validate the system maintains consistent performance over a 30-minute continuous session without degradation. Traces to: AC-07 (real-time), AC-08 (memory < 8GB) Metric: Per-frame latency, GPS_INPUT rate, position accuracy over time
Preconditions:
- System running on Jetson Orin Nano Super
- Camera-replay looping flight-sequence-60 (re-starts after frame 60)
- Satellite tiles available
Steps:
| Step | Consumer Action | Measurement |
|---|---|---|
| 1 | Run for 30 minutes, collect per-minute stats | Latency, rate, accuracy |
| 2 | Compare first-5-min stats vs last-5-min stats | No degradation >10% |
| 3 | Monitor for any position output gaps > 1s | Count gaps |
Pass criteria: No latency degradation >10% over 30 min; GPS_INPUT rate remains 5-10Hz; no output gaps >1s Duration: 30 minutes