mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-04-23 03:16:36 +00:00
Revise skills documentation to incorporate updated directory structure and terminology. Replace references to integration tests with blackbox tests in SKILL.md files and templates. Adjust paths in planning and deployment documentation to align with the new _docs/02_document/ structure, ensuring consistency and clarity throughout the documentation.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,503 @@
|
||||
# Blackbox Tests
|
||||
|
||||
## Positive Scenarios
|
||||
|
||||
### FT-P-01: End-to-End Position Accuracy — 50m Threshold
|
||||
|
||||
**Summary**: Validate that ≥80% of frame positions are within 50m of ground truth GPS across a full 60-frame flight sequence.
|
||||
**Traces to**: AC-01 (80% within 50m)
|
||||
**Category**: Position Accuracy
|
||||
|
||||
**Preconditions**:
|
||||
- System running with SITL ArduPilot (GPS_TYPE=14)
|
||||
- Camera replay serving flight-sequence-60 at 0.7fps
|
||||
- Satellite tiles for test area loaded
|
||||
- System has completed startup (first satellite match done)
|
||||
|
||||
**Input data**: flight-sequence-60 (60 frames), coordinates.csv (ground truth), position_accuracy.csv (thresholds)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Start session via POST /sessions | HTTP 201 with session ID |
|
||||
| 2 | Subscribe to SSE stream GET /sessions/{id}/stream | SSE events begin at ~1Hz |
|
||||
| 3 | Wait for camera-replay to complete all 60 frames (~86s at 0.7fps) | Position events for each processed frame |
|
||||
| 4 | Collect all position events with lat/lon | 60 position estimates (some frames may have multiple updates) |
|
||||
| 5 | For each frame: compute haversine distance between estimated and ground truth position | Distance array |
|
||||
| 6 | Count frames where distance < 50m, compute percentage | ≥80% |
|
||||
|
||||
**Expected outcome**: ≥48 of 60 frames have position error < 50m from ground truth in coordinates.csv
|
||||
**Max execution time**: 120s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-02: End-to-End Position Accuracy — 20m Threshold
|
||||
|
||||
**Summary**: Validate that ≥60% of frame positions are within 20m of ground truth GPS.
|
||||
**Traces to**: AC-02 (60% within 20m)
|
||||
**Category**: Position Accuracy
|
||||
|
||||
**Preconditions**: Same as FT-P-01
|
||||
|
||||
**Input data**: flight-sequence-60, coordinates.csv, position_accuracy.csv
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Reuse position data from FT-P-01 run (or re-run) | 60 position estimates |
|
||||
| 2 | Count frames where distance < 20m, compute percentage | ≥60% |
|
||||
|
||||
**Expected outcome**: ≥36 of 60 frames have position error < 20m
|
||||
**Max execution time**: 120s (shared with FT-P-01)
|
||||
|
||||
---
|
||||
|
||||
### FT-P-03: No Single Frame Exceeds Maximum Error
|
||||
|
||||
**Summary**: Validate that no individual frame position estimate exceeds 100m error.
|
||||
**Traces to**: AC-01, AC-02 (implicit: no catastrophic outliers)
|
||||
**Category**: Position Accuracy
|
||||
|
||||
**Preconditions**: Same as FT-P-01
|
||||
|
||||
**Input data**: flight-sequence-60, coordinates.csv, position_accuracy.csv
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Reuse position data from FT-P-01 | 60 position estimates |
|
||||
| 2 | Find max error across all frames | max(distances) ≤ 100m |
|
||||
|
||||
**Expected outcome**: Maximum position error across all 60 frames ≤ 100m
|
||||
**Max execution time**: 120s (shared with FT-P-01)
|
||||
|
||||
---
|
||||
|
||||
### FT-P-04: VO Drift Between Satellite Anchors
|
||||
|
||||
**Summary**: Validate cumulative VO drift stays below 100m between consecutive satellite correction events.
|
||||
**Traces to**: AC-03 (drift < 100m between anchors)
|
||||
**Category**: Position Accuracy
|
||||
|
||||
**Preconditions**: Same as FT-P-01; satellite matching active on keyframes
|
||||
|
||||
**Input data**: flight-sequence-60 SSE stream (includes drift_from_anchor field)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Subscribe to SSE stream | Events with drift_from_anchor field |
|
||||
| 2 | Record drift_from_anchor values over the full sequence | Array of drift values |
|
||||
| 3 | Find maximum drift_from_anchor value | max(drift) < 100m |
|
||||
|
||||
**Expected outcome**: drift_from_anchor never exceeds 100m during the 60-frame sequence
|
||||
**Max execution time**: 120s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-05: GPS_INPUT Message Correctness — Normal Tracking
|
||||
|
||||
**Summary**: Validate GPS_INPUT message fields are correctly populated during normal satellite-anchored tracking.
|
||||
**Traces to**: AC-08 (GPS_INPUT to FC via MAVLink), AC-04 (confidence score)
|
||||
**Category**: Flight Controller Integration
|
||||
|
||||
**Preconditions**: System tracking normally with recent satellite match (<30s)
|
||||
|
||||
**Input data**: Normal frame + satellite match; MAVLink capture from mavlink-inspector
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Read captured GPS_INPUT messages from mavlink-inspector | GPS_INPUT messages at 5-10Hz |
|
||||
| 2 | Verify field: fix_type | fix_type == 3 |
|
||||
| 3 | Verify field: horiz_accuracy | 1.0 ≤ horiz_accuracy ≤ 50.0 |
|
||||
| 4 | Verify field: satellites_visible | satellites_visible == 10 |
|
||||
| 5 | Verify fields: lat, lon | Non-zero, within operational area bounds |
|
||||
| 6 | Verify fields: vn, ve, vd | Populated (non-NaN), magnitude consistent with ~50-70 km/h flight |
|
||||
|
||||
**Expected outcome**: All GPS_INPUT fields populated correctly per specification
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-06: Image Registration Rate
|
||||
|
||||
**Summary**: Validate that ≥95% of frames in a normal flight are successfully registered by the VO pipeline.
|
||||
**Traces to**: AC-05 (registration > 95%)
|
||||
**Category**: Image Processing Quality
|
||||
|
||||
**Preconditions**: System running with full 60-frame sequence
|
||||
|
||||
**Input data**: flight-sequence-60 SSE stream (vo_status field)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Subscribe to SSE stream | Events with vo_status field |
|
||||
| 2 | Count frames where vo_status == "tracking" | ≥57 of 60 |
|
||||
| 3 | Compute registration rate | ≥95% |
|
||||
|
||||
**Expected outcome**: ≥57 of 60 frames report vo_status "tracking"
|
||||
**Max execution time**: 120s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-07: Confidence Tier — HIGH
|
||||
|
||||
**Summary**: Validate HIGH confidence tier when satellite match is recent and covariance is low.
|
||||
**Traces to**: AC-04 (confidence score per estimate)
|
||||
**Category**: Confidence Scoring
|
||||
|
||||
**Preconditions**: System running, satellite match completed <30s ago
|
||||
|
||||
**Input data**: SSE stream during normal tracking
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Read SSE event immediately after satellite match | confidence field |
|
||||
| 2 | Verify confidence == "HIGH" | "HIGH" |
|
||||
| 3 | Read GPS_INPUT fix_type from mavlink-inspector | fix_type == 3 |
|
||||
|
||||
**Expected outcome**: Confidence tier is HIGH, fix_type is 3
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-08: Confidence Tier — MEDIUM (VO-only, No Recent Satellite Match)
|
||||
|
||||
**Summary**: Validate MEDIUM confidence tier when VO is tracking but no satellite match in >30s.
|
||||
**Traces to**: AC-04
|
||||
**Category**: Confidence Scoring
|
||||
|
||||
**Preconditions**: System running; satellite tile server paused (returns 503) to prevent new matches; >30s since last match
|
||||
|
||||
**Input data**: SSE stream during VO-only tracking
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Pause satellite-tile-server (Docker pause) | No new satellite matches possible |
|
||||
| 2 | Wait >30s after last satellite match | Confidence should transition |
|
||||
| 3 | Read SSE event | confidence == "MEDIUM" |
|
||||
| 4 | Read GPS_INPUT fix_type | fix_type == 3 |
|
||||
|
||||
**Expected outcome**: Confidence transitions to MEDIUM; fix_type remains 3
|
||||
**Max execution time**: 60s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-09: GPS_INPUT Output Rate
|
||||
|
||||
**Summary**: Validate GPS_INPUT messages are sent at 5-10Hz continuously.
|
||||
**Traces to**: AC-08 (GPS_INPUT via MAVLink), AC-09 (frame-by-frame streaming)
|
||||
**Category**: Flight Controller Integration
|
||||
|
||||
**Preconditions**: System running and producing position estimates
|
||||
|
||||
**Input data**: MAVLink capture from mavlink-inspector (10s window)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Capture GPS_INPUT messages for 10 seconds | N messages |
|
||||
| 2 | Compute rate: N / 10 | 5 ≤ rate ≤ 10 |
|
||||
| 3 | Verify no gaps > 300ms between consecutive messages | max gap ≤ 300ms |
|
||||
|
||||
**Expected outcome**: Rate is 5-10Hz, no gap exceeds 300ms
|
||||
**Max execution time**: 15s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-10: Object Localization
|
||||
|
||||
**Summary**: Validate object GPS localization from pixel coordinates via the FastAPI endpoint.
|
||||
**Traces to**: AC-16 (object localization), AC-17 (trigonometric calculation)
|
||||
**Category**: Object Localization
|
||||
|
||||
**Preconditions**: System running with known UAV position (from GPS-denied estimate); known object ground truth GPS
|
||||
|
||||
**Input data**: pixel_x, pixel_y (center of frame = nadir), gimbal_pan_deg=0, gimbal_tilt_deg=-90, zoom_factor=1.0
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /objects/locate with pixel at frame center, gimbal pointing straight down | JSON: { lat, lon, alt, accuracy_m, confidence } |
|
||||
| 2 | Compute haversine distance between response lat/lon and current UAV position | Should be < accuracy_m (nadir point ≈ UAV position) |
|
||||
| 3 | Verify accuracy_m is consistent with current system accuracy | accuracy_m > 0, accuracy_m < 100m |
|
||||
|
||||
**Expected outcome**: Object location at nadir matches UAV position within accuracy_m
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-11: Coordinate Transform Round-Trip
|
||||
|
||||
**Summary**: Validate GPS→NED→pixel→GPS round-trip error is <0.1m.
|
||||
**Traces to**: AC-18 (WGS84 output)
|
||||
**Category**: Coordinate Transforms
|
||||
|
||||
**Preconditions**: System running, position known
|
||||
|
||||
**Input data**: Known GPS coordinate within operational area
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Query system for current position via SSE | lat, lon |
|
||||
| 2 | POST /objects/locate with frame center pixel, straight-down gimbal | Returned lat, lon |
|
||||
| 3 | Compute haversine distance between original UAV lat/lon and round-trip result | distance < 0.1m |
|
||||
|
||||
**Expected outcome**: Round-trip error < 0.1m
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-12: Startup — GPS_INPUT Within 60 Seconds
|
||||
|
||||
**Summary**: Validate the system begins outputting GPS_INPUT messages within 60s of boot.
|
||||
**Traces to**: AC-11 (startup from last GPS)
|
||||
**Category**: Startup & Failsafe
|
||||
|
||||
**Preconditions**: Fresh system start; SITL ArduPilot running with GLOBAL_POSITION_INT available
|
||||
|
||||
**Input data**: MAVLink capture from mavlink-inspector
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Start gps-denied-system container | System boots |
|
||||
| 2 | Monitor mavlink-inspector for first GPS_INPUT message | Timestamp of first GPS_INPUT |
|
||||
| 3 | Compute elapsed time from container start to first GPS_INPUT | ≤ 60s |
|
||||
|
||||
**Expected outcome**: First GPS_INPUT message arrives within 60s of system start
|
||||
**Max execution time**: 90s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-13: Telemetry Output Rate
|
||||
|
||||
**Summary**: Validate telemetry NAMED_VALUE_FLOAT messages are sent at 1Hz.
|
||||
**Traces to**: AC-14 (telemetry to ground station)
|
||||
**Category**: Telemetry
|
||||
|
||||
**Preconditions**: System running normally
|
||||
|
||||
**Input data**: MAVLink capture from mavlink-inspector (10s window)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Capture NAMED_VALUE_FLOAT messages for "gps_conf", "gps_drift", "gps_hacc" over 10s | N messages per name |
|
||||
| 2 | Verify rate: ~1Hz per metric (8-12 messages per name in 10s) | 0.8-1.2 Hz |
|
||||
|
||||
**Expected outcome**: Each telemetry metric sent at ~1Hz
|
||||
**Max execution time**: 15s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-14: SSE Stream Schema
|
||||
|
||||
**Summary**: Validate SSE position events contain all required fields with correct types.
|
||||
**Traces to**: AC-14 (streaming to ground station)
|
||||
**Category**: API & Communication
|
||||
|
||||
**Preconditions**: Active session with SSE stream
|
||||
|
||||
**Input data**: SSE events from /sessions/{id}/stream
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Subscribe to SSE stream | Events at ~1Hz |
|
||||
| 2 | Parse event JSON | Valid JSON |
|
||||
| 3 | Verify fields: type (string), timestamp (ISO8601), lat (float), lon (float), alt (float), accuracy_h (float), confidence (string), drift_from_anchor (float), vo_status (string), last_satellite_match_age_s (float) | All present with correct types |
|
||||
|
||||
**Expected outcome**: Every SSE event conforms to the specified schema
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
## Negative Scenarios
|
||||
|
||||
### FT-N-01: Trajectory Direction Change (Frames 32-43)
|
||||
|
||||
**Summary**: Validate system continues producing position estimates through a trajectory direction change.
|
||||
**Traces to**: AC-07 (disconnected segments core to system)
|
||||
**Category**: Resilience & Edge Cases
|
||||
|
||||
**Preconditions**: System running; camera-replay set to serve frames 32-43 (direction change area)
|
||||
|
||||
**Input data**: Frames AD000032-043.jpg, coordinates for frames 32-43
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Camera-replay serves frames 32-43 at 0.7fps | System processes frames |
|
||||
| 2 | Collect SSE position events for each frame | ≥12 position estimates (one per frame minimum) |
|
||||
| 3 | Verify no gap >5s without a position update | Continuous output |
|
||||
|
||||
**Expected outcome**: System produces position estimates for all frames in the direction-change segment; no prolonged output gap
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-02: Outlier Frame Handling (350m Gap)
|
||||
|
||||
**Summary**: Validate system handles a 350m outlier between consecutive photos without position corruption.
|
||||
**Traces to**: AC-06 (350m outlier tolerance)
|
||||
**Category**: Resilience & Edge Cases
|
||||
|
||||
**Preconditions**: System running with normal tracking established; fault injection: camera-replay skips frames to simulate 350m gap
|
||||
|
||||
**Input data**: Normal frames followed by a frame 350m away (simulated by frame skip in camera-replay)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Normal tracking for 10 frames | Position estimates with <50m error |
|
||||
| 2 | Camera-replay jumps forward ~350m (skips multiple frames) | System detects discontinuity |
|
||||
| 3 | Collect position estimates for next 5 frames after the gap | Recovery within 3-5 frames |
|
||||
| 4 | Verify position error of recovered frames | Error < 100m for first valid frame after recovery |
|
||||
|
||||
**Expected outcome**: System recovers from 350m outlier; post-recovery position error < 100m
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-03: Invalid Object Localization Request
|
||||
|
||||
**Summary**: Validate API rejects invalid pixel coordinates with HTTP 422.
|
||||
**Traces to**: AC-16 (object localization)
|
||||
**Category**: API Error Handling
|
||||
|
||||
**Preconditions**: System running with active session
|
||||
|
||||
**Input data**: POST /objects/locate with pixel_x=-100, pixel_y=-100
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /objects/locate with negative pixel coordinates | HTTP 422 |
|
||||
| 2 | Verify response body contains error description | JSON with "error" or "detail" field |
|
||||
|
||||
**Expected outcome**: HTTP 422 with validation error
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-04: Unauthenticated API Access
|
||||
|
||||
**Summary**: Validate API rejects unauthenticated requests with HTTP 401.
|
||||
**Traces to**: AC-14 (security — JWT auth)
|
||||
**Category**: API Security
|
||||
|
||||
**Preconditions**: System running
|
||||
|
||||
**Input data**: POST /sessions with no Authorization header
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /sessions without JWT token | HTTP 401 |
|
||||
| 2 | GET /sessions/{id}/stream without JWT | HTTP 401 |
|
||||
| 3 | POST /objects/locate without JWT | HTTP 401 |
|
||||
| 4 | GET /health (no auth required) | HTTP 200 |
|
||||
|
||||
**Expected outcome**: Protected endpoints return 401; /health remains accessible
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-05: 3-Consecutive-Failure Re-Localization Request
|
||||
|
||||
**Summary**: Validate that after VO loss + 3 consecutive satellite match failures, the system sends a re-localization request to the ground station.
|
||||
**Traces to**: AC-08 (3 consecutive failures → re-localization request)
|
||||
**Category**: Resilience & Edge Cases
|
||||
|
||||
**Preconditions**: System running; camera-replay set to serve featureless frames (VO will fail); satellite-tile-server returning 404 (tile not found)
|
||||
|
||||
**Input data**: Featureless frames (e.g., blank/uniform images), satellite tile server offline
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Camera-replay serves featureless frames | VO tracking lost |
|
||||
| 2 | Satellite-tile-server returns 404 | Satellite matching fails |
|
||||
| 3 | Wait for 3 camera frames (3 × 1.43s ≈ 4.3s) | 3 consecutive failures |
|
||||
| 4 | Check mavlink-inspector for STATUSTEXT | Message matches `RELOC_REQ: last_lat=.* last_lon=.* uncertainty=.*m` |
|
||||
| 5 | Verify GPS_INPUT fix_type | fix_type == 0 |
|
||||
| 6 | Verify GPS_INPUT horiz_accuracy | horiz_accuracy == 999.0 |
|
||||
|
||||
**Expected outcome**: RELOC_REQ sent via STATUSTEXT; GPS_INPUT reports no-fix with 999.0 accuracy
|
||||
**Max execution time**: 15s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-06: IMU-Only Dead Reckoning (VO Lost, No Satellite)
|
||||
|
||||
**Summary**: Validate system degrades gracefully to IMU-only ESKF prediction when VO and satellite matching both fail.
|
||||
**Traces to**: AC-06 (VO lost behavior), AC-04 (confidence score reflects state)
|
||||
**Category**: Resilience & Edge Cases
|
||||
|
||||
**Preconditions**: System running; camera-replay paused (no frames); satellite-tile-server paused
|
||||
|
||||
**Input data**: No camera frames, no satellite tiles; only IMU from SITL
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Pause camera-replay and satellite-tile-server | System loses VO and satellite inputs |
|
||||
| 2 | Read SSE events over 5s | confidence transitions from HIGH/MEDIUM to LOW |
|
||||
| 3 | Read GPS_INPUT from mavlink-inspector | fix_type == 2 |
|
||||
| 4 | Read horiz_accuracy over time | horiz_accuracy ≥ 50m and increasing |
|
||||
| 5 | Verify GPS_INPUT continues at 5-10Hz | Messages continue (IMU-driven ESKF prediction) |
|
||||
|
||||
**Expected outcome**: System continues GPS_INPUT at 5-10Hz via IMU; confidence drops; accuracy degrades but output never stops
|
||||
**Max execution time**: 15s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-07: Operator Re-Localization Hint Accepted
|
||||
|
||||
**Summary**: Validate the system accepts an operator re-localization hint and recovers position.
|
||||
**Traces to**: AC-08 (re-localization), AC-15 (ground station commands)
|
||||
**Category**: Ground Station Integration
|
||||
|
||||
**Preconditions**: System in FAILED confidence state (3 consecutive failures); satellite-tile-server restored
|
||||
|
||||
**Input data**: Operator hint: approximate lat/lon (from coordinates.csv ground truth ± 200m offset)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Trigger 3-consecutive-failure state (FT-N-05 preconditions) | RELOC_REQ sent |
|
||||
| 2 | Restore satellite-tile-server | Tiles available again |
|
||||
| 3 | POST /sessions/{id}/anchor with approximate lat/lon | HTTP 200 |
|
||||
| 4 | Wait for satellite match attempt (~3-5s) | System searches in new area |
|
||||
| 5 | Read SSE events | confidence transitions back to HIGH/MEDIUM |
|
||||
| 6 | Read GPS_INPUT fix_type | fix_type == 3 |
|
||||
|
||||
**Expected outcome**: System accepts operator hint, searches satellite tiles in new area, recovers position, confidence returns to HIGH/MEDIUM
|
||||
**Max execution time**: 30s
|
||||
@@ -0,0 +1,149 @@
|
||||
# Test Environment
|
||||
|
||||
## Overview
|
||||
|
||||
**System under test**: GPS-Denied Visual Navigation System — a real-time position estimation service running on Jetson Orin Nano Super. Public interfaces: FastAPI REST/SSE endpoints (port 8000), MAVLink GPS_INPUT messages over serial/UDP, MAVLink telemetry messages.
|
||||
|
||||
**Consumer app purpose**: Standalone Python test runner (pytest) that exercises the GPS-denied system through its public interfaces (HTTP API, MAVLink message inspection, SSE stream consumption) without access to internal modules, ESKF state, or GPU buffers.
|
||||
|
||||
## Test Modes
|
||||
|
||||
This is embedded robotics software targeting Jetson Orin Nano Super. A pure Docker environment cannot exercise GPU-dependent paths (TRT inference, cuVSLAM, CUDA streams). The test environment supports two modes:
|
||||
|
||||
**Mode 1 — Docker SITL (CI/dev)**: Full system in Docker containers with ArduPilot SITL providing MAVLink + IMU at 200Hz. Camera images replayed from input_data/. Satellite tiles served from a mock HTTP server. GPS-denied system runs in CPU-mode with stubbed TRT/cuVSLAM inference (functionally equivalent but slower). Tests all integration paths, API, MAVLink, resilience, and security.
|
||||
|
||||
**Mode 2 — Jetson Hardware (nightly/pre-deploy)**: GPS-denied system runs natively on Jetson Orin Nano Super with real CUDA/TRT/cuVSLAM. ArduPilot SITL runs on the Jetson or a companion x86 host, connected via UART or UDP. Camera frames injected via USB camera emulator or replay service. Tests real-time performance, GPU memory, thermal, TRT correctness, and CUDA stream isolation.
|
||||
|
||||
## Docker Environment (Mode 1)
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Image / Build | Purpose | Ports |
|
||||
|---------|--------------|---------|-------|
|
||||
| gps-denied-system | `./Dockerfile` (build context: project root) | GPS-denied navigation system in CPU-mode (TRT stubs, cuVSLAM stub returning synthetic VO poses derived from ground truth trajectory) | 8000 (FastAPI), 14550/udp (MAVLink) |
|
||||
| ardupilot-sitl | `ardupilot/sitl:plane-4.5` (fixed-wing) | ArduPilot SITL: flies waypoint mission following coordinates.csv trajectory, generates IMU at 200Hz via MAVLink, GPS_TYPE=14 accepts GPS_INPUT, provides GLOBAL_POSITION_INT at startup | 5760 (MAVLink TCP), 14551/udp |
|
||||
| camera-replay | `./tests/docker/camera-replay/Dockerfile` | Replays AD000001-060.jpg from input_data/ at configurable FPS (default 0.7fps) with timestamps synchronized to SITL clock. Supports fault injection: frame skip, corrupted JPEG, pause. | 8001 (frame server) |
|
||||
| satellite-tile-server | `./tests/docker/tile-server/Dockerfile` | HTTP server for pre-cached satellite tiles (zoom 18, ~50-100 tiles covering test area). Supports fault injection: 404 for specific tiles, 503 for full outage, slow responses. | 8002 |
|
||||
| mavlink-inspector | `./tests/docker/mavlink-inspector/Dockerfile` | Passively captures all MAVLink traffic (GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT, COMMAND_LONG) for post-test assertion. Can inject operator re-localization hints. | 14552/udp |
|
||||
| e2e-consumer | `./tests/e2e/Dockerfile` | Black-box test runner (pytest). Communicates only via HTTP API + MAVLink inspector. | — |
|
||||
|
||||
### Networks
|
||||
|
||||
| Network | Services | Purpose |
|
||||
|---------|----------|---------|
|
||||
| e2e-net | all | Isolated test network; MAVLink UDP multicast between gps-denied-system, ardupilot-sitl, mavlink-inspector |
|
||||
|
||||
### Volumes
|
||||
|
||||
| Volume | Mounted to | Purpose |
|
||||
|--------|-----------|---------|
|
||||
| input-data | camera-replay:/data, e2e-consumer:/test-data | Camera frames (AD000001-060.jpg), coordinates.csv, data_parameters.md, gmaps reference images |
|
||||
| satellite-tiles | satellite-tile-server:/tiles, gps-denied-system:/tiles | Pre-processed satellite tiles for test area (zoom 18, 48.249-48.276°N, 37.340-37.386°E) |
|
||||
| sitl-mission | ardupilot-sitl:/mission | Waypoint mission file derived from coordinates.csv (SITL flies this trajectory, generating physically consistent 200Hz IMU data) |
|
||||
| test-results | e2e-consumer:/results | Test result CSV output |
|
||||
| mavlink-capture | mavlink-inspector:/capture | Recorded MAVLink messages for post-test assertions |
|
||||
|
||||
### IMU Data Flow
|
||||
|
||||
ArduPilot SITL is the primary source of IMU data. It flies a waypoint mission derived from coordinates.csv and internally generates physically consistent accelerometer + gyroscope readings at 200Hz, delivered to the GPS-denied system via MAVLink (RAW_IMU, SCALED_IMU2). This eliminates the need for pre-recorded IMU data files and ensures IMU/trajectory consistency.
|
||||
|
||||
```
|
||||
coordinates.csv → mission_generator script → ArduPilot waypoint file
|
||||
↓
|
||||
ArduPilot SITL flies trajectory
|
||||
↓
|
||||
IMU @ 200Hz + heartbeat + GLOBAL_POSITION_INT
|
||||
↓ (MAVLink UDP)
|
||||
gps-denied-system receives IMU for ESKF
|
||||
```
|
||||
|
||||
### docker-compose structure
|
||||
|
||||
```yaml
|
||||
services:
|
||||
ardupilot-sitl:
|
||||
# ArduPilot SITL fixed-wing, outputs IMU at 200Hz via MAVLink
|
||||
# GPS_TYPE=14 (MAVLink), pre-configured for GPS_INPUT acceptance
|
||||
satellite-tile-server:
|
||||
# HTTP tile server with tiles for test area (48.249-48.276°N, 37.340-37.386°E)
|
||||
camera-replay:
|
||||
# Replays AD000001-060.jpg at 0.7fps, serves via HTTP or shared volume
|
||||
depends_on:
|
||||
- satellite-tile-server
|
||||
gps-denied-system:
|
||||
# The system under test
|
||||
depends_on:
|
||||
- ardupilot-sitl
|
||||
- satellite-tile-server
|
||||
- camera-replay
|
||||
mavlink-inspector:
|
||||
# Captures GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT messages
|
||||
depends_on:
|
||||
- ardupilot-sitl
|
||||
e2e-consumer:
|
||||
# pytest runner — executes after system reaches steady state
|
||||
depends_on:
|
||||
- gps-denied-system
|
||||
- mavlink-inspector
|
||||
```
|
||||
|
||||
## Consumer Application
|
||||
|
||||
**Tech stack**: Python 3.11, pytest, httpx (HTTP client), pymavlink (MAVLink inspection), sseclient-py (SSE stream)
|
||||
**Entry point**: `pytest tests/e2e/ --tb=short --csv=results/report.csv`
|
||||
|
||||
### Communication with system under test
|
||||
|
||||
| Interface | Protocol | Endpoint / Topic | Authentication |
|
||||
|-----------|----------|-----------------|----------------|
|
||||
| Position API | HTTP REST | http://gps-denied-system:8000/sessions | JWT token |
|
||||
| Position stream | HTTP SSE | http://gps-denied-system:8000/sessions/{id}/stream | JWT token |
|
||||
| Object localization | HTTP REST | http://gps-denied-system:8000/objects/locate | JWT token |
|
||||
| Health check | HTTP REST | http://gps-denied-system:8000/health | None |
|
||||
| GPS_INPUT inspection | MAVLink UDP | mavlink-inspector:14552 (recorded messages) | None |
|
||||
| Telemetry inspection | MAVLink UDP | mavlink-inspector:14552 (NAMED_VALUE_FLOAT, STATUSTEXT) | None |
|
||||
|
||||
### What the consumer does NOT have access to
|
||||
|
||||
- No direct access to ESKF internal state, covariance matrices, or error vectors
|
||||
- No direct access to cuVSLAM tracking state or feature maps
|
||||
- No direct access to GPU memory, CUDA streams, or TRT engine internals
|
||||
- No direct access to the system's file system or configuration files
|
||||
- No direct database or state store access
|
||||
|
||||
## Jetson Hardware Environment (Mode 2)
|
||||
|
||||
Tests tagged `@pytest.mark.jetson` require actual Jetson Orin Nano Super hardware. These run natively (no Docker for the GPS-denied system) to exercise real GPU paths.
|
||||
|
||||
**Hardware setup**:
|
||||
- Jetson Orin Nano Super (JetPack 6.2, CUDA 12.x, TensorRT 10.3)
|
||||
- ArduPilot SITL on same Jetson (or x86 companion connected via UART/UDP)
|
||||
- Camera frames injected via: USB camera emulator (v4l2loopback feeding frames from input_data/) or HTTP replay service
|
||||
- Satellite tiles on local SSD (same path as production deployment)
|
||||
- Active cooling attached (required for sustained load tests)
|
||||
|
||||
**Tests in this mode**:
|
||||
- NFT-PERF-01 through NFT-PERF-06 (real GPU latency, throughput)
|
||||
- NFT-RES-LIM-01 (GPU+CPU shared memory monitoring via tegrastats)
|
||||
- NFT-RES-LIM-02 (thermal monitoring via thermal_zone sysfs)
|
||||
- NFT-RES-LIM-05 (CUDA stream isolation with real concurrent GPU work)
|
||||
- TRT engine build and inference correctness (expected_results #42-44)
|
||||
|
||||
**Jetson CI runner**: Self-hosted GitHub Actions runner on a dedicated Jetson Orin Nano Super, triggered for nightly builds and pre-deploy gates.
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
**When to run**: On every PR to `dev`, nightly full suite, before production deploy
|
||||
**Pipeline stages**:
|
||||
1. Unit tests (no Docker, no hardware) — on every commit
|
||||
2. Docker blackbox tests (SITL + CPU mode) — on PR merge to dev
|
||||
3. Hardware tests (Jetson runner) — nightly + pre-deploy
|
||||
|
||||
**Gate behavior**: Docker tests block merge; hardware tests are advisory (nightly) or blocking (pre-deploy)
|
||||
**Timeout**: Docker suite: 15 minutes; Hardware suite: 30 minutes
|
||||
|
||||
## Reporting
|
||||
|
||||
**Format**: CSV
|
||||
**Columns**: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message (if FAIL)
|
||||
**Output path**: `./tests/e2e-results/report.csv`
|
||||
@@ -0,0 +1,138 @@
|
||||
# Performance Tests
|
||||
|
||||
### NFT-PERF-01: End-to-End Per-Frame Latency
|
||||
|
||||
**Summary**: Validate total pipeline latency from camera capture to GPS_INPUT output is <400ms.
|
||||
**Traces to**: AC-07 (< 400ms end-to-end per frame)
|
||||
**Metric**: End-to-end latency (camera frame timestamp → GPS_INPUT message timestamp)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super (GPU-mode)
|
||||
- Camera-replay serving frames at 0.7fps
|
||||
- System in steady state (warm-up: ≥10 frames processed)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Camera-replay sends frame with known timestamp | Record t_capture |
|
||||
| 2 | Monitor GPS_INPUT messages at mavlink-inspector | Record t_gps_input for first GPS_INPUT update after t_capture |
|
||||
| 3 | Compute latency = t_gps_input - t_capture | Per-frame latency |
|
||||
| 4 | Repeat for 30 consecutive frames | Array of 30 latency values |
|
||||
|
||||
**Pass criteria**: p95 latency < 400ms; max latency < 500ms
|
||||
**Duration**: 50s (~30 frames at 0.7fps + warm-up)
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-02: GPS_INPUT Output Rate Consistency
|
||||
|
||||
**Summary**: Validate GPS_INPUT messages are delivered at a sustained 5-10Hz with no gaps.
|
||||
**Traces to**: AC-08 (GPS_INPUT via MAVLink at 5-10Hz)
|
||||
**Metric**: Message rate (Hz), maximum inter-message gap (ms)
|
||||
|
||||
**Preconditions**:
|
||||
- System in steady state
|
||||
- Camera-replay active
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Capture GPS_INPUT messages for 60 seconds | Count messages, record timestamps |
|
||||
| 2 | Compute rate: count / 60 | 5 ≤ rate ≤ 10 Hz |
|
||||
| 3 | Compute max gap between consecutive messages | max_gap ≤ 250ms |
|
||||
| 4 | Compute jitter: std_dev of inter-message intervals | jitter < 50ms |
|
||||
|
||||
**Pass criteria**: Rate 5-10Hz; max gap ≤ 250ms; jitter < 50ms
|
||||
**Duration**: 60s
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-03: cuVSLAM Visual Odometry Processing Time
|
||||
|
||||
**Summary**: Validate cuVSLAM processes each frame within 20ms.
|
||||
**Traces to**: AC-07 (real-time processing budget)
|
||||
**Metric**: Per-frame cuVSLAM inference time (ms)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super
|
||||
- Steady state (≥10 frames processed)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Replay 30 frames, read processing time from SSE events or health endpoint metrics | Per-frame VO time |
|
||||
| 2 | Compute p95 of VO time | p95 ≤ 20ms |
|
||||
|
||||
**Pass criteria**: p95 cuVSLAM inference time ≤ 20ms
|
||||
**Duration**: 50s
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-04: Satellite Matching Latency (Async)
|
||||
|
||||
**Summary**: Validate satellite matching completes within 330ms per keyframe (async, does not block VO).
|
||||
**Traces to**: AC-07 (within frame budget), solution processing time budget
|
||||
**Metric**: Per-keyframe satellite matching latency (ms)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super
|
||||
- Satellite tiles loaded
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Monitor satellite match events over 60s (expect ~4-8 matches at 0.07-0.14Hz) | Per-match latency from health/metrics endpoint |
|
||||
| 2 | Verify no VO frame was blocked during satellite matching | VO timestamps maintain 0.7fps cadence |
|
||||
|
||||
**Pass criteria**: p95 satellite matching ≤ 330ms; VO cadence unaffected
|
||||
**Duration**: 60s
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-05: TRT Engine Load Time
|
||||
|
||||
**Summary**: Validate all TensorRT engines load within 10 seconds total.
|
||||
**Traces to**: AC-11 (startup), solution startup sequence
|
||||
**Metric**: Engine load time (seconds)
|
||||
|
||||
**Preconditions**:
|
||||
- Cold start on Jetson Orin Nano Super
|
||||
- Engines pre-built and available on storage
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Start system, monitor startup log for engine load timestamps | t_start_load, t_end_load per engine |
|
||||
| 2 | Compute total: sum of all engine load times | ≤ 10s total |
|
||||
|
||||
**Pass criteria**: Total TRT engine load time ≤ 10s
|
||||
**Duration**: 30s (includes boot time)
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-06: Sustained 30-Minute Processing
|
||||
|
||||
**Summary**: Validate the system maintains consistent performance over a 30-minute continuous session without degradation.
|
||||
**Traces to**: AC-07 (real-time), AC-08 (memory < 8GB)
|
||||
**Metric**: Per-frame latency, GPS_INPUT rate, position accuracy over time
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super
|
||||
- Camera-replay looping flight-sequence-60 (re-starts after frame 60)
|
||||
- Satellite tiles available
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Run for 30 minutes, collect per-minute stats | Latency, rate, accuracy |
|
||||
| 2 | Compare first-5-min stats vs last-5-min stats | No degradation >10% |
|
||||
| 3 | Monitor for any position output gaps > 1s | Count gaps |
|
||||
|
||||
**Pass criteria**: No latency degradation >10% over 30 min; GPS_INPUT rate remains 5-10Hz; no output gaps >1s
|
||||
**Duration**: 30 minutes
|
||||
@@ -0,0 +1,169 @@
|
||||
# Resilience Tests
|
||||
|
||||
### NFT-RES-01: Mid-Flight Reboot Recovery
|
||||
|
||||
**Summary**: Validate the system recovers from a companion computer reboot within 70 seconds and restores position accuracy.
|
||||
**Traces to**: AC-12 (mid-flight reboot recovery)
|
||||
|
||||
**Preconditions**:
|
||||
- System running in steady state with good position accuracy
|
||||
- SITL ArduPilot continues running (FC stays up during companion computer reboot)
|
||||
|
||||
**Fault injection**:
|
||||
- Kill gps-denied-system process (docker stop or SIGKILL)
|
||||
- Restart after 5s delay (simulates Jetson reboot time)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Record current position accuracy and confidence | Baseline metrics |
|
||||
| 2 | Kill gps-denied-system container | GPS_INPUT messages stop |
|
||||
| 3 | Verify SITL continues running (heartbeat present) | FC still alive, using IMU dead reckoning |
|
||||
| 4 | Restart gps-denied-system container after 5s | System starts recovery sequence |
|
||||
| 5 | Monitor time from restart to first GPS_INPUT | ≤ 70s |
|
||||
| 6 | Wait for first satellite match | Position accuracy restored |
|
||||
| 7 | Verify position error after recovery | Error ≤ 50m after first satellite match |
|
||||
|
||||
**Pass criteria**: Recovery time ≤ 70s; post-recovery position error ≤ 50m after satellite match
|
||||
**Duration**: 120s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-02: Tracking Loss and Satellite Re-Localization
|
||||
|
||||
**Summary**: Validate the system recovers from cuVSLAM tracking loss via satellite-based re-localization.
|
||||
**Traces to**: AC-07 (disconnected segments), AC-06 (sharp turn handling)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking (HIGH confidence)
|
||||
- Satellite tiles available
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay sends featureless/blurred frames (simulates VO tracking loss from sharp turn)
|
||||
- Then resumes normal frames
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking established | confidence: HIGH, vo_status: tracking |
|
||||
| 2 | Camera-replay serves 3 featureless frames | cuVSLAM reports tracking_lost |
|
||||
| 3 | System enters TRACKING_LOST state | Satellite matching switches to every frame |
|
||||
| 4 | Camera-replay resumes normal frames | Satellite match succeeds |
|
||||
| 5 | Monitor SSE: vo_status returns to "tracking" | cuVSLAM restarted |
|
||||
| 6 | Monitor SSE: confidence returns to HIGH | Position re-anchored |
|
||||
| 7 | Verify position accuracy after recovery | Error ≤ 50m |
|
||||
|
||||
**Pass criteria**: Recovery within 5 frames after normal frames resume; position error ≤ 50m post-recovery
|
||||
**Duration**: 30s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-03: Sustained IMU-Only Operation
|
||||
|
||||
**Summary**: Validate the system continues producing position estimates during extended IMU-only periods without crashing.
|
||||
**Traces to**: AC-08 (system continues during failure), AC-12 (failsafe)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Pause both camera-replay (no VO) and satellite-tile-server (no satellite matching)
|
||||
- Duration: 30s
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Establish normal tracking baseline | GPS_INPUT at 5-10Hz, confidence HIGH |
|
||||
| 2 | Pause camera-replay and satellite-tile-server | VO and satellite inputs stop |
|
||||
| 3 | Monitor GPS_INPUT for 30s | Messages continue at 5-10Hz (IMU-driven ESKF prediction) |
|
||||
| 4 | Verify horiz_accuracy grows over time | accuracy increases monotonically |
|
||||
| 5 | Verify fix_type transitions to 2 | Degraded but present |
|
||||
| 6 | Verify confidence transitions to LOW | Reflects IMU-only state |
|
||||
| 7 | Resume camera-replay and satellite-tile-server | System recovers to normal tracking |
|
||||
| 8 | Verify recovery to HIGH confidence | Satellite match re-anchors position |
|
||||
|
||||
**Pass criteria**: GPS_INPUT never stops during 30s IMU-only period; system recovers when inputs resume
|
||||
**Duration**: 60s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-04: Satellite Tile Server Failure
|
||||
|
||||
**Summary**: Validate the system continues operating when satellite tile server becomes unavailable, with graceful accuracy degradation.
|
||||
**Traces to**: AC-07 (resilience), solution risk: Google Maps quality
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Stop satellite-tile-server container (simulates tile unavailability)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking with satellite corrections | confidence: HIGH |
|
||||
| 2 | Stop satellite-tile-server | Satellite matching returns errors |
|
||||
| 3 | Monitor for 60s | System falls back to VO-only; confidence drops to MEDIUM after 30s |
|
||||
| 4 | Verify GPS_INPUT continues | Messages at 5-10Hz, fix_type remains 3 (VO tracking OK) |
|
||||
| 5 | Restart satellite-tile-server | Satellite matching resumes |
|
||||
| 6 | Verify confidence returns to HIGH | Position re-anchored |
|
||||
|
||||
**Pass criteria**: No crash or hang; GPS_INPUT continues; confidence degrades gracefully and recovers when tiles return
|
||||
**Duration**: 90s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-05: Corrupted Camera Frame
|
||||
|
||||
**Summary**: Validate the system handles a corrupted camera frame without crashing.
|
||||
**Traces to**: AC-06 (outlier tolerance)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay injects a truncated/corrupted JPEG between normal frames
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking for 5 frames | Baseline established |
|
||||
| 2 | Camera-replay sends corrupted JPEG | System logs warning, skips frame |
|
||||
| 3 | Camera-replay sends next normal frame | VO continues processing |
|
||||
| 4 | Verify no crash, no hang | GPS_INPUT continues at 5-10Hz |
|
||||
| 5 | Verify position accuracy on next valid frame | Error < 50m |
|
||||
|
||||
**Pass criteria**: System skips corrupted frame gracefully; no crash; next frame processed normally
|
||||
**Duration**: 15s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-06: Camera Feed Interruption (No Frames for 10s)
|
||||
|
||||
**Summary**: Validate the system survives a 10-second camera feed interruption.
|
||||
**Traces to**: AC-12 (failsafe — N seconds no estimate), AC-08 (continued operation)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay pauses for 10s (no frames delivered)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking baseline | GPS_INPUT at 5-10Hz |
|
||||
| 2 | Pause camera-replay for 10s | No new camera frames |
|
||||
| 3 | Monitor GPS_INPUT | Messages continue via IMU prediction |
|
||||
| 4 | Monitor confidence | Transitions to LOW after VO timeout |
|
||||
| 5 | Resume camera-replay | VO restarts, satellite matching resumes |
|
||||
| 6 | Verify recovery | confidence returns to HIGH within 10 frames |
|
||||
|
||||
**Pass criteria**: GPS_INPUT never stops; recovery within 10 frames after camera feed resumes
|
||||
**Duration**: 30s
|
||||
@@ -0,0 +1,90 @@
|
||||
# Resource Limit Tests
|
||||
|
||||
### NFT-RES-LIM-01: Memory Usage Under 8GB
|
||||
|
||||
**Summary**: Validate system memory usage stays below 8GB shared memory (CPU + GPU) during sustained operation.
|
||||
**Traces to**: AC-08 (memory < 8GB), RESTRICT-09 (8GB shared LPDDR5)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super (8GB shared memory)
|
||||
- Full pipeline active: cuVSLAM + satellite matching + ESKF + GPS_INPUT + FastAPI
|
||||
|
||||
**Monitoring**:
|
||||
- Total system memory (RSS + GPU allocated) via `tegrastats` or `/sys/devices/platform/host1x/*/memory`
|
||||
- Poll every 5s via GET /health (memory_mb field)
|
||||
|
||||
**Duration**: 30 minutes
|
||||
**Pass criteria**: Peak memory < 8192MB; no memory leak (growth < 50MB over 30 minutes after first 2 minutes warm-up)
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-02: GPU Thermal Envelope
|
||||
|
||||
**Summary**: Validate SoC junction temperature stays below 80°C under sustained processing load.
|
||||
**Traces to**: AC-08 (thermal), RESTRICT-10 (25W TDP, thermal throttling at 80°C)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super with active cooling
|
||||
- Full pipeline active
|
||||
|
||||
**Monitoring**:
|
||||
- SoC junction temperature via `tegrastats` or `/sys/devices/virtual/thermal/thermal_zone*/temp`
|
||||
- Poll every 10s via GET /health (gpu_temp_c field)
|
||||
|
||||
**Duration**: 30 minutes
|
||||
**Pass criteria**: SoC junction temperature < 80°C throughout; no thermal throttling events
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-03: Satellite Tile Storage
|
||||
|
||||
**Summary**: Validate pre-loaded satellite tile storage stays within calculated budget for a test mission area.
|
||||
**Traces to**: AC-19 (satellite imagery pre-loaded), RESTRICT-08 (onboard storage limited)
|
||||
|
||||
**Preconditions**:
|
||||
- Satellite tiles pre-processed for the test flight area (48.249-48.276°N, 37.340-37.386°E)
|
||||
- Zoom 18 primary + zoom 19 for ±500m along flight path
|
||||
|
||||
**Monitoring**:
|
||||
- Total tile storage size on disk
|
||||
- RAM usage for preloaded tiles (±2km buffer)
|
||||
|
||||
**Duration**: Static check
|
||||
**Pass criteria**: Total tile storage ≤ 1000MB on disk; RAM usage for preloaded tiles ≤ 200MB
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-04: Long Flight Simulation (3000 Frames)
|
||||
|
||||
**Summary**: Validate the system handles a maximum-length flight of 3000 frames without resource exhaustion.
|
||||
**Traces to**: RESTRICT-04 (up to 3000 photos per flight)
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson (or Docker with CPU-mode for functional test)
|
||||
- Camera-replay looping flight-sequence-60 to generate 3000 frames (50 loops)
|
||||
|
||||
**Monitoring**:
|
||||
- Memory usage every 60s
|
||||
- GPS_INPUT rate every 60s
|
||||
- Position accuracy sampled every 100 frames
|
||||
|
||||
**Duration**: ~71 minutes (3000 frames at 0.7fps)
|
||||
**Pass criteria**: Memory stays < 8GB; GPS_INPUT rate stays 5-10Hz; no crash or hang over full 3000-frame sequence
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-05: CUDA Stream Isolation
|
||||
|
||||
**Summary**: Validate that Stream B (satellite matching) does not block Stream A (cuVSLAM) under concurrent load.
|
||||
**Traces to**: AC-07 (< 400ms per frame), solution CUDA stream pipelining
|
||||
|
||||
**Preconditions**:
|
||||
- System running on Jetson Orin Nano Super
|
||||
- Satellite matching triggered on a keyframe while VO is processing the next frame
|
||||
|
||||
**Monitoring**:
|
||||
- VO frame-to-frame timing during satellite matching
|
||||
- Satellite matching does not extend VO latency
|
||||
|
||||
**Duration**: 30s (capture ~5 satellite matching events concurrent with VO)
|
||||
**Pass criteria**: VO per-frame time ≤ 20ms even when satellite matching is running concurrently; no frame drops
|
||||
@@ -0,0 +1,88 @@
|
||||
# Security Tests
|
||||
|
||||
### NFT-SEC-01: JWT Authentication Required on Protected Endpoints
|
||||
|
||||
**Summary**: Validate all protected endpoints reject requests without a valid JWT token.
|
||||
**Traces to**: AC-14 (security — JWT auth on API)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | POST /sessions with no Authorization header | HTTP 401 |
|
||||
| 2 | GET /sessions/{id}/stream with no Authorization header | HTTP 401 |
|
||||
| 3 | POST /sessions/{id}/anchor with no Authorization header | HTTP 401 |
|
||||
| 4 | DELETE /sessions/{id} with no Authorization header | HTTP 401 |
|
||||
| 5 | POST /objects/locate with no Authorization header | HTTP 401 |
|
||||
| 6 | GET /health with no Authorization header | HTTP 200 (health is public) |
|
||||
|
||||
**Pass criteria**: All protected endpoints return 401; /health returns 200 without auth
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-02: Expired JWT Token Rejection
|
||||
|
||||
**Summary**: Validate the system rejects expired JWT tokens.
|
||||
**Traces to**: AC-14 (security)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Generate a JWT token with exp set to 1 hour ago | Expired token |
|
||||
| 2 | POST /sessions with expired token in Authorization header | HTTP 401 |
|
||||
| 3 | POST /objects/locate with expired token | HTTP 401 |
|
||||
|
||||
**Pass criteria**: Expired tokens are rejected with 401
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-03: Invalid JWT Signature Rejection
|
||||
|
||||
**Summary**: Validate the system rejects JWT tokens signed with the wrong key.
|
||||
**Traces to**: AC-14 (security)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Generate a JWT token with a different signing key | Invalid signature token |
|
||||
| 2 | POST /sessions with invalid-signature token | HTTP 401 |
|
||||
|
||||
**Pass criteria**: Invalid-signature tokens are rejected with 401
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-04: Malformed API Request Handling
|
||||
|
||||
**Summary**: Validate the system handles malformed API payloads without crashing or leaking internal details.
|
||||
**Traces to**: AC-14 (security), AC-16 (API robustness)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | POST /objects/locate with empty body | HTTP 422 with generic error (no stack trace) |
|
||||
| 2 | POST /objects/locate with body `{"pixel_x": "not_a_number"}` | HTTP 422 with validation error |
|
||||
| 3 | POST /sessions with body exceeding 1MB | HTTP 413 or 422 (no crash) |
|
||||
| 4 | POST /sessions/{id}/anchor with body `{"lat": 999, "lon": 999}` | HTTP 422 (invalid coordinates) |
|
||||
| 5 | Verify system continues operating after all malformed requests | GET /health returns 200 |
|
||||
|
||||
**Pass criteria**: All malformed requests return 4xx errors with safe error messages (no stack traces, no internal paths); system remains operational
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-05: MAVLink Injection Resistance
|
||||
|
||||
**Summary**: Validate the system ignores unexpected or malformed MAVLink messages on the MAVLink channel.
|
||||
**Traces to**: AC-15 (ground station commands), solution security analysis
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Send unexpected MAVLink message types to the system's MAVLink port | System ignores (no crash, no state corruption) |
|
||||
| 2 | Send malformed COMMAND_LONG with invalid lat/lon in re-localization hint | System rejects or ignores invalid coordinates |
|
||||
| 3 | Verify GPS_INPUT output continues normally | No disruption |
|
||||
|
||||
**Pass criteria**: System ignores unexpected messages; continues normal operation; does not process invalid re-localization coordinates
|
||||
@@ -0,0 +1,95 @@
|
||||
# Test Data Management
|
||||
|
||||
## Seed Data Sets
|
||||
|
||||
| Data Set | Description | Used by Tests | How Loaded | Cleanup |
|
||||
|----------|-------------|---------------|-----------|---------|
|
||||
| flight-sequence-60 | 60 aerial images (AD000001-060.jpg) with ground truth GPS from coordinates.csv, captured at ~1 photo per 2-3s from a fixed-wing UAV at 400m altitude | FT-P-01 through FT-P-06, FT-N-01, FT-N-02, NFT-PERF-01, NFT-RES-01 | Volume mount to camera-replay service; coordinates.csv loaded by e2e-consumer for ground truth comparison | Container restart between test groups |
|
||||
| camera-params | Camera parameters: ADTi Surveyor Lite 26S v2, 26MP (6252x4168), 25mm focal length, 23.5mm sensor width | All position accuracy tests, object localization tests | Volume mount; read by gps-denied-system at startup | N/A (read-only) |
|
||||
| satellite-tiles-test | Pre-processed satellite tiles (zoom 18) covering test flight area: 48.249-48.276°N, 37.340-37.386°E | All tests requiring satellite matching | Volume mount to satellite-tile-server and gps-denied-system | Container restart |
|
||||
| ardupilot-params | ArduPilot SITL parameters: GPS1_TYPE=14, GPS_RATE=5, EK3_SRC1_POSXY=1, EK3_SRC1_VELXY=1, fixed-wing frame | All tests requiring flight controller interaction | Baked into ardupilot-sitl Docker image | Container restart |
|
||||
| imu-replay-data | Synthetic IMU data (accelerometer + gyroscope at 200Hz) matching the flight-sequence-60 trajectory. Generated from coordinates.csv ground truth positions by computing velocities/accelerations between frames, interpolating to 200Hz, adding noise consistent with ICM-42688-P datasheet (gyro: 3.0e-3 °/s/√Hz, accel: 70 µg/√Hz). Stored as `input_data/imu_synthetic_200hz.csv` with columns: timestamp_us, accel_x, accel_y, accel_z, gyro_x, gyro_y, gyro_z. Alternative: SITL ArduPilot flies a waypoint mission following the coordinates.csv trajectory and internally generates physically consistent IMU data at 200Hz. | FT-P-01 through FT-P-06, NFT-PERF-01, NFT-RES-01 through NFT-RES-05 | Primary: SITL ArduPilot flies the trajectory and generates IMU internally via MAVLink. Fallback: pre-generated CSV replayed via MAVLink injector | Container restart |
|
||||
| invalid-inputs | Malformed images (truncated JPEG, wrong resolution), invalid API payloads, corrupted IMU streams | FT-N-03 through FT-N-06, NFT-SEC-01 through NFT-SEC-04 | Volume mount to e2e-consumer; injected via API calls | Container restart |
|
||||
|
||||
## Data Isolation Strategy
|
||||
|
||||
Each test group runs against a fresh container restart of the gps-denied-system and ardupilot-sitl services. The camera-replay service is restarted and configured per test group (different frame subsets, different replay speeds, or different fault injection modes). MAVLink capture logs are isolated per test run via timestamped directories.
|
||||
|
||||
## Input Data Mapping
|
||||
|
||||
| Input Data File | Source Location | Description | Covers Scenarios |
|
||||
|-----------------|----------------|-------------|-----------------|
|
||||
| AD000001-060.jpg | `_docs/00_problem/input_data/` | 60 aerial images (6252x4168, 26MP) from a fixed-wing UAV at 400m altitude. Images taken at ~1 photo/2-3s (wider spacing than real 0.7fps but usable for functional tests) | FT-P-01 to FT-P-06, FT-N-01, FT-N-02, NFT-PERF-01, NFT-RES-01, NFT-RES-LIM-01 |
|
||||
| AD000001_gmaps.png, AD000002_gmaps.png | `_docs/00_problem/input_data/` | Google Maps satellite reference images for frames 1-2 (used as sample satellite tile data for satellite matching validation) | FT-P-01, NFT-RES-02, NFT-RES-04 |
|
||||
| coordinates.csv | `_docs/00_problem/input_data/coordinates.csv` | Ground truth GPS (lat, lon) for each of the 60 images | FT-P-01, FT-P-02, FT-P-03, FT-P-04 (comparison baseline) |
|
||||
| data_parameters.md | `_docs/00_problem/input_data/data_parameters.md` | Camera specs: 400m altitude, ADTi Surveyor Lite 26S v2, 26MP, 25mm focal length, 23.5mm sensor | Test environment configuration |
|
||||
| position_accuracy.csv | `_docs/00_problem/input_data/expected_results/position_accuracy.csv` | Per-frame ground truth with acceptance thresholds | FT-P-01, FT-P-02 (expected result comparison) |
|
||||
| imu_synthetic_200hz.csv | `_docs/00_problem/input_data/` (TO BE GENERATED) | Synthetic 200Hz IMU data (accel + gyro) derived from coordinates.csv trajectory. Matches ICM-42688-P noise characteristics. Required for ESKF sensor fusion testing outside SITL. | FT-P-01 to FT-P-06, NFT-PERF-01, NFT-RES-01 to NFT-RES-06 |
|
||||
|
||||
### IMU Data Generation
|
||||
|
||||
No real IMU recordings exist for the 60-image flight sequence. Two approaches for providing IMU data during tests:
|
||||
|
||||
**Approach A — SITL-generated (primary)**: ArduPilot SITL flies a waypoint mission following the coordinates.csv trajectory. SITL's internal physics engine generates physically consistent IMU data at 200Hz, delivered via MAVLink to the GPS-denied system. This is the most realistic approach and requires no pre-generated files.
|
||||
|
||||
**Approach B — Synthetic CSV (fallback/replay)**: Generate `imu_synthetic_200hz.csv` offline from coordinates.csv:
|
||||
1. Compute inter-frame velocities from GPS positions and timestamps
|
||||
2. Interpolate position/velocity to 200Hz using cubic splines
|
||||
3. Compute accelerations (body frame) accounting for gravity + flight dynamics
|
||||
4. Add sensor noise matching ICM-42688-P specs (gyro: 3.0e-3 °/s/√Hz, accel: 70 µg/√Hz)
|
||||
5. Add bias random walks (gyro: 5.0e-5 °/s²/√Hz, accel: 2.0e-3 m/s³/√Hz)
|
||||
6. Replay via MAVLink injector service at 200Hz
|
||||
|
||||
Approach A is recommended for integration tests. Approach B is useful for deterministic unit-level ESKF tests where reproducible IMU streams are needed.
|
||||
|
||||
### Satellite Tile Data
|
||||
|
||||
Only 2 Google Maps screenshots exist (AD000001_gmaps.png, AD000002_gmaps.png). Full satellite tile coverage for the test area must be prepared:
|
||||
1. Download Google Maps tiles at zoom 18 for the bounding box: 48.249-48.276°N, 37.340-37.386°E
|
||||
2. Store as 256x256 JPEG tiles with geohash-based naming
|
||||
3. Load into satellite-tile-server Docker service
|
||||
4. Estimated: ~50-100 tiles for the test area (~1-2MB total)
|
||||
|
||||
## Expected Results Mapping
|
||||
|
||||
| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Expected Result Source |
|
||||
|-----------------|------------|-----------------|-------------------|-----------|----------------------|
|
||||
| FT-P-01 | flight-sequence-60 (60 frames) | ≥80% of frames within 50m of ground truth | percentage | ≥80% | `expected_results/position_accuracy.csv` |
|
||||
| FT-P-02 | flight-sequence-60 (60 frames) | ≥60% of frames within 20m of ground truth | percentage | ≥60% | `expected_results/position_accuracy.csv` |
|
||||
| FT-P-03 | flight-sequence-60 (60 frames) | No single frame exceeds 100m error | threshold_max | ≤100m | `expected_results/position_accuracy.csv` |
|
||||
| FT-P-04 | flight-sequence-60 (selected satellite anchor pairs) | VO drift between satellite anchors <100m | threshold_max | ≤100m | inline |
|
||||
| FT-P-05 | Single frame + satellite match | GPS_INPUT: fix_type=3, horiz_accuracy 5-20m, satellites_visible=10 | exact + range | fix_type==3, accuracy∈[1,50] | `expected_results/results_report.md` #5 |
|
||||
| FT-P-06 | flight-sequence-60 | ≥57 of 60 frames registered (≥95%) | percentage | ≥95% | inline |
|
||||
| FT-P-07 | Normal operation, satellite match <30s | Confidence tier: HIGH | exact | N/A | inline |
|
||||
| FT-P-08 | VO tracking, no satellite >30s | Confidence tier: MEDIUM | exact | N/A | inline |
|
||||
| FT-P-09 | GPS_INPUT stream | Messages at 5-10Hz | range | [5,10] Hz | inline |
|
||||
| FT-P-10 | POST /objects/locate (known pixel, gimbal, zoom, UAV position) | lat/lon within accuracy_m of ground truth | numeric_tolerance | within accuracy_m | `expected_results/results_report.md` #27 |
|
||||
| FT-P-11 | Known GPS → NED → pixel → GPS | Round-trip error <0.1m | threshold_max | ≤0.1m | inline |
|
||||
| FT-P-12 | System boot + GLOBAL_POSITION_INT | GPS_INPUT output within 60s | threshold_max | ≤60s | inline |
|
||||
| FT-N-01 | Frames 32-43 (direction change area) | System continues producing estimates | threshold_min | ≥1 output per frame | inline |
|
||||
| FT-N-02 | Simulated 350m gap between frames | System handles outlier, next valid frame <100m error | threshold_max | ≤100m | inline |
|
||||
| FT-N-03 | POST /objects/locate with invalid pixels | HTTP 422 | exact | status==422 | inline |
|
||||
| FT-N-04 | Unauthenticated request to /sessions | HTTP 401 | exact | status==401 | inline |
|
||||
| FT-N-05 | VO lost + 3 satellite failures | fix_type=0, horiz_accuracy=999.0, RELOC_REQ sent | exact + regex | fix_type==0, matches `RELOC_REQ:.*` | inline |
|
||||
| FT-N-06 | VO lost + IMU-only | fix_type=2, horiz_accuracy≥50m growing | exact + threshold_min | fix_type==2, accuracy≥50 | inline |
|
||||
|
||||
## External Dependency Mocks
|
||||
|
||||
| External Service | Mock/Stub | How Provided | Behavior |
|
||||
|-----------------|-----------|-------------|----------|
|
||||
| Flight Controller (ArduPilot) | ArduPilot SITL | Docker service (ardupilot-sitl) | Full MAVLink protocol: heartbeat, GLOBAL_POSITION_INT, IMU data at 200Hz, accepts GPS_INPUT, responds to COMMAND_LONG |
|
||||
| Camera hardware (ADTI 20L V1) | Frame replay server | Docker service (camera-replay) | Serves frames from input_data/ at configurable rate (0.7fps default); supports fault injection (frame drop, delayed frame, corrupted JPEG) |
|
||||
| Satellite imagery (Google Maps) | Static tile server | Docker service (satellite-tile-server) | Serves pre-cached tiles via HTTP; supports fault injection (404 for missing tiles, slow response) |
|
||||
| Ground station | MAVLink inspector | Docker service (mavlink-inspector) | Captures STATUSTEXT and NAMED_VALUE_FLOAT; can inject COMMAND_LONG (operator re-localization hint) |
|
||||
| GPU (CUDA/TensorRT) | CPU fallback or Jetson hardware | Conditional | Docker: CPU-mode stubs for TRT inference (slower but functionally equivalent). Jetson: real GPU |
|
||||
|
||||
## Data Validation Rules
|
||||
|
||||
| Data Type | Validation | Invalid Examples | Expected System Behavior |
|
||||
|-----------|-----------|-----------------|------------------------|
|
||||
| Camera frame (JPEG) | Valid JPEG, resolution ≥ 1280x720 | Truncated JPEG, 0-byte file, BMP format | Log warning, skip frame, continue with IMU-only ESKF prediction |
|
||||
| GPS coordinate | lat ∈ [-90, 90], lon ∈ [-180, 180] | lat=999, lon=NaN | Reject, use last valid position |
|
||||
| IMU data | Acceleration ∈ [-160, 160] m/s², gyro ∈ [-35, 35] rad/s | All zeros, NaN values, extreme spikes | Filter outliers via ESKF process noise, log warning |
|
||||
| Satellite tile | Valid JPEG/PNG, 256x256 px | Missing tile (404), corrupted image | Skip tile, expand search radius, fall back to wider area |
|
||||
| API request body | Valid JSON, required fields present | Missing pixel_x, non-numeric zoom_factor | HTTP 422 with validation error details |
|
||||
| JWT token | Valid signature, not expired | Expired token, invalid signature, missing token | HTTP 401 Unauthorized |
|
||||
@@ -0,0 +1,69 @@
|
||||
# Traceability Matrix
|
||||
|
||||
## Acceptance Criteria Coverage
|
||||
|
||||
| AC ID | Acceptance Criterion | Test IDs | Coverage |
|
||||
|-------|---------------------|----------|----------|
|
||||
| AC-01 | 80% of frames within 50m of real GPS | FT-P-01, FT-P-03 | Covered |
|
||||
| AC-02 | 60% of frames within 20m of real GPS | FT-P-02 | Covered |
|
||||
| AC-03 | Cumulative VO drift between satellite anchors < 100m | FT-P-04 | Covered |
|
||||
| AC-04 | Confidence score per position estimate (high/low) | FT-P-07, FT-P-08, FT-N-06 | Covered |
|
||||
| AC-05 | Image registration rate > 95% | FT-P-06 | Covered |
|
||||
| AC-06 | System handles 350m outlier between consecutive photos | FT-N-02 | Covered |
|
||||
| AC-07 | System handles sharp turns with <5% overlap, <200m drift, <70° angle via satellite re-localization | FT-N-01, NFT-RES-02 | Covered |
|
||||
| AC-08 | System handles disconnected route segments (core feature) | FT-N-01, NFT-RES-02, NFT-RES-03 | Covered |
|
||||
| AC-09 | 3 consecutive failures → re-localization request to ground station | FT-N-05, FT-N-07 | Covered |
|
||||
| AC-10 | < 400ms end-to-end per frame | NFT-PERF-01 | Covered |
|
||||
| AC-11 | Memory < 8GB shared | NFT-RES-LIM-01, NFT-RES-LIM-04 | Covered |
|
||||
| AC-12 | GPS_INPUT via MAVLink (MAVSDK) to flight controller | FT-P-05, FT-P-09 | Covered |
|
||||
| AC-13 | Frame-by-frame streaming, no batch/delay | FT-P-09, NFT-PERF-02 | Covered |
|
||||
| AC-14 | System initializes from last known GPS position | FT-P-12 | Covered |
|
||||
| AC-15 | Complete failure for N seconds → FC falls back to IMU, system logs | NFT-RES-03, FT-N-06 | Covered |
|
||||
| AC-16 | Mid-flight reboot → re-initialize from FC's IMU-extrapolated position | NFT-RES-01 | Covered |
|
||||
| AC-17 | Position + confidence streamed to ground station via telemetry | FT-P-13, FT-P-14 | Covered |
|
||||
| AC-18 | Ground station can send commands (re-localization hint) | FT-N-07 | Covered |
|
||||
| AC-19 | Output coordinates in WGS84 | FT-P-05, FT-P-11 | Covered |
|
||||
| AC-20 | Other onboard AI can request object GPS coordinates | FT-P-10 | Covered |
|
||||
| AC-21 | Object coordinates via trigonometric calculation (gimbal, zoom, altitude) | FT-P-10, FT-P-11 | Covered |
|
||||
| AC-22 | Object accuracy consistent with frame-center accuracy | FT-P-10 | Covered |
|
||||
| AC-23 | Satellite imagery ≥ 0.5 m/pixel | NFT-RES-LIM-03 | Covered (tile storage validation uses zoom 18 = 0.6m/px) |
|
||||
| AC-24 | Satellite imagery pre-loaded before flight | NFT-RES-LIM-03 | Covered |
|
||||
| AC-25 | MRE < 1.0 pixels | — | NOT COVERED — requires cuVSLAM internal reprojection metric; not observable via black-box interfaces. Covered at component test level (Step 5). |
|
||||
| AC-26 | Satellite imagery < 2 years old | — | NOT COVERED — operational procurement constraint; not runtime-testable. Validated during offline tile preparation. |
|
||||
|
||||
## Restrictions Coverage
|
||||
|
||||
| Restriction ID | Restriction | Test IDs | Coverage |
|
||||
|---------------|-------------|----------|----------|
|
||||
| RESTRICT-01 | Fixed-wing UAV only | FT-P-01 through FT-P-06 (test data from fixed-wing flight) | Covered |
|
||||
| RESTRICT-02 | Camera pointing downwards, not autostabilized | FT-P-10, FT-P-11 (coordinate transforms account for non-stabilized mount) | Covered |
|
||||
| RESTRICT-03 | Eastern/southern Ukraine operational area | FT-P-01 (test coordinates at 48.25-48.28°N, 37.34-37.39°E) | Covered |
|
||||
| RESTRICT-04 | Altitude ≤ 1km, terrain height negligible | FT-P-01, FT-P-10 (test data at 400m altitude, flat terrain assumed) | Covered |
|
||||
| RESTRICT-05 | Mostly sunny weather | — | NOT COVERED — environmental condition; cannot be tested in Docker. Mitigated by feature-based matching robustness. |
|
||||
| RESTRICT-06 | Sharp turns (no common points possible, exceptional) | FT-N-01, FT-N-02, NFT-RES-02 | Covered |
|
||||
| RESTRICT-07 | Up to 3000 photos per flight | NFT-RES-LIM-04 | Covered |
|
||||
| RESTRICT-08 | Two cameras: navigation (fixed nadir) + AI (configurable) | FT-P-10 (object localization uses AI camera angles) | Covered |
|
||||
| RESTRICT-09 | Navigation camera: FullHD to 6252x4168, known parameters | FT-P-01 (test data at 6252x4168) | Covered |
|
||||
| RESTRICT-10 | Jetson Orin Nano Super: 67 TOPS, 8GB, 25W TDP | NFT-RES-LIM-01, NFT-RES-LIM-02, NFT-PERF-01 | Covered |
|
||||
| RESTRICT-11 | Thermal throttling at sustained GPU load | NFT-RES-LIM-02 | Covered |
|
||||
| RESTRICT-12 | IMU data via flight controller (MAVLink) | FT-P-05, FT-P-12 (SITL provides IMU via MAVLink) | Covered |
|
||||
| RESTRICT-13 | GPS_INPUT message to flight controller | FT-P-05, FT-P-09 | Covered |
|
||||
| RESTRICT-14 | Telemetry link bandwidth-limited | FT-P-13 (1Hz telemetry rate is bandwidth-appropriate) | Covered |
|
||||
| RESTRICT-15 | Google Maps satellite (potentially outdated) | NFT-RES-04 (tile server failure resilience) | Covered |
|
||||
| RESTRICT-16 | Onboard storage limited for satellite tiles | NFT-RES-LIM-03 | Covered |
|
||||
|
||||
## Coverage Summary
|
||||
|
||||
| Category | Total Items | Covered | Not Covered | Coverage % |
|
||||
|----------|-----------|---------|-------------|-----------|
|
||||
| Acceptance Criteria | 26 | 24 | 2 | 92% |
|
||||
| Restrictions | 16 | 15 | 1 | 94% |
|
||||
| **Total** | **42** | **39** | **3** | **93%** |
|
||||
|
||||
## Uncovered Items Analysis
|
||||
|
||||
| Item | Reason Not Covered | Risk | Mitigation |
|
||||
|------|-------------------|------|-----------|
|
||||
| AC-25 (MRE < 1.0 pixels) | cuVSLAM reprojection error is an internal metric not exposed via public API or MAVLink; cannot be observed at black-box level | LOW — MRE is a proxy for VO quality; position accuracy tests (FT-P-01/02) validate the end result | Covered at component test level (Step 5) with cuVSLAM-specific unit tests |
|
||||
| AC-26 (Satellite imagery < 2 years old) | Operational procurement constraint validated during offline tile preparation, not at runtime | LOW — outdated imagery degrades matching but is caught by satellite matching accuracy tests | Validated during pre-flight tile download workflow; documented in operational procedures |
|
||||
| RESTRICT-05 (Sunny weather) | Environmental condition that cannot be reproduced in Docker or controlled test environment | LOW — system uses feature-based matching (LiteSAM/XFeat) robust to lighting variation; not a software test | Validated during real-world flight tests; feature-based matching provides robustness |
|
||||
Reference in New Issue
Block a user