gps-denied-onboard/_docs/02_document/tests/environment.md

Test Environment

Overview

System under test: GPS-Denied Visual Navigation System — a real-time position estimation service running on Jetson Orin Nano Super. Public interfaces: FastAPI REST/SSE endpoints (port 8000), MAVLink GPS_INPUT messages over serial/UDP, MAVLink telemetry messages.

Consumer app purpose: Standalone Python test runner (pytest) that exercises the GPS-denied system through its public interfaces (HTTP API, MAVLink message inspection, SSE stream consumption) without access to internal modules, ESKF state, or GPU buffers.

Test Modes

This is embedded robotics software targeting Jetson Orin Nano Super. A pure Docker environment cannot exercise GPU-dependent paths (TRT inference, cuVSLAM, CUDA streams). The test environment supports two modes:

Mode 1 — Docker SITL (CI/dev): Full system in Docker containers with ArduPilot SITL providing MAVLink + IMU at 200Hz. Camera images replayed from input_data/. Satellite tiles served from a mock HTTP server. GPS-denied system runs in CPU-mode with stubbed TRT/cuVSLAM inference (functionally equivalent but slower). Tests all integration paths, API, MAVLink, resilience, and security.

Mode 2 — Jetson Hardware (nightly/pre-deploy): GPS-denied system runs natively on Jetson Orin Nano Super with real CUDA/TRT/cuVSLAM. ArduPilot SITL runs on the Jetson or a companion x86 host, connected via UART or UDP. Camera frames injected via USB camera emulator or replay service. Tests real-time performance, GPU memory, thermal, TRT correctness, and CUDA stream isolation.

Docker Environment (Mode 1)

Services

Service	Image / Build	Purpose	Ports
gps-denied-system	./Dockerfile (build context: project root)	GPS-denied navigation system in CPU-mode (TRT stubs, cuVSLAM stub returning synthetic VO poses derived from ground truth trajectory)	8000 (FastAPI), 14550/udp (MAVLink)
ardupilot-sitl	ardupilot/sitl:plane-4.5 (fixed-wing)	ArduPilot SITL: flies waypoint mission following coordinates.csv trajectory, generates IMU at 200Hz via MAVLink, GPS_TYPE=14 accepts GPS_INPUT, provides GLOBAL_POSITION_INT at startup	5760 (MAVLink TCP), 14551/udp
camera-replay	./tests/docker/camera-replay/Dockerfile	Replays AD000001-060.jpg from input_data/ at configurable FPS (default 0.7fps) with timestamps synchronized to SITL clock. Supports fault injection: frame skip, corrupted JPEG, pause.	8001 (frame server)
satellite-tile-server	./tests/docker/tile-server/Dockerfile	HTTP server for pre-cached satellite tiles (zoom 18, ~50-100 tiles covering test area). Supports fault injection: 404 for specific tiles, 503 for full outage, slow responses.	8002
mavlink-inspector	./tests/docker/mavlink-inspector/Dockerfile	Passively captures all MAVLink traffic (GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT, COMMAND_LONG) for post-test assertion. Can inject operator re-localization hints.	14552/udp
e2e-consumer	./tests/e2e/Dockerfile	Black-box test runner (pytest). Communicates only via HTTP API + MAVLink inspector.	—

Networks

Network	Services	Purpose
e2e-net	all	Isolated test network; MAVLink UDP multicast between gps-denied-system, ardupilot-sitl, mavlink-inspector

Volumes

Volume	Mounted to	Purpose
input-data	camera-replay:/data, e2e-consumer:/test-data	Camera frames (AD000001-060.jpg), coordinates.csv, data_parameters.md, gmaps reference images
satellite-tiles	satellite-tile-server:/tiles, gps-denied-system:/tiles	Pre-processed satellite tiles for test area (zoom 18, 48.249-48.276°N, 37.340-37.386°E)
sitl-mission	ardupilot-sitl:/mission	Waypoint mission file derived from coordinates.csv (SITL flies this trajectory, generating physically consistent 200Hz IMU data)
test-results	e2e-consumer:/results	Test result CSV output
mavlink-capture	mavlink-inspector:/capture	Recorded MAVLink messages for post-test assertions

IMU Data Flow

ArduPilot SITL is the primary source of IMU data. It flies a waypoint mission derived from coordinates.csv and internally generates physically consistent accelerometer + gyroscope readings at 200Hz, delivered to the GPS-denied system via MAVLink (RAW_IMU, SCALED_IMU2). This eliminates the need for pre-recorded IMU data files and ensures IMU/trajectory consistency.

coordinates.csv → mission_generator script → ArduPilot waypoint file
                                              ↓
                          ArduPilot SITL flies trajectory
                                              ↓
                          IMU @ 200Hz + heartbeat + GLOBAL_POSITION_INT
                                              ↓ (MAVLink UDP)
                          gps-denied-system receives IMU for ESKF

docker-compose structure

services:
  ardupilot-sitl:
    # ArduPilot SITL fixed-wing, outputs IMU at 200Hz via MAVLink
    # GPS_TYPE=14 (MAVLink), pre-configured for GPS_INPUT acceptance
  satellite-tile-server:
    # HTTP tile server with tiles for test area (48.249-48.276°N, 37.340-37.386°E)
  camera-replay:
    # Replays AD000001-060.jpg at 0.7fps, serves via HTTP or shared volume
    depends_on:
      - satellite-tile-server
  gps-denied-system:
    # The system under test
    depends_on:
      - ardupilot-sitl
      - satellite-tile-server
      - camera-replay
  mavlink-inspector:
    # Captures GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT messages
    depends_on:
      - ardupilot-sitl
  e2e-consumer:
    # pytest runner — executes after system reaches steady state
    depends_on:
      - gps-denied-system
      - mavlink-inspector

Consumer Application

Tech stack: Python 3.11, pytest, httpx (HTTP client), pymavlink (MAVLink inspection), sseclient-py (SSE stream) Entry point: pytest tests/e2e/ --tb=short --csv=results/report.csv

Communication with system under test

Interface	Protocol	Endpoint / Topic	Authentication
Position API	HTTP REST	http://gps-denied-system:8000/sessions	JWT token
Position stream	HTTP SSE	http://gps-denied-system:8000/sessions/{id}/stream	JWT token
Object localization	HTTP REST	http://gps-denied-system:8000/objects/locate	JWT token
Health check	HTTP REST	http://gps-denied-system:8000/health	None
GPS_INPUT inspection	MAVLink UDP	mavlink-inspector:14552 (recorded messages)	None
Telemetry inspection	MAVLink UDP	mavlink-inspector:14552 (NAMED_VALUE_FLOAT, STATUSTEXT)	None

What the consumer does NOT have access to

• No direct access to ESKF internal state, covariance matrices, or error vectors
• No direct access to cuVSLAM tracking state or feature maps
• No direct access to GPU memory, CUDA streams, or TRT engine internals
• No direct access to the system's file system or configuration files
• No direct database or state store access

Jetson Hardware Environment (Mode 2)

Tests tagged @pytest.mark.jetson require actual Jetson Orin Nano Super hardware. These run natively (no Docker for the GPS-denied system) to exercise real GPU paths.

Hardware setup:

• Jetson Orin Nano Super (JetPack 6.2, CUDA 12.x, TensorRT 10.3)
• ArduPilot SITL on same Jetson (or x86 companion connected via UART/UDP)
• Camera frames injected via: USB camera emulator (v4l2loopback feeding frames from input_data/) or HTTP replay service
• Satellite tiles on local SSD (same path as production deployment)
• Active cooling attached (required for sustained load tests)

Tests in this mode:

• NFT-PERF-01 through NFT-PERF-06 (real GPU latency, throughput)
• NFT-RES-LIM-01 (GPU+CPU shared memory monitoring via tegrastats)
• NFT-RES-LIM-02 (thermal monitoring via thermal_zone sysfs)
• NFT-RES-LIM-05 (CUDA stream isolation with real concurrent GPU work)
• TRT engine build and inference correctness (expected_results #42-44)

Jetson CI runner: Self-hosted GitHub Actions runner on a dedicated Jetson Orin Nano Super, triggered for nightly builds and pre-deploy gates.

CI/CD Integration

When to run: On every PR to dev, nightly full suite, before production deploy Pipeline stages:

1. Unit tests (no Docker, no hardware) — on every commit
2. Docker blackbox tests (SITL + CPU mode) — on PR merge to dev
3. Hardware tests (Jetson runner) — nightly + pre-deploy

Gate behavior: Docker tests block merge; hardware tests are advisory (nightly) or blocking (pre-deploy) Timeout: Docker suite: 15 minutes; Hardware suite: 30 minutes

Reporting

Format: CSV Columns: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message (if FAIL) Output path: ./tests/e2e-results/report.csv