gps-denied-onboard/_docs/02_document/tests/environment.md

# Test Environment

## Overview

**System under test**: GPS-Denied Visual Navigation System — a real-time position estimation service running on Jetson Orin Nano Super. Public interfaces: FastAPI REST/SSE endpoints (port 8000), MAVLink GPS_INPUT messages over serial/UDP, MAVLink telemetry messages.

**Consumer app purpose**: Standalone Python test runner (pytest) that exercises the GPS-denied system through its public interfaces (HTTP API, MAVLink message inspection, SSE stream consumption) without access to internal modules, ESKF state, or GPU buffers.

## Test Modes

This is embedded robotics software targeting Jetson Orin Nano Super. A pure Docker environment cannot exercise GPU-dependent paths (TRT inference, cuVSLAM, CUDA streams). The test environment supports two modes:

**Mode 1 — Docker SITL (CI/dev)**: Full system in Docker containers with ArduPilot SITL providing MAVLink + IMU at 200Hz. Camera images replayed from input_data/. Satellite tiles served from a mock HTTP server. GPS-denied system runs in CPU-mode with stubbed TRT/cuVSLAM inference (functionally equivalent but slower). Tests all integration paths, API, MAVLink, resilience, and security.

**Mode 2 — Jetson Hardware (nightly/pre-deploy)**: GPS-denied system runs natively on Jetson Orin Nano Super with real CUDA/TRT/cuVSLAM. ArduPilot SITL runs on the Jetson or a companion x86 host, connected via UART or UDP. Camera frames injected via USB camera emulator or replay service. Tests real-time performance, GPU memory, thermal, TRT correctness, and CUDA stream isolation.

## Docker Environment (Mode 1)

### Services

| Service | Image / Build | Purpose | Ports |
|---------|--------------|---------|-------|
| gps-denied-system | `./Dockerfile` (build context: project root) | GPS-denied navigation system in CPU-mode (TRT stubs, cuVSLAM stub returning synthetic VO poses derived from ground truth trajectory) | 8000 (FastAPI), 14550/udp (MAVLink) |
| ardupilot-sitl | `ardupilot/sitl:plane-4.5` (fixed-wing) | ArduPilot SITL: flies waypoint mission following coordinates.csv trajectory, generates IMU at 200Hz via MAVLink, GPS_TYPE=14 accepts GPS_INPUT, provides GLOBAL_POSITION_INT at startup | 5760 (MAVLink TCP), 14551/udp |
| camera-replay | `./tests/docker/camera-replay/Dockerfile` | Replays AD000001-060.jpg from input_data/ at configurable FPS (default 0.7fps) with timestamps synchronized to SITL clock. Supports fault injection: frame skip, corrupted JPEG, pause. | 8001 (frame server) |
| satellite-tile-server | `./tests/docker/tile-server/Dockerfile` | HTTP server for pre-cached satellite tiles (zoom 18, ~50-100 tiles covering test area). Supports fault injection: 404 for specific tiles, 503 for full outage, slow responses. | 8002 |
| mavlink-inspector | `./tests/docker/mavlink-inspector/Dockerfile` | Passively captures all MAVLink traffic (GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT, COMMAND_LONG) for post-test assertion. Can inject operator re-localization hints. | 14552/udp |
| e2e-consumer | `./tests/e2e/Dockerfile` | Black-box test runner (pytest). Communicates only via HTTP API + MAVLink inspector. | — |

### Networks

| Network | Services | Purpose |
|---------|----------|---------|
| e2e-net | all | Isolated test network; MAVLink UDP multicast between gps-denied-system, ardupilot-sitl, mavlink-inspector |

### Volumes

| Volume | Mounted to | Purpose |
|--------|-----------|---------|
| input-data | camera-replay:/data, e2e-consumer:/test-data | Camera frames (AD000001-060.jpg), coordinates.csv, data_parameters.md, gmaps reference images |
| satellite-tiles | satellite-tile-server:/tiles, gps-denied-system:/tiles | Pre-processed satellite tiles for test area (zoom 18, 48.249-48.276°N, 37.340-37.386°E) |
| sitl-mission | ardupilot-sitl:/mission | Waypoint mission file derived from coordinates.csv (SITL flies this trajectory, generating physically consistent 200Hz IMU data) |
| test-results | e2e-consumer:/results | Test result CSV output |
| mavlink-capture | mavlink-inspector:/capture | Recorded MAVLink messages for post-test assertions |

### IMU Data Flow

ArduPilot SITL is the primary source of IMU data. It flies a waypoint mission derived from coordinates.csv and internally generates physically consistent accelerometer + gyroscope readings at 200Hz, delivered to the GPS-denied system via MAVLink (RAW_IMU, SCALED_IMU2). This eliminates the need for pre-recorded IMU data files and ensures IMU/trajectory consistency.

```
coordinates.csv → mission_generator script → ArduPilot waypoint file
                                              ↓
                          ArduPilot SITL flies trajectory
                                              ↓
                          IMU @ 200Hz + heartbeat + GLOBAL_POSITION_INT
                                              ↓ (MAVLink UDP)
                          gps-denied-system receives IMU for ESKF
```

### docker-compose structure

```yaml
services:
  ardupilot-sitl:
    # ArduPilot SITL fixed-wing, outputs IMU at 200Hz via MAVLink
    # GPS_TYPE=14 (MAVLink), pre-configured for GPS_INPUT acceptance
  satellite-tile-server:
    # HTTP tile server with tiles for test area (48.249-48.276°N, 37.340-37.386°E)
  camera-replay:
    # Replays AD000001-060.jpg at 0.7fps, serves via HTTP or shared volume
    depends_on:
      - satellite-tile-server
  gps-denied-system:
    # The system under test
    depends_on:
      - ardupilot-sitl
      - satellite-tile-server
      - camera-replay
  mavlink-inspector:
    # Captures GPS_INPUT, NAMED_VALUE_FLOAT, STATUSTEXT messages
    depends_on:
      - ardupilot-sitl
  e2e-consumer:
    # pytest runner — executes after system reaches steady state
    depends_on:
      - gps-denied-system
      - mavlink-inspector
```

## Consumer Application

**Tech stack**: Python 3.11, pytest, httpx (HTTP client), pymavlink (MAVLink inspection), sseclient-py (SSE stream)
**Entry point**: `pytest tests/e2e/ --tb=short --csv=results/report.csv`

### Communication with system under test

| Interface | Protocol | Endpoint / Topic | Authentication |
|-----------|----------|-----------------|----------------|
| Position API | HTTP REST | http://gps-denied-system:8000/sessions | JWT token |
| Position stream | HTTP SSE | http://gps-denied-system:8000/sessions/{id}/stream | JWT token |
| Object localization | HTTP REST | http://gps-denied-system:8000/objects/locate | JWT token |
| Health check | HTTP REST | http://gps-denied-system:8000/health | None |
| GPS_INPUT inspection | MAVLink UDP | mavlink-inspector:14552 (recorded messages) | None |
| Telemetry inspection | MAVLink UDP | mavlink-inspector:14552 (NAMED_VALUE_FLOAT, STATUSTEXT) | None |

### What the consumer does NOT have access to

- No direct access to ESKF internal state, covariance matrices, or error vectors
- No direct access to cuVSLAM tracking state or feature maps
- No direct access to GPU memory, CUDA streams, or TRT engine internals
- No direct access to the system's file system or configuration files
- No direct database or state store access

## Jetson Hardware Environment (Mode 2)

Tests tagged `@pytest.mark.jetson` require actual Jetson Orin Nano Super hardware. These run natively (no Docker for the GPS-denied system) to exercise real GPU paths.

**Hardware setup**:
- Jetson Orin Nano Super (JetPack 6.2, CUDA 12.x, TensorRT 10.3)
- ArduPilot SITL on same Jetson (or x86 companion connected via UART/UDP)
- Camera frames injected via: USB camera emulator (v4l2loopback feeding frames from input_data/) or HTTP replay service
- Satellite tiles on local SSD (same path as production deployment)
- Active cooling attached (required for sustained load tests)

**Tests in this mode**:
- NFT-PERF-01 through NFT-PERF-06 (real GPU latency, throughput)
- NFT-RES-LIM-01 (GPU+CPU shared memory monitoring via tegrastats)
- NFT-RES-LIM-02 (thermal monitoring via thermal_zone sysfs)
- NFT-RES-LIM-05 (CUDA stream isolation with real concurrent GPU work)
- TRT engine build and inference correctness (expected_results #42-44)

**Jetson CI runner**: Self-hosted GitHub Actions runner on a dedicated Jetson Orin Nano Super, triggered for nightly builds and pre-deploy gates.

## CI/CD Integration

**When to run**: On every PR to `dev`, nightly full suite, before production deploy
**Pipeline stages**:
1. Unit tests (no Docker, no hardware) — on every commit
2. Docker blackbox tests (SITL + CPU mode) — on PR merge to dev
3. Hardware tests (Jetson runner) — nightly + pre-deploy

**Gate behavior**: Docker tests block merge; hardware tests are advisory (nightly) or blocking (pre-deploy)
**Timeout**: Docker suite: 15 minutes; Hardware suite: 30 minutes

## Reporting

**Format**: CSV
**Columns**: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message (if FAIL)
**Output path**: `./tests/e2e-results/report.csv`