# ADR 0001 — E2E Validation on Public UAV Datasets

**Date:** 2026-04-16
**Status:** Accepted
**Supersedes:** —
**Superseded by:** —

## Context

The `next_steps.md` roadmap (item 3) requires bringing the codebase under the `autopilot existing-code` flow: build an end-to-end test harness that runs the full product as a black box on real UAV data *before* further refactoring (VO → cuVSLAM, `src/gps_denied/` → `src/`). Without that harness, any refactor is a blind change — unit tests would stay green while the integrated pipeline could degrade silently.

The obvious data source — a synchronised IMU + downward-camera log from the target tactical fixed-wing UAV — is not available:

- Denys Popov (Mavic) was asked for logs but delivery is open-ended.
- In-house Mavic operators would need to be convinced to perform non-mission flights with nadir-oriented cameras at high altitude, which is not their normal pattern.
- The DJI Mavic flight in `/home/yuzviak/Azaion/Data/` has no raw IMU (DJI only exports fused attitude + velocities), so the ESKF path is not exercised.

Blocking the harness until someone flies a mission to spec costs weeks. Meanwhile several mature public UAV datasets exist that ship synchronised camera + IMU + ground truth.

## Decision

Build the e2e harness on **public UAV datasets** as the primary validation substrate. Three datasets, three test tiers:

| Tier | Dataset | Role | Platform match |
|---|---|---|---|
| Primary | **VPAIR** (sample) | Fixed-wing, nadir, 300–400 m — closest to target envelope | Fixed-wing light aircraft ≈ tactical fixed-wing dynamics (constant forward motion, banks) |
| Stress | **MARS-LVIG** | Rotary UAV with explicit *featureless* sequences — stress-test VO on low-texture terrain flagged as CRITICAL RISK in solution_draft02 | DJI M300 RTK (rotary) — dynamics mismatch accepted, chosen for the terrain variety |
| CI regression | **EuRoC MH_01** | Industry-standard VIO benchmark; every published VIO algorithm reports EuRoC numbers, so harness output can be sanity-checked against literature | Indoor micro-MAV — dynamics mismatch accepted, chosen for reproducibility and paper-comparability |

Proprietary data collection remains on the roadmap but is deprioritised to a later phase (see `next_steps.md` §4). It becomes relevant once (a) we need to validate on our specific airframe's IMU vibration spectrum and camera intrinsics, or (b) VO+ESKF tuning stabilises and we want a "final" dataset captured at our operational envelope.

All adapters share a single `DatasetAdapter` interface (`src/gps_denied/testing/datasets/base.py`) with capability flags (`has_raw_imu`, `has_rtk_gt`, `has_loop_closures`, `platform_class`), so integration tests auto-skip paths the current adapter can't exercise rather than failing them.

## Consequences

**Positive:**

- Harness and refactor work unblocked on day one — no person-dependent delay.
- Three datasets cover three distinct failure modes: fixed-wing dynamics (VPAIR), low-texture terrain (MARS-LVIG featureless), indoor benchmark comparability (EuRoC).
- Public numbers for EuRoC mean we can cross-check our ATE against OpenVINS / VINS-Fusion — catches gross regressions immediately.
- Capability-flag pattern means an adapter that ships poses but not raw IMU (VPAIR) still contributes: VO+GPR+graph paths run, ESKF path is explicitly skipped.

**Negative / accepted trade-offs:**

- VPAIR and MARS-LVIG are academic-use-only licences. Fine for R&D and internal CI, not for shipping benchmarks in commercial material. ALTO (BSD-3) is the escape hatch if commercial-license benchmarks become a requirement.
- VPAIR sample has no raw IMU — full ESKF+VO path is not exercised by it. EuRoC and MARS-LVIG cover that gap.
- Altitude envelope of public datasets (indoor EuRoC, 80–130 m MARS-LVIG, 300–400 m VPAIR) undershoots the target 200–1500 m tactical envelope. Extrapolating upward is a leap of faith; real-target-altitude validation stays on the roadmap.
- Dataset formats vary wildly (EuRoC ASL, VPAIR ECEF+Euler text, MARS-LVIG ROS bags). Each adapter is custom. Mitigation: shared `coord.py` helpers (ECEF→WGS84, Euler→quaternion) and a small design contract enforced by the ABC.

**First real-run evidence**:

- **VPAIR sample** (2026-04-16, 200 fixed-wing nadir frames): pipeline completes without crashing, ATE RMSE ~1770 km — VO alone diverges because VPAIR ships poses only, no raw IMU, so ESKF never engages.
- **EuRoC MH_01** (2026-04-17, first 100 indoor MAV frames, ~30 s wall-time): pipeline completes, ATE RMSE ~10.9 km. Raw IMU is present (200 Hz ADIS16448) so ESKF *does* run, but the satellite-matching anchor never fires on an indoor scene. Still diverges — one order of magnitude less than VPAIR, and for a different underlying reason. Both are xfail-gated.

These are the starting measurements. Improvements will come from tuning VO+ESKF for each regime and wiring a substitute anchor (e.g. synthetic ArUco markers for indoor, GPR-against-satellite at the right resolution for VPAIR).

## Alternatives considered

- **Wait for Denys / internal pilots to collect data.** Rejected: unbounded delay, no guarantee of raw IMU, coordinates nothing we can start on today.
- **Use only EuRoC.** Rejected: indoor micro-MAV dynamics are nothing like a tactical fixed-wing, and it has no place-recognition-against-satellite angle. Good for CI, insufficient for primary validation.
- **Preprocess datasets into a single normalised format and load via one reader.** Rejected on YAGNI — three adapters with a shared ABC stay clearer than a converter pipeline and don't lose dataset-specific metadata.
- **Use Mid-Air (synthetic).** Rejected: synthetic extremely-low-altitude quadcopter flights. Adds no signal over `SyntheticAdapter` which we already have for harness self-test.

## References

- Roadmap context: [next_steps.md](../../../next_steps.md) §3
- Harness architecture: [src/gps_denied/testing/README.md](../../../src/gps_denied/testing/README.md)
- Target system: [_docs/01_solution/solution.md](../solution.md), §Testing Strategy
- Low-texture CRITICAL RISK: [_docs/01_solution/solution_draft02.md](../solution_draft02.md)
- Local working drafts (not in repo): `.planning/brainstorms/2026-04-16-e2e-datasets-design.md`, `.planning/brainstorms/2026-04-16-e2e-datasets-plan.md`