From 5128ac17ba81493229955529d469a9a8303522a5 Mon Sep 17 00:00:00 2001 From: Yuzviak Date: Fri, 17 Apr 2026 13:55:29 +0300 Subject: [PATCH] docs(solution): reference e2e harness + ADR 0001 from testing strategy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an "E2E Test Harness (Public UAV Datasets)" subsection to §Testing Strategy explaining the three-tier adapter layout and pointing readers at testing/README.md for architecture and ADR 0001 for selection rationale. Updates Related Artifacts to list the new in-repo docs. next_steps.md cross-links the ADR as the authoritative decision record (brainstorm drafts stay local). Co-Authored-By: Claude Opus 4.7 (1M context) --- _docs/01_solution/solution.md | 15 +++++++++++++++ next_steps.md | 7 ++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/_docs/01_solution/solution.md b/_docs/01_solution/solution.md index 5438fe7..cf79d02 100644 --- a/_docs/01_solution/solution.md +++ b/_docs/01_solution/solution.md @@ -577,6 +577,18 @@ Unchanged from draft05. VO frame: ~17-22ms. Satellite matching: ≤210ms async. - **ADTI sustained capture rate**: Verify 0.7fps sustained >30 min without buffer overflow - **Confidence tier transitions**: Verify fix_type and accuracy change correctly across HIGH → MEDIUM → LOW → FAILED transitions +### E2E Test Harness (Public UAV Datasets) + +Since the original "fly with real GPS recording" primary validation is blocked on proprietary data collection, an e2e test harness runs the full `FlightProcessor` pipeline as a black box against public UAV datasets. Three tiers: + +- **VPAIR sample** — fixed-wing light aircraft, downward camera, 300–400 m altitude over Bonn/Eifel. Closest public match to the target tactical fixed-wing envelope. Pose-only (no raw IMU), so exercises VO+GPR+graph paths but not ESKF. Nominal e2e tier. +- **MARS-LVIG** — rotary UAV (DJI M300 RTK) with explicit *featureless* sequences. Used as stress test for the low-texture-terrain failure mode flagged below in §Key Risks. Ships raw IMU + RTK GT. +- **EuRoC MH_01** — indoor MAV with 200 Hz IMU and Vicon GT. CI-tier regression, chosen because every published VIO algorithm reports EuRoC ATE/RPE numbers, so harness output can be cross-checked against literature. + +Adapters share a `DatasetAdapter` interface with capability flags (`has_raw_imu`, `has_rtk_gt`, `platform_class`); tests auto-skip paths the adapter can't exercise. Datasets live at `./datasets//` (gitignored), tests skip when datasets are absent. Implementation in `src/gps_denied/testing/`; architecture guide in that directory's `README.md`; selection rationale in [ADR 0001](decisions/0001-e2e-dataset-strategy.md). + +Proprietary flight data (Mavic logs, in-house operator flights) remains on the roadmap but is deprioritised — it becomes relevant once VO+ESKF tuning stabilises and we need to validate on our specific airframe's IMU spectrum and camera intrinsics. + ### Non-Functional Tests - **End-to-end accuracy** (primary validation): Fly with real GPS recording → run GPS-denied system in parallel → compare estimated vs real positions → verify 80% within 50m, 60% within 20m @@ -619,4 +631,7 @@ Unchanged from draft05. VO frame: ~17-22ms. Satellite matching: ≤210ms async. - Tech stack evaluation: `_docs/01_solution/tech_stack.md` (needs sync with draft05 corrections) - Security analysis: `_docs/01_solution/security_analysis.md` - Previous draft: `_docs/01_solution/solution_draft05.md` +- E2E harness architecture: `src/gps_denied/testing/README.md` +- Architecture decisions: `_docs/01_solution/decisions/` (ADR 0001 — e2e dataset strategy) +- Roadmap and decision log: `next_steps.md` diff --git a/next_steps.md b/next_steps.md index cc6af29..a9d52f0 100644 --- a/next_steps.md +++ b/next_steps.md @@ -40,10 +40,11 @@ ### Статус виконання -**Design**: `.planning/brainstorms/2026-04-16-e2e-datasets-design.md` -**Plan**: `.planning/brainstorms/2026-04-16-e2e-datasets-plan.md` +**Design** (локально, gitignored): `.planning/brainstorms/2026-04-16-e2e-datasets-design.md` +**Plan** (локально, gitignored): `.planning/brainstorms/2026-04-16-e2e-datasets-plan.md` +**In-repo docs**: [src/gps_denied/testing/README.md](src/gps_denied/testing/README.md) (harness architecture), [_docs/01_solution/decisions/0001-e2e-dataset-strategy.md](_docs/01_solution/decisions/0001-e2e-dataset-strategy.md) (ADR — selection rationale) -[decision 2026-04-16: замість збору власних даних з Mavic-а — використати публічні UAV датасети. Причина: блокер на проприєтарні дані (Денис, мавікісти), а нам треба рухатись вже зараз. Public датасети: VPAIR (fixed-wing, downward, 300–400 м) + EuRoC (indoor MAV, IMU+GT, індустрійний benchmark) + MARS-LVIG (rotary, featureless сіквенси як stress-test). Деталі — у design-doc.] +[decision 2026-04-16: замість збору власних даних з Mavic-а — використати публічні UAV датасети. Причина: блокер на проприєтарні дані (Денис, мавікісти), а нам треба рухатись вже зараз. Public датасети: VPAIR (fixed-wing, downward, 300–400 м) + EuRoC (indoor MAV, IMU+GT, індустрійний benchmark) + MARS-LVIG (rotary, featureless сіквенси як stress-test). Деталі — у ADR 0001.] - [x] Побудувати `DatasetAdapter` ABC + capability flags (has_raw_imu, has_rtk_gt, platform_class) — `src/gps_denied/testing/datasets/base.py` - [x] `SyntheticAdapter` для harness self-test (`synthetic.py`)