docs(solution): reference e2e harness + ADR 0001 from testing strategy

Adds an "E2E Test Harness (Public UAV Datasets)" subsection to
§Testing Strategy explaining the three-tier adapter layout and
pointing readers at testing/README.md for architecture and ADR 0001
for selection rationale. Updates Related Artifacts to list the new
in-repo docs. next_steps.md cross-links the ADR as the authoritative
decision record (brainstorm drafts stay local).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yuzviak
2026-04-17 13:55:29 +03:00
parent 560dc38f0a
commit 5128ac17ba
2 changed files with 19 additions and 3 deletions
+15
View File
@@ -577,6 +577,18 @@ Unchanged from draft05. VO frame: ~17-22ms. Satellite matching: ≤210ms async.
- **ADTI sustained capture rate**: Verify 0.7fps sustained >30 min without buffer overflow
- **Confidence tier transitions**: Verify fix_type and accuracy change correctly across HIGH → MEDIUM → LOW → FAILED transitions
### E2E Test Harness (Public UAV Datasets)
Since the original "fly with real GPS recording" primary validation is blocked on proprietary data collection, an e2e test harness runs the full `FlightProcessor` pipeline as a black box against public UAV datasets. Three tiers:
- **VPAIR sample** — fixed-wing light aircraft, downward camera, 300400 m altitude over Bonn/Eifel. Closest public match to the target tactical fixed-wing envelope. Pose-only (no raw IMU), so exercises VO+GPR+graph paths but not ESKF. Nominal e2e tier.
- **MARS-LVIG** — rotary UAV (DJI M300 RTK) with explicit *featureless* sequences. Used as stress test for the low-texture-terrain failure mode flagged below in §Key Risks. Ships raw IMU + RTK GT.
- **EuRoC MH_01** — indoor MAV with 200 Hz IMU and Vicon GT. CI-tier regression, chosen because every published VIO algorithm reports EuRoC ATE/RPE numbers, so harness output can be cross-checked against literature.
Adapters share a `DatasetAdapter` interface with capability flags (`has_raw_imu`, `has_rtk_gt`, `platform_class`); tests auto-skip paths the adapter can't exercise. Datasets live at `./datasets/<name>/` (gitignored), tests skip when datasets are absent. Implementation in `src/gps_denied/testing/`; architecture guide in that directory's `README.md`; selection rationale in [ADR 0001](decisions/0001-e2e-dataset-strategy.md).
Proprietary flight data (Mavic logs, in-house operator flights) remains on the roadmap but is deprioritised — it becomes relevant once VO+ESKF tuning stabilises and we need to validate on our specific airframe's IMU spectrum and camera intrinsics.
### Non-Functional Tests
- **End-to-end accuracy** (primary validation): Fly with real GPS recording → run GPS-denied system in parallel → compare estimated vs real positions → verify 80% within 50m, 60% within 20m
@@ -619,4 +631,7 @@ Unchanged from draft05. VO frame: ~17-22ms. Satellite matching: ≤210ms async.
- Tech stack evaluation: `_docs/01_solution/tech_stack.md` (needs sync with draft05 corrections)
- Security analysis: `_docs/01_solution/security_analysis.md`
- Previous draft: `_docs/01_solution/solution_draft05.md`
- E2E harness architecture: `src/gps_denied/testing/README.md`
- Architecture decisions: `_docs/01_solution/decisions/` (ADR 0001 — e2e dataset strategy)
- Roadmap and decision log: `next_steps.md`
+4 -3
View File
@@ -40,10 +40,11 @@
### Статус виконання
**Design**: `.planning/brainstorms/2026-04-16-e2e-datasets-design.md`
**Plan**: `.planning/brainstorms/2026-04-16-e2e-datasets-plan.md`
**Design** (локально, gitignored): `.planning/brainstorms/2026-04-16-e2e-datasets-design.md`
**Plan** (локально, gitignored): `.planning/brainstorms/2026-04-16-e2e-datasets-plan.md`
**In-repo docs**: [src/gps_denied/testing/README.md](src/gps_denied/testing/README.md) (harness architecture), [_docs/01_solution/decisions/0001-e2e-dataset-strategy.md](_docs/01_solution/decisions/0001-e2e-dataset-strategy.md) (ADR — selection rationale)
[decision 2026-04-16: замість збору власних даних з Mavic-а — використати публічні UAV датасети. Причина: блокер на проприєтарні дані (Денис, мавікісти), а нам треба рухатись вже зараз. Public датасети: VPAIR (fixed-wing, downward, 300400 м) + EuRoC (indoor MAV, IMU+GT, індустрійний benchmark) + MARS-LVIG (rotary, featureless сіквенси як stress-test). Деталі — у design-doc.]
[decision 2026-04-16: замість збору власних даних з Mavic-а — використати публічні UAV датасети. Причина: блокер на проприєтарні дані (Денис, мавікісти), а нам треба рухатись вже зараз. Public датасети: VPAIR (fixed-wing, downward, 300400 м) + EuRoC (indoor MAV, IMU+GT, індустрійний benchmark) + MARS-LVIG (rotary, featureless сіквенси як stress-test). Деталі — у ADR 0001.]
- [x] Побудувати `DatasetAdapter` ABC + capability flags (has_raw_imu, has_rtk_gt, platform_class) — `src/gps_denied/testing/datasets/base.py`
- [x] `SyntheticAdapter` для harness self-test (`synthetic.py`)