Oleksandr Bezdieniezhnykh
64542d32fc
Transitioned the autodev state to phase 21, reflecting the completion of Step 5 and the drafting of Step 6 epics. Revised the architecture documentation to clarify the roles of the Tile Manager and its components, ensuring accurate representation of the system's operational flow. Updated glossary entries for Flight State and Operator to incorporate recent changes and enhance clarity on component interactions and responsibilities.
8.4 KiB
Test Specification — C1 Visual / Visual-Inertial Odometry
This file is component-scoped. AC-level coverage is in the suite-level test docs (_docs/02_document/tests/*.md) and the canonical traceability map is _docs/02_document/tests/traceability-matrix.md. The tables below cite test IDs from those files; the Component-Internal Tests section adds C1-specific unit/contract tests that the suite-level scenarios do not cover.
Acceptance Criteria Traceability
| AC ID | Acceptance Criterion (one-line) | Test IDs (suite-level + this file) | Coverage |
|---|---|---|---|
| AC-1.3 | Cumulative drift between satellite-anchored fixes <100 m visual / <50 m IMU-fused | FT-P-02, C1-IT-01 | Covered |
| AC-1.4 | Estimate reports 95% covariance + source label | FT-P-03, C1-IT-02 | Covered |
| AC-2.1a | Frame-to-frame registration ≥95% on normal segments | FT-P-04, C1-IT-03 | Covered |
| AC-2.2 (frame-to-frame portion) | MRE <1 px frame-to-frame | FT-P-05, C1-IT-04 | Covered |
| AC-3.2 | Tolerate sharp turns; recovery via satellite re-loc | FT-P-07, FT-N-02 | Covered (C1 contributes track-loss detection) |
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01 (Tier-2), C1-PT-01 | Covered |
| AC-5.1 | Init from FC EKF's last valid GPS + IMU-extrapolated | FT-P-11, C1-IT-05 | Covered |
| AC-5.3 | On reboot, re-init from FC IMU-extrapolated pose | NFT-RES-02, C1-IT-06 | Covered |
Component-Internal Tests
C1-IT-01: VioStrategy contract — process_frame honest covariance under degradation
Summary: every concrete VioStrategy implementation (Okvis2, VinsMono, KltRansac) must produce a 6×6 covariance whose norm grows monotonically when feature tracking degrades.
Traces to: AC-1.3, AC-1.4
Description: feed each strategy a synthetic 60 s nav-camera + IMU sequence with a controlled feature-loss event at t=30 s (50% feature drop). Assert that pose_covariance_6x6 Frobenius norm before t=30 s is below the steady-state threshold and rises monotonically for ≥3 s after the event. No strategy may emit a tightened covariance during a degradation event (catches honest-covariance-violation regressions).
Input data: tests/fixtures/synthetic_vio/normal_then_feature_drop_60s/ (nav-cam frames at 3 Hz + IMU at 200 Hz; feature-loss event injected via image masking).
Expected result: ||cov||_F curve has a rising shoulder ≥1.5× steady-state norm within 3 s of the event for all three strategies; VioHealth.state transitions TRACKING → DEGRADED within the same window.
Max execution time: 30 s per strategy on Tier-1.
Dependencies: helper ImuPreintegrator (shared with C5).
C1-IT-02: VioOutput schema invariants
Summary: every VioOutput carries a 6×6 SPD covariance and a non-empty feature_quality.
Traces to: AC-1.4
Description: drive each strategy through 100 frames of a synthetic loop; for each emitted VioOutput, assert (a) pose_covariance_6x6 is symmetric and positive-definite, (b) feature_quality.tracked + new ≥ 0, (c) frame_id matches the input NavCameraFrame.frame_id. Any single violation fails the test.
Input data: tests/fixtures/synthetic_vio/loop_100f/.
Expected result: 100/100 frames pass all invariants per strategy.
Max execution time: 10 s.
C1-IT-03: KltRansac mandatory simple-baseline registration ≥95%
Summary: the simple-baseline strategy must hit AC-2.1a's 95% threshold on the Derkachi normal segment so the engine rule's mandatory baseline is met.
Traces to: AC-2.1a (engine rule)
Description: replay the Derkachi normal-segment fixture (the same 60 stills C8 fixture that FT-P-04 uses) through the KltRansac strategy only; count frames with VioHealth.state == TRACKING at emission. Pass if ≥95%.
Input data: tests/fixtures/flight_derkachi/normal_segment_60_stills/.
Expected result: tracked frame ratio ≥ 0.95.
Max execution time: 60 s.
C1-IT-04: Frame-to-frame MRE bound
Summary: each strategy's per-frame mean reprojection error stays under 1 px on normal segments.
Traces to: AC-2.2 (frame-to-frame portion)
Description: same Derkachi normal-segment fixture as C1-IT-03. Compute MRE per frame from the strategy's internal residual; assert MRE p95 < 1 px per AC-2.2.
Input data: as above.
Expected result: MRE p95 < 1 px for Okvis2 + KltRansac (production-default + simple-baseline). VinsMono is research-only and exempt from MRE bound (only IT-12 comparative-study coverage).
Max execution time: 60 s.
C1-IT-05: Warm-start from WarmStartPose converges within configured budget
Summary: reset_to_warm_start followed by 5 frames of input must converge to state == TRACKING for every strategy.
Traces to: AC-5.1
Description: prepare a WarmStartPose derived from the FC EKF's last-valid-GPS fixture; call reset_to_warm_start, then push 5 frames of normal-segment input; assert health transitions INIT → TRACKING within 5 frames.
Input data: tests/fixtures/flight_derkachi/takeoff_warmstart/.
Expected result: TRACKING by frame 5 for all three strategies.
Max execution time: 5 s.
C1-IT-06: F8 reboot recovery via warm-start hint
Summary: simulate a mid-flight reboot — the strategy must re-init from the warm-start hint without crashing or producing covariance < pre-reboot.
Traces to: AC-5.3
Description: run normal-segment input for 30 s; capture last VioOutput; reset the strategy with that pose as WarmStartPose; resume input; assert next 5 emitted VioOutput have pose_covariance_6x6 Frobenius norm ≥ the pre-reboot value (no fake confidence after reboot).
Input data: as C1-IT-03.
Expected result: pass per assertion above for all three strategies.
Max execution time: 60 s.
Performance Tests
C1-PT-01: per-frame latency budget on Tier-2
Summary: process_frame p95 latency on Jetson under nominal thermal.
Traces to: AC-4.1 (component-level partition; suite-level NFT-PERF-01 owns the e2e budget).
Load scenario:
- Single ingest thread, 3 Hz frame rate, 10 min replay of Derkachi normal segment.
- Concurrent C2 backbone forward pass running on the same Jetson (realistic load contention).
Expected results:
| Metric | Target | Failure Threshold |
|---|---|---|
process_frame latency p50 |
≤ 25 ms (Okvis2 production-default) | 60 ms |
process_frame latency p95 |
≤ 80 ms | 120 ms |
| Throughput | ≥ 3 Hz sustained | < 2.5 Hz |
Resource limits:
- CPU: ≤ 30% of one core (Okvis2 is multi-threaded internally; bound at 30% per ADR-002 budget partition).
- Memory: ≤ 1.5 GB resident.
Security Tests
C1 has no externally-reachable surface (internal-only component); suite-level NFT-SEC-02 (no in-flight egress) and NFT-SEC-05 (DNS blackholing) cover the airborne process broadly. No C1-specific security tests.
Acceptance Tests
C1 contributes to AC-1.3 / 1.4 / 2.1a / 5.1 / 5.3 via the suite-level FT scenarios cited in the traceability table. No additional C1-only acceptance tests are needed.
Test Data Management
Required test data:
| Data Set | Description | Source | Size |
|---|---|---|---|
synthetic_vio/normal_then_feature_drop_60s/ |
60 s nav-cam + IMU with controlled feature-loss event at t=30 s | generated by scripts/gen_synthetic_vio.py (deterministic seed) |
~50 MB |
synthetic_vio/loop_100f/ |
100-frame synthetic closed loop for invariant checks | generated, deterministic | ~30 MB |
flight_derkachi/normal_segment_60_stills/ |
the project's canonical normal-segment fixture | curated subset of Derkachi raw drop | ~80 MB |
flight_derkachi/takeoff_warmstart/ |
last-valid-GPS + IMU window from FC EKF for warm-start tests | recorded once, replayed | ~5 MB |
Setup procedure:
- Run
scripts/gen_synthetic_vio.pyonce per fixture to populatetests/fixtures/synthetic_vio/. - Mount
tests/fixtures/flight_derkachi/from the project's data archive (read-only).
Teardown procedure:
- Synthetic fixtures persist between runs (deterministic; no per-run mutation).
- The Derkachi fixture is read-only; nothing to clean up.
Data isolation strategy: every test runs in its own temp directory under tests/tmp/c1/<test-id>/; per-strategy state is constructed fresh in each test (no shared state across tests).