mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 08:21:13 +00:00
Update autodev state, architecture documentation, and glossary terms
Transitioned the autodev state to phase 21, reflecting the completion of Step 5 and the drafting of Step 6 epics. Revised the architecture documentation to clarify the roles of the Tile Manager and its components, ensuring accurate representation of the system's operational flow. Updated glossary entries for Flight State and Operator to incorporate recent changes and enhance clarity on component interactions and responsibilities.
This commit is contained in:
@@ -0,0 +1,136 @@
|
||||
# C1 — Visual / Visual-Inertial Odometry
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: produce a per-frame relative pose SE(3) + 6×6 covariance + IMU bias estimate + feature-quality summary from the nav-camera frame and the FC IMU/attitude window, fusing visual and inertial cues without any external (satellite) reference.
|
||||
|
||||
**Architectural Pattern**: Strategy — `VioStrategy` interface with three concrete implementations (Okvis2 production-default, VinsMono research-only, KltRansac mandatory simple-baseline), constructor-injected at the composition root (ADR-009), build-time gated by per-implementation CMake `BUILD_*` flags (ADR-002), runtime selection by config at startup (ADR-001), not hot-swappable mid-flight.
|
||||
|
||||
**Upstream dependencies**:
|
||||
- Camera ingest thread → `NavCameraFrame` (3 Hz nominal, drop-oldest queue).
|
||||
- C8 FC adapter inbound side → `ImuWindow` (100–200 Hz, time-aligned to frame timestamp).
|
||||
- Camera calibration artifact (loaded once at startup; passed in via constructor).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C5 StateEstimator (consumes `VioOutput` for the iSAM2 `BetweenFactorPose3` + IMU bias prior).
|
||||
- F8 Companion-reboot recovery (uses last `VioOutput` as warm-start hint when re-entering the per-frame loop).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `VioStrategy`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `process_frame` | `NavCameraFrame, ImuWindow, CameraCalibration` | `VioOutput` | No (called on the camera ingest hot path) | `VioInitializingError`, `VioDegradedError`, `VioFatalError` |
|
||||
| `reset_to_warm_start` | `WarmStartPose` | `None` | No | `VioFatalError` |
|
||||
| `health_snapshot` | `()` | `VioHealth` | No | — |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
NavCameraFrame:
|
||||
frame_id: uuid (required) — monotonic per-flight
|
||||
capture_timestamp: monotonic_ns (required) — companion clock; FC clock cross-sync via C8
|
||||
pixels: ndarray[H=3648, W=5472, C=3, dtype=uint8] (required) — 3 Hz nominal
|
||||
camera_id: string (required) — matches the loaded calibration artifact
|
||||
|
||||
ImuWindow:
|
||||
start_t_ns: monotonic_ns (required)
|
||||
end_t_ns: monotonic_ns (required) — should bracket the frame timestamp
|
||||
samples: list[ImuSample] (required) — accel + gyro at 100–200 Hz
|
||||
|
||||
WarmStartPose:
|
||||
body_T_world: SE3 (required) — initial pose hint, e.g. from F2 takeoff load (AC-5.1)
|
||||
velocity_b: Vector3 (required, m/s)
|
||||
bias: ImuBias (required) — accel + gyro bias seed
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
VioOutput:
|
||||
frame_id: uuid — echoes input
|
||||
relative_pose_T: SE3 — body-frame motion since last keyframe
|
||||
pose_covariance_6x6: Matrix6 — honest ESKF or factor-graph covariance per concrete strategy
|
||||
imu_bias: ImuBias — current accel + gyro bias estimate
|
||||
feature_quality: FeatureQuality — tracked/lost feature counts, mean parallax, MRE
|
||||
emitted_at: monotonic_ns
|
||||
|
||||
VioHealth:
|
||||
state: enum {INIT, TRACKING, DEGRADED, LOST}
|
||||
consecutive_lost: int
|
||||
bias_norm: float — used by C5 quality_metadata + AC-NEW-8 spoof gate
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable — internal-only component, no HTTP/gRPC surface.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
Stateless w.r.t. persistent storage. Each strategy holds **in-memory** state only:
|
||||
- Sliding window of N keyframes (concrete strategy decides N).
|
||||
- IMU bias and velocity state.
|
||||
- Feature track buffer.
|
||||
|
||||
No database access, no cache layer beyond the in-process keyframe window.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: per-frame cost is dominated by feature extraction + matching; `O(F)` in feature count for KltRansac, `O(F·log K)` for Okvis2 sliding-window optimisation across K keyframes (D-C5-3 sets K=10–20).
|
||||
|
||||
**State Management**: per-instance in-memory (window of keyframes, IMU bias, velocity). The strategy lives for the duration of a flight; reset on `reset_to_warm_start` for F8 reboot recovery.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| OKVIS2 (C++) | upstream HEAD pinned per Plan-phase | Production-default tightly-coupled VIO; BSD-3-Clause |
|
||||
| VINS-Mono (C++) | upstream HEAD pinned per Plan-phase | Research-only loosely-coupled VIO for IT-12 comparative study; behind `BUILD_VINS_MONO` |
|
||||
| OpenCV | ≥ 4.12.0 (CVE-2025-53644 mitigation) | KLT pyramidal optical flow + RANSAC for the simple-baseline strategy |
|
||||
| Eigen | matches OKVIS2 / GTSAM pin | Lie-algebra math for SE(3) + 6×6 covariance |
|
||||
| pybind11 | matches OKVIS2 / VINS-Mono build | Python bindings for the C++ strategies |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `VioInitializingError`: state = INIT, no `VioOutput` emitted, C5 falls back to FC IMU prior — no MAVLink emission.
|
||||
- `VioDegradedError`: state = DEGRADED, `VioOutput` emitted with inflated covariance, C5 down-weights.
|
||||
- `VioFatalError`: state = LOST after configurable consecutive frames; AC-5.2 fallback path triggered (FC IMU-only after 3 s).
|
||||
- No retries inside `process_frame` — the caller is responsible for handling drop-oldest queue semantics on the hot path.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `ImuPreintegrator` | shared GTSAM `CombinedImuFactor` preintegration buffer | C1, C5 (both consume the same IMU window) |
|
||||
| `SE3Utils` | SE(3) ↔ pose-matrix conversion, Lie-algebra exponential/logarithm | C1, C4, C5 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- Pure VIO drifts unbounded over time; AC-1.3 cumulative-drift bound (<100 m visual / <50 m IMU-fused between satellite anchors) is met *only* in cooperation with C2/C3/C4 anchors, not by C1 alone.
|
||||
- Sharp turns with <5% frame overlap (RESTRICT-UAV-3) cause feature-track loss in all three strategies; F6 satellite re-localization is the recovery path.
|
||||
|
||||
**Potential race conditions**:
|
||||
- The camera ingest thread is the sole producer; C5 is the sole consumer. Concurrent calls to `process_frame` on a single strategy instance are forbidden — enforce in the composition root by binding one strategy instance to the camera ingest thread.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Okvis2 sliding-window optimisation can spike to 80–120 ms on a thermally-throttled Jetson; D-CROSS-LATENCY-1 hybrid auto-degrades C4 covariance recovery (not C1) to free budget.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C7 (TRT/ONNX runtime is not on C1's path, but `ImuPreintegrator` shares GTSAM with C5 and is built alongside C5), C13 (FDR sink for VioHealth telemetry).
|
||||
|
||||
**Can be implemented in parallel with**: C2, C3, C6 — independent code paths.
|
||||
|
||||
**Blocks**: C5 (no fusion without `VioOutput`), C4 (no per-frame relative-pose prior), F3 / F5 / F6 (every per-frame flow consumes `VioOutput`).
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `VioFatalError` raised; AC-5.2 path imminent | `VIO LOST after 9 consecutive frames; strategy=okvis2` |
|
||||
| WARN | `VioDegradedError`; covariance inflation > 2× steady-state | `VIO degraded: parallax=0.02, mre=4.1px, bias_norm=0.18` |
|
||||
| INFO | Strategy init complete; warm-start applied; state transitions | `VIO ready: strategy=okvis2, calibration=adti20.unit-7` |
|
||||
| DEBUG | Per-frame keyframe decision, feature-track count | `VIO frame=12345 tracked=187 new=42 keyframe=true` |
|
||||
|
||||
**Log format**: structured JSON via the project's shared logger; no plaintext.
|
||||
|
||||
**Log storage**: stdout (Tier-1) / journald (Tier-2 dev) / FDR via C13 (production). Per-frame DEBUG logs are never persisted to FDR — they go to stdout/journald only.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Test Specification — C1 Visual / Visual-Inertial Odometry
|
||||
|
||||
This file is component-scoped. AC-level coverage is in the suite-level test docs (`_docs/02_document/tests/*.md`) and the canonical traceability map is `_docs/02_document/tests/traceability-matrix.md`. The tables below cite test IDs from those files; the `Component-Internal Tests` section adds C1-specific unit/contract tests that the suite-level scenarios do not cover.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs (suite-level + this file) | Coverage |
|
||||
|-------|---------------------------------|-----------------------------------|----------|
|
||||
| AC-1.3 | Cumulative drift between satellite-anchored fixes <100 m visual / <50 m IMU-fused | FT-P-02, **C1-IT-01** | Covered |
|
||||
| AC-1.4 | Estimate reports 95% covariance + source label | FT-P-03, **C1-IT-02** | Covered |
|
||||
| AC-2.1a | Frame-to-frame registration ≥95% on normal segments | FT-P-04, **C1-IT-03** | Covered |
|
||||
| AC-2.2 (frame-to-frame portion) | MRE <1 px frame-to-frame | FT-P-05, **C1-IT-04** | Covered |
|
||||
| AC-3.2 | Tolerate sharp turns; recovery via satellite re-loc | FT-P-07, FT-N-02 | Covered (C1 contributes track-loss detection) |
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01 (Tier-2), **C1-PT-01** | Covered |
|
||||
| AC-5.1 | Init from FC EKF's last valid GPS + IMU-extrapolated | FT-P-11, **C1-IT-05** | Covered |
|
||||
| AC-5.3 | On reboot, re-init from FC IMU-extrapolated pose | NFT-RES-02, **C1-IT-06** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C1-IT-01: VioStrategy contract — `process_frame` honest covariance under degradation
|
||||
|
||||
**Summary**: every concrete `VioStrategy` implementation (Okvis2, VinsMono, KltRansac) must produce a 6×6 covariance whose norm grows monotonically when feature tracking degrades.
|
||||
|
||||
**Traces to**: AC-1.3, AC-1.4
|
||||
|
||||
**Description**: feed each strategy a synthetic 60 s nav-camera + IMU sequence with a controlled feature-loss event at t=30 s (50% feature drop). Assert that `pose_covariance_6x6` Frobenius norm before t=30 s is below the steady-state threshold and rises monotonically for ≥3 s after the event. No strategy may emit a tightened covariance during a degradation event (catches honest-covariance-violation regressions).
|
||||
|
||||
**Input data**: `tests/fixtures/synthetic_vio/normal_then_feature_drop_60s/` (nav-cam frames at 3 Hz + IMU at 200 Hz; feature-loss event injected via image masking).
|
||||
|
||||
**Expected result**: `||cov||_F` curve has a rising shoulder ≥1.5× steady-state norm within 3 s of the event for all three strategies; `VioHealth.state` transitions `TRACKING → DEGRADED` within the same window.
|
||||
|
||||
**Max execution time**: 30 s per strategy on Tier-1.
|
||||
**Dependencies**: helper `ImuPreintegrator` (shared with C5).
|
||||
|
||||
---
|
||||
|
||||
### C1-IT-02: `VioOutput` schema invariants
|
||||
|
||||
**Summary**: every `VioOutput` carries a 6×6 SPD covariance and a non-empty `feature_quality`.
|
||||
|
||||
**Traces to**: AC-1.4
|
||||
|
||||
**Description**: drive each strategy through 100 frames of a synthetic loop; for each emitted `VioOutput`, assert (a) `pose_covariance_6x6` is symmetric and positive-definite, (b) `feature_quality.tracked + new ≥ 0`, (c) `frame_id` matches the input `NavCameraFrame.frame_id`. Any single violation fails the test.
|
||||
|
||||
**Input data**: `tests/fixtures/synthetic_vio/loop_100f/`.
|
||||
|
||||
**Expected result**: 100/100 frames pass all invariants per strategy.
|
||||
|
||||
**Max execution time**: 10 s.
|
||||
|
||||
---
|
||||
|
||||
### C1-IT-03: KltRansac mandatory simple-baseline registration ≥95%
|
||||
|
||||
**Summary**: the simple-baseline strategy must hit AC-2.1a's 95% threshold on the Derkachi normal segment so the engine rule's mandatory baseline is met.
|
||||
|
||||
**Traces to**: AC-2.1a (engine rule)
|
||||
|
||||
**Description**: replay the Derkachi normal-segment fixture (the same 60 stills C8 fixture that FT-P-04 uses) through the KltRansac strategy only; count frames with `VioHealth.state == TRACKING` at emission. Pass if ≥95%.
|
||||
|
||||
**Input data**: `tests/fixtures/flight_derkachi/normal_segment_60_stills/`.
|
||||
|
||||
**Expected result**: tracked frame ratio ≥ 0.95.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C1-IT-04: Frame-to-frame MRE bound
|
||||
|
||||
**Summary**: each strategy's per-frame mean reprojection error stays under 1 px on normal segments.
|
||||
|
||||
**Traces to**: AC-2.2 (frame-to-frame portion)
|
||||
|
||||
**Description**: same Derkachi normal-segment fixture as C1-IT-03. Compute MRE per frame from the strategy's internal residual; assert MRE p95 < 1 px per AC-2.2.
|
||||
|
||||
**Input data**: as above.
|
||||
|
||||
**Expected result**: MRE p95 < 1 px for Okvis2 + KltRansac (production-default + simple-baseline). VinsMono is research-only and exempt from MRE bound (only IT-12 comparative-study coverage).
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C1-IT-05: Warm-start from `WarmStartPose` converges within configured budget
|
||||
|
||||
**Summary**: `reset_to_warm_start` followed by 5 frames of input must converge to `state == TRACKING` for every strategy.
|
||||
|
||||
**Traces to**: AC-5.1
|
||||
|
||||
**Description**: prepare a `WarmStartPose` derived from the FC EKF's last-valid-GPS fixture; call `reset_to_warm_start`, then push 5 frames of normal-segment input; assert health transitions `INIT → TRACKING` within 5 frames.
|
||||
|
||||
**Input data**: `tests/fixtures/flight_derkachi/takeoff_warmstart/`.
|
||||
|
||||
**Expected result**: TRACKING by frame 5 for all three strategies.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C1-IT-06: F8 reboot recovery via warm-start hint
|
||||
|
||||
**Summary**: simulate a mid-flight reboot — the strategy must re-init from the warm-start hint without crashing or producing covariance < pre-reboot.
|
||||
|
||||
**Traces to**: AC-5.3
|
||||
|
||||
**Description**: run normal-segment input for 30 s; capture last `VioOutput`; reset the strategy with that pose as `WarmStartPose`; resume input; assert next 5 emitted `VioOutput` have `pose_covariance_6x6` Frobenius norm ≥ the pre-reboot value (no fake confidence after reboot).
|
||||
|
||||
**Input data**: as C1-IT-03.
|
||||
|
||||
**Expected result**: pass per assertion above for all three strategies.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C1-PT-01: per-frame latency budget on Tier-2
|
||||
|
||||
**Summary**: `process_frame` p95 latency on Jetson under nominal thermal.
|
||||
|
||||
**Traces to**: AC-4.1 (component-level partition; suite-level NFT-PERF-01 owns the e2e budget).
|
||||
|
||||
**Load scenario**:
|
||||
- Single ingest thread, 3 Hz frame rate, 10 min replay of Derkachi normal segment.
|
||||
- Concurrent C2 backbone forward pass running on the same Jetson (realistic load contention).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `process_frame` latency p50 | ≤ 25 ms (Okvis2 production-default) | 60 ms |
|
||||
| `process_frame` latency p95 | ≤ 80 ms | 120 ms |
|
||||
| Throughput | ≥ 3 Hz sustained | < 2.5 Hz |
|
||||
|
||||
**Resource limits**:
|
||||
- CPU: ≤ 30% of one core (Okvis2 is multi-threaded internally; bound at 30% per ADR-002 budget partition).
|
||||
- Memory: ≤ 1.5 GB resident.
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
C1 has no externally-reachable surface (internal-only component); suite-level NFT-SEC-02 (no in-flight egress) and NFT-SEC-05 (DNS blackholing) cover the airborne process broadly. No C1-specific security tests.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
C1 contributes to AC-1.3 / 1.4 / 2.1a / 5.1 / 5.3 via the suite-level FT scenarios cited in the traceability table. No additional C1-only acceptance tests are needed.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
**Required test data**:
|
||||
|
||||
| Data Set | Description | Source | Size |
|
||||
|----------|-------------|--------|------|
|
||||
| `synthetic_vio/normal_then_feature_drop_60s/` | 60 s nav-cam + IMU with controlled feature-loss event at t=30 s | generated by `scripts/gen_synthetic_vio.py` (deterministic seed) | ~50 MB |
|
||||
| `synthetic_vio/loop_100f/` | 100-frame synthetic closed loop for invariant checks | generated, deterministic | ~30 MB |
|
||||
| `flight_derkachi/normal_segment_60_stills/` | the project's canonical normal-segment fixture | curated subset of Derkachi raw drop | ~80 MB |
|
||||
| `flight_derkachi/takeoff_warmstart/` | last-valid-GPS + IMU window from FC EKF for warm-start tests | recorded once, replayed | ~5 MB |
|
||||
|
||||
**Setup procedure**:
|
||||
1. Run `scripts/gen_synthetic_vio.py` once per fixture to populate `tests/fixtures/synthetic_vio/`.
|
||||
2. Mount `tests/fixtures/flight_derkachi/` from the project's data archive (read-only).
|
||||
|
||||
**Teardown procedure**:
|
||||
1. Synthetic fixtures persist between runs (deterministic; no per-run mutation).
|
||||
2. The Derkachi fixture is read-only; nothing to clean up.
|
||||
|
||||
**Data isolation strategy**: every test runs in its own temp directory under `tests/tmp/c1/<test-id>/`; per-strategy state is constructed fresh in each test (no shared state across tests).
|
||||
@@ -0,0 +1,138 @@
|
||||
# C2 — Visual Place Recognition
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: given the current `NavCameraFrame`, retrieve the top-K=10 candidate satellite tiles from the pre-cached corpus by descriptor similarity. C2 owns the *retrieval* step; C2.5 narrows K=10 → N=3 via inlier-based re-rank.
|
||||
|
||||
**Architectural Pattern**: Strategy — `VprStrategy` interface; concrete implementations (UltraVPR primary, MegaLoc secondary, MixVPR / SelaVPR / EigenPlaces / NetVLAD / SALAD additional candidates) selected at startup by config (ADR-001); build-time gated per-implementation by `BUILD_*` flags (ADR-002); composition-root wired (ADR-009).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- Camera ingest thread → `NavCameraFrame` (parallel fan-out with C1; same frame, distinct queue depth).
|
||||
- C7 InferenceRuntime → backbone forward pass (TRT/ONNX/PyTorch per active runtime).
|
||||
- C6 DescriptorIndex → FAISS HNSW lookup over pre-cached tile descriptors.
|
||||
- Camera calibration artifact — for backbone input preprocessing (resize/crop/normalise).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C2.5 ReRanker (consumes `VprResult`).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `VprStrategy`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `embed_query` | `NavCameraFrame, CameraCalibration` | `VprQuery` | No | `VprBackboneError` |
|
||||
| `retrieve_topk` | `VprQuery, k: int` | `VprResult` | No | `IndexUnavailableError`, `VprBackboneError` |
|
||||
| `descriptor_dim` | `()` | `int` | No | — |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
NavCameraFrame: see C1 spec — same DTO
|
||||
|
||||
VprQuery:
|
||||
frame_id: uuid (required)
|
||||
embedding: ndarray[D, dtype=float16|float32] (required) — D depends on backbone
|
||||
produced_at: monotonic_ns
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
VprResult:
|
||||
frame_id: uuid
|
||||
candidates: list[VprCandidate] (length = k, ranked by descriptor distance ascending)
|
||||
retrieved_at: monotonic_ns
|
||||
backbone_label: string — for FDR provenance
|
||||
|
||||
VprCandidate:
|
||||
tile_id: composite (zoomLevel, lat, lon)
|
||||
descriptor_distance: float — backbone-specific metric (cosine for L2-normalised embeddings)
|
||||
descriptor_dim: int
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable — internal-only component.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
### Queries
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| FAISS HNSW top-K=10 search | 3 Hz (per nav frame) | Yes | Yes — pre-built HNSW (C6) |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| Backbone weights | TRT engine on disk + GPU resident | flight lifetime | Manifest content-hash gate (D-C10-3) at takeoff |
|
||||
| FAISS HNSW index | mmap (C6 owns the file) | flight lifetime | Same as above |
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
C2 itself stores no persistent data; it consumes C6's descriptor index. Sizing belongs in C6.
|
||||
|
||||
### Data Management
|
||||
|
||||
C2 is read-only against C6 during F3/F4/F6. Pre-flight, F1 triggers C10 (after C11 `TileDownloader` has populated C6) to call `embed_query` on every staged tile to populate the descriptor matrix consumed by C6.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: HNSW search is `O(log N)` in corpus size for k=10; backbone forward pass is `O(1)` per frame (GPU-bound).
|
||||
|
||||
**State Management**: stateless per-frame; the only persistent state is the loaded backbone weights and the FAISS index pointer (held by C6 and passed in via constructor).
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| FAISS (Python + C++) | upstream HEAD pinned per Plan-phase | HNSW retrieval; consumed via C6 |
|
||||
| TensorRT | 10.3 (JetPack 6.2 pin) | Primary inference backend; consumed via C7 |
|
||||
| ONNX Runtime + TRT EP | matches C7 | Fallback backend |
|
||||
| PyTorch | matches simple-baseline track | FP16 baseline (NetVLAD / MixVPR mandatory) |
|
||||
| UltraVPR (research code drop) | upstream HEAD pinned per Plan-phase | Documentary Lead PRIMARY backbone |
|
||||
| MegaLoc, MixVPR, SelaVPR, EigenPlaces, NetVLAD | upstream HEAD pinned per Plan-phase | Secondary + mandatory simple-baselines |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `VprBackboneError`: backbone forward pass failed (CUDA OOM, TRT engine deserialize mismatch). C2 emits no `VprResult`; C5 falls back to VIO-only with provenance label `visual_propagated` (AC-1.4).
|
||||
- `IndexUnavailableError`: FAISS index handle invalid (e.g., post-F8 reboot before warm-up). Same fallback as above; F8 recovery flow re-mmaps the index.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `BackbonePreprocessor` | resize / crop / normalise per backbone's input contract | C2 only — keep inside the component, not a shared helper |
|
||||
| `DescriptorNormaliser` | L2-normalise descriptors so cosine similarity aligns with Euclidean | C2 (query side), C10 (corpus side at cache artifact build) |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- VPR is sensitive to scene change between cache build and flight time — AC-NEW-6 freshness gating is the project-level mitigation, not a C2 concern.
|
||||
- Backbone choice is constrained by ADR-002: only the linked-in implementations are selectable at runtime.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Concurrent `embed_query` calls on a single strategy instance can race on the GPU stream. Bind one strategy instance to one ingest thread — composition root enforces.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Backbone forward pass is the dominant cost (~30–80 ms on Jetson per backbone). FAISS HNSW search is sub-millisecond for 100k-tile corpora.
|
||||
- D-CROSS-LATENCY-1 hybrid does not change C2 behaviour — C2's budget is fixed; the auto-degrade happens at C4.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C6 (descriptor index), C7 (inference runtime), C10 (descriptor population at cache artifact build).
|
||||
|
||||
**Can be implemented in parallel with**: C1, C8 — independent paths.
|
||||
|
||||
**Blocks**: C2.5 (no candidates without `VprResult`), F3 / F6.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `VprBackboneError` or `IndexUnavailableError` | `VPR backbone OOM: backbone=ultravpr, frame=12345` |
|
||||
| WARN | top-1 distance exceeds drift threshold (potential false-positive retrieval) | `VPR top-1 distance 0.42 above warn threshold 0.30; backbone=ultravpr` |
|
||||
| INFO | Strategy ready; backbone loaded | `VPR ready: backbone=ultravpr, dim=512, corpus_size=87654` |
|
||||
| DEBUG | Per-frame top-K distances | `VPR frame=12345 top10_distances=[0.12, 0.14, ...]` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN only).
|
||||
@@ -0,0 +1,144 @@
|
||||
# Test Specification — C2 Visual Place Recognition
|
||||
|
||||
Component-scoped. Suite-level coverage is in `_docs/02_document/tests/*.md`; canonical traceability is `tests/traceability-matrix.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-2.1b | Satellite-anchor registration meets AC-1.1/1.2/2.2/8.2/8.6 | FT-P-05, FT-P-19, **C2-IT-01** | Covered |
|
||||
| AC-2.2 (cross-domain portion) | MRE <2.5 px cross-domain | FT-P-06, **C2-IT-02** (recall floor only — MRE owned by C3) | Covered |
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01, **C2-PT-01** | Covered |
|
||||
| AC-NEW-7 | Cache poisoning safety budget | NFT-SEC-01 (onboard-side), **C2-IT-03** | Covered (relaxed) |
|
||||
| AC-8.6 (scale-ratio portion) | Scale-ratio satellite relocalization | FT-P-19, **C2-IT-04** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C2-IT-01: top-K=10 recall at p=10 on Derkachi
|
||||
|
||||
**Summary**: the primary backbone (UltraVPR) achieves recall@10 ≥ 0.95 on the Derkachi normal segment against the pre-built corpus.
|
||||
|
||||
**Traces to**: AC-2.1b
|
||||
|
||||
**Description**: for each query frame in `flight_derkachi/normal_segment_60_stills/`, embed via UltraVPR; query the FAISS HNSW index; assert that the ground-truth tile (per recorded GPS + sector classification) is within the top-10 candidates ≥95% of frames.
|
||||
|
||||
**Input data**: `flight_derkachi/normal_segment_60_stills/` + the Derkachi corpus FAISS index built by C10 (consumed read-only here).
|
||||
|
||||
**Expected result**: recall@10 ≥ 0.95 for UltraVPR; recall@10 ≥ 0.85 for the mandatory simple-baseline NetVLAD (engine rule check).
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
### C2-IT-02: VprResult schema invariants
|
||||
|
||||
**Summary**: every `VprResult` carries `len(candidates) == k`, monotonically-non-decreasing `descriptor_distance`, and a non-empty `backbone_label`.
|
||||
|
||||
**Traces to**: AC-2.2 (downstream-coverage prerequisite)
|
||||
|
||||
**Description**: 100 frames through `retrieve_topk(k=10)`; assert (a) length, (b) sorted-ascending distance, (c) `backbone_label` non-empty.
|
||||
|
||||
**Input data**: `tests/fixtures/synthetic_vpr/diverse_100f/`.
|
||||
|
||||
**Expected result**: 100/100 results pass all invariants.
|
||||
|
||||
**Max execution time**: 10 s.
|
||||
|
||||
---
|
||||
|
||||
### C2-IT-03: cache-poisoning seed rejection at retrieval
|
||||
|
||||
**Summary**: when the corpus contains a poisoned tile injected during NFT-SEC-01 setup, the top-1 distance to that tile is bounded so downstream RANSAC + voting can reject it within the AC-NEW-7 relaxed budget.
|
||||
|
||||
**Traces to**: AC-NEW-7 (component-level partition)
|
||||
|
||||
**Description**: load the NFT-SEC-01 setup corpus (3-flight cumulative dataset with deflated covariance × 1.5–3); query each Derkachi frame; record the top-1 candidate for each. Pass if the poisoned tile is top-1 in fewer than the relaxed-CI threshold (the relaxation per AC-text 2026-05-09).
|
||||
|
||||
**Input data**: NFT-SEC-01 setup corpus.
|
||||
|
||||
**Expected result**: poisoned-tile top-1 rate within AC-NEW-7 relaxed CI.
|
||||
|
||||
**Max execution time**: 5 min.
|
||||
|
||||
---
|
||||
|
||||
### C2-IT-04: scale-ratio invariance on satellite re-loc
|
||||
|
||||
**Summary**: when the nav-camera scale changes (e.g., altitude shift), VPR top-K still surfaces the correct tile.
|
||||
|
||||
**Traces to**: AC-8.6 (scale-ratio half)
|
||||
|
||||
**Description**: synthetically scale the Derkachi normal-segment frames by ±20% (mimicking altitude variation); assert recall@10 stays ≥ 0.85 across the scale sweep.
|
||||
|
||||
**Input data**: `flight_derkachi/scaled_+20%/` and `flight_derkachi/scaled_-20%/` (generated via deterministic resize).
|
||||
|
||||
**Expected result**: recall@10 ≥ 0.85 at both scale extremes for UltraVPR.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C2-PT-01: backbone forward + HNSW lookup budget on Tier-2
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: 3 Hz frame rate, 10 min replay; corpus size 87 654 tiles (Derkachi area at 0.5 m/px).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `embed_query` p95 | ≤ 60 ms (UltraVPR / FP16) | 100 ms |
|
||||
| `retrieve_topk(k=10)` p95 | ≤ 2 ms (HNSW) | 10 ms |
|
||||
| Combined p95 | ≤ 65 ms | 110 ms |
|
||||
|
||||
**Resource limits**:
|
||||
- GPU memory: ≤ 600 MB resident for backbone weights.
|
||||
- System memory: ≤ 200 MB for the mmap'd FAISS index handle.
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C2-ST-01: index-handle invalidation safety
|
||||
|
||||
**Summary**: after C10 rebuilds the FAISS index (post-takeoff is FORBIDDEN, but unit-level safety check), the previous handle held by C2 must not silently return stale results.
|
||||
|
||||
**Traces to**: defensive — no AC trace; backstops a code-injection / config-drift mode that AC-NEW-7 already covers at the suite level.
|
||||
|
||||
**Test procedure**:
|
||||
1. Build a tiny 100-tile FAISS index; mmap it through C2.
|
||||
2. Replace the underlying file (simulating an out-of-band rebuild, which is FORBIDDEN at flight time per D-C10-3 but defended-in-depth here).
|
||||
3. Call `retrieve_topk` and assert C2 raises `IndexUnavailableError` rather than returning stale candidates.
|
||||
|
||||
**Pass criteria**: `IndexUnavailableError` raised; no candidates returned.
|
||||
**Fail criteria**: any candidate returned.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
C2 contributes to AC-2.1b / AC-8.6 via the suite-level FT scenarios. No additional C2-only acceptance tests.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Description | Source | Size |
|
||||
|----------|-------------|--------|------|
|
||||
| `synthetic_vpr/diverse_100f/` | 100 diverse synthetic frames for invariant checks | generated, deterministic | ~50 MB |
|
||||
| `flight_derkachi/normal_segment_60_stills/` | shared with C1 | curated | shared |
|
||||
| `flight_derkachi/scaled_±20%/` | scale-ratio sweep | generated from above | ~40 MB |
|
||||
| Derkachi FAISS corpus | C10's output, consumed read-only | C10 build artifact | ~200 MB |
|
||||
|
||||
**Setup procedure**:
|
||||
1. C10 must have built the Derkachi FAISS index (this is a Step-5-test-side prereq; in the test runner, the corpus is staged from `tests/fixtures/cache_artifacts/`).
|
||||
2. Synthetic + scaled fixtures generated by deterministic scripts.
|
||||
|
||||
**Teardown**: corpus is read-only; nothing to clean up.
|
||||
|
||||
**Data isolation**: each test gets its own `tests/tmp/c2/<test-id>/`.
|
||||
@@ -0,0 +1,111 @@
|
||||
# C2.5 — Inlier-based Re-rank
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: re-rank C2's top-K=10 VPR candidates down to top-N=3 by single-pair LightGlue inlier count, producing a higher-precision input for the cross-domain matcher (C3). The re-rank step is the architectural boundary between cheap descriptor retrieval (C2) and expensive cross-domain matching (C3) — it pays a small extra cost so C3 only operates on the most promising candidates.
|
||||
|
||||
**Architectural Pattern**: Strategy (single concrete implementation today: `InlierCountReRanker`). Future re-rank algorithms can be added as additional `ReRankStrategy` implementations behind the same interface.
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C2 → `VprResult` (top-K=10 candidates).
|
||||
- Shared `LightGlueRuntime` helper (used in single-pair mode for inlier counting; the same matcher object is shared with C3 — owned by the helper, not by C3, so neither component depends on the other at build time).
|
||||
- C6 TileStore → fetch tile pixels for each candidate (cheap, in-memory page-cache hit during a flight).
|
||||
- Camera calibration artifact — for nav-frame preprocessing.
|
||||
|
||||
**Downstream consumers**:
|
||||
- C3 CrossDomainMatcher (consumes `RerankResult`).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `ReRankStrategy`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `rerank` | `NavCameraFrame, VprResult, n: int` | `RerankResult` | No | `RerankBackboneError`, `TileFetchError` |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
NavCameraFrame: see C1
|
||||
VprResult: see C2
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
RerankResult:
|
||||
frame_id: uuid
|
||||
candidates: list[RerankCandidate] (length = n=3, ranked by inlier_count descending)
|
||||
reranked_at: monotonic_ns
|
||||
|
||||
RerankCandidate:
|
||||
tile_id: composite (zoomLevel, lat, lon)
|
||||
inlier_count: int — single-pair LightGlue inliers
|
||||
descriptor_distance: float — carried forward from C2 for FDR provenance
|
||||
tile_pixels_handle: Tile pixel reference (do not copy — page-cache hit)
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| Tile pixel fetch from C6 (10 tiles per frame) | 3 Hz × 10 = 30 Hz | Yes | tile filesystem already mmap-backed in C6 |
|
||||
|
||||
No caching layer beyond C6's mmap. The same tile may be fetched repeatedly across frames; OS page cache absorbs that cost.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: `O(K)` LightGlue forward passes per frame (K=10), each `O(M_tile · M_query)` in feature counts. The whole step is GPU-bound on the same engine that C3 uses — hence the shared LightGlue runtime.
|
||||
|
||||
**State Management**: stateless per-frame. Holds a reference to the shared LightGlue object owned by C3.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| LightGlue (Python) | upstream HEAD pinned per Plan-phase | Single-pair matching for inlier count |
|
||||
| TensorRT | matches C7 | LightGlue inference engine reuse |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `RerankBackboneError`: LightGlue forward pass failed on one or more candidates. The candidate is dropped from the rerank set; if fewer than N=3 candidates survive, C2.5 returns whatever it has and C3 proceeds with reduced N.
|
||||
- `TileFetchError`: C6 read failure for a candidate tile. Same drop-and-continue behaviour as above.
|
||||
- Hard failure (zero candidates left after rerank): emit no `RerankResult`; C5 falls back to VIO-only with provenance label `visual_propagated`.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `LightGlueRuntime` | shared LightGlue inference handle (one engine, many call sites) | C2.5, C3 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- The re-rank correctness depends on LightGlue inlier-count being a meaningful proxy for cross-domain match quality at single-pair resolution. If a backbone in C2 returns visually-similar-but-geographically-wrong candidates, C2.5's inlier count can still rank them above the true match — AC-NEW-7 cache-poisoning safety budget catches this downstream.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Shared LightGlue runtime is the same object as C3 uses. Serial access from a single ingest thread; concurrent calls forbidden.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- 10 LightGlue passes per frame is non-trivial; budget allocation lives in `tests/performance-tests.md` NFT-PERF-01 partition.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C2 (input), shared `LightGlueRuntime` helper (which both C2.5 and C3 consume), C6 (tile pixels), C7 (inference runtime). C2.5 does **not** depend on C3 at build time — they are sibling consumers of the helper, and the data flow is C2.5 → C3 (not the other way).
|
||||
|
||||
**Can be implemented in parallel with**: C1 (independent path).
|
||||
|
||||
**Blocks**: C3 (no `RerankResult`, C3 has no input), F3 / F6.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | Zero candidates surviving rerank | `Re-rank produced 0 candidates; frame=12345; falling back to visual_propagated` |
|
||||
| WARN | <N=3 candidates surviving | `Re-rank produced 1 candidate of N=3; frame=12345` |
|
||||
| INFO | Strategy ready | `Re-rank ready: strategy=inlier_count, N=3, K=10` |
|
||||
| DEBUG | Per-frame inlier counts | `Re-rank frame=12345 inlier_counts=[412, 287, 198, ...]` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN only).
|
||||
@@ -0,0 +1,107 @@
|
||||
# Test Specification — C2.5 Re-rank
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-2.1b | Satellite-anchor registration | FT-P-05, **C2.5-IT-01** | Covered |
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01, **C2.5-PT-01** | Covered |
|
||||
| AC-NEW-7 | Cache poisoning (rerank-side filter) | NFT-SEC-01, **C2.5-IT-02** | Covered (relaxed) |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C2.5-IT-01: Top-K=10 → Top-N=3 promotion stability
|
||||
|
||||
**Summary**: when C2's top-1 is the ground-truth tile, C2.5's top-1 stays the ground-truth tile.
|
||||
|
||||
**Traces to**: AC-2.1b
|
||||
|
||||
**Description**: for the Derkachi normal segment where C2 already picks the correct top-1 (per C2-IT-01), assert that C2.5's `RerankResult.candidates[0].tile_id` matches C2's top-1 in ≥98% of frames. The remaining ≤2% are accepted (rerank can legitimately demote a top-1 candidate when a top-K=2..10 candidate has more inliers — this is the design intent).
|
||||
|
||||
**Input data**: shared with C2-IT-01.
|
||||
|
||||
**Expected result**: top-1 promotion rate ≥ 0.98 (i.e., rerank rarely overrides a correct C2 top-1).
|
||||
|
||||
**Max execution time**: 2 min (10 LightGlue passes per frame on Tier-1 with the LightGlue runtime).
|
||||
|
||||
---
|
||||
|
||||
### C2.5-IT-02: drop-and-continue on per-candidate failure
|
||||
|
||||
**Summary**: if one of the K=10 candidates raises `RerankBackboneError`, C2.5 returns N=2 candidates instead of zero — never crashes.
|
||||
|
||||
**Traces to**: AC-NEW-7 (defensive — keeps the pipeline alive on the resilience path)
|
||||
|
||||
**Description**: monkey-patch the LightGlue runtime to raise `RerankBackboneError` on the 5th candidate of every frame; run 100 frames; assert (a) C2.5 emits a `RerankResult` for every frame, (b) `len(candidates) ∈ {2, 3}`, (c) error logged at WARN level.
|
||||
|
||||
**Input data**: `synthetic_vpr/diverse_100f/`.
|
||||
|
||||
**Expected result**: 100/100 frames produce a result; counts match assertion.
|
||||
|
||||
**Max execution time**: 2 min.
|
||||
|
||||
---
|
||||
|
||||
### C2.5-IT-03: shared LightGlue runtime serial-access invariant
|
||||
|
||||
**Summary**: concurrent calls to the shared `LightGlueRuntime` from C2.5 and C3 must serialize without deadlock or corrupt output.
|
||||
|
||||
**Traces to**: helper invariant (no AC trace; backstops the helper-ownership decision per R14)
|
||||
|
||||
**Description**: spawn two threads — one running C2.5 rerank, the other running C3 match — sharing the same `LightGlueRuntime`. Run 50 iterations each; assert no exceptions, all outputs produced, and the result determinism holds (compare against single-threaded baseline; bit-identical match).
|
||||
|
||||
**Input data**: synthetic batch.
|
||||
|
||||
**Expected result**: no deadlock; outputs bit-identical to single-threaded run.
|
||||
|
||||
**Max execution time**: 2 min.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C2.5-PT-01: 10 LightGlue passes per frame budget on Tier-2
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: 3 Hz, K=10 single-pair LightGlue per frame, 10 min replay.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `rerank` p95 | ≤ 80 ms (10 single-pair LightGlue) | 150 ms |
|
||||
| Inference engine reuse | 1 engine across all 10 calls | regression on engine-reuse causes test failure |
|
||||
|
||||
**Resource limits**:
|
||||
- GPU memory: ≤ 300 MB for the shared LightGlue engine (counted once, not 10×).
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
C2.5 has no externally-reachable surface; defensive coverage flows through the helper-ownership invariant (C2.5-IT-03).
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
C2.5 has no operator-facing behaviour; covered transitively via FT-P-05 / FT-P-06.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| `synthetic_vpr/diverse_100f/` | shared with C2 | shared |
|
||||
| `flight_derkachi/normal_segment_60_stills/` | shared with C1/C2 | shared |
|
||||
| Derkachi corpus + LightGlue engine | C10/C7 build artifacts | shared |
|
||||
|
||||
**Setup**: same as C2.
|
||||
**Teardown**: corpus + engines are read-only.
|
||||
**Data isolation**: per-test temp dirs under `tests/tmp/c2_5/<test-id>/`.
|
||||
@@ -0,0 +1,127 @@
|
||||
# C3 — Cross-domain Matcher
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: produce 2D-3D correspondences between the current `NavCameraFrame` and the top-N=3 satellite tiles from C2.5, with RANSAC-filtered inliers and reprojection residual statistics. C3 is the **cross-domain** step (nav-camera ↔ satellite-imagery domain gap) and is the dominant compute cost in F3.
|
||||
|
||||
**Architectural Pattern**: Strategy — `CrossDomainMatcher` interface, with concrete implementations DISK+LightGlue (D-C3-1 = (a) primary), ALIKED+LightGlue (secondary), XFeat (alternate). Selection at startup by config (ADR-001); build-time gating by `BUILD_*` flags (ADR-002); composition-root wired (ADR-009).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C2.5 → `RerankResult` (top-N=3 candidates).
|
||||
- C7 InferenceRuntime → backbone forward pass.
|
||||
- Camera calibration artifact — for nav-frame preprocessing.
|
||||
- C6 TileStore — for tile pixels (handle carried in `RerankCandidate`).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C3.5 ConditionalRefiner (consumes `MatchResult`; passthrough or AdHoP refinement).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `CrossDomainMatcher`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `match` | `NavCameraFrame, RerankResult, CameraCalibration` | `MatchResult` | No | `MatcherBackboneError`, `InsufficientInliersError` |
|
||||
| `health_snapshot` | `()` | `MatcherHealth` | No | — |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
NavCameraFrame: see C1
|
||||
RerankResult: see C2.5
|
||||
CameraCalibration: see C5
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
MatchResult:
|
||||
frame_id: uuid
|
||||
per_candidate: list[CandidateMatchSet] (length up to N=3, drop on failure)
|
||||
best_candidate_idx: int — argmax(inlier_count) among per_candidate
|
||||
reprojection_residual_px: float — best candidate's median residual
|
||||
matched_at: monotonic_ns
|
||||
matcher_label: string — for FDR provenance
|
||||
|
||||
CandidateMatchSet:
|
||||
tile_id: composite (zoomLevel, lat, lon)
|
||||
inlier_count: int
|
||||
inlier_correspondences: ndarray[I, 4, dtype=float32] — (px_query, py_query, px_tile, py_tile)
|
||||
ransac_outlier_count: int
|
||||
per_candidate_residual_px: float
|
||||
|
||||
MatcherHealth:
|
||||
consecutive_low_inlier: int
|
||||
mean_inliers_60s: float
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| Tile pixel access (3 tiles per frame) | 3 Hz × 3 = 9 Hz | Yes | C6 mmap |
|
||||
|
||||
No additional caching beyond C6.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: `O(N · F · F)` for the matching pass per backbone (N=3 candidates, F features per image), plus RANSAC `O(I · trials)` on inliers. Dominant cost is the backbone forward pass.
|
||||
|
||||
**State Management**: stateless per-frame. Holds the shared LightGlue / DISK runtime handle.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| DISK (Python) | upstream HEAD pinned per Plan-phase | Primary feature extractor (D-C3-1 = (a)) |
|
||||
| LightGlue (Python) | upstream HEAD pinned per Plan-phase | Primary matcher; replaces SuperPoint+SuperGlue (Magic Leap noncommercial) |
|
||||
| ALIKED (Python) | upstream HEAD pinned per Plan-phase | Secondary feature extractor |
|
||||
| XFeat (Python) | upstream HEAD pinned per Plan-phase | Alternate (lightweight) feature+matcher |
|
||||
| OpenCV | ≥ 4.12.0 | RANSAC + reprojection residual computation |
|
||||
| TensorRT | matches C7 | Backbone engine compilation/runtime |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `MatcherBackboneError`: backbone forward pass failed on a candidate. Candidate dropped from `per_candidate`; if all N=3 candidates fail, emit `InsufficientInliersError`.
|
||||
- `InsufficientInliersError`: RANSAC inlier count below configurable threshold on every candidate. C5 falls back to VIO-only with provenance label `visual_propagated`. F6 satellite re-localization may trigger if the condition persists.
|
||||
- Catastrophic backbone failure (engine unloadable): treated as `MatcherBackboneError` for every frame until F8 reboot recovery.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `LightGlueRuntime` | shared LightGlue inference handle | C2.5, C3 |
|
||||
| `RansacFilter` | RANSAC + reprojection residual computation thin wrapper | C3, C3.5, C4 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- The cross-domain gap (nav-camera vs satellite tile) is the hardest step in the pipeline. Backbone choice depends on the deployment camera's spectral and resolution characteristics; the current default (DISK+LightGlue) is locked per Mode B Fact #110 / D-C3-1 = (a) pending IT-12 verdict.
|
||||
- D-C2-12 (DINOv2-feature-based matcher) is a carryforward research item that may displace DISK in a future cycle.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Shared LightGlue runtime with C2.5; serial access from one ingest thread.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- C3 dominates the F3 latency budget. The D-CROSS-LATENCY-1 hybrid does NOT change C3 (K=N=3 stays); it changes C4 covariance recovery.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C2.5 (input), C7 (inference runtime).
|
||||
|
||||
**Can be implemented in parallel with**: C1, C6 — independent paths.
|
||||
|
||||
**Blocks**: C3.5 / C4 / C5, F3 / F6.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `InsufficientInliersError`; all N=3 candidates failed | `C3 zero-inlier on all candidates; frame=12345; backbone=disk_lightglue` |
|
||||
| WARN | reprojection residual above threshold (will trigger AdHoP at C3.5) | `C3 residual=4.2px > threshold=2.5px; frame=12345; will refine` |
|
||||
| INFO | Strategy ready | `C3 ready: matcher=disk_lightglue` |
|
||||
| DEBUG | per-candidate inlier + residual | `C3 frame=12345 candidates=[(412,1.1px),(287,1.4px),(198,2.0px)]` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN only).
|
||||
@@ -0,0 +1,144 @@
|
||||
# Test Specification — C3 Cross-Domain Matcher
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-1.1 | Frame-center GPS within 50 m for ≥80% of normal-flight photos | FT-P-01, **C3-IT-01** (inlier-budget partition) | Covered |
|
||||
| AC-2.1b | Satellite-anchor registration | FT-P-05, **C3-IT-02** | Covered |
|
||||
| AC-2.2 (cross-domain portion) | MRE <2.5 px cross-domain | FT-P-06, **C3-IT-03** | Covered |
|
||||
| AC-3.1 | Tolerate up to 350 m outliers, tilt ±20° | FT-N-01, **C3-IT-04** | Covered |
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01, **C3-PT-01** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C3-IT-01: per-candidate inlier-count floor on Derkachi
|
||||
|
||||
**Summary**: on the Derkachi normal segment, the best candidate's RANSAC inlier count is at least the configured floor (default ≥ 80 inliers per frame).
|
||||
|
||||
**Traces to**: AC-1.1 (component-level partition feeding AC-1.1's accuracy budget)
|
||||
|
||||
**Description**: for the Derkachi normal-segment fixture, run C3 with the production-default DISK+LightGlue backbone; record `MatchResult.per_candidate[best_candidate_idx].inlier_count`; assert p5 ≥ 80 (i.e., ≥95% of frames clear the floor).
|
||||
|
||||
**Input data**: `flight_derkachi/normal_segment_60_stills/` + C10-built tile descriptors (read-only).
|
||||
|
||||
**Expected result**: p5 inlier count ≥ 80 for DISK+LightGlue.
|
||||
|
||||
**Max execution time**: 4 min on Tier-1 (CPU fallback) / 90 s on Tier-2.
|
||||
|
||||
---
|
||||
|
||||
### C3-IT-02: best-candidate selection determinism
|
||||
|
||||
**Summary**: `best_candidate_idx == argmax(inlier_count)` for every emitted `MatchResult`.
|
||||
|
||||
**Traces to**: AC-2.1b
|
||||
|
||||
**Description**: 100 frames through `match`; assert (a) `best_candidate_idx` equals the index of the largest `inlier_count` in `per_candidate`, (b) ties are broken deterministically (lowest tile_id wins; check against a known-tie synthetic fixture).
|
||||
|
||||
**Input data**: `synthetic_matcher/known_tie_10f/` (synthetic frames where two candidates have identical inlier counts).
|
||||
|
||||
**Expected result**: deterministic tie-breaking; no `best_candidate_idx` mismatch in 100/100 frames.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C3-IT-03: cross-domain MRE bound
|
||||
|
||||
**Summary**: median per-frame reprojection residual stays under 2.5 px for the production-default matcher on the Derkachi normal segment.
|
||||
|
||||
**Traces to**: AC-2.2
|
||||
|
||||
**Description**: same Derkachi fixture; record `MatchResult.reprojection_residual_px`; assert p95 < 2.5 px.
|
||||
|
||||
**Input data**: as C3-IT-01.
|
||||
|
||||
**Expected result**: p95 < 2.5 px for DISK+LightGlue. ALIKED+LightGlue (secondary) and XFeat (alternate) tested on a smoke subset only — comparative-study verdict belongs to IT-12.
|
||||
|
||||
**Max execution time**: 4 min.
|
||||
|
||||
---
|
||||
|
||||
### C3-IT-04: tilt + outlier robustness
|
||||
|
||||
**Summary**: under ±20° tilt and synthetic 350 m outliers, the matcher still produces a usable inlier set (≥40 inliers).
|
||||
|
||||
**Traces to**: AC-3.1
|
||||
|
||||
**Description**: synthetically tilt the Derkachi frames by {−20°, −10°, 0°, +10°, +20°}; inject 350 m position outliers into the candidate tile metadata for 5% of frames; assert C3 emits a `MatchResult` with `best_candidate.inlier_count ≥ 40` for ≥90% of frames in each tilt bucket.
|
||||
|
||||
**Input data**: `flight_derkachi/tilted_±20°/` (deterministic synthetic tilt).
|
||||
|
||||
**Expected result**: per-bucket inlier-count p10 ≥ 40 for DISK+LightGlue.
|
||||
|
||||
**Max execution time**: 6 min.
|
||||
|
||||
---
|
||||
|
||||
### C3-IT-05: `InsufficientInliersError` propagation
|
||||
|
||||
**Summary**: when all N=3 candidates fall below the inlier floor, C3 raises `InsufficientInliersError` and emits no `MatchResult`.
|
||||
|
||||
**Traces to**: AC-3.5 (defensive — keeps the spoof-fallback path clean)
|
||||
|
||||
**Description**: synthetic frames with deliberately mismatched candidate tiles (cross-region pulls); assert `match` raises `InsufficientInliersError` for every frame and the downstream consumer (C3.5 / C4) receives no `MatchResult`.
|
||||
|
||||
**Input data**: `synthetic_matcher/cross_region_mismatch_20f/`.
|
||||
|
||||
**Expected result**: 20/20 frames raise the error.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C3-PT-01: per-frame match latency on Tier-2 (dominant cost)
|
||||
|
||||
**Traces to**: AC-4.1 (C3 owns the largest single partition of the budget)
|
||||
|
||||
**Load scenario**: 3 Hz, N=3 candidates, 10 min replay; concurrent C2 backbone + C5 iSAM2 update on the same Jetson.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `match` p95 | ≤ 180 ms (DISK+LightGlue) | 280 ms |
|
||||
| Per-candidate p95 | ≤ 60 ms | 95 ms |
|
||||
| Throughput | ≥ 3 Hz sustained | < 2.5 Hz |
|
||||
|
||||
**Resource limits**:
|
||||
- GPU memory: ≤ 800 MB for backbone + matcher engines combined.
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
C3 has no externally-reachable surface; defensive coverage at the cache-poisoning level (NFT-SEC-01) and via shared-runtime invariants (C2.5-IT-03).
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-01, FT-P-05, FT-P-06.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| `flight_derkachi/normal_segment_60_stills/` | shared | shared |
|
||||
| `flight_derkachi/tilted_±20°/` | generated | ~150 MB |
|
||||
| `synthetic_matcher/known_tie_10f/` | generated | ~5 MB |
|
||||
| `synthetic_matcher/cross_region_mismatch_20f/` | generated | ~10 MB |
|
||||
| C10-built tile descriptors | C10 artifact | shared |
|
||||
|
||||
**Setup**: C10 must have built tile descriptors; matchers' TRT engines must be compiled (consumes ~5 min on Tier-2 first run; cached after).
|
||||
**Teardown**: read-only.
|
||||
**Data isolation**: per-test temp dirs.
|
||||
@@ -0,0 +1,107 @@
|
||||
# C3.5 — AdHoP-conditional Refinement
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: conditionally refine `MatchResult` via OrthoLoC AdHoP (method-agnostic perspective preconditioning) when the initial reprojection residual exceeds a configurable threshold; bypass otherwise. The conditional invocation preserves the AC-4.1 latency budget on the steady-state path while keeping the refinement option for hard frames.
|
||||
|
||||
**Architectural Pattern**: Strategy with two concrete implementations: `AdHoPRefiner` (real refinement) and `PassthroughRefiner` (no-op for the non-conditional baseline / smoke tests). Selection at startup by config (ADR-001); both implementations linked into the deployment binary by default (refinement is conditionally invoked at runtime, not gated at build time).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C3 → `MatchResult`.
|
||||
- C7 InferenceRuntime — AdHoP backbone forward pass when invoked.
|
||||
|
||||
**Downstream consumers**:
|
||||
- C4 PoseEstimator (consumes the possibly-refined `MatchResult`).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `ConditionalRefiner`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `refine_if_needed` | `NavCameraFrame, MatchResult, residual_threshold_px: float` | `MatchResult` (possibly enriched with refined inliers) | No | `RefinerBackboneError` |
|
||||
| `was_invoked` | `()` | `bool` — last-call flag for FDR provenance | No | — |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
NavCameraFrame: see C1
|
||||
MatchResult: see C3
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
MatchResult (refined):
|
||||
Same shape as C3's MatchResult, with the following enrichments when refinement was invoked:
|
||||
per_candidate[i].inlier_correspondences: refined coordinates from AdHoP perspective preconditioning
|
||||
per_candidate[i].per_candidate_residual_px: post-refinement residual
|
||||
refinement_label: "adhop" | "passthrough"
|
||||
refinement_added_latency_ms: float — for FDR latency partition
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| Nav-frame pixel access (1 frame per invocation) | conditional, <3 Hz | When invoked | already in memory from C1/C2 path |
|
||||
|
||||
No additional caching.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: AdHoP perspective preconditioning is `O(F)` in feature count plus one extra backbone forward pass. Steady-state cost is zero (passthrough); worst-case adds ~30–90 ms per invocation on the Jetson.
|
||||
|
||||
**State Management**: stateless per-frame.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| OrthoLoC AdHoP (research code drop) | upstream HEAD pinned per Plan-phase | Conditional refinement |
|
||||
| OpenCV | ≥ 4.12.0 | Reprojection residual computation, perspective transforms |
|
||||
| TensorRT | matches C7 | AdHoP backbone engine when invoked |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `RefinerBackboneError`: AdHoP backbone failure. Fall through to passthrough — emit C3's original `MatchResult` unchanged; downstream pose estimation may fail subsequent quality gates and trigger F6 satellite re-localization.
|
||||
- The conditional gate is a configuration parameter (`adhop.residual_threshold_px`); no runtime decision logic outside that comparison.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `RansacFilter` | shared with C3 | C3, C3.5, C4 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- The threshold is a tunable; setting it too low triggers AdHoP every frame and breaks the AC-4.1 budget; too high lets bad matches through to C4. Calibration via NFT-PERF-01 invocation-rate measurement during planning's NFT validation cycle.
|
||||
- D-CROSS-LATENCY-1 hybrid does not directly change C3.5's behaviour; under thermal throttle, the elevated frame-to-frame latency may push the threshold to be raised at the operator-tooling level pre-flight (C12 concern).
|
||||
|
||||
**Potential race conditions**:
|
||||
- Stateless; concurrent calls are safe in principle but the hot path is single-threaded by design.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- AdHoP invocation is the variable cost in the F3 budget. NFT-PERF-01 measures the invocation rate; an invocation rate above ~30% suggests the threshold needs revisiting.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C3 (input), C7 (inference runtime).
|
||||
|
||||
**Can be implemented in parallel with**: C1, C6 — independent paths.
|
||||
|
||||
**Blocks**: C4, F3 / F6.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `RefinerBackboneError` | `AdHoP backbone failure; frame=12345; falling through to passthrough` |
|
||||
| WARN | invocation rate over rolling 60 s exceeds budget | `AdHoP invocation rate 38% > target 25%; frame=12345` |
|
||||
| INFO | Strategy ready | `Refiner ready: strategy=adhop, threshold=2.5px` |
|
||||
| DEBUG | per-frame invoked/passthrough decision | `Refiner frame=12345 invoked=true added_latency_ms=42` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN only). DEBUG goes to FDR's per-frame estimate stream as a flag (`refinement_label`).
|
||||
@@ -0,0 +1,104 @@
|
||||
# Test Specification — C3.5 AdHoP-Conditional Refinement
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-2.2 (cross-domain MRE on hard frames) | MRE < 2.5 px after refinement | FT-P-06, **C3.5-IT-01** | Covered |
|
||||
| AC-4.1 | E2E latency budget under conditional invocation | NFT-PERF-01, **C3.5-PT-01** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C3.5-IT-01: residual reduction under refinement
|
||||
|
||||
**Summary**: when `refine_if_needed` is invoked (residual > threshold), the post-refinement residual is strictly lower than the pre-refinement residual on ≥90% of invocations.
|
||||
|
||||
**Traces to**: AC-2.2 (hard-frame portion)
|
||||
|
||||
**Description**: prepare a fixture of "hard frames" (synthetic perspective skew that pushes the C3 residual above the threshold); run C3 → C3.5; assert `MatchResult.per_candidate[best].per_candidate_residual_px` post-refinement < pre-refinement value in ≥90% of invocations. The remaining ≤10% are accepted (refinement can legitimately fail to improve a frame; passthrough is the documented fallback).
|
||||
|
||||
**Input data**: `synthetic_matcher/hard_frames_perspective_skew_50f/`.
|
||||
|
||||
**Expected result**: improvement rate ≥ 0.90.
|
||||
|
||||
**Max execution time**: 5 min on Tier-1.
|
||||
|
||||
---
|
||||
|
||||
### C3.5-IT-02: passthrough fall-through on `RefinerBackboneError`
|
||||
|
||||
**Summary**: if the AdHoP backbone fails, C3.5 returns C3's original `MatchResult` unchanged with `refinement_label = "passthrough"`.
|
||||
|
||||
**Traces to**: defensive (no AC trace; backstops the conditional-refinement design contract)
|
||||
|
||||
**Description**: monkey-patch the AdHoP runtime to raise `RefinerBackboneError`; run 20 hard frames; assert (a) every frame produces a `MatchResult`, (b) `refinement_label == "passthrough"`, (c) error logged at ERROR level, (d) inlier coordinates equal C3's input coordinates bit-for-bit (no silent corruption).
|
||||
|
||||
**Input data**: as C3.5-IT-01.
|
||||
|
||||
**Expected result**: 20/20 passthrough; bit-identical correspondences.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C3.5-IT-03: invocation rate stays under target on the steady-state path
|
||||
|
||||
**Summary**: on the Derkachi normal segment, the AdHoP invocation rate is below the 25% target threshold; if it exceeds 30%, the test fails (signals a threshold mis-tune).
|
||||
|
||||
**Traces to**: NFT-PERF-01 invocation-rate budget partition
|
||||
|
||||
**Description**: replay Derkachi normal segment; count `was_invoked()` true returns over 60 s rolling windows; assert max invocation rate < 0.30 (warning at 0.25 per the component spec § 9 logging table).
|
||||
|
||||
**Input data**: `flight_derkachi/normal_segment_60_stills/`.
|
||||
|
||||
**Expected result**: max invocation rate < 0.30.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C3.5-PT-01: refinement-added latency budget
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: 3 Hz, mixed normal + hard frames (25% hard) replicating realistic invocation rate.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `refine_if_needed` p95 (when invoked) | ≤ 90 ms (one extra backbone forward + perspective remap) | 150 ms |
|
||||
| `refine_if_needed` p95 (passthrough) | ≤ 0.5 ms | 2 ms |
|
||||
| Frame-aggregated added latency p95 | ≤ 25 ms (≤25% × 90 ms invocation; ≥75% × 0.5 ms passthrough) | 40 ms |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
C3.5 has no externally-reachable surface.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-06.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| `synthetic_matcher/hard_frames_perspective_skew_50f/` | generated, deterministic | ~50 MB |
|
||||
| `flight_derkachi/normal_segment_60_stills/` | shared | shared |
|
||||
| AdHoP TRT engine | C7-built artifact | shared |
|
||||
|
||||
**Setup**: AdHoP engine must be compiled (deterministic; cached after first build).
|
||||
**Teardown**: read-only.
|
||||
**Data isolation**: per-test temp dirs.
|
||||
@@ -0,0 +1,115 @@
|
||||
# C4 — Pose Estimation
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: convert `MatchResult` (2D-3D correspondences) into a `PoseEstimate` — WGS84 position + 6×6 covariance + provenance label + `last_satellite_anchor_age_ms` — using OpenCV `solvePnPRansac` (IPPE) wrapped in GTSAM `Marginals` for native 6×6 posterior covariance recovery (D-C4-2 = (b)). Under thermal throttle, auto-degrades to Jacobian-based covariance (D-C4-2 = (a)) per the D-CROSS-LATENCY-1 hybrid.
|
||||
|
||||
**Architectural Pattern**: single concrete implementation `OpenCVGtsamPoseEstimator` behind the `PoseEstimator` interface. The pose estimator and the state estimator (C5) **share the GTSAM substrate**; the C4 factor is added directly to C5's iSAM2 graph rather than computed in isolation.
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C3.5 → `MatchResult` (refined or passthrough).
|
||||
- C5 StateEstimator — supplies the GTSAM iSAM2 handle so C4 can add its factor in-graph (architecture principle: shared substrate per ADR-003).
|
||||
- Camera calibration artifact — for intrinsics + distortion + body-to-camera extrinsics.
|
||||
- C7 InferenceRuntime — only indirectly via the LightGlue inliers fed in from C3 / C3.5; C4 itself is OpenCV+GTSAM, not GPU-bound.
|
||||
|
||||
**Downstream consumers**:
|
||||
- C5 StateEstimator (consumes `PoseEstimate`).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `PoseEstimator`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `estimate` | `MatchResult, CameraCalibration, ThermalState` | `PoseEstimate` | No | `PnpFailureError`, `CovarianceDegradedWarning` |
|
||||
| `current_covariance_mode` | `()` | `CovarianceMode` enum {MARGINALS, JACOBIAN} | No | — |
|
||||
|
||||
**Input DTOs**:
|
||||
```
|
||||
MatchResult: see C3 / C3.5
|
||||
CameraCalibration: see C5
|
||||
ThermalState: see C7 (telemetry from jetson-stats)
|
||||
```
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
PoseEstimate:
|
||||
frame_id: uuid
|
||||
position_wgs84: LatLonAlt — degrees, degrees, metres MSL
|
||||
orientation_world_T_body: Quat (w, x, y, z)
|
||||
covariance_6x6: Matrix6 — position (3x3) + orientation (3x3) sub-matrices
|
||||
covariance_mode: CovarianceMode {MARGINALS, JACOBIAN}
|
||||
source_label: enum {satellite_anchored, visual_propagated, dead_reckoned}
|
||||
last_satellite_anchor_age_ms: int — bin input for AC-1.3
|
||||
emitted_at: monotonic_ns
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
Stateless w.r.t. persistent storage; reads camera calibration once at construction.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: `solvePnPRansac` is `O(I · trials)` in inlier count and RANSAC trials; `Marginals.marginalCovariance(pose_key)` is `O(K^3)` in keyframe-window size for the steady-state path (D-C5-3 K=10–20). Jacobian-degraded mode is `O(I)`.
|
||||
|
||||
**State Management**:
|
||||
- Stateless w.r.t. flight history (C5 owns history).
|
||||
- Holds the GTSAM `Marginals` factor handle and the OpenCV `solvePnPRansac` configuration.
|
||||
- Holds a reference to the shared GTSAM iSAM2 graph owned by C5 — does **not** own it.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| OpenCV | ≥ 4.12.0 (CVE-2025-53644 mitigation) | `solvePnPRansac` with `SOLVEPNP_IPPE` flag; D-C4-1 = (b) |
|
||||
| GTSAM (Python + C++) | per Plan-phase pin | `Marginals.marginalCovariance(pose_key)` for native 6×6 covariance |
|
||||
| Eigen | matches GTSAM | Lie-algebra math |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `PnpFailureError`: RANSAC convergence failure or degenerate match geometry. Emit no `PoseEstimate`; C5 falls back to VIO-only with provenance label `visual_propagated`.
|
||||
- `CovarianceDegradedWarning`: thermal-throttle telemetry crossed the configurable threshold; auto-switch to Jacobian-based covariance (D-CROSS-LATENCY-1). Emit `PoseEstimate` with `covariance_mode = JACOBIAN`. NOT a fatal condition.
|
||||
- The thermal-throttle decision is per-frame; once telemetry returns below threshold, switch back to MARGINALS on the next frame.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `SE3Utils` | shared with C1, C5 | C1, C4, C5 |
|
||||
| `WgsConverter` | local-tangent-plane ↔ WGS84 latitude/longitude/altitude | C4, C8 |
|
||||
| `RansacFilter` | shared RANSAC + reprojection residual | C3, C3.5, C4 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- Posterior covariance accuracy depends on the GTSAM substrate being healthy. C5's iSAM2 instability propagates into C4's covariance honesty.
|
||||
- Jacobian-degraded covariance is a known accuracy trade (~5–10% loss per ADR-006); accepted under thermal throttle, never accepted on the steady-state path.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Concurrent calls to `estimate` would race on the shared GTSAM graph. Single-threaded hot path; composition root binds C4 + C5 to the same thread.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- `Marginals.marginalCovariance(pose_key)` is the dominant cost in steady state (~30–90 ms). The D-CROSS-LATENCY-1 hybrid trades this for ~5–15 ms Jacobian under thermal throttle.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C3.5 (input), C5 (shared GTSAM substrate; circular at the design level — both must be co-developed but C5's iSAM2 graph must be constructable without C4 calling into it).
|
||||
|
||||
**Can be implemented in parallel with**: C1, C6 — independent paths.
|
||||
|
||||
**Blocks**: C5 (graph factor add depends on C4), F3 / F6 / F7.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `PnpFailureError` | `C4 PnP failure on frame=12345; mode=marginals; falling back to visual_propagated` |
|
||||
| WARN | `CovarianceDegradedWarning`; thermal throttle entered | `C4 covariance degraded to JACOBIAN; thermal_throttle=true; clock=mhz=750` |
|
||||
| INFO | Strategy ready | `C4 ready: estimator=opencv_gtsam, default_covariance=MARGINALS` |
|
||||
| DEBUG | per-frame inlier count + residual + chosen mode | `C4 frame=12345 inliers=412 residual=0.9px mode=MARGINALS` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN always; DEBUG only via FDR per-frame estimate row).
|
||||
@@ -0,0 +1,123 @@
|
||||
# Test Specification — C4 Pose Estimator
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-1.1 | Frame-center GPS within 50 m for ≥80% of normal-flight photos | FT-P-01, **C4-IT-01** | Covered |
|
||||
| AC-1.2 | Frame-center GPS within 20 m for ≥50% | FT-P-01, **C4-IT-01** | Covered |
|
||||
| AC-1.4 | Estimate reports 95% covariance + source label | FT-P-03, **C4-IT-02** | Covered |
|
||||
| AC-4.1 | E2E latency <400 ms p95 (incl. Marginals) | NFT-PERF-01, **C4-PT-01** | Covered |
|
||||
| AC-NEW-5 | Operating envelope; thermal-throttle-driven covariance degradation hybrid | NFT-LIM-04, **C4-IT-03** | Covered (workstation portion) |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C4-IT-01: PnP convergence + WGS84 accuracy on Derkachi
|
||||
|
||||
**Summary**: for the Derkachi normal segment, p80 of frame-center positions land within 50 m of ground truth, p50 within 20 m.
|
||||
|
||||
**Traces to**: AC-1.1, AC-1.2
|
||||
|
||||
**Description**: feed `MatchResult` from C3 (Derkachi normal-segment fixture) into `estimate`; convert `position_wgs84` to local frame; compute distance to recorded GPS ground truth; assert p80 ≤ 50 m and p50 ≤ 20 m.
|
||||
|
||||
**Input data**: `flight_derkachi/normal_segment_60_stills/` (with recorded GPS ground truth) + corresponding C3 `MatchResult` outputs (replayed from a recorded fixture so the test is C4-isolated).
|
||||
|
||||
**Expected result**: p80 ≤ 50 m, p50 ≤ 20 m.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C4-IT-02: 6×6 covariance is SPD and honest under match degradation
|
||||
|
||||
**Summary**: every emitted `PoseEstimate` carries an SPD `covariance_6x6`; under inlier degradation (synthetically reduced inlier count), the covariance norm rises monotonically.
|
||||
|
||||
**Traces to**: AC-1.4
|
||||
|
||||
**Description**: replay 100 frames; for each, assert (a) covariance is symmetric and positive-definite, (b) when a fixture-injected inlier-degradation event occurs at frame 50, the cov norm rises and stays elevated for ≥10 frames. Also assert `source_label` reflects the gate state — `satellite_anchored` only when C5's gate confirms; `visual_propagated` otherwise.
|
||||
|
||||
**Input data**: `synthetic_matcher/inlier_degradation_at_f50/`.
|
||||
|
||||
**Expected result**: 100/100 frames pass SPD invariant; cov norm rises ≥1.5× steady-state for the degradation interval.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C4-IT-03: D-CROSS-LATENCY-1 hybrid auto-degrade switch
|
||||
|
||||
**Summary**: when `ThermalState.throttle == true`, `current_covariance_mode()` returns `JACOBIAN`; when false, returns `MARGINALS`. The switch is per-frame.
|
||||
|
||||
**Traces to**: AC-NEW-5 (workstation-baseline portion; hot-soak chamber deferred)
|
||||
|
||||
**Description**: drive a 60 s replay alternating thermal flag every 5 s; assert `covariance_mode` in each emitted `PoseEstimate` matches the input thermal flag for that frame; assert no jitter or hysteresis (the spec calls for per-frame decision).
|
||||
|
||||
**Input data**: synthetic frames + a `ThermalState` injection harness.
|
||||
|
||||
**Expected result**: 100% match between thermal input and `covariance_mode` output.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
### C4-IT-04: shared-graph integration with C5
|
||||
|
||||
**Summary**: factors added by C4 to C5's iSAM2 graph survive an `update`/`marginalCovariance` cycle without corrupting prior keyframes.
|
||||
|
||||
**Traces to**: AC-1.4 (defensive — backstops the C4↔C5 shared-substrate co-dependency per ADR-003)
|
||||
|
||||
**Description**: in a single test process, instantiate C5 with a 10-keyframe synthetic prior; have C4 add a `GenericProjectionFactorCal3DS2` for keyframe 11; trigger iSAM2 update; assert (a) the previously-known keyframe poses change by less than configurable tolerance (10 cm position, 0.5° rotation), (b) marginals on every keyframe remain SPD.
|
||||
|
||||
**Input data**: synthetic 10-keyframe iSAM2 prior + a known correspondence set.
|
||||
|
||||
**Expected result**: prior-keyframe perturbations within tolerance; SPD invariant holds.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C4-PT-01: Marginals vs Jacobian path latency on Tier-2
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: 3 Hz, 10 min replay; thermal flag toggled every 30 s to exercise both modes.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `estimate` p95 (MARGINALS, K=15) | ≤ 90 ms | 130 ms |
|
||||
| `estimate` p95 (JACOBIAN) | ≤ 15 ms | 25 ms |
|
||||
| Mode-switch latency | < 1 frame (i.e., next-frame switch) | > 1 frame |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
C4 has no externally-reachable surface.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-01 / FT-P-03 / FT-P-09-AP / FT-P-09-iNav.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Replayed C3 `MatchResult` for Derkachi | recorded once via fixture-build script | ~30 MB |
|
||||
| `synthetic_matcher/inlier_degradation_at_f50/` | generated | ~10 MB |
|
||||
| Synthetic 10-keyframe iSAM2 prior | scripted | <1 MB |
|
||||
|
||||
**Setup**: replay fixture must be re-recorded if C3's output schema changes (versioned in `tests/fixtures/c4_inputs/<schema-version>/`).
|
||||
**Teardown**: read-only.
|
||||
**Data isolation**: per-test temp dirs.
|
||||
@@ -0,0 +1,134 @@
|
||||
# C5 — State Estimator
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: own the GTSAM `iSAM2` + `IncrementalFixedLagSmoother` (K=10–20 keyframes per D-C5-3) state. Fuse `VioOutput` (C1), `PoseEstimate` (C4), and FC IMU/attitude windows (C8 inbound) into the posterior pose with native 6×6 covariance via `Marginals` (D-C5-5 = (c)). Emit the smoothed corrected current frame to C8 for FC delivery; emit smoothed past-keyframes to C13 (FDR only — AC-4.5 internal smoothing, NOT FC retroactive correction).
|
||||
|
||||
**Architectural Pattern**: Strategy with two concrete implementations: `GtsamIsam2StateEstimator` (production-default) and `EskfStateEstimator` (mandatory simple-baseline). Selection at startup (ADR-001), `BUILD_*` gating (ADR-002), composition-root wired (ADR-009).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C1 → `VioOutput` (relative pose + IMU bias).
|
||||
- C4 → `PoseEstimate` (absolute satellite-anchored pose); C4 adds factors directly to C5's iSAM2 graph (shared substrate).
|
||||
- C8 inbound side → FC `ImuWindow` + `AttitudeWindow` + `GpsHealth` (for warm-start AC-5.1, blackout AC-NEW-8, spoofing-promotion AC-NEW-2 / F7).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C8 outbound side (per-FC encoder) → `EmittedExternalPosition` (5 Hz periodic to FC).
|
||||
- C6 (mid-flight tile gen via orthorectifier; C5 supplies the `PoseEstimate` + quality_metadata for tile emission).
|
||||
- C13 FDR (smoothed past-keyframe estimates, source-set switch events, spoofing-rejection events).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `StateEstimator`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `add_vio` | `VioOutput` | `None` | No | `EstimatorDegradedError`, `EstimatorFatalError` |
|
||||
| `add_pose_anchor` | `PoseEstimate` | `None` | No | `EstimatorDegradedError`, `EstimatorFatalError` |
|
||||
| `add_fc_imu` | `ImuWindow` | `None` | No | `EstimatorDegradedError` |
|
||||
| `current_estimate` | `()` | `EstimatorOutput` (smoothed current keyframe) | No | — |
|
||||
| `smoothed_history` | `n_keyframes: int` | `list[EstimatorOutput]` | No | — |
|
||||
| `health_snapshot` | `()` | `EstimatorHealth` | No | — |
|
||||
|
||||
**Input DTOs**: see C1, C4, C8.
|
||||
|
||||
**Output DTOs**:
|
||||
```
|
||||
EstimatorOutput:
|
||||
frame_id: uuid
|
||||
position_wgs84: LatLonAlt
|
||||
orientation_world_T_body: Quat (w, x, y, z)
|
||||
velocity_world: Vector3 (m/s)
|
||||
covariance_6x6: Matrix6
|
||||
source_label: enum {satellite_anchored, visual_propagated, dead_reckoned}
|
||||
last_satellite_anchor_age_ms: int
|
||||
smoothed: bool — true for entries from `smoothed_history`
|
||||
emitted_at: monotonic_ns
|
||||
|
||||
EstimatorHealth:
|
||||
isam2_state: enum {INIT, TRACKING, DEGRADED, LOST}
|
||||
keyframe_count: int
|
||||
cov_norm_growing_for_s: float — AC-NEW-8 monotonicity check
|
||||
spoof_promotion_blocked: bool — AC-NEW-2 / AC-NEW-8 gate state
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
C5 holds the GTSAM iSAM2 state in memory; persistent storage is only via FDR writes (C13 owns the file). No DB queries.
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| In-memory keyframe window | up to 20 keyframes resident | ~2 KB / keyframe (factors + values) | ~40 KB | bounded by IncrementalFixedLagSmoother K=10–20 |
|
||||
|
||||
C5 is bounded by design — no unbounded growth.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**:
|
||||
- iSAM2 update on factor add: amortised `O(K)` in keyframe count for the typical case; `O(K^2)` worst-case on relinearisation.
|
||||
- `Marginals.marginalCovariance(pose_key)`: `O(K^3)` in keyframe-window size; the dominant per-frame cost (~30–90 ms steady-state).
|
||||
- `IncrementalFixedLagSmoother` keeps the active window bounded — older keyframes are marginalised out.
|
||||
|
||||
**State Management**:
|
||||
- iSAM2 graph + Values + Marginals lifecycle for the flight.
|
||||
- Source-label state machine: tracks the AC-NEW-2 / AC-NEW-8 spoofing-promotion gate (≥10 s + visual consistency check before re-promoting a previously-spoofed FC GPS source).
|
||||
- Last-anchor-age timer for AC-1.3 binning.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| GTSAM (Python + C++) | per Plan-phase pin | iSAM2 + `CombinedImuFactor` + `BetweenFactorPose3` + `GenericProjectionFactorCal3DS2` + `Marginals` |
|
||||
| `gtsam_unstable.IncrementalFixedLagSmoother` | per Plan-phase pin | Bounded keyframe window (D-C5-3 K=10–20) |
|
||||
| Eigen | matches GTSAM | Lie-algebra math |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `EstimatorDegradedError`: factor add yielded poor convergence; covariance inflated; emit `EstimatorOutput` with degraded label.
|
||||
- `EstimatorFatalError`: iSAM2 numerical failure, KEYFRAME_LIMIT exceeded, etc.; emit no `EstimatorOutput` for this tick. AC-5.2 fallback (3 s no estimate → FC IMU-only) applies.
|
||||
- Spoof-promotion gate: never re-introduce a previously-spoofed FC GPS source until BOTH (i) FC `gps_health == STABLE_NON_SPOOFED` for ≥ 10 s AND (ii) the next satellite-anchored frame agrees with the FC GPS within a configurable tolerance (AC-NEW-8). Document every reject in FDR + GCS STATUSTEXT.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `ImuPreintegrator` | shared with C1 | C1, C5 |
|
||||
| `SE3Utils` | shared with C1, C4 | C1, C4, C5 |
|
||||
| `WgsConverter` | shared with C4, C8 | C4, C5, C8 |
|
||||
| `SourceLabelStateMachine` | spoofing-promotion gate logic | C5 only — keep inside the component |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- AC-4.5 internal smoothing is **onboard only**; the FC log is forward-time only. The smoothed past-keyframe estimates go to FDR, not back to the FC.
|
||||
- iSAM2 + `IncrementalFixedLagSmoother` requires careful key management; missing keys cause silent factor-add failures — the implementation MUST log every `add_*` call's success/failure status.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Single writer thread for the iSAM2 graph by design. C1 + C4 + C8-inbound deliver to a timestamp-ordered merge queue ahead of C5's writer thread.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- `Marginals.marginalCovariance(pose_key)` is the per-frame hot spot. D-CROSS-LATENCY-1 hybrid degrades C4's covariance recovery (not C5's) under thermal throttle.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C1 (input), C4 (input + shared graph), C8 inbound (FC IMU prior).
|
||||
|
||||
**Can be implemented in parallel with**: C6, C13 — independent paths.
|
||||
|
||||
**Blocks**: C8 outbound (no per-frame estimate), F3 / F4 / F5 / F7 / F9 / F10.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `EstimatorFatalError`; iSAM2 numerical failure; AC-5.2 path imminent | `C5 fatal iSAM2 failure; frame=12345; AC-5.2 fallback` |
|
||||
| WARN | `EstimatorDegradedError`; spoofing-promotion blocked; cov norm growing >2× steady | `C5 degraded: cov_inflation=3.1, spoof_block=true` |
|
||||
| INFO | Strategy ready; warm-start applied; spoof-promotion gate state changes | `C5 ready: estimator=gtsam_isam2, K=15` |
|
||||
| DEBUG | per-frame factor adds + smoothed history depth | `C5 frame=12345 vio_added=true pose_added=true imu_added=true smoothed_n=15` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN always; smoothed past-keyframe entries always go to FDR per AC-4.5; spoofing-promotion-block events go to FDR + GCS STATUSTEXT).
|
||||
@@ -0,0 +1,189 @@
|
||||
# Test Specification — C5 State Estimator
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-1.3 | Cumulative drift between satellite-anchored fixes | FT-P-02, **C5-IT-01** | Covered |
|
||||
| AC-1.4 | 95% covariance + source label | FT-P-03, **C5-IT-02** | Covered |
|
||||
| AC-3.5 | Visual blackout + spoofed-GPS failsafe | FT-N-04, **C5-IT-03** | Covered |
|
||||
| AC-4.5 (revised) | Internal smoothing of past keyframes (NOT FC retroactive correction) | FT-P-10, **C5-IT-04** | Covered |
|
||||
| AC-5.2 | On >3 s without estimate, FC IMU-only fallback | NFT-RES-01, **C5-IT-05** | Covered |
|
||||
| AC-NEW-2 | Spoofing-promotion latency <3 s p95 | NFT-PERF-04, **C5-IT-06** | Covered |
|
||||
| AC-NEW-8 | Visual blackout + spoof degraded-mode escalation | FT-N-04, NFT-RES-04, **C5-IT-07** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C5-IT-01: source_label state machine produces correct `last_satellite_anchor_age_ms`
|
||||
|
||||
**Summary**: after a satellite-anchored frame, `last_satellite_anchor_age_ms` resets to the frame's age; under visual-propagated frames it monotonically increases.
|
||||
|
||||
**Traces to**: AC-1.3 (binning input)
|
||||
|
||||
**Description**: scripted sequence — 3 satellite-anchored frames, then 30 s of visual-propagated, then another satellite-anchored. Assert `last_satellite_anchor_age_ms` resets at the anchored events and rises monotonically between them with ms-level resolution.
|
||||
|
||||
**Input data**: scripted `EstimatorOutput` sequence.
|
||||
|
||||
**Expected result**: monotonic between resets; resets within 100 ms of the anchored frame's `emitted_at`.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-02: smoothed-current estimate honest covariance
|
||||
|
||||
**Summary**: `current_estimate()` produces an SPD 6×6 covariance whose norm reflects the iSAM2 graph's actual posterior — not a fake-confidence value.
|
||||
|
||||
**Traces to**: AC-1.4
|
||||
|
||||
**Description**: build a synthetic graph where the keyframe-11 absolute factor is intentionally noisy (3× covariance); assert C5's emitted covariance norm is at least 2× the steady-state norm of a clean graph. Repeat with a clean factor; assert ≤1.2× steady-state.
|
||||
|
||||
**Input data**: synthetic factor-graph fixtures.
|
||||
|
||||
**Expected result**: noisy → ≥2× norm; clean → ≤1.2× norm.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-03: VIO-only fallback under cross-domain matcher failure
|
||||
|
||||
**Summary**: when C4 stops emitting `PoseEstimate` (matcher failure), C5 continues with VIO-only and labels `source_label = visual_propagated`.
|
||||
|
||||
**Traces to**: AC-3.5
|
||||
|
||||
**Description**: feed `add_vio` for 60 s while withholding `add_pose_anchor` calls; assert (a) `current_estimate` keeps emitting, (b) `source_label == visual_propagated`, (c) `cov_norm_growing_for_s` rises monotonically.
|
||||
|
||||
**Input data**: scripted VIO-only fixture.
|
||||
|
||||
**Expected result**: 60 s of visual_propagated estimates; cov norm monotonically rising.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-04: smoothed past-keyframe history is NOT forwarded to FC
|
||||
|
||||
**Summary**: `smoothed_history(n)` reflects past-keyframe smoothing per AC-4.5 (revised), but the FC emission path uses `current_estimate` only — the smoothing must NOT alter what C8 emits as `GPS_INPUT` / `MSP2_SENSOR_GPS`.
|
||||
|
||||
**Traces to**: AC-4.5 (revised)
|
||||
|
||||
**Description**: trigger an iSAM2 relinearisation that materially shifts a 5-keyframe-old pose; assert (a) `smoothed_history(10)` shows the shift, (b) the next 10 calls to `current_estimate` are unaffected, (c) the FDR record stream contains the smoothed history (per AC-4.5) AND the unshifted FC emissions in the same flight log.
|
||||
|
||||
**Input data**: synthetic graph with a deliberately-late loop closure.
|
||||
|
||||
**Expected result**: history shows shift; current_estimate unaffected; FDR has both streams.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-05: 3 s no-estimate threshold triggers AC-5.2 fallback
|
||||
|
||||
**Summary**: when no `add_vio` and no `add_pose_anchor` arrive for >3 s, `current_estimate` returns a `dead_reckoned`-labeled output (or refuses to emit) so C8 can fall to FC IMU-only.
|
||||
|
||||
**Traces to**: AC-5.2
|
||||
|
||||
**Description**: prime C5 with a normal warm-start; cease all input for 4 s; observe `current_estimate` over the gap; assert (a) at < 3 s gap, label is `visual_propagated`, (b) at ≥ 3 s gap, label transitions to `dead_reckoned` or no emission, (c) the transition timestamp is logged at ERROR level.
|
||||
|
||||
**Input data**: scripted gap fixture.
|
||||
|
||||
**Expected result**: transition at gap ≥ 3 s; ERROR logged.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-06: spoof-promotion gate enforces ≥10 s + visual consistency
|
||||
|
||||
**Summary**: a previously-spoofed FC GPS source can only be re-promoted to trusted after BOTH (i) `gps_health == STABLE_NON_SPOOFED` for ≥10 s AND (ii) the next satellite-anchored frame agrees with FC GPS within tolerance.
|
||||
|
||||
**Traces to**: AC-NEW-2, AC-NEW-8
|
||||
|
||||
**Description**: scripted scenario — initial trust → spoof event (gps_health = SPOOFED) → recovery to STABLE_NON_SPOOFED at t=0; satellite-anchored agreement frames at t=5 s, t=11 s. Assert promotion blocks until t=11 s + agreement; reject every promotion attempt before then; log every reject in FDR + STATUSTEXT.
|
||||
|
||||
**Input data**: scripted `gps_health` + `EstimatorOutput` sequence.
|
||||
|
||||
**Expected result**: promotion at t=11 s; rejects logged before then.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C5-IT-07: visual blackout + spoof escalation
|
||||
|
||||
**Summary**: simultaneous visual blackout (no C4 anchors) and spoofed FC GPS escalates to `dead_reckoned` source label and AC-NEW-8 STATUSTEXT.
|
||||
|
||||
**Traces to**: AC-NEW-8
|
||||
|
||||
**Description**: combine the visual-blackout fixture (no `add_pose_anchor` for 5 s) with a `gps_health == SPOOFED` event; assert the next emission is `dead_reckoned` and an AC-NEW-8 STATUSTEXT is published via the C8 path (mocked C8 in the test harness records the outgoing message).
|
||||
|
||||
**Input data**: combined scripted fixture.
|
||||
|
||||
**Expected result**: `dead_reckoned` label; STATUSTEXT recorded in mock C8 harness.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C5-PT-01: iSAM2 + Marginals throughput on Tier-2
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: 3 Hz `add_pose_anchor` + 200 Hz `add_fc_imu` + 3 Hz `add_vio`; 10 min replay.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `add_pose_anchor` + `current_estimate` p95 | ≤ 60 ms | 100 ms |
|
||||
| `add_fc_imu` p95 | ≤ 1 ms (preintegration buffer-add only) | 5 ms |
|
||||
| `add_vio` p95 | ≤ 5 ms | 15 ms |
|
||||
|
||||
**Resource limits**:
|
||||
- Memory: bounded by `IncrementalFixedLagSmoother K=10–20`; ≤ 100 MB resident.
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C5-ST-01: spoof-rejection logging cannot be silenced
|
||||
|
||||
**Summary**: every spoof-promotion-block event lands in FDR + GCS STATUSTEXT — the system has no config knob that disables this.
|
||||
|
||||
**Traces to**: AC-NEW-2 / AC-NEW-8 (defensive)
|
||||
|
||||
**Test procedure**:
|
||||
1. Configure C5 with the production-default config.
|
||||
2. Inject a spoof-promotion-block event.
|
||||
3. Assert the FDR record stream contains an entry with `kind = "spoof_promotion_block"` AND a STATUSTEXT was issued.
|
||||
4. Search the codebase for any config flag that could disable either path; assert no such flag exists.
|
||||
|
||||
**Pass criteria**: FDR + STATUSTEXT both recorded; no disabling config flag found.
|
||||
**Fail criteria**: either path missing or a disabling flag exists.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-02 / FT-P-03 / FT-N-04 / NFT-RES-01.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Synthetic factor-graph fixtures | scripted | <5 MB |
|
||||
| `flight_derkachi/normal_segment_60_stills/` (replayed VIO + pose feeds) | shared | shared |
|
||||
| `gps_health` event fixtures | scripted | <1 MB |
|
||||
|
||||
**Setup**: in-process; no external services.
|
||||
**Teardown**: per-test temp dirs.
|
||||
**Data isolation**: each test instantiates a fresh `StateEstimator`.
|
||||
@@ -0,0 +1,172 @@
|
||||
# C6 — Tile Cache + Spatial Index
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: own the on-companion persistent imagery store (tiles + descriptor index + tile metadata) in a layout that is byte-identical to `satellite-provider`'s on-disk format, so the C11 `TileUploader` post-landing upload (F10) is a straight copy. Provide read access to C2 (descriptor index), C2.5 / C3 (tile pixels), and write access to the orthorectifier path during F4 (mid-flight tile generation), to the C11 `TileDownloader` during F1 (pre-flight tile fetch), and to C10 during F1 (Manifest + FAISS index write).
|
||||
|
||||
**Architectural Pattern**: Repository — three concrete stores behind separate interfaces (`TileStore` for pixel + metadata I/O; `TileMetadataStore` for the Postgres spatial index; `DescriptorIndex` for FAISS HNSW). Single concrete implementation per interface today (`PostgresFilesystemStore`, `FaissDescriptorIndex`); future variants (e.g., RocksDB-backed metadata for resource-constrained tiers) can be added behind the same interfaces.
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C11 `TileDownloader` (writes `tiles` rows + JPEGs during F1 pre-flight provisioning, source='googlemaps').
|
||||
- C10 CacheProvisioner (writes Manifest + FAISS index during F1 pre-flight provisioning, after C11 has populated tiles).
|
||||
- C5 → orthorectifier → C6 (writes during F4 mid-flight tile generation, source='onboard_ingest').
|
||||
- C11 `TileUploader` (reads during F10).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C2 VPR (reads descriptor index).
|
||||
- C2.5 ReRanker (reads tile pixels).
|
||||
- C3 CrossDomainMatcher (reads tile pixels).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `TileStore`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `read_tile_pixels` | `tile_id: TileId` | `TilePixelHandle` (mmap-backed) | No | `TileNotFoundError`, `TileFsError` |
|
||||
| `write_tile` | `tile_blob: bytes, metadata: TileMetadata` | `None` | No | `TileFsError`, `TileMetadataError` |
|
||||
| `tile_exists` | `tile_id: TileId` | `bool` | No | — |
|
||||
|
||||
### Interface: `TileMetadataStore`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `query_by_bbox` | `bbox, zoom` | `list[TileMetadata]` | No | `TileMetadataError` |
|
||||
| `insert_metadata` | `TileMetadata` | `None` | No | `TileMetadataError` |
|
||||
| `mark_voting_status` | `tile_id, status: VotingStatus` | `None` | No | `TileMetadataError` |
|
||||
|
||||
### Interface: `DescriptorIndex`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `search_topk` | `query: ndarray[D], k: int` | `list[(tile_id, distance)]` | No | `IndexUnavailableError` |
|
||||
| `descriptor_dim` | `()` | `int` | No | — |
|
||||
| `mmap_handle` | `()` | filesystem path / FAISS reader handle | No | — |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
```
|
||||
TileId: composite (zoomLevel: int, lat: float, lon: float)
|
||||
|
||||
TileMetadata:
|
||||
tile_id: TileId
|
||||
tile_size_meters: float
|
||||
tile_size_pixels: int
|
||||
capture_timestamp: ISO 8601 datetime
|
||||
source: enum {googlemaps, onboard_ingest}
|
||||
freshness_label: enum {fresh, stale_active_conflict, stale_rear, downgraded}
|
||||
flight_id: uuid (optional, set for onboard_ingest)
|
||||
companion_id: string (optional, set for onboard_ingest)
|
||||
quality_metadata: TileQualityMetadata (optional, set for onboard_ingest)
|
||||
voting_status: enum {pending, trusted, rejected} (default pending for onboard_ingest)
|
||||
|
||||
TileQualityMetadata: see data_model.md (TileQualityMetadata entity)
|
||||
|
||||
TilePixelHandle: opaque (filesystem path + mmap pointer; consumer must not copy)
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable — internal-only; C11 `TileUploader` reads via `TileStore` for upload to `satellite-provider` over an external HTTP API, and C11 `TileDownloader` writes via `TileStore` after fetching from `satellite-provider`. C11 owns those API calls, not C6.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
### Queries
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| `read_tile_pixels` (3 candidates per frame from C2.5/C3) | 9 Hz | Yes | Filesystem mmap (page-cache backed) |
|
||||
| `search_topk` from C2 (top-K=10) | 3 Hz | Yes | FAISS HNSW (.index file) |
|
||||
| `query_by_bbox` from C10/C11/C12 (cache build + post-landing) | offline / pre-flight | No | btree spatial index on (zoomLevel, lat, lon) |
|
||||
| `insert_metadata` + `write_tile` from F4 orthorectifier | bursty during flight (≤ a few Hz on average) | No (background) | btree spatial index |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| Tile pixels | OS page cache (filesystem mmap) | flight lifetime | None — files are append-only during flight; F1 wipes for next flight |
|
||||
| FAISS descriptor index | mmap (`IO_FLAG_MMAP_IFC`) | flight lifetime | F1 rebuild + content-hash gate |
|
||||
| Tile metadata rows | Postgres internal page cache + connection pool | flight lifetime | F4 inserts append; F1 may truncate-and-reload |
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| `tiles` (Postgres rows) | ~ tens to hundreds of thousands per cached area | ~256 B per row | up to ~100 MB | bounded by 10 GB AC-8.3 cache budget overall |
|
||||
| `tiles/{zoomLevel}/{x}/{y}.jpg` | matches `tiles` row count | 50–200 KB / tile | up to ~9 GB (after metadata/index overheads) | bounded by AC-8.3 |
|
||||
| FAISS HNSW `.index` | one file per provisioning | 200 MB–1 GB depending on backbone descriptor dim | bounded by AC-8.3 carve-out per D-C2-6/9/10 | F1 atomic rebuild |
|
||||
| Onboard mid-flight ingest tiles (F4) | a few hundred per flight | same as above | bounded by AC-8.3 carve-out | per flight |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: F1 pre-flight provisioning is the seeding step (operator runs C12 → C11 `TileDownloader` (tiles) → C10 (engines + descriptors + manifest) → C6).
|
||||
|
||||
**Rollback**: F1 is idempotent (D-C10-1 manifest-hash check). Mid-flight writes during F4 are append-only; on F8 reboot, partially-written tiles are detected via the SHA-256 content-hash gate (D-C10-3) at takeoff, and the corrupted tile is dropped.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**:
|
||||
- HNSW search: `O(log N)` in corpus size.
|
||||
- bbox query: btree index `O(log N)` per probe; result-set scan `O(R)` in matched rows.
|
||||
|
||||
**State Management**:
|
||||
- `TileStore` is stateless (filesystem-backed).
|
||||
- `TileMetadataStore` holds a Postgres connection pool.
|
||||
- `DescriptorIndex` holds the FAISS reader + mmap'd index file handle; lifetime = flight.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| PostgreSQL (server + libpq) | 16.x (mirror of `satellite-provider`'s pin) | Spatial metadata index |
|
||||
| psycopg / asyncpg | per project pin | Python Postgres client |
|
||||
| FAISS (Python + C++) | upstream HEAD pinned per Plan-phase | HNSW retrieval |
|
||||
| atomicwrites | latest | Atomic file replacement for `.index` rebuild (D-C10-3) |
|
||||
| hashlib (stdlib) | stdlib | SHA-256 content-hash sidecars |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `TileNotFoundError`: tile pixel file missing despite metadata row present. Log + drop the candidate; signal cache inconsistency to C13 for FDR.
|
||||
- `TileFsError`: I/O error on filesystem read/write. F4 writes retry once with backoff; reads do not retry (the candidate is dropped).
|
||||
- `TileMetadataError`: Postgres failure. Pre-flight: F1 fails fast; takeoff blocked. Mid-flight: F4 writes drop the tile (logged); reads for C2's top-K fail to enrich and downstream uses descriptor distances only (degraded but not fatal).
|
||||
- `IndexUnavailableError`: FAISS handle invalid (e.g., file replaced concurrently). Treated as fatal in flight; F8 reboot recovery re-mmaps.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `WgsConverter` | shared with C4, C5, C8 | C4, C5, C6 (bbox queries), C8 |
|
||||
| `Sha256Sidecar` | atomic write + SHA-256 content-hash sidecar pattern (D-C10-3) | C6, C10 |
|
||||
| `OrthorectifierUtils` | minimal orthorectification math (used by F4 write path) | C5, C6 — keep inside the F4 boundary; if needed by more components, promote |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- The on-disk layout MUST mirror `satellite-provider` exactly so F10 upload is byte-identical. Any deviation breaks AC-8.4. Verified by NFT-SEC-01 and IT-7.
|
||||
- Postgres on a Jetson is heavyweight; runtime resource budget validated by NFT-LIM-01 (8 h replay).
|
||||
|
||||
**Potential race conditions**:
|
||||
- F4 writes can race with C2/C2.5/C3 reads on the same tile filesystem. Solution: F4 writes use `atomicwrites` for atomic rename; readers see either the old version or the new version, never partial.
|
||||
- FAISS `.index` mmap is read-only during a flight; F1 would never overwrite during a flight (F1 happens pre-flight only).
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Postgres metadata writes during F4 are bursty; if bursts approach 10 Hz, the Postgres connection becomes a bottleneck. Bound by configurable backpressure on the F4 path.
|
||||
- FAISS HNSW first query pays the page-in cost (multi-second). F2 takeoff load forces a warm-up query so F3 first frame is cheap.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: external `satellite-provider` on-disk layout reference (read its README + table schema; mirror exactly).
|
||||
|
||||
**Can be implemented in parallel with**: C7 — independent paths.
|
||||
|
||||
**Blocks**: C2, C2.5, C3, C10, C11 (both `TileDownloader` and `TileUploader`), F1, F2, F3, F4, F8, F10.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `TileNotFoundError` (cache inconsistency); `IndexUnavailableError` | `Tile pixel missing for tile_id=(z=18,lat=49.71,lon=37.45)` |
|
||||
| WARN | F4 write retried; bbox query slower than threshold | `F4 tile write retry: tile_id=…; reason=fs_busy` |
|
||||
| INFO | Provisioning complete; index loaded | `C6 ready: tiles=87654, faiss_dim=512, index_size_mb=412` |
|
||||
| DEBUG | per-query timing | `C6 search_topk frame=12345 took=0.4ms; bbox_query took=2.1ms` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN only; the FDR also captures every successful F4 write as part of the mid-flight-tile-gen log).
|
||||
@@ -0,0 +1,168 @@
|
||||
# Test Specification — C6 Tile Cache + Spatial Index
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-8.1 | Imagery via Suite Sat Service offline cache, ≥0.5 m/px | FT-P-15, FT-P-16, **C6-IT-01** | Covered |
|
||||
| AC-8.2 | Tile freshness <6 mo (active-conflict) / <12 mo (rear) | FT-N-05, **C6-IT-02** | Covered |
|
||||
| AC-8.4 | Mid-flight tile generation with quality metadata | FT-P-17, **C6-IT-03** | Covered |
|
||||
| AC-NEW-3 | FDR ≤64 GB / flight, no silent drops (mid-flight tiles count toward C13 — C6 must not block) | NFT-LIM-02, **C6-IT-04** | Covered |
|
||||
| AC-NEW-6 | System rejects/downgrades stale tiles | FT-N-05, FT-N-06, **C6-IT-05** | Covered |
|
||||
| RESTRICT-SAT-2 | Cache budget 10 GB across operational area | NFT-LIM-03, **C6-IT-06** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C6-IT-01: filesystem layout byte-identical to `satellite-provider`
|
||||
|
||||
**Summary**: tiles written by C11 `TileDownloader` and tiles emitted at mid-flight by C5/C6 produce identical filesystem paths and JPEG byte content for the same `(zoomLevel, lat, lon)` coordinate.
|
||||
|
||||
**Traces to**: AC-8.1 (interop with `satellite-provider` for the upload return path)
|
||||
|
||||
**Description**: download a known tile via C11 TileDownloader (against the real `satellite-provider` mock fixture); generate the same tile via the mid-flight orthorectification path (C5 + C6); assert (a) filesystem path equality, (b) JPEG bytes equality, (c) `tiles` table row equality on `(zoomLevel, lat, lon, source, content_sha256)`.
|
||||
|
||||
**Input data**: a single Derkachi-area tile (`zoom=18, lat≈49.94, lon≈36.31`).
|
||||
|
||||
**Expected result**: byte-identical layout.
|
||||
|
||||
**Max execution time**: 10 s.
|
||||
|
||||
---
|
||||
|
||||
### C6-IT-02: freshness gate at write-time
|
||||
|
||||
**Summary**: a tile with `produced_at < now - active_conflict_max_age` is rejected at C6 insert if the target sector is `active_conflict`.
|
||||
|
||||
**Traces to**: AC-8.2
|
||||
|
||||
**Description**: insert a synthetic tile with `produced_at = now - 7 months` into an `active_conflict` sector; assert C6 raises a freshness-rejection error and does NOT write to disk or DB. Repeat with `stable_rear` — assert the tile is written but flagged `freshness_downgraded`.
|
||||
|
||||
**Input data**: synthetic tile fixtures with controlled timestamps.
|
||||
|
||||
**Expected result**: rejection in active_conflict; downgrade-write in stable_rear.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C6-IT-03: mid-flight tile insert with quality metadata
|
||||
|
||||
**Summary**: every tile written via the F4 mid-flight gen path has full `quality_metadata` (covariance, inlier count, source-label provenance).
|
||||
|
||||
**Traces to**: AC-8.4
|
||||
|
||||
**Description**: simulate F4 — feed C5 a 60 s sequence; the orthorectifier emits N tiles; assert each tile's row in `tiles` has non-null `quality_metadata.covariance_norm`, `quality_metadata.inlier_count`, `quality_metadata.source_label`. Empty quality metadata fails the test.
|
||||
|
||||
**Input data**: scripted F4 fixture.
|
||||
|
||||
**Expected result**: 100% of mid-flight tiles have full metadata.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
### C6-IT-04: write throughput exceeds peak F4 demand
|
||||
|
||||
**Summary**: C6 sustains the peak mid-flight write rate (1 tile / 2 s typical, ~5/s peak) without queueing-induced backpressure on C5.
|
||||
|
||||
**Traces to**: AC-NEW-3 (no-silent-drop gate)
|
||||
|
||||
**Description**: synthetic burst — 100 tiles enqueued at 5 Hz; assert (a) all 100 land in `tiles` table within 30 s, (b) no tile dropped, (c) C5's write-side queue depth never exceeds the configured ceiling.
|
||||
|
||||
**Input data**: synthetic burst fixture.
|
||||
|
||||
**Expected result**: 100/100 written; no drops.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C6-IT-05: stale-tile reject + downgrade across the F1 / F4 / F5 boundary
|
||||
|
||||
**Summary**: in F5 (steady-state retrieval), a tile flagged `freshness_downgraded` is returned with the downgrade flag, and any consumer that depends on freshness (e.g., spoof-rejection check) sees the flag.
|
||||
|
||||
**Traces to**: AC-NEW-6
|
||||
|
||||
**Description**: write a `freshness_downgraded` tile via C6-IT-02's stable_rear path; query via the C2/C3 fetch path; assert the returned tile carries the downgrade flag and that the consumer-side label propagates into the `EstimatorOutput.source_label`.
|
||||
|
||||
**Input data**: as C6-IT-02.
|
||||
|
||||
**Expected result**: downgrade flag survives the round-trip.
|
||||
|
||||
**Max execution time**: 10 s.
|
||||
|
||||
---
|
||||
|
||||
### C6-IT-06: 10 GB cache budget enforcement
|
||||
|
||||
**Summary**: when the operational-area working set hits 10 GB, C6 evicts oldest tiles by LRU; never silently dropping a tile that would put the system over budget.
|
||||
|
||||
**Traces to**: RESTRICT-SAT-2
|
||||
|
||||
**Description**: synthetic load that fills the cache to 10 GB - 50 MB; insert another 100 MB of tiles; assert (a) total disk usage stays ≤ 10 GB, (b) eviction count matches insert count above the threshold, (c) every eviction is logged at INFO with the evicted tile_id.
|
||||
|
||||
**Input data**: synthetic tile generator.
|
||||
|
||||
**Expected result**: 10 GB cap held; LRU eviction visible in logs.
|
||||
|
||||
**Max execution time**: 5 min on Tier-1 (disk-bound).
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C6-PT-01: per-tile read latency on Tier-2 (mmap hit)
|
||||
|
||||
**Traces to**: AC-4.1 (read-side; mmap is the design assumption for C2/C2.5/C3 hot paths)
|
||||
|
||||
**Load scenario**: 3 Hz × 10 candidate tiles per frame × 10 min replay; OS page cache warm.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `get_tile_pixels` p95 (warm) | ≤ 0.5 ms | 5 ms |
|
||||
| `get_tile_pixels` p95 (cold first read) | ≤ 50 ms | 200 ms |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C6-ST-01: content-sha256 mismatch rejects insert
|
||||
|
||||
**Summary**: a tile insert with a content_sha256 that doesn't match the actual JPEG bytes is rejected at C6 (defends against the cache-poisoning path D-C10-3 covers at takeoff).
|
||||
|
||||
**Traces to**: defensive (D-C10-3 + AC-NEW-7 cumulative)
|
||||
|
||||
**Test procedure**:
|
||||
1. Compute SHA-256 of a known JPEG.
|
||||
2. Submit insert with the JPEG bytes + a deliberately-wrong SHA-256.
|
||||
3. Assert insert is rejected and the DB row is not created.
|
||||
|
||||
**Pass criteria**: insert rejection + no row.
|
||||
**Fail criteria**: insert accepted or partial row created.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-15 / FT-P-16 / FT-P-17 / FT-N-05 / FT-N-06.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Single Derkachi tile (`zoom=18`) | curated | <1 MB |
|
||||
| Synthetic tile fixtures with controlled timestamps | scripted | ~50 MB |
|
||||
| Synthetic 10 GB cache fill set | scripted, deterministic | 10 GB on disk |
|
||||
| `flight_derkachi/normal_segment_60_stills/` for F4 simulation | shared | shared |
|
||||
|
||||
**Setup**: ephemeral Postgres + filesystem under `tests/tmp/c6/<test-id>/db/` and `…/tiles/`.
|
||||
**Teardown**: drop tmp directory.
|
||||
**Data isolation**: per-test ephemeral DB + filesystem.
|
||||
@@ -0,0 +1,155 @@
|
||||
# C7 — On-Jetson Inference Runtime
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: provide a single inference-runtime abstraction that all GPU-using components (C1 selectively, C2, C2.5, C3, C3.5) consume. Owns engine compilation (Polygraphy / trtexec / IBuilderConfig hybrid), engine deserialization at takeoff load, GPU memory management, INT8 calibration cache trust, and the thermal-throttle telemetry feed that drives the D-CROSS-LATENCY-1 hybrid in C4.
|
||||
|
||||
**Architectural Pattern**: Strategy — `InferenceRuntime` interface with three concrete implementations: `TensorrtRuntime` (production-default per D-C7-9 JetPack 6.2 + TensorRT 10.3 lock), `OnnxTrtEpRuntime` (fallback), `PytorchFp16Runtime` (mandatory simple-baseline). Selection at startup by config (ADR-001), build-time gating by `BUILD_*` flags (ADR-002), composition-root wired (ADR-009).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C10 CacheProvisioner → during F1 (after C11 `TileDownloader` has populated C6) triggers engine compilation when no cached engine matches the `(SM, JP, TRT, precision)` tuple.
|
||||
- F2 takeoff load → triggers `deserialize_cached_engine` for every model used by C1/C2/C2.5/C3/C3.5.
|
||||
- jetson-stats / NVML → thermal-throttle telemetry source.
|
||||
|
||||
**Downstream consumers**:
|
||||
- C2 VPR (backbone forward pass).
|
||||
- C2.5 ReRanker (LightGlue forward pass).
|
||||
- C3 CrossDomainMatcher (DISK / LightGlue / ALIKED / XFeat forward passes).
|
||||
- C3.5 AdHoP (conditional refinement backbone).
|
||||
- C1 (only the strategies that have a CUDA path; KltRansac is CPU-only).
|
||||
- C4 (consumes `ThermalState` for the D-CROSS-LATENCY-1 covariance-mode decision).
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `InferenceRuntime`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `compile_engine` | `model_path: Path, build_config: BuildConfig` | `EngineCacheEntry` | No (offline) | `EngineBuildError`, `CalibrationCacheError` |
|
||||
| `deserialize_engine` | `EngineCacheEntry` | `EngineHandle` | No | `EngineDeserializeError` |
|
||||
| `infer` | `EngineHandle, inputs: dict[str, Tensor]` | `dict[str, Tensor]` | No (sync GPU stream) | `InferenceError`, `OutOfMemoryError` |
|
||||
| `release_engine` | `EngineHandle` | `None` | No | — |
|
||||
| `thermal_state` | `()` | `ThermalState` | No | `TelemetryUnavailableError` |
|
||||
| `current_runtime_label` | `()` | `string` | No | — |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
```
|
||||
BuildConfig:
|
||||
precision: enum {fp16, int8, mixed}
|
||||
workspace_mb: int
|
||||
calibration_dataset: Path (required for int8)
|
||||
optimization_profiles: list[(input_name, min_shape, opt_shape, max_shape)]
|
||||
|
||||
EngineCacheEntry: see data_model.md
|
||||
EngineHandle: opaque GPU-resident handle
|
||||
|
||||
ThermalState:
|
||||
cpu_temp_c: float
|
||||
gpu_temp_c: float
|
||||
thermal_throttle_active: bool
|
||||
measured_clock_mhz: int
|
||||
measured_at: monotonic_ns
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
### Queries
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| `infer` for VPR backbone | 3 Hz | Yes | n/a |
|
||||
| `infer` for LightGlue (×10 in C2.5, ×3 in C3) | 3 Hz × 13 = 39 Hz | Yes | n/a |
|
||||
| `infer` for AdHoP (conditional) | <1 Hz typical | Yes (when invoked) | n/a |
|
||||
| `thermal_state` poll | 1 Hz from C4 | No (sampled, not per-frame) | n/a |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| Compiled `.engine` files | filesystem keyed by `(SM, JP, TRT, precision)` (D-C10-7) | bounded by JetPack/TRT version stability | manifest content-hash gate at takeoff (D-C10-3) |
|
||||
| INT8 calibration cache | filesystem alongside `.engine` (D-C10-6) | bounded by calibration dataset version | rebuild when calibration dataset hash changes |
|
||||
| Resident engine handles | GPU memory | flight lifetime | F8 reboot recovery re-deserialises |
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| `.engine` files | one per (model × precision × backbone) | 50 MB – 500 MB / engine | up to ~1.5 GB across all backbones for a deployment binary | bounded by AC-8.3 carve-out |
|
||||
| INT8 calibration caches | one per engine | 1–10 MB | <50 MB | as above |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: pre-flight F1 provisioning compiles engines (or reuses cached). No mid-flight compilation.
|
||||
|
||||
**Rollback**: D-C10-7 self-describing filename schema (`<model>__sm<SM>_jp<JP>_trt<TRT>_<precision>.engine`) makes stale engines visually obvious; F2 takeoff load refuses to deserialize an engine whose metadata doesn't match the host's current `(SM, JP, TRT)` tuple.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: per-model forward pass cost is the design driver. Engine builds are `O(complexity_of_optimizer_search)` — minutes for INT8 with calibration; sub-minute for FP16.
|
||||
|
||||
**State Management**:
|
||||
- Owns the CUDA stream(s) for the runtime; one stream per concurrent consumer (typically one stream because the F3 hot path is single-threaded).
|
||||
- Owns the resident engine handles for the duration of a flight.
|
||||
- Owns the polling loop for thermal-throttle telemetry (1 Hz background thread).
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| TensorRT (C++ + Python) | 10.3 (JetPack 6.2 pin) per D-C7-9 | Primary engine compile + deserialize + infer |
|
||||
| Polygraphy | matches TensorRT | Engine build orchestration |
|
||||
| trtexec | bundled with TensorRT | Alternate engine build path |
|
||||
| ONNX Runtime + TRT EP | per project pin | Fallback runtime |
|
||||
| PyTorch | per simple-baseline pin | FP16 baseline (mandatory) |
|
||||
| jetson-stats / pynvml | latest | Thermal-throttle telemetry source |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `EngineBuildError`: surface to C10/operator pre-flight; takeoff blocked. **Never silently fall back** between runtimes — if the configured runtime can't build, the operator must explicitly switch.
|
||||
- `EngineDeserializeError` at takeoff: refuse takeoff with explicit `(SM, JP, TRT, precision)` mismatch detail.
|
||||
- `InferenceError` mid-flight (rare; e.g., transient CUDA fault): emit no result for that frame; the consumer (C2/C3) reports its own degraded path.
|
||||
- `OutOfMemoryError`: same as above; surface to C13 FDR and C12 operator-tooling for post-flight investigation.
|
||||
- `TelemetryUnavailableError`: jetson-stats hung or unavailable. Default to "thermal_throttle_active = false" (D-CROSS-LATENCY-1 stays on the steady-state path); log WARN.
|
||||
- `CalibrationCacheError`: per D-C10-6, calibration cache trust is critical; if the cache hash mismatches, refuse to use it and force a rebuild.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `EngineFilenameSchema` | self-describing filename per D-C10-7 | C7, C10 |
|
||||
| `Sha256Sidecar` | atomic write + content-hash sidecar pattern | C6, C7, C10 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- TensorRT engines are NOT portable across `(SM, JP, TRT, precision)` tuples; Tier-1 (workstation Docker) cannot reuse Tier-2 (Jetson) engines. CI emits both tiers' engines as artifacts.
|
||||
- INT8 calibration cache trust is the lurking foot-gun; D-C10-6 manifest-hash gate is the only protection. Any deviation breaks NFT-PERF-01 / NFT-LIM-01.
|
||||
|
||||
**Potential race conditions**:
|
||||
- The thermal-throttle polling thread MUST be reentrant-safe with the F3 hot path's `infer` calls. Use a lock-free atomic snapshot for `thermal_state`.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Per-frame inference cost is the F3 hot path's largest contributor. NFT-PERF-01 partition is the source of truth.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: nothing internal — C7 is foundational.
|
||||
|
||||
**Can be implemented in parallel with**: C6, C13.
|
||||
|
||||
**Blocks**: C1 (CUDA strategies), C2, C2.5, C3, C3.5, C4 (consumes `ThermalState`), C10, F1, F2, F3, F6, F8.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `EngineBuildError`, `EngineDeserializeError`, `OutOfMemoryError`, `CalibrationCacheError` | `C7 OOM during infer; backbone=ultravpr; frame=12345` |
|
||||
| WARN | thermal-throttle entered/exited; telemetry unavailable | `C7 thermal throttle active; gpu_temp=83C; clock=750mhz` |
|
||||
| INFO | Strategy ready; engine deserialised; backbone resident | `C7 ready: runtime=tensorrt, engines=[ultravpr@fp16, lightglue@fp16, disk@fp16]` |
|
||||
| DEBUG | per-frame infer timing per backbone | `C7 infer backbone=ultravpr frame=12345 took=37ms` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN always; thermal-state transitions always to FDR).
|
||||
@@ -0,0 +1,170 @@
|
||||
# Test Specification — C7 On-Jetson Inference Runtime
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01 (Tier-2), **C7-PT-01** | Covered |
|
||||
| AC-4.2 | Memory <8 GB on Jetson | NFT-LIM-01, **C7-PT-02** | Covered |
|
||||
| AC-NEW-1 | Cold-start TTFF <30 s p95 | NFT-PERF-03, **C7-IT-01** | Covered |
|
||||
| AC-NEW-5 | Operating envelope; thermal telemetry feed | NFT-LIM-04, **C7-IT-02** | Covered (workstation portion) |
|
||||
| D-C10-3 | Manifest content-hash takeoff gate | (gate is C10-owned, but the engine deserialise call is C7) | **C7-IT-03** | Covered |
|
||||
| D-C10-7 | Engine filename schema (SM/JP/TRT/precision) | Helper-doc cited; **C7-IT-04** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C7-IT-01: cold-start engine load + warm-up budget
|
||||
|
||||
**Summary**: from a cold (zero-resident-engines) Jetson process, every required engine deserialises and warms up in under the AC-NEW-1 30 s p95 budget.
|
||||
|
||||
**Traces to**: AC-NEW-1
|
||||
|
||||
**Description**: kill the companion process; restart; measure wall-clock from process start to "all engines warm" event in the FDR record stream. Repeat 10 times; assert p95 ≤ 30 s.
|
||||
|
||||
**Input data**: pre-built engine cache for the Derkachi fixture profile.
|
||||
|
||||
**Expected result**: p95 ≤ 30 s; no engine fails to warm.
|
||||
|
||||
**Max execution time**: 6 min (10 × ~30 s + overhead).
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-02: thermal telemetry feeds C4's hybrid
|
||||
|
||||
**Summary**: `ThermalState` from `jetson-stats` is published at ≥1 Hz and is observable to C4; under simulated throttle, `throttle == true` is reported within 1 s of the throttle event.
|
||||
|
||||
**Traces to**: AC-NEW-5 (workstation-baseline portion; chamber portion deferred per traceability matrix)
|
||||
|
||||
**Description**: simulate a thermal-throttle event by spoofing the `jetson-stats` sysfs reading; assert (a) `ThermalState` updates carry `throttle == true` within 1 s, (b) C4's `current_covariance_mode` flips to JACOBIAN within 1 frame after that.
|
||||
|
||||
**Input data**: scripted sysfs spoof.
|
||||
|
||||
**Expected result**: 1 s telemetry latency; 1-frame C4 reaction.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-03: D-C10-3 takeoff gate refuses mismatched engine
|
||||
|
||||
**Summary**: when the manifest's content-hash for an engine does not match the on-disk engine's hash, C7 refuses to deserialise and the F2 takeoff aborts.
|
||||
|
||||
**Traces to**: D-C10-3
|
||||
|
||||
**Description**: corrupt one byte of a deployed engine after the manifest has been signed; trigger F2 takeoff load; assert (a) C7 raises `EngineHashMismatchError`, (b) the airborne process refuses to open the FC adapter, (c) the failure is logged at ERROR.
|
||||
|
||||
**Input data**: a deployed engine + its corrupted twin.
|
||||
|
||||
**Expected result**: takeoff aborts; ERROR logged.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-04: SM / JetPack / TRT / precision filename schema enforcement
|
||||
|
||||
**Summary**: an engine file whose `<sm>/<jp>/<trt>/<precision>` quadruple in the filename does not match the running Jetson's actual quadruple is refused at deserialise time.
|
||||
|
||||
**Traces to**: D-C10-7
|
||||
|
||||
**Description**: copy a valid engine file but rename it with a mismatched SM (e.g., `sm86` instead of `sm87`); call `load_engine`; assert `EngineSchemaMismatchError` and no GPU memory allocated.
|
||||
|
||||
**Input data**: a valid engine + a renamed copy.
|
||||
|
||||
**Expected result**: engine refused at filename-parse time.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-05: ONNX-RT fallback when TRT engine unavailable
|
||||
|
||||
**Summary**: if the primary TRT engine is missing or unloadable, C7 falls back to ONNX-RT + TRT-EP and continues without dropping the request.
|
||||
|
||||
**Traces to**: defensive (engine-rule simple-baseline path)
|
||||
|
||||
**Description**: rename the TRT engine for one model away (so deserialise fails); call `infer`; assert the call succeeds via ONNX-RT path with a degraded-latency warning logged.
|
||||
|
||||
**Input data**: TRT engine + ONNX model side-by-side.
|
||||
|
||||
**Expected result**: successful inference; degraded-latency warning.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C7-PT-01: per-call inference latency p95 by model
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: scripted call rate matching production — UltraVPR @ 3 Hz, LightGlue @ 9 Hz (3 cands × 3 Hz), AdHoP conditional (~25%).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Model | Mode | p95 latency target | Failure threshold |
|
||||
|-------|------|--------------------|-------------------|
|
||||
| UltraVPR | TRT FP16 | ≤ 60 ms | 100 ms |
|
||||
| LightGlue | TRT FP16 | ≤ 30 ms | 60 ms |
|
||||
| AdHoP | TRT FP16 | ≤ 90 ms | 150 ms |
|
||||
| DISK | TRT FP16 | ≤ 50 ms | 90 ms |
|
||||
|
||||
---
|
||||
|
||||
### C7-PT-02: aggregate GPU memory budget
|
||||
|
||||
**Traces to**: AC-4.2
|
||||
|
||||
**Load scenario**: all production-default engines resident concurrently.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| GPU resident memory (all engines) | ≤ 4 GB | 5 GB |
|
||||
| System RAM (process resident) | ≤ 1.5 GB | 2 GB |
|
||||
|
||||
(remaining 8 GB shared LPDDR5 budget partition belongs to OS + ROS-equivalents + scratch; tracked at the system level by NFT-LIM-01.)
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C7-ST-01: engine deserialise refuses files with no SHA-256 sidecar
|
||||
|
||||
**Summary**: per Helper `Sha256Sidecar`, every engine has a sidecar `.sha256` file; deserialising an engine without one is refused.
|
||||
|
||||
**Traces to**: D-C10-3 (defensive)
|
||||
|
||||
**Test procedure**:
|
||||
1. Delete the sidecar for one valid engine.
|
||||
2. Call `load_engine` on it.
|
||||
3. Assert refusal with `EngineSidecarMissingError`.
|
||||
|
||||
**Pass criteria**: refusal + no GPU memory allocated.
|
||||
**Fail criteria**: load succeeds.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
C7 has no operator-facing behaviour; covered transitively via NFT-PERF-01 / NFT-PERF-03.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Pre-built engine cache for Derkachi profile | C10 build artifact | ~600 MB |
|
||||
| Spoofed `jetson-stats` sysfs harness | scripted | <1 MB |
|
||||
| Corrupted-engine fixture | scripted | varies |
|
||||
|
||||
**Setup**: C10 must have built engines for SM 87 / JP 6.2 / TRT 10.3 / FP16 once before C7 tests can run on Tier-2.
|
||||
**Teardown**: read-only.
|
||||
**Data isolation**: per-test temp dirs.
|
||||
@@ -0,0 +1,153 @@
|
||||
# C8 — Flight-Controller Adapter
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: own the per-FC inbound + outbound communication. Inbound: subscribe to FC IMU/attitude/GPS-health/MAV_STATE telemetry; publish `ImuWindow` / `AttitudeWindow` / `GpsHealth` / `FlightStateSignal` for upstream consumers. Outbound: encode `EstimatorOutput` into the per-FC external-position contract (`GPS_INPUT` for ArduPilot Plane, `MSP2_SENSOR_GPS` for iNav) at 5 Hz periodic with honest 6×6 → 2×2 covariance projection. Owns MAVLink 2.0 message signing on the AP wired channel (D-C8-9 = (d)) and the per-flight key rotation. Also feeds the GCS link with downsampled telemetry (1–2 Hz per AC-6.1).
|
||||
|
||||
**Architectural Pattern**: Strategy — `FcAdapter` interface with two concrete implementations: `PymavlinkArdupilotAdapter`, `Msp2InavAdapter`. Plus a `GcsAdapter` (single concrete `QgcTelemetryAdapter` today). All selected at startup by config (ADR-001), build-time gating per `BUILD_*` flags (ADR-002, both adapters typically linked into the deployment binary so a single image can target both FCs by configuration), composition-root wired (ADR-009).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- C5 StateEstimator → `EstimatorOutput` (5 Hz periodic emit driver).
|
||||
- Hardware: UART/USB to FC; UART (or USB) to GCS (often shared or via FC mavlink-routing).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C5 StateEstimator (consumes `ImuWindow`, `AttitudeWindow`, `GpsHealth`, `FlightStateSignal`).
|
||||
- C1 VIO (consumes `ImuWindow`).
|
||||
- C13 FDR (consumes raw inbound + emitted outbound MAVLink/MSP2 streams; signing key rotation events; spoof-promotion events).
|
||||
- C11 `TileUploader` (consumes `FlightStateSignal == ON_GROUND` confirmation; runs on a different process / image, so the signal flows out-of-band via the FDR or a small bus the operator tool subscribes to post-flight). The C11 `TileDownloader` does NOT depend on `FlightStateSignal` — it runs pre-flight when the companion is plugged into the operator workstation.
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `FcAdapter`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `open` | `port_config: PortConfig, signing_key: bytes (AP only)` | `None` | No | `FcOpenError`, `SigningHandshakeError` |
|
||||
| `subscribe_telemetry` | `callback: Callable[[FcTelemetryFrame], None]` | `Subscription` | No | — |
|
||||
| `emit_external_position` | `EstimatorOutput` | `None` | No | `FcEmitError` |
|
||||
| `emit_status_text` | `string, severity: enum {INFO, WARN, ERROR}` | `None` | No | `FcEmitError` |
|
||||
| `request_source_set_switch` (AP only) | `()` | `None` | No | `SourceSetSwitchError` |
|
||||
| `current_flight_state` | `()` | `FlightStateSignal` | No | — |
|
||||
|
||||
### Interface: `GcsAdapter`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `emit_summary` | `EstimatorOutput` (downsampled to 1–2 Hz) | `None` | No | `GcsEmitError` |
|
||||
| `subscribe_operator_commands` | `callback` | `Subscription` | No | — |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
```
|
||||
PortConfig:
|
||||
device: string (e.g. /dev/ttyTHS1)
|
||||
baud: int
|
||||
fc_kind: enum {ardupilot_plane, inav}
|
||||
|
||||
FcTelemetryFrame:
|
||||
kind: enum {imu_sample, attitude, gps_health, mav_state}
|
||||
payload: union per kind
|
||||
received_at: monotonic_ns
|
||||
signed: bool — true only for AP signed frames
|
||||
|
||||
EmittedExternalPosition:
|
||||
per_fc_payload: enum (GPS_INPUT for AP / MSP2_SENSOR_GPS for iNav)
|
||||
horiz_accuracy_m: float (AP) — projection of covariance_6x6 → 2x2 → equivalent_radius
|
||||
hPosAccuracy_mm: int (iNav) — same source, mm units (D-C8-8 = (b))
|
||||
source_label: STATUSTEXT / NAMED_VALUE_FLOAT (out-of-band)
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
C8 exposes no HTTP/gRPC API. Its **external** surfaces are:
|
||||
|
||||
- **MAVLink 2.0 messages** to/from ArduPilot Plane, signed (D-C8-9 = (d)). Specifically:
|
||||
- In: `RAW_IMU` / `SCALED_IMU2`, `ATTITUDE`, `GPS_RAW_INT`, `GPS2_RAW`, `HEARTBEAT`, `STATUSTEXT`.
|
||||
- Out: `GPS_INPUT` (5 Hz), `STATUSTEXT`, `NAMED_VALUE_FLOAT` (provenance label), `COMMAND_LONG MAV_CMD_SET_EKF_SOURCE_SET` (D-C8-2 source-set switch on F7).
|
||||
- **MSP2 messages** to/from iNav over UART (unsigned, accepted residual risk):
|
||||
- In: `MSP2_INAV_ANALOG`, attitude+IMU stream.
|
||||
- Out: `MSP2_SENSOR_GPS` (5 Hz). MAVLink outbound for telemetry only (iNav speaks both).
|
||||
- **MAVLink 2.0 messages** to/from QGroundControl on the GCS link (1–2 Hz downsampled summary out, operator commands in).
|
||||
|
||||
Refer to `architecture.md` § 5 External Integrations for the per-message rate/auth/failure-mode table; this spec inherits those values.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
C8 holds no persistent storage. All communication is in-memory streaming.
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| Per-flight signing key (AP) | in-memory, generated at takeoff | flight lifetime | new key on next takeoff |
|
||||
| Last-emitted external position (for downsampling 5 Hz → 1–2 Hz GCS) | in-memory ring | flight lifetime | n/a |
|
||||
| FC telemetry rolling window (for `ImuWindow`, `AttitudeWindow` consumers) | in-memory bounded ring | bounded by C1/C5 consumption | drop-oldest |
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: per-message encode/decode is `O(message_size)`; the per-frame work is dominated by the covariance projection (3×3 sub-matrix → equivalent_radius), which is constant-time.
|
||||
|
||||
**State Management**:
|
||||
- AP path: holds the MAVLink 2.0 signing state (key, key generation timestamp, signature counter).
|
||||
- iNav path: holds the MSP2 channel state (sequence numbers).
|
||||
- Both paths: hold the bounded telemetry rings and a backpressure budget for the 5 Hz outbound emit.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| pymavlink | bundled unmodified per D-C8-3 | AP MAVLink 2.0 + signing |
|
||||
| YAMSPy | per project pin | iNav MSP2 |
|
||||
| INAV-Toolkit | per project pin | iNav MSP2 message definitions for `MSP2_SENSOR_GPS` |
|
||||
| pyserial | per project pin | UART transport |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `FcOpenError` at takeoff: refuse takeoff with explicit error.
|
||||
- `SigningHandshakeError` at takeoff (AP): refuse takeoff. Mid-flight signing failure: FC ignores unsigned messages, AC-5.2 takes over (3 s no estimate → FC IMU-only). Companion logs the rotation/expiry event to FDR.
|
||||
- `FcEmitError` mid-flight: log + continue; the FC's own staleness detection will eventually trigger AC-5.2.
|
||||
- `SourceSetSwitchError` (AP only, F7): try once; on failure log + STATUSTEXT to GCS; the system continues to emit `GPS_INPUT` and the operator can manually switch via RC aux per D-C8-2-FALLBACK.
|
||||
- iNav unsigned: NOT an error — accepted residual risk per Mode B Source #129. Documented in `tests/security-tests.md` NFT-SEC-03.
|
||||
|
||||
**Covariance projection** (D-C8-8 = (b)):
|
||||
- Project 6×6 GTSAM `Marginals` covariance → 3×3 position sub-matrix → 2×2 horizontal sub-matrix → `equivalent_radius` per per-FC contract:
|
||||
- AP `horiz_accuracy` (m): `sqrt(0.5 * (sigma_xx + sigma_yy + sqrt((sigma_xx - sigma_yy)^2 + 4*sigma_xy^2)))` (largest eigenvalue of 2×2; documented as the "honest covariance projection" in IT-10).
|
||||
- iNav `hPosAccuracy` (mm): same value, converted to mm.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `WgsConverter` | shared with C4, C5, C6 | C4, C5, C6, C8 |
|
||||
| `CovarianceProjector` | 6×6 → 2×2 → equivalent_radius — keep inside C8 (per-FC unit conversion is variant-specific per coderule SRP) | C8 only |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- iNav has no signing equivalent; accepted residual risk per Mode B Source #129. Plan-phase carryforward proposes an iNav firmware feature request — out of scope for this cycle.
|
||||
- D-C8-2 source-set switch is firmware-supported but not deployed-precedent; ADR-008 makes IT-3 (ArduPilot SITL validation) the lock gate.
|
||||
|
||||
**Potential race conditions**:
|
||||
- The 5 Hz outbound emit thread MUST not block on the 100–200 Hz inbound telemetry decode thread. Bounded rings + drop-oldest semantics on the inbound side; outbound-emit timer is independent.
|
||||
- The AP signing key rotation MUST happen between flights, not mid-flight. The key generation is at takeoff load (F2).
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- pymavlink's signing handshake is sub-second on a wired UART/USB link. F2 budget is generous.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: hardware UART config (operator workstation pre-flight stage).
|
||||
|
||||
**Can be implemented in parallel with**: C6, C13.
|
||||
|
||||
**Blocks**: C1 (FC IMU prior), C5 (factor inputs), F2 / F3 / F5 / F7 / F8 / F9 / F10.
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `FcOpenError`, `SigningHandshakeError`, persistent `FcEmitError` | `C8 AP signing handshake failed; refusing takeoff` |
|
||||
| WARN | one-off `FcEmitError`; spoofing-promotion blocked event surfaced via STATUSTEXT | `C8 GPS_INPUT emit dropped; reason=uart_busy; will retry next tick` |
|
||||
| INFO | adapter ready; key rotation event; F7 source-set switch executed | `C8 AP signing key rotated; flight_id=…; key_age_s=0` |
|
||||
| DEBUG | per-emit timing | `C8 GPS_INPUT emit frame=12345 took=2.1ms; horiz_accuracy_m=4.2` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout / journald / FDR via C13 (ERROR + WARN always; AP signing key rotation events ALWAYS to FDR; F7 source-set switch events ALWAYS to FDR + GCS STATUSTEXT; STATUSTEXT broadcasts ALWAYS mirrored to FDR).
|
||||
@@ -0,0 +1,224 @@
|
||||
# Test Specification — C8 FC + GCS Adapter
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-4.3 | FC output: GPS_INPUT (AP) + MSP2_SENSOR_GPS (iNav) with honest covariance | FT-P-03, FT-P-09-AP, FT-P-09-iNav, **C8-IT-01** | Covered |
|
||||
| AC-4.4 | Estimates streamed frame-by-frame | NFT-PERF-02, **C8-IT-02** | Covered |
|
||||
| AC-5.1 | Init from FC EKF's last valid GPS + IMU-extrapolated | FT-P-11, **C8-IT-03** | Covered |
|
||||
| AC-5.2 | On >3 s without estimate, FC IMU-only fallback; SUT logs | NFT-RES-01 | Covered (C8 emission path) |
|
||||
| AC-6.1 | GCS stream at 1–2 Hz | FT-P-12, **C8-IT-04** | Covered |
|
||||
| AC-6.2 | GCS may send commands via standard MAVLink | FT-P-13, **C8-IT-05** | Covered |
|
||||
| AC-6.3 | WGS84 output | FT-P-14, **C8-IT-06** | Covered |
|
||||
| AC-NEW-2 | Spoofing-promotion latency <3 s p95 (source-set switch) | NFT-PERF-04, **C8-IT-07** | Covered |
|
||||
| RESTRICT-COMM-2 | iNav has no MAVLink signing; documented asymmetry | NFT-SEC-03, **C8-IT-08** | Covered |
|
||||
| D-C8-9 = (d) | MAVLink 2.0 per-flight signing on AP | IT-3, **C8-ST-01** | Open (gated by IT-3 SITL pass) |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C8-IT-01: per-FC encoder produces honest 6×6 → 2×2 covariance projection
|
||||
|
||||
**Summary**: `EstimatorOutput.covariance_6x6`'s position 3×3 block projects to the FC's 2×2 horizontal covariance with the same Frobenius norm scale (no fake-confidence regression).
|
||||
|
||||
**Traces to**: AC-4.3
|
||||
|
||||
**Description**: 100 emitted `EstimatorOutput` records with varying covariances; capture the encoded `GPS_INPUT` bytes; decode via pymavlink; assert the FC-side 2×2 covariance Frobenius norm equals the source 3×3 horizontal-block norm to within 1% (small numerical tolerance for unit conversion). Same assertion against `MSP2_SENSOR_GPS` for the iNav adapter.
|
||||
|
||||
**Input data**: scripted `EstimatorOutput` sequence with controlled covariances.
|
||||
|
||||
**Expected result**: norm equality within 1% for both adapters.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-02: 5 Hz periodic emission stays within ±5% of nominal interval
|
||||
|
||||
**Summary**: `emit_external_position` runs at exactly 5 Hz with jitter ≤ ±5%.
|
||||
|
||||
**Traces to**: AC-4.4
|
||||
|
||||
**Description**: drive the emit path for 60 s; record emission timestamps; compute inter-emission interval distribution; assert mean = 200 ms ± 1 ms; std-dev ≤ 10 ms; max gap ≤ 220 ms.
|
||||
|
||||
**Input data**: scripted `EstimatorOutput` source at 5 Hz.
|
||||
|
||||
**Expected result**: jitter within bound.
|
||||
|
||||
**Max execution time**: 90 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-03: warm-start GPS from FC EKF feeds C5
|
||||
|
||||
**Summary**: at takeoff, the FC EKF's last-valid-GPS is read by C8 and surfaced via `current_flight_state` for C5's warm-start.
|
||||
|
||||
**Traces to**: AC-5.1
|
||||
|
||||
**Description**: stage a SITL with a known last-valid-GPS; bring up C8; call `current_flight_state`; assert `FlightStateSignal` includes the EKF's pose hint with timestamp ≤ 1 s old.
|
||||
|
||||
**Input data**: ArduPilot SITL with scripted EKF state.
|
||||
|
||||
**Expected result**: warm-start hint exposed within 1 s of C8 ready.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-04: 1–2 Hz GCS downsampling
|
||||
|
||||
**Summary**: GCS stream is downsampled from C5's 5 Hz to 1–2 Hz per AC-6.1.
|
||||
|
||||
**Traces to**: AC-6.1
|
||||
|
||||
**Description**: drive 5 Hz `EstimatorOutput`; capture GCS-side stream; assert rate is between 1 and 2 Hz with consistent downsample factor.
|
||||
|
||||
**Input data**: scripted source.
|
||||
|
||||
**Expected result**: 1–2 Hz GCS stream.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-05: GCS commands are accepted and routed
|
||||
|
||||
**Summary**: standard MAVLink commands sent from QGC reach C8 and the system handles them per AC-6.2 (e.g., parameter read, mode set).
|
||||
|
||||
**Traces to**: AC-6.2
|
||||
|
||||
**Description**: connect QGC SITL to the C8 GCS adapter; send a parameter-read; assert the response is well-formed and the parameter values match the running config.
|
||||
|
||||
**Input data**: QGC SITL + C8 adapter.
|
||||
|
||||
**Expected result**: parameter read returns values.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-06: WGS84 conversion round-trip identity
|
||||
|
||||
**Summary**: `WgsConverter` (helper) round-trips a local-tangent-plane → WGS84 → local-tangent-plane within 1 cm position residual.
|
||||
|
||||
**Traces to**: AC-6.3 (defensive — backstops the helper that C4 + C8 both rely on)
|
||||
|
||||
**Description**: 1000 random local-tangent-plane points; round-trip through the helper; assert max residual < 1 cm position; max angular residual < 0.001°.
|
||||
|
||||
**Input data**: scripted random points within ±10 km of an origin.
|
||||
|
||||
**Expected result**: round-trip residual within bound.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-07: source-set switch latency under spoof event
|
||||
|
||||
**Summary**: when the spoof gate clears, `request_source_set_switch` fires the AP `MAV_CMD_SET_EKF_SOURCE_SET` within the AC-NEW-2 3 s budget.
|
||||
|
||||
**Traces to**: AC-NEW-2
|
||||
|
||||
**Description**: scripted scenario from C5-IT-06 (spoof recovery); assert C8 issues the `MAV_CMD_SET_EKF_SOURCE_SET` within 3 s of the gate-clear event; assert SITL acknowledges the command.
|
||||
|
||||
**Input data**: ArduPilot SITL + scripted gate-clear.
|
||||
|
||||
**Expected result**: command issued + ACK within 3 s.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C8-IT-08: iNav signing-asymmetry assertion
|
||||
|
||||
**Summary**: the iNav adapter never attempts MAVLink signing (iNav doesn't support it per RESTRICT-COMM-2); the AP adapter always does.
|
||||
|
||||
**Traces to**: RESTRICT-COMM-2
|
||||
|
||||
**Description**: bring up the iNav adapter; assert no signing handshake is attempted (capture wire bytes, check no MAVLink2 signed-flag set). Bring up AP adapter; assert signing handshake completes.
|
||||
|
||||
**Input data**: iNav SITL + AP SITL.
|
||||
|
||||
**Expected result**: per the assertion above.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C8-PT-01: emission latency under load
|
||||
|
||||
**Traces to**: AC-4.4 / AC-4.1 (emission portion)
|
||||
|
||||
**Load scenario**: 5 Hz emit + 200 Hz inbound IMU subscribe + GCS at 2 Hz; 10 min replay.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `emit_external_position` p95 | ≤ 5 ms | 15 ms |
|
||||
| Inbound IMU callback p95 | ≤ 1 ms | 5 ms |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C8-ST-01: MAVLink 2.0 per-flight signing handshake (D-C8-9 = (d), gated by IT-3)
|
||||
|
||||
**Summary**: at takeoff, C8 (AP adapter) generates a per-flight ephemeral key and completes the MAVLink 2.0 signing handshake with ArduPilot SITL.
|
||||
|
||||
**Traces to**: D-C8-9 = (d) (R03 risk)
|
||||
|
||||
**Test procedure**:
|
||||
1. Bring up SITL with no pre-shared key.
|
||||
2. C8 generates a fresh per-flight key, completes handshake.
|
||||
3. Send 100 signed messages; assert SITL accepts them.
|
||||
4. Replay one of those messages with a tampered signature; assert SITL rejects it.
|
||||
|
||||
**Pass criteria**: handshake succeeds; tampered messages rejected.
|
||||
**Fail criteria**: handshake fails (escalate to D-C8-2-FALLBACK per ADR-008) OR tampered messages accepted.
|
||||
|
||||
**Status**: gated by IT-3 SITL pass; this test IS the gate.
|
||||
|
||||
---
|
||||
|
||||
### C8-ST-02: signing key never persists across flights
|
||||
|
||||
**Summary**: per-flight ephemeral key is destroyed in memory at landing; never written to disk.
|
||||
|
||||
**Traces to**: defensive (D-C8-9 = (d))
|
||||
|
||||
**Test procedure**:
|
||||
1. Run a flight; capture the key in memory at takeoff via test harness.
|
||||
2. Trigger landing.
|
||||
3. Inspect process memory + disk for any residue of the key.
|
||||
4. Assert no on-disk persistence; in-memory residue cleared by the configured zeroisation routine.
|
||||
|
||||
**Pass criteria**: no on-disk + memory zeroised.
|
||||
**Fail criteria**: any persistence.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-09-AP / FT-P-09-iNav / FT-P-12 / FT-P-13 / FT-P-14.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| ArduPilot SITL fixture (Plane) | upstream SITL Docker | image |
|
||||
| iNav SITL fixture | upstream SITL Docker | image |
|
||||
| QGC SITL | upstream Docker | image |
|
||||
| Scripted `EstimatorOutput` sequences | scripted | <5 MB each |
|
||||
|
||||
**Setup**: SITL containers must be running; C8 connects via simulated UART.
|
||||
**Teardown**: stop containers; clean ephemeral keys.
|
||||
**Data isolation**: per-test SITL container.
|
||||
@@ -0,0 +1,151 @@
|
||||
# C10 — Pre-flight Cache Provisioning
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: build the **model-derived** pre-flight cache artifacts on top of an already-populated tile store, and verify them at takeoff. After C11 `TileDownloader` has fetched tiles into C6, C10 orchestrates: compile/deserialize TensorRT engines via C7 → batch each tile through C2's backbone for descriptors → atomically write FAISS HNSW index with SHA-256 sidecars (D-C10-3) → write Manifest with hash of (model + calibration + corpus + sector_class) for D-C10-1 idempotence. At F2 takeoff load, run `verify_manifest` (D-C10-3 SHA-256 content-hash gate) before allowing the system to arm.
|
||||
|
||||
**C10 does NOT touch `satellite-provider`.** Tile I/O — both download (F1 inbound) and post-landing upload (F10) — lives in C11 (Tile Manager). C10 reads tiles from C6, writes engines + descriptors + manifest to filesystem and Postgres. The split is operational: C11 carries the operator-side network identity (TLS API key for download, per-flight signing key for upload) and the airborne-exclusion property (ADR-004); C10 carries the model identity and the takeoff-load verifier — neither of which need to leave the workstation/companion enclave at runtime.
|
||||
|
||||
**Architectural Pattern**: Coordinator — single concrete implementation `CacheProvisioner` behind two interfaces (`CacheProvisioner` for the F1 build phase, `ManifestVerifier` for F2's content-hash gate). The interfaces are split because F2 only needs the verifier and shouldn't pull in the full provisioning code path.
|
||||
|
||||
**Upstream dependencies**:
|
||||
|
||||
- C12 OperatorTooling → triggers `build_cache_artifacts(...)` after C11 `TileDownloader` has populated C6.
|
||||
- C6 TileStore + TileMetadataStore + DescriptorIndex → read source (tiles + metadata), write target (FAISS index).
|
||||
- C7 InferenceRuntime → engine compile + deserialize.
|
||||
- C2 backbone (via C7 engine) → descriptor batched generation.
|
||||
|
||||
**Downstream consumers**:
|
||||
|
||||
- F2 takeoff load → consumes `verify_manifest` outcome.
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `CacheProvisioner`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `build_cache_artifacts` | `BuildRequest` | `BuildReport` | No (offline; minutes) | `EngineBuildError`, `DescriptorBatchError`, `ManifestWriteError`, `IdempotentNoOp` |
|
||||
| `compile_engines_for_corpus` | `BackboneList` | `list[EngineCacheEntry]` | No | `EngineBuildError`, `CalibrationCacheError` |
|
||||
|
||||
### Interface: `ManifestVerifier`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `verify_manifest` | `manifest_path: Path` | `VerificationResult` | No | `ManifestNotFoundError`, `ContentHashMismatchError` |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
|
||||
```
|
||||
BuildRequest:
|
||||
bbox: BoundingBox (lat_min, lon_min, lat_max, lon_max) # scopes which C6 tiles are in the manifest
|
||||
zoom_levels: list[int]
|
||||
sector_class: enum {active_conflict, stable_rear} # baked into manifest
|
||||
calibration_path: Path
|
||||
cache_root: Path
|
||||
|
||||
BuildReport:
|
||||
engines_built: int
|
||||
engines_reused: int
|
||||
descriptors_generated: int
|
||||
manifest_hash: sha256
|
||||
outcome: enum {success, failure, idempotent_no_op}
|
||||
failure_reason: string (optional)
|
||||
|
||||
Manifest: see data_model.md
|
||||
EngineCacheEntry: see data_model.md
|
||||
|
||||
VerificationResult:
|
||||
manifest_hash_match: bool
|
||||
per_artifact_hash_match: dict[Path, bool]
|
||||
outcome: enum {pass, fail}
|
||||
fail_reasons: list[string]
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable. C10 has no network surface — all I/O is local filesystem + local Postgres.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
C10 reads `tiles` rows from C6 (scoped to the build's bbox + zoom_levels), writes the FAISS `.index` to filesystem via `Sha256Sidecar`, and writes Manifest + `manifests` row to Postgres via C6.
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| Manifest | one per build per cached area | ~10 KB (YAML/JSON) | negligible | per build |
|
||||
| SHA-256 sidecars | one per artifact (.index, calibration JSON, manifest, .engine) | 64 B (hex digest) | negligible | per build |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: none — C10 writes from scratch (or D-C10-1 idempotently no-ops). Tiles must already be in C6 (placed there by C11 `TileDownloader`); a missing-tiles condition is a build error, not a download trigger.
|
||||
|
||||
**Rollback**: D-C10-1 manifest-hash check makes provisioning idempotent. Atomic writes (atomicwrites package) prevent partial states; on partial failure, the previous-good cache remains until the new one is fully written.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: dominated by descriptor batched generation on Jetson (GPU-bound). Worst-case ~400 km² provisioning is ≤ tens of minutes (offline, not time-critical per AC-8.3). Tile network bandwidth is **not** in C10's budget — that cost is in C11.
|
||||
|
||||
**State Management**: stateless w.r.t. flight lifetime. No connection state — all dependencies are local.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| atomicwrites | latest | Atomic file replacement for `.index` + Manifest (D-C10-3) |
|
||||
| hashlib (stdlib) | stdlib | SHA-256 content-hash sidecars |
|
||||
| PyYAML / orjson | per project pin | Manifest serialization |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
|
||||
- `EngineBuildError` / `CalibrationCacheError`: surfaced from C7 — never silently fall back; operator must intervene.
|
||||
- `DescriptorBatchError`: CUDA OOM during descriptor generation. Halve batch size and retry once; if still OOM, surface to operator.
|
||||
- `ManifestWriteError`: filesystem error or atomic-write rollback. Cache marked invalid; operator must re-run.
|
||||
- `IdempotentNoOp`: D-C10-1 manifest-hash matched the prior build's hash; skip rebuild; emit no-op report.
|
||||
- `ContentHashMismatchError` (F2): refuse takeoff; STATUSTEXT to GCS; FDR records the event; operator must re-run F1.
|
||||
- **Missing tiles in C6 for the requested bbox/zoom**: surface as `BuildReport.failure` with explicit instruction to run C11 `TileDownloader` first; do **not** fall back to a network fetch — that responsibility lives in C11.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `Sha256Sidecar` | atomic write + content-hash sidecar pattern | C6, C7, C10 |
|
||||
| `EngineFilenameSchema` | self-describing filename per D-C10-7 | C7, C10 |
|
||||
| `WgsConverter` | bbox math | C4, C5, C6, C8, C10 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
|
||||
- C10 depends on C6 already containing the tiles for the requested bbox + zoom levels. The F1 cache-build workflow (C12) sequences `C11 TileDownloader → C10 build_cache_artifacts`; C10 alone is not a complete F1.
|
||||
- D-C10-3 SHA-256 content-hash gate must cover EVERY artifact: every tile (the per-tile hash is computed at C11 download time and stored in C6), the FAISS `.index`, the calibration JSON, and the Manifest itself. Missing sidecars are a release-blocking defect.
|
||||
|
||||
**Potential race conditions**:
|
||||
|
||||
- Concurrent `build_cache_artifacts` invocations on the same cache root would corrupt state. Single-process operator-tool wraps with a filesystem lockfile (the same lockfile C11 honours); if a second invocation tries to start, fail with explicit error.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
|
||||
- Descriptor batched generation is GPU-bound; batching is the main lever (D-C7-1 INT8/FP16 mix decision applies).
|
||||
- Engine compile is workspace-bound on Jetson; D-C10-6 calibration cache reuse is the main lever.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C6 (read source for tiles, write target for FAISS), C7 (engine + descriptor runtime), C2 (backbone interface for descriptor generation; called via C7).
|
||||
|
||||
**Can be implemented in parallel with**: C8, C13.
|
||||
|
||||
**Blocks**: C12 (operator can't sequence F1 without C10 ready), F1, F2 (verify_manifest), F8 (warm-cache verify on reboot recovery).
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `EngineBuildError`, `DescriptorBatchError`, `ManifestWriteError`, `ContentHashMismatchError` (F2) | `C10 engine build failed: backbone=disk; takeoff blocked` |
|
||||
| WARN | engine cache miss falls through to build | `C10 engine cache miss: model=ultra_vpr; sm=87, jp=6.2, trt=10.3, fp16; rebuild` |
|
||||
| INFO | Build start/end + report; verify_manifest pass | `C10 build complete: engines=4, descriptors=87654, manifest_hash=…; outcome=success` |
|
||||
| DEBUG | per-tile descriptor batch progress | `C10 descriptor batch progress: 12345/87654 (14%)` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: stdout (operator tool); journald (companion verify); FDR via C13 (only for F2 verify_manifest events — provisioning is offline and goes to operator-facing logs, not flight FDR).
|
||||
@@ -0,0 +1,151 @@
|
||||
# Test Specification — C10 Pre-flight Cache Provisioning (engines + descriptors + manifest)
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`. C10 was narrowed in this Plan cycle: it builds model-derived artifacts (TensorRT engines, VPR descriptors, signed Manifest) from an **already-populated** C6 tile cache. Tile fetch is C11 `TileDownloader`'s concern.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-8.3 | Imagery pre-loaded onto companion before flight (the manifest gate) | FT-P-15, FT-P-16, **C10-IT-01** | Covered |
|
||||
| AC-NEW-1 | Cold-start TTFF <30 s p95 (pre-built engines required) | NFT-PERF-03, **C10-IT-02** | Covered |
|
||||
| D-C10-1 | Manifest-hash idempotence on repeated build | **C10-IT-03** | Covered |
|
||||
| D-C10-3 | Takeoff content-hash gate refuses mismatch | covered at C7-IT-03; **C10-IT-04** asserts the manifest signing path | Covered |
|
||||
| D-C10-6 | Engine cache hardware-tied (SM 87 / JP 6.2 / TRT 10.3 / FP16) | C7-IT-04, **C10-IT-05** | Covered |
|
||||
| D-C10-7 | Engine filename schema enforcement | covered at C7-IT-04 | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C10-IT-01: end-to-end build from a pre-populated C6
|
||||
|
||||
**Summary**: given a C6 tile cache populated by C11 `TileDownloader` (10 GB Derkachi area), C10 produces (a) all required TensorRT engines, (b) the FAISS HNSW index over VPR descriptors, (c) a signed Manifest, in under the operator-tooling time budget.
|
||||
|
||||
**Traces to**: AC-8.3, AC-NEW-1
|
||||
|
||||
**Description**: stage a C6 with the Derkachi corpus already populated; run `CacheProvisioner.build_artifacts`; assert (a) the engine set under `cache_artifacts/engines/` matches the configured model list, (b) `descriptor_index.faiss` is non-empty and queryable, (c) the Manifest is signed with the operator's signing key and content-hashes every artifact.
|
||||
|
||||
**Input data**: pre-populated C6 (`tests/fixtures/c6_populated_derkachi/`).
|
||||
|
||||
**Expected result**: all artifacts present + signed Manifest.
|
||||
|
||||
**Max execution time**: 12 min on Tier-1 (CPU TRT compile is slow; Tier-2 takes ~4 min and is the production path).
|
||||
|
||||
---
|
||||
|
||||
### C10-IT-02: ManifestVerifier refuses unsigned / wrong-signature Manifest
|
||||
|
||||
**Summary**: `ManifestVerifier.verify` rejects a Manifest whose signature doesn't validate against the operator's public key.
|
||||
|
||||
**Traces to**: AC-NEW-1, D-C10-3
|
||||
|
||||
**Description**: build a valid Manifest; copy it; tamper one byte; call `verify`; assert `ManifestSignatureError`. Repeat: copy + replace signature with one signed by an unauthorized key; assert `ManifestSignatureError`.
|
||||
|
||||
**Input data**: valid Manifest + 2 tampered copies.
|
||||
|
||||
**Expected result**: both tampered Manifests rejected.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C10-IT-03: idempotence on repeated build
|
||||
|
||||
**Summary**: re-running `build_artifacts` against an unchanged C6 produces the same Manifest content-hash and skips already-built engines.
|
||||
|
||||
**Traces to**: D-C10-1
|
||||
|
||||
**Description**: run build once; record Manifest content-hash + engine compile timestamps. Re-run with no C6 changes; assert (a) Manifest content-hash unchanged, (b) engines reused (no recompile, asserted via timestamp comparison), (c) total wall-clock < 1 min on Tier-1.
|
||||
|
||||
**Input data**: as C10-IT-01.
|
||||
|
||||
**Expected result**: idempotent — same hash, no recompile.
|
||||
|
||||
**Max execution time**: 90 s (second-run only).
|
||||
|
||||
---
|
||||
|
||||
### C10-IT-04: Manifest covers every shipped artifact
|
||||
|
||||
**Summary**: the Manifest's content-hash table includes every file under `cache_artifacts/`; an artifact present on disk but missing from the Manifest is a build failure.
|
||||
|
||||
**Traces to**: D-C10-3 (no smuggled artifacts can pass the takeoff gate)
|
||||
|
||||
**Description**: after a successful build, plant an extra file in `cache_artifacts/`; re-run `build_artifacts` (or call the build's post-step audit hook); assert build refuses to sign — output `ManifestCoverageError` listing the orphan file.
|
||||
|
||||
**Input data**: as C10-IT-01 plus an extra file.
|
||||
|
||||
**Expected result**: build fails with `ManifestCoverageError`.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C10-IT-05: Tier-2 hardware-tied engine compile produces SM-87 / JP-6.2 / TRT-10.3 binary
|
||||
|
||||
**Summary**: when run on the bench Jetson, C10 produces engines whose internal TRT metadata reports `SM=87, JetPack=6.2, TRT=10.3, precision=FP16`.
|
||||
|
||||
**Traces to**: D-C10-6
|
||||
|
||||
**Description**: run `build_artifacts` on the bench Jetson; for each engine, parse the internal TRT version footer; assert the quadruple matches.
|
||||
|
||||
**Input data**: bench Jetson + Derkachi C6 fixture.
|
||||
|
||||
**Expected result**: all engines tagged correctly.
|
||||
|
||||
**Max execution time**: 6 min on Tier-2.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C10-PT-01: build wall-clock budget on Tier-1 (operator-tooling laptop)
|
||||
|
||||
**Traces to**: operator-tooling UX (no AC trace; an operator-tooling SLO)
|
||||
|
||||
**Load scenario**: full Derkachi corpus (10 GB, ~87 654 tiles).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| Cold build wall-clock | ≤ 12 min on a developer laptop with NVIDIA GPU | 25 min |
|
||||
| Warm idempotent re-run | ≤ 1 min | 3 min |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C10-ST-01: signing-key path uses operator-controlled key (not a baked-in dev key)
|
||||
|
||||
**Summary**: the build refuses to sign the Manifest if the configured signing-key path points to the baked-in dev key (caught via a hash-list check).
|
||||
|
||||
**Traces to**: defensive (production-key safety)
|
||||
|
||||
**Test procedure**:
|
||||
1. Configure C10 with the dev-key path that's hard-coded into the dev fixtures.
|
||||
2. Run `build_artifacts`.
|
||||
3. Assert refusal with `OperatorKeyRequiredError`.
|
||||
|
||||
**Pass criteria**: refusal.
|
||||
**Fail criteria**: build succeeds with the dev key.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-15 / FT-P-16 (operator workflow tests).
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| `tests/fixtures/c6_populated_derkachi/` | C11 `TileDownloader` build artifact | ~10 GB on disk |
|
||||
| Operator signing key (test-only) | generated per test run | <1 KB |
|
||||
| Dev key (for the negative test) | curated, in-repo | <1 KB |
|
||||
|
||||
**Setup**: C11 `TileDownloader` integration test (under C11) populates C6 once; that artifact is reused.
|
||||
**Teardown**: per-test temp dirs for `cache_artifacts/` build outputs.
|
||||
**Data isolation**: per-test temp `cache_artifacts/`.
|
||||
@@ -0,0 +1,201 @@
|
||||
# C11 — Tile Manager
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: own the operator-side network I/O against `satellite-provider` for the onboard tile corpus, in **both directions**:
|
||||
|
||||
- **Download** (pre-flight, F1): fetch tiles from `satellite-provider` for the operational area, apply AC-NEW-6 freshness gating, and write into C6 (`TileStore` + `TileMetadataStore`). C11 is the **only** path that crosses the workstation/companion enclave to the parent suite for tile pixels — C10 reads from the populated C6 store and never touches `satellite-provider` itself.
|
||||
- **Upload** (post-landing, F10): when `flight_state == ON_GROUND` is confirmed, read pending mid-flight tiles from C6 and POST to `satellite-provider`'s ingest endpoint (D-PROJ-2 contract sketch).
|
||||
|
||||
C11 is a **separate operator-side binary / image**. The airborne companion image's CMake target deliberately excludes the entire `c11_tilemanager/` source tree so the airborne process cannot accidentally execute either the download path or the upload path even via reflection or config error (ADR-004 process-level isolation, AC-8.4). Both directions of tile I/O are operator-driven on the operator workstation; the companion only consumes the populated C6 store while airborne.
|
||||
|
||||
**Architectural Pattern**: Pipeline behind two interfaces (`TileDownloader`, `TileUploader`) under one component, consistent with C8's multi-interface shape (FC-AP, FC-iNav, GCS adapters under one component). The two interfaces are bundled into C11 because they share auth (TLS + service-internal API key for download, per-flight onboard signing key for upload), HTTP client, network configuration, deployment unit (operator-tooling tarball), and the airborne-exclusion property — splitting them into two components would duplicate all of that. They are kept as **two interfaces** so SRP is preserved at the call-site level: C12 binds `TileDownloader` for the F1 cache-build workflow, `TileUploader` for the F10 post-landing trigger; neither is forced to depend on the other.
|
||||
|
||||
**Upstream dependencies**:
|
||||
|
||||
- C12 OperatorTooling → invokes `TileDownloader.download_tiles_for_area(...)` during F1 and `TileUploader.upload_pending_tiles(...)` post-landing.
|
||||
- C6 TileStore + TileMetadataStore → write target during download (`source = googlemaps`); read source during upload (`source = onboard_ingest`, `voting_status = pending`).
|
||||
- Operator workstation OS → invocation entry point (CLI / tray app, owned by C12).
|
||||
- `satellite-provider` (external) → `GET /api/satellite/tiles?bbox=…&zoom=…` for download; `POST /api/satellite/tiles/ingest` for upload (D-PROJ-2 design task #1, **planned, not yet implemented service-side**).
|
||||
|
||||
**Downstream consumers**:
|
||||
|
||||
- C10 CacheProvisioner reads the populated C6 store after a `TileDownloader` run completes; C10 does not call C11 directly. C12 sequences the two steps.
|
||||
- On the upload side: none on the onboard side; the parent-suite voting layer (D-PROJ-2 design task #2) consumes the uploaded tiles asynchronously.
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `TileDownloader`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `download_tiles_for_area` | `DownloadRequest` | `DownloadBatchReport` | No (offline; minutes) | `SatelliteProviderError`, `FreshnessRejectionError`, `ResolutionRejectionError`, `CacheBudgetExceededError`, `IdempotentNoOp` |
|
||||
| `enumerate_remote_coverage` | `bbox: BoundingBox, zoom_levels: list[int]` | `list[TileSummary]` | No | `SatelliteProviderError` |
|
||||
|
||||
### Interface: `TileUploader`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `confirm_flight_state` | `()` | `FlightStateSignal` (must be ON_GROUND) | No | `FlightStateNotOnGroundError` |
|
||||
| `enumerate_pending_tiles` | `flight_id: uuid (optional)` | `list[TileMetadata]` | No | `TileMetadataError` |
|
||||
| `upload_pending_tiles` | `UploadRequest` | `UploadBatchReport` | No | `SatelliteProviderError`, `RateLimitedError`, `SignatureRejectedError` |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
|
||||
```
|
||||
DownloadRequest:
|
||||
bbox: BoundingBox (lat_min, lon_min, lat_max, lon_max)
|
||||
zoom_levels: list[int]
|
||||
sector_class: enum {active_conflict, stable_rear}
|
||||
satellite_provider_url: URL
|
||||
service_api_key: string
|
||||
cache_root: Path
|
||||
|
||||
DownloadBatchReport:
|
||||
tiles_downloaded: int
|
||||
tiles_rejected_freshness: int
|
||||
tiles_rejected_resolution: int # RESTRICT-SAT-4 < 0.5 m/px
|
||||
tiles_downgraded: int
|
||||
freshness_summary: dict[freshness_label, count]
|
||||
outcome: enum {success, failure, idempotent_no_op}
|
||||
failure_reason: string (optional)
|
||||
|
||||
UploadRequest:
|
||||
flight_id: uuid (optional; defaults to all flights with pending tiles)
|
||||
batch_size: int
|
||||
satellite_provider_url: URL
|
||||
|
||||
FlightStateSignal: see C8 — must be ON_GROUND for any upload to proceed
|
||||
|
||||
UploadBatchReport:
|
||||
batch_uuid: uuid (assigned by satellite-provider per D-PROJ-2 contract)
|
||||
per_tile_status: list[(tile_id, status: enum {queued, rejected, duplicate, superseded})]
|
||||
retry_count: int
|
||||
next_retry_at_s: int (when retried)
|
||||
outcome: enum {success, partial, failure}
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
C11 is a **client** of `satellite-provider`'s REST surface in both directions.
|
||||
|
||||
### 3.1 Download — read path (existing `satellite-provider` API)
|
||||
|
||||
| Endpoint | Method | Auth | Rate Limit | Description |
|
||||
|----------|--------|------|------------|-------------|
|
||||
| `/api/satellite/tiles?bbox=…&zoom=…` | GET | TLS + service-internal API key | parent-suite enforces | Paged tile blobs + metadata for a bounding box at the given zoom level(s). |
|
||||
|
||||
C11 honours `Retry-After` on 429s, fails fast on TLS / auth errors, retries with backoff on 5xx. Resolution below 0.5 m/px (RESTRICT-SAT-4) is rejected at the C11 boundary, not pushed downstream.
|
||||
|
||||
### 3.2 Upload — write path (D-PROJ-2 contract sketch, **planned**)
|
||||
|
||||
| Endpoint | Method | Auth | Rate Limit | Description |
|
||||
|----------|--------|------|------------|-------------|
|
||||
| `/api/satellite/tiles/ingest` (parent-suite, **planned**) | POST | Per-flight onboard signing key (D-C8-9 = (d) family); each tile carries the signature | parent-suite enforces | Multipart upload of one or more tiles; response 202 with batch UUID + per-tile status. |
|
||||
|
||||
**Request schema** (multipart fields per tile):
|
||||
|
||||
- `tile_blob` (JPEG body, byte-identical to `satellite-provider`'s existing tile format)
|
||||
- `zoomLevel` (int)
|
||||
- `latitude` / `longitude` (double)
|
||||
- `tile_size_meters` (double), `tile_size_pixels` (int)
|
||||
- `capture_timestamp` (ISO 8601), `flight_id` (UUID), `companion_id` (string)
|
||||
- `quality_metadata` (JSON; `TileQualityMetadata` per data_model.md)
|
||||
- `signature` (per-flight onboard signing key signature over the payload)
|
||||
|
||||
**Response**: 202 Accepted + `{batch_uuid: UUID, per_tile_status: [...]}`.
|
||||
|
||||
Test substitute during NFT-SEC-01 / FT-P-17 / IT runs: the e2e-test `mock-suite-sat-service` fixture (under `tests/fixtures/mock-suite-sat-service/`) implements the planned POST surface so upload integration tests can run before D-PROJ-2 ships service-side. Download integration tests run against the **real** `satellite-provider` (its existing GET surface is already implemented). The mock is not a component and is never reached in production.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
C11 reads from / writes to C6 (the local store) and reads from / writes to `satellite-provider` (network). It owns no relational state of its own beyond a small download-progress journal and a small upload-progress journal.
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| Download-progress journal | filesystem alongside the operator workstation cache root | until a `download_tiles_for_area` run completes | per-area run on completion |
|
||||
| Pending-upload journal | filesystem alongside the operator workstation cache root | until upload acknowledged | per-batch acknowledgment removes from journal |
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| Download-progress journal | a few hundred rows per area provisioning | ~256 B / row | <1 MB | per provisioning run |
|
||||
| Pending-upload journal | a few hundred per flight | ~256 B / row | <1 MB | per flight |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: none — both journals are empty until the operator triggers a download or an upload run.
|
||||
|
||||
**Rollback**: the download path is idempotent — re-running `download_tiles_for_area` for an unchanged `(bbox, zoom_levels, sector_class)` triggers C10's manifest-hash check (D-C10-1) downstream and the engine/descriptor build is skipped. The upload path is idempotent on the service side via the `(zoomLevel, lat, lon, capture_timestamp, companion_id, flight_id)` dedup key.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**:
|
||||
|
||||
- Download: linear in tile count; bandwidth-bound by the operator workstation's link to `satellite-provider`.
|
||||
- Upload: linear in pending tile count; bandwidth-bound; bursty post-landing.
|
||||
|
||||
**State Management**: stateless except for the two journals.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| httpx | per project pin | GET (download) + multipart POST (upload) to `satellite-provider` |
|
||||
| atomicwrites | latest | Journal updates |
|
||||
| cryptography | per project pin | Per-flight signing key (upload payload signing); the production `satellite-provider` ingest endpoint and the e2e-test `mock-suite-sat-service` fixture both verify with the same key family |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
|
||||
- `SatelliteProviderError`: HTTP timeout / 5xx / TLS failure on either direction. Retry-with-backoff on 5xx; fail fast on TLS / auth. On download, surface to operator + takeoff blocked. On upload, leave tiles in the pending-upload journal and surface to operator. **Do not delete uploaded tiles from C6** until acknowledged.
|
||||
- `RateLimitedError` (429): obey `Retry-After`; the operator can also re-invoke later. Same handling either direction.
|
||||
- `FreshnessRejectionError` / `ResolutionRejectionError`: download-side only. Per AC-NEW-6 / RESTRICT-SAT-4 — never silently downgrade fresh-required tiles in `active_conflict` sectors. Surface counts in the `DownloadBatchReport`.
|
||||
- `CacheBudgetExceededError`: download-side only. Pre-flight free-space check against AC-8.3 (≤ 10 GB). Fail fast with explicit budget delta; no partial write.
|
||||
- `FlightStateNotOnGroundError`: upload-side only. Refuse to start; log + show explicit reason. ADR-004 process-level isolation means C11 should never run when the FC believes it's airborne — this error is a defense-in-depth, not the primary control.
|
||||
- `SignatureRejectedError`: upload-side only. Per-flight signing key was rejected by `satellite-provider`. This is a security-critical event — do NOT silently drop; surface to operator + log to FDR.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `TileSignaturePayloadBuilder` | constructs the signed payload for D-PROJ-2 contract (upload) | C11 only — keep inside the component |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
|
||||
- D-PROJ-2 ingest endpoint is NOT yet implemented service-side. Until parent-suite delivers the endpoint, C11 will fail every upload — the pending-upload journal accumulates. Operator workflow tolerates this.
|
||||
- The e2e-test `mock-suite-sat-service` fixture implements only the planned POST contract (per the leftover file). Download integration tests run against the real `satellite-provider`. Production runs reach `satellite-provider` directly in both directions; the fixture is never on the production path.
|
||||
- `TileDownloader` requires the operator workstation to have network reach to `satellite-provider` (the only path that crosses out of the workstation enclave). Pre-flight network configuration is an operator concern owned by C12; C11 fails fast if reachability is missing.
|
||||
|
||||
**Potential race conditions**:
|
||||
|
||||
- If the operator launches two `TileDownloader` runs concurrently against the same cache root, a filesystem lockfile (operated by C12 tooling) prevents corrupting C6's tile rows. Same lockfile gates concurrent `TileUploader` invocations.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
|
||||
- Download: bandwidth-bound by the operator workstation's `satellite-provider` link; descriptor / engine work is downstream in C10 (offline, minutes).
|
||||
- Upload: bandwidth-bound. Per-flight upload volume is bounded by the F4 mid-flight tile gen cap (typically a few hundred tiles, each 50–200 KB → tens of MB per flight).
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C6 (read source for upload, write target for download), `satellite-provider` (download path; existing) + D-PROJ-2 endpoint (upload path; the e2e-test `mock-suite-sat-service` fixture covers tests until the real endpoint ships).
|
||||
|
||||
**Can be implemented in parallel with**: anything except C6 changes.
|
||||
|
||||
**Blocks**: F1 (pre-flight cache build cannot start without `TileDownloader`), F10 (post-landing upload cannot start without `TileUploader`).
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `FlightStateNotOnGroundError`, `SignatureRejectedError`, persistent `SatelliteProviderError`, `CacheBudgetExceededError` | `C11 refused to start: flight_state=IN_AIR; safeguard active` |
|
||||
| WARN | one-off network failure, scheduled retry, freshness-driven rejections (counts) | `C11 batch upload retry: batch_uuid=…; next_retry_in_s=30` |
|
||||
| INFO | session start/end; per-batch report (download + upload) | `C11 download complete: 87654 tiles, 12 stale-rejected; bbox=…` |
|
||||
| DEBUG | per-tile request/response | `C11 tile uploaded: tile_id=(z=18,lat=…,lon=…); status=queued` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: operator workstation log file (e.g. `~/.azaion/onboard/c11-tilemanager.log`); also writes per-run summaries (download report, upload report) to the operator workstation cache root for audit. The companion's FDR is NOT involved (C11 doesn't run on the companion).
|
||||
@@ -0,0 +1,213 @@
|
||||
# Test Specification — C11 Tile Manager (TileDownloader + TileUploader)
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`. C11 was renamed and expanded in this Plan cycle: it now owns BOTH operator-side network I/O directions against `satellite-provider`. Strict ADR-004 enforcement: never loaded into the airborne companion image.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-8.1 | Imagery via Suite Sat Service offline cache, ≥0.5 m/px | FT-P-15, **C11-IT-01** (download) | Covered |
|
||||
| AC-8.2 | Tile freshness <6 mo (active-conflict) / <12 mo (rear) | FT-N-05, **C11-IT-02** (download-side gate) | Covered |
|
||||
| AC-8.3 | Imagery pre-loaded onto companion before flight | FT-P-15, FT-P-16, **C11-IT-01** | Covered |
|
||||
| AC-NEW-6 | System rejects/downgrades stale tiles | FT-N-05, FT-N-06, **C11-IT-02** | Covered |
|
||||
| AC-8.4 + D-PROJ-2 (POST contract) | Mid-flight tile generation + post-landing upload to satellite-provider | FT-P-17, **C11-IT-03** (upload) | Covered (against e2e-test fixture until D-PROJ-2 ships) |
|
||||
| RESTRICT-SAT-1 | Onboard cache offline-only; no in-flight Service calls | NFT-SEC-02, NFT-SEC-05, **C11-ST-01** | Covered |
|
||||
| ADR-004 | Process isolation; C11 never airborne | **C11-ST-01**, **C11-ST-02** | Covered (R02 enforcement) |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C11-IT-01: TileDownloader fetch + sector-classified freshness gate + write to C6
|
||||
|
||||
**Summary**: given a Derkachi-area request, `TileDownloader.fetch` retrieves tiles from the real `satellite-provider` (or fixture in tests), enforces the per-sector freshness gate, and writes accepted tiles into C6 with byte-identical filesystem layout.
|
||||
|
||||
**Traces to**: AC-8.1, AC-8.2, AC-8.3, AC-NEW-6
|
||||
|
||||
**Description**: configure the fetch job with the Derkachi bbox + zoom range + sector classification (active_conflict + stable_rear mix); run `fetch`; assert (a) all returned tiles within the bbox land in C6, (b) tiles with `produced_at` older than the per-sector threshold are downgrade-flagged or rejected, (c) the `DownloadBatchReport` reports counts that sum to the request size.
|
||||
|
||||
**Input data**: scripted fetch request + the real `satellite-provider` (or its test Docker fixture under `tests/fixtures/satellite-provider/`; this is the REAL service's existing GET surface, not the mock).
|
||||
|
||||
**Expected result**: tiles in C6 match request scope; freshness flags applied per sector.
|
||||
|
||||
**Max execution time**: 4 min on Tier-1 (network-dependent).
|
||||
|
||||
---
|
||||
|
||||
### C11-IT-02: freshness rejection counts surface in DownloadBatchReport
|
||||
|
||||
**Summary**: when the source has stale tiles, `DownloadBatchReport.stale_rejected` is non-zero and matches the count rejected at the C6 boundary.
|
||||
|
||||
**Traces to**: AC-NEW-6 (operator visibility)
|
||||
|
||||
**Description**: run a fetch that hits a known-stale tile range; assert (a) `stale_rejected > 0`, (b) the value equals C6's freshness-rejection count for that batch.
|
||||
|
||||
**Input data**: a synthetic `satellite-provider` response with stale tiles.
|
||||
|
||||
**Expected result**: counts match.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C11-IT-03: TileUploader posts mid-flight tiles + signs payload
|
||||
|
||||
**Summary**: `TileUploader.upload_pending` reads mid-flight tiles from C6's pending-upload set, packages them per the D-PROJ-2 POST contract sketch, signs each payload with the per-flight key, and posts to the configured `/api/satellite/tiles/ingest` endpoint.
|
||||
|
||||
**Traces to**: AC-8.4, D-PROJ-2 contract
|
||||
|
||||
**Description**: stage C6 with 50 mid-flight tiles flagged pending-upload; bring up the e2e-test `mock-suite-sat-service` fixture (under `tests/fixtures/mock-suite-sat-service/`); call `upload_pending`; assert (a) all 50 tiles POSTed, (b) each payload signature verifies against the test public key, (c) on 202 Accepted, the C6 row is marked uploaded, (d) the fixture's request log shows all 50 tiles in arrival.
|
||||
|
||||
**Input data**: scripted C6 + e2e-test mock fixture.
|
||||
|
||||
**Expected result**: all 50 uploaded + acknowledged + marked.
|
||||
|
||||
**Max execution time**: 5 min.
|
||||
|
||||
---
|
||||
|
||||
### C11-IT-04: TileUploader gates on `flight_state == ON_GROUND`
|
||||
|
||||
**Summary**: `TileUploader.upload_pending` refuses to run if `FlightStateSignal != ON_GROUND` (defense-in-depth atop ADR-004 process isolation).
|
||||
|
||||
**Traces to**: AC-8.4 (defensive — ADR-004's secondary guard)
|
||||
|
||||
**Description**: call `upload_pending` with `FlightStateSignal == IN_FLIGHT`; assert `UploadGateBlockedError`. Same with `UNKNOWN`. Set `ON_GROUND` and assert upload proceeds.
|
||||
|
||||
**Input data**: scripted FlightStateSignal source.
|
||||
|
||||
**Expected result**: upload blocked except in `ON_GROUND`.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C11-IT-05: idempotent uploads on retry
|
||||
|
||||
**Summary**: re-running `upload_pending` after a partial-success batch only POSTs the tiles that weren't acknowledged before.
|
||||
|
||||
**Traces to**: D-PROJ-2 contract resilience
|
||||
|
||||
**Description**: stage 50 pending-upload tiles; call `upload_pending` with the mock configured to return 202 for first 30, 5xx for last 20; assert C6 marks 30 as uploaded. Reset mock to 202 for all; call `upload_pending` again; assert only the remaining 20 are POSTed and the first 30 are NOT re-sent.
|
||||
|
||||
**Input data**: scripted with controlled mock failure profile.
|
||||
|
||||
**Expected result**: per assertion above.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C11-PT-01: download throughput
|
||||
|
||||
**Traces to**: operator-tooling SLO (AC-8.3 supports this).
|
||||
|
||||
**Load scenario**: 10 GB Derkachi area, parallel fetch.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| Throughput | ≥ 50 MB/s on a 1 Gbps link | < 20 MB/s |
|
||||
| `DownloadBatchReport` produced | yes | no report = test failure |
|
||||
|
||||
---
|
||||
|
||||
### C11-PT-02: upload throughput
|
||||
|
||||
**Traces to**: post-landing operator UX (AC-8.4 + D-PROJ-2 timing assumption).
|
||||
|
||||
**Load scenario**: 1000 mid-flight tiles, sequential POST + sign.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| Throughput | ≥ 20 tile/s with signing | < 10 tile/s |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C11-ST-01: airborne process cannot import `c11_tilemanager`
|
||||
|
||||
**Summary**: per ADR-004 R02 enforcement, the airborne `production-binary` artifact has no path to import the C11 module.
|
||||
|
||||
**Traces to**: ADR-004, R02
|
||||
|
||||
**Test procedure**:
|
||||
1. Build the airborne `production-binary`.
|
||||
2. Inside a sandbox running the artifact, attempt `import c11_tilemanager` (and any submodule).
|
||||
3. Assert `ModuleNotFoundError` for every variant.
|
||||
4. Run the `runtime_root.py` self-check; assert it does NOT panic (because `find_spec` returns `None`).
|
||||
|
||||
**Pass criteria**: import fails everywhere; self-check passes silently.
|
||||
**Fail criteria**: any successful import.
|
||||
|
||||
---
|
||||
|
||||
### C11-ST-02: NFT-SEC-02 network-egress test
|
||||
|
||||
**Summary**: the airborne process running in a no-route-to-`satellite-provider` network namespace cannot reach `satellite-provider` over TCP.
|
||||
|
||||
**Traces to**: ADR-004 R02 enforcement; covers `RESTRICT-SAT-1`.
|
||||
|
||||
**Test procedure**: as documented in `tests/security-tests.md` NFT-SEC-02. Listed here for traceability.
|
||||
|
||||
**Pass criteria**: zero outbound connection attempts to `satellite-provider`'s host:port from the airborne process during a 10-min replay.
|
||||
**Fail criteria**: any attempt.
|
||||
|
||||
---
|
||||
|
||||
### C11-ST-03: per-flight signing key zeroised after upload completes
|
||||
|
||||
**Summary**: after `upload_pending` completes (success or final failure), the per-flight key is zeroised in memory.
|
||||
|
||||
**Traces to**: D-C8-9 = (d) (operator-side analogue)
|
||||
|
||||
**Test procedure**:
|
||||
1. Run an upload.
|
||||
2. Capture the key buffer post-call via test harness.
|
||||
3. Assert the buffer is zero-filled.
|
||||
|
||||
**Pass criteria**: zero-filled.
|
||||
**Fail criteria**: any non-zero residue.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
### C11-AT-01: operator runs F1 download + F10 upload via CLI
|
||||
|
||||
**Summary**: end-to-end operator flow exercises both interfaces via the C12-driven CLI.
|
||||
|
||||
**Traces to**: AC-8.3, AC-8.4
|
||||
|
||||
**Preconditions**:
|
||||
- Operator workstation with Docker + the operator-tooling tarball.
|
||||
- A `satellite-provider` (real or test fixture) reachable.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Result |
|
||||
|------|--------|-----------------|
|
||||
| 1 | `operator-tool download --area derkachi.geojson --since 2026-01` | `DownloadBatchReport` printed; tiles in C6 |
|
||||
| 2 | `operator-tool build-cache` | C10 builds engines + descriptors + Manifest |
|
||||
| 3 | (simulate flight) | (covered by other tests) |
|
||||
| 4 | `operator-tool upload-pending` | Pending-upload tiles POSTed; report printed |
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Real `satellite-provider` Docker fixture (download) | upstream parent-suite Docker | image |
|
||||
| e2e-test `mock-suite-sat-service` fixture (upload) | `tests/fixtures/mock-suite-sat-service/` | <50 MB image |
|
||||
| Scripted Derkachi bbox + sector classification | scripted | <1 MB |
|
||||
|
||||
**Setup**: bring up the appropriate fixture per test (real for download; e2e-test mock for upload until D-PROJ-2 ships).
|
||||
**Teardown**: stop fixture containers; clean per-test C6.
|
||||
**Data isolation**: per-test C6 + per-test fixture instance.
|
||||
@@ -0,0 +1,145 @@
|
||||
# C12 — Operator Pre-flight Tooling
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: operator-facing tooling on the workstation that **sequences** the F1 cache-build workflow (calls C11 `TileDownloader` then C10 `CacheProvisioner`), drives sector classification, freshness pipeline, calibration loading, and (post-flight) triggers the C11 `TileUploader`. Provides the human-in-the-loop entry point for everything that needs operator judgement (active-conflict vs stable-rear sector classification, AC-3.4 ≥3-frame outage operator re-loc requests, network configuration to `satellite-provider`).
|
||||
|
||||
**Architectural Pattern**: Coordinator (single concrete `OperatorTooling` today). Two interfaces: `CacheBuildWorkflow` (pre-flight UX) and `OperatorReLocService` (mid-flight operator re-loc requests, AC-3.4). The latter is consumed via the GCS link by C8 (operator commands subscription).
|
||||
|
||||
**Upstream dependencies**:
|
||||
- Operator (human input).
|
||||
- C11 `TileDownloader` (operator-workstation-side) — invoked first in F1 to populate C6 from `satellite-provider`.
|
||||
- C10 CacheProvisioner (companion-side) — invoked second in F1, over USB/Eth.
|
||||
- Camera calibration artifact (operator workstation filesystem).
|
||||
|
||||
**Downstream consumers**:
|
||||
- C11 `TileDownloader` (workstation) — invoked at F1 start.
|
||||
- C10 CacheProvisioner (companion) — invoked via USB/Eth after C11 download completes.
|
||||
- C11 `TileUploader` — invoked post-landing.
|
||||
- C8 inbound operator-commands subscription — receives AC-3.4 re-loc requests over the GCS link.
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `CacheBuildWorkflow`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `build_cache` | `bbox, sector_class, calibration_path, satellite_provider_url, api_key` | `CacheBuildReport` (wraps C11 `DownloadBatchReport` + C10 `BuildReport`) | No (operator-facing; minutes) | `CacheBuildError` (wraps SatelliteProviderError, EngineBuildError, etc.) |
|
||||
| `trigger_post_landing_upload` | `flight_id` | C11 `UploadBatchReport` | No (operator-facing; minutes) | `CacheBuildError` wrapper around `FlightStateNotOnGroundError`, `SignatureRejectedError`, etc. |
|
||||
| `verify_companion_ready` | `companion_address` | `ReadinessReport` | No | `CompanionUnreachableError`, `ContentHashMismatchError` |
|
||||
| `set_sector_classification` | `area, sector_class` | `None` | No | — |
|
||||
| `apply_freshness_threshold` | `sector_class` | `int (months)` | No | — |
|
||||
|
||||
### Interface: `OperatorReLocService`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `request_reloc` | `reloc_hint: ReLocHint` | `None` (sent over GCS link to companion) | No | `GcsLinkError` |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
```
|
||||
CacheBuildReport:
|
||||
download_report: DownloadBatchReport (see C11 spec)
|
||||
build_report: BuildReport (see C10 spec)
|
||||
outcome: enum {success, failure, idempotent_no_op}
|
||||
failure_phase: enum {download, build, none}
|
||||
failure_reason: string (optional)
|
||||
|
||||
ReadinessReport:
|
||||
manifest_present: bool
|
||||
content_hashes_pass: bool
|
||||
engines_present: bool
|
||||
calibration_present: bool
|
||||
outcome: enum {ready, not_ready}
|
||||
not_ready_reasons: list[string]
|
||||
|
||||
ReLocHint:
|
||||
approximate_position_wgs84: LatLonAlt (operator's best guess)
|
||||
confidence_radius_m: float
|
||||
reason: string (e.g. "lost track at waypoint 3")
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
C12 itself does NOT expose HTTP. It **consumes**:
|
||||
|
||||
- `satellite-provider`'s REST API **only via C11 `TileDownloader`** — C12 itself never holds the TLS / API-key credentials. The credential boundary is C11 (operator-workstation-side); C12 sequences and reports.
|
||||
- The GCS link's MAVLink path (operator commands path), for AC-3.4 re-loc requests.
|
||||
|
||||
C12 may eventually expose a thin local HTTP API for an operator GUI. **Plan-phase carryforward, deferred** — the current cycle ships only a CLI.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
C12 holds the operator workstation's local cache staging area + per-area sector classification metadata.
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| Sector classification persistence | tens of areas | ~1 KB / area | <100 KB | per operator deployment |
|
||||
| Local cache staging area | matches the area being prepared | up to ~10 GB pre-stage | bounded by operator workstation NVM | per operation |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: none — operator drives every decision.
|
||||
|
||||
**Rollback**: rebuilding the cache from scratch is the rollback action; D-C10-1 idempotence makes it cheap if no inputs changed.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: dominated by C11 `TileDownloader` bandwidth + C10's descriptor-batching cost (both offline). C12's own logic is constant-time per operator decision.
|
||||
|
||||
**State Management**: holds a session-local view of the operator's classifications + active provisioning job. No long-lived state on the companion.
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| Click / Typer | per project pin | CLI framework |
|
||||
| paramiko / ssh / fabric | per project pin | Companion bring-up over SSH (USB/Eth) |
|
||||
| pymavlink | bundled per D-C8-3 | GCS-link operator commands (AC-3.4) |
|
||||
| (httpx is NOT a direct C12 dep) | — | All `satellite-provider` HTTP traffic is owned by C11; C12 calls C11 in-process |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `CacheBuildError`: wraps the underlying error from C11/C10/C7/C6 with operator-friendly text + remediation hint. Includes `failure_phase: download | build` so the operator knows which step to retry.
|
||||
- `CompanionUnreachableError`: pre-flight bring-up failure; operator must check the wire/SSH config.
|
||||
- `ContentHashMismatchError`: post-stage tampering detected; refuse to mark as ready; operator must re-run F1.
|
||||
- `GcsLinkError` (AC-3.4 path): GCS link drop. Re-loc request is best-effort; operator may need to re-issue when link recovers.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `WgsConverter` | shared with C4, C5, C6, C8, C10 | C4, C5, C6, C8, C10, C12 |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- Operator UX is CLI-only this cycle; GUI is a Plan-phase carryforward (deferred).
|
||||
- AC-3.4 operator re-loc is best-effort over a bandwidth-limited GCS link; the companion must remain useful even when no re-loc hint arrives.
|
||||
|
||||
**Potential race conditions**:
|
||||
- Operator re-runs of F1 while a previous F1 is still in progress: filesystem lockfile in the cache staging area prevents concurrency.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Pre-flight tile download (C11) + descriptor build (C10) is the long pole; C12 just shows progress and surfaces failures.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: C11 (both `TileDownloader` and `TileUploader`), C10 (cache artifact build), C8 inbound operator-commands subscription (for AC-3.4).
|
||||
|
||||
**Can be implemented in parallel with**: anything not in the above list.
|
||||
|
||||
**Blocks**: F1 (without C12 there is no operator entry point), F10 (without C12 there is no post-landing trigger).
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `CacheBuildError`, `CompanionUnreachableError`, `ContentHashMismatchError`, `GcsLinkError` | `C12 cache build failed: reason=satellite_provider_5xx; retried=3` |
|
||||
| WARN | freshness downgrade applied; non-fatal stage warning | `C12 freshness downgrade: area=Derkachi; stale_count=42` |
|
||||
| INFO | start/end of provisioning; sector classification set; AC-3.4 hint sent | `C12 sector classification updated: area=Derkachi; class=active_conflict` |
|
||||
| DEBUG | per-step progress mirrors of C11 (download) and C10 (build) | `C12 progress: download=12345/87654 (14%); build=engine=2/4` |
|
||||
|
||||
**Log format**: structured JSON.
|
||||
**Log storage**: operator workstation log file (e.g. `~/.azaion/onboard/c12-tooling.log`).
|
||||
@@ -0,0 +1,143 @@
|
||||
# Test Specification — C12 Operator Pre-flight Tooling
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`. C12 sequences the F1 (C11 download → C10 build) and F10 (C11 upload trigger) operator-side flows.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-3.4 | Operator-relocalization on visual loss | FT-N-03, **C12-IT-01** | Covered |
|
||||
| AC-8.3 | Imagery pre-loaded onto companion before flight | FT-P-15, FT-P-16, **C12-IT-02** | Covered |
|
||||
| AC-8.4 | Mid-flight tile upload trigger (post-landing) | FT-P-17, **C12-IT-03** | Covered |
|
||||
| AC-NEW-1 | Cold-start TTFF (cache must be in place pre-flight) | NFT-PERF-03, **C12-IT-02** | Covered (pre-flight side) |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C12-IT-01: operator re-localization workflow returns SUT to satellite-anchored
|
||||
|
||||
**Summary**: when the SUT requests operator re-loc per AC-3.4, the operator workflow exposes the request, lets the operator confirm a candidate location, and returns the system to `satellite_anchored` within the configured budget.
|
||||
|
||||
**Traces to**: AC-3.4
|
||||
|
||||
**Description**: scripted scenario — SUT publishes a re-loc request via FDR + GCS STATUSTEXT; operator-side C12 catches it; an automated test "operator" confirms the candidate; assert (a) the system transitions back to `satellite_anchored` within 30 s, (b) the confirmation event lands in FDR.
|
||||
|
||||
**Input data**: scripted re-loc request fixture.
|
||||
|
||||
**Expected result**: per assertion.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C12-IT-02: build-cache orchestrates C11 download then C10 build
|
||||
|
||||
**Summary**: `build_cache(area, sector_classification)` calls `C11.TileDownloader.fetch` first, then `C10.CacheProvisioner.build_artifacts`, and produces a `CacheBuildReport` wrapping both sub-reports.
|
||||
|
||||
**Traces to**: AC-8.3, AC-NEW-1
|
||||
|
||||
**Description**: with a fresh empty C6, call `build_cache` for a small Derkachi sub-area; assert (a) C11's `DownloadBatchReport` is in the result, (b) C10's `BuildReport` is in the result, (c) call ordering: C11 completes before C10 starts (verifiable via a test-side hook on each component), (d) on a download failure (mock 5xx), C10 is NOT invoked and the error surfaces in `CacheBuildReport`.
|
||||
|
||||
**Input data**: small Derkachi sub-area + scripted fetch + a controllable mock layer.
|
||||
|
||||
**Expected result**: per assertion.
|
||||
|
||||
**Max execution time**: 5 min.
|
||||
|
||||
---
|
||||
|
||||
### C12-IT-03: trigger_post_landing_upload invokes C11 TileUploader on confirmed ON_GROUND
|
||||
|
||||
**Summary**: `trigger_post_landing_upload` reads the most recent `FlightStateSignal` from the post-flight FDR; if `ON_GROUND` is confirmed for ≥ a configurable safety threshold (default 30 s), it invokes `C11.TileUploader.upload_pending`. If `ON_GROUND` is not confirmed, it refuses and returns a clear error.
|
||||
|
||||
**Traces to**: AC-8.4
|
||||
|
||||
**Description**: stage two flight FDR fixtures — one ending with confirmed ON_GROUND for 60 s, one ending with `IN_FLIGHT` (incomplete log). Call `trigger_post_landing_upload`; assert (a) first case invokes upload, (b) second case refuses with `FlightStateNotConfirmedError`.
|
||||
|
||||
**Input data**: 2 scripted FDR fixtures.
|
||||
|
||||
**Expected result**: per assertion.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C12-IT-04: CacheBuildReport surfaces operator-actionable failures
|
||||
|
||||
**Summary**: when C10 reports a `ManifestSignatureError` or C11 reports stale-tile rejections beyond a configurable threshold, `CacheBuildReport` flags the build as actionable and returns a non-zero exit code from the CLI.
|
||||
|
||||
**Traces to**: AC-8.2 + operator UX
|
||||
|
||||
**Description**: simulate (a) C10 manifest signature failure, (b) C11 stale-rejection rate > 30%; assert in each case the CLI exits non-zero and the report includes a clear next-action ("re-run fetch with --since=…" or "rotate signing key").
|
||||
|
||||
**Input data**: scripted failure injection.
|
||||
|
||||
**Expected result**: actionable failure messages + non-zero exit.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C12-PT-01: end-to-end operator workflow wall-clock
|
||||
|
||||
**Traces to**: AC-NEW-1 (pre-flight side; post-landing side covered by C11-PT-02).
|
||||
|
||||
**Load scenario**: full Derkachi area; cold C6.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| `build_cache` wall-clock | ≤ 18 min on a developer laptop with NVIDIA GPU (10 min download + 8 min build with overlap) | 35 min |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C12-ST-01: CLI rejects writes to airborne images
|
||||
|
||||
**Summary**: the operator-tool CLI has no command path that writes into the airborne `production-binary` image (defends against operator-side mistakes that would defeat ADR-004).
|
||||
|
||||
**Traces to**: ADR-004 R02 enforcement (C12 side)
|
||||
|
||||
**Test procedure**:
|
||||
1. Enumerate every CLI subcommand.
|
||||
2. For each, assert the implementation does not import `c11_tilemanager` into a context that gets packaged into the airborne image (this is structural — verified at build time by the SBOM diff and at code-review time).
|
||||
3. Run a runtime smoke test: invoke each subcommand with `--help`; assert no warning about C11 being loaded.
|
||||
|
||||
**Pass criteria**: no command path crosses into the airborne package boundary.
|
||||
**Fail criteria**: any cross-boundary import.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
### C12-AT-01: full operator workflow on a fresh workstation
|
||||
|
||||
**Summary**: an operator with a fresh workstation can install the operator-tooling tarball and complete download → build → simulated-flight → upload in one session.
|
||||
|
||||
**Traces to**: AC-8.3, AC-8.4
|
||||
|
||||
**Preconditions**:
|
||||
- Fresh workstation with Docker installed.
|
||||
- Operator-tooling tarball present.
|
||||
- Real `satellite-provider` reachable (for download); e2e-test mock fixture (for upload) bundled in the tarball.
|
||||
|
||||
**Steps**: see `C11-AT-01` (C12 is the orchestrator that AT-01 exercises).
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Operator-tooling tarball | CI build artifact | varies |
|
||||
| FDR fixtures (ON_GROUND-confirmed and IN_FLIGHT) | scripted | <100 MB each |
|
||||
| Small Derkachi sub-area for C12-IT-02 | scripted | <500 MB |
|
||||
|
||||
**Setup**: extract operator-tooling tarball; bring up Docker compose.
|
||||
**Teardown**: stop containers; clean per-test C6.
|
||||
**Data isolation**: per-test workspace under `tests/tmp/c12/<test-id>/`.
|
||||
@@ -0,0 +1,136 @@
|
||||
# C13 — Flight Data Recorder (FDR)
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose**: persist a per-flight ≤ 64 GB record of every payload class onboard (estimates, IMU traces, emitted MAVLink, system health, mid-flight tiles, ≤0.1 Hz failed-tile thumbnails) without silently dropping data (AC-NEW-3). Exclude raw nav/AI-cam frames (AC-8.5; only the failed-tile thumbnail forensic exception is allowed). The FDR is the system's audit log: every safety-critical decision, every emitted frame, every signing key rotation, every spoof-promotion-block lands here.
|
||||
|
||||
**Architectural Pattern**: single concrete `FileFdrWriter` behind a `FdrWriter` interface. Single writer thread fed by lock-free in-process queues from every component. Lossy on writer-thread overrun **only by logging the rollover event**, never silently.
|
||||
|
||||
**Upstream dependencies**: every component publishes to C13 via in-process pub/sub (drop-oldest-with-rollover-log on overrun).
|
||||
|
||||
**Downstream consumers**:
|
||||
- Post-flight: operator workstation (read via C12 retrieval).
|
||||
- Real-time: nothing — C13 is write-only at runtime.
|
||||
|
||||
## 2. Internal Interfaces
|
||||
|
||||
### Interface: `FdrWriter`
|
||||
|
||||
| Method | Input | Output | Async | Error Types |
|
||||
|--------|-------|--------|-------|-------------|
|
||||
| `open_flight` | `FlightHeader` | `None` | No (called once at takeoff) | `FdrOpenError` |
|
||||
| `write_record` | `FdrRecord` | `None` | No (lock-free enqueue) | `FdrQueueOverrunError` (logged but does not raise) |
|
||||
| `close_flight` | `()` | `FlightFooter` | No (called once at landing) | — |
|
||||
| `current_size_bytes` | `()` | `int` | No | — |
|
||||
| `is_rolling` | `()` | `bool` | No | — |
|
||||
|
||||
**Input/Output DTOs**:
|
||||
```
|
||||
FlightHeader:
|
||||
flight_id: uuid
|
||||
flight_started_at: ISO 8601 + monotonic_ns
|
||||
config_snapshot: JSON
|
||||
signing_key_rotation_event: record
|
||||
manifest_content_hashes: dict[Path, sha256]
|
||||
|
||||
FdrRecord: see data_model.md (FdrRecord; tagged union over payload classes)
|
||||
|
||||
FlightFooter:
|
||||
flight_ended_at: ISO 8601 + monotonic_ns
|
||||
records_written: int
|
||||
records_dropped_overrun: int
|
||||
bytes_written: int
|
||||
rollover_count: int
|
||||
```
|
||||
|
||||
## 3. External API Specification
|
||||
|
||||
Not applicable.
|
||||
|
||||
## 4. Data Access Patterns
|
||||
|
||||
| Query | Frequency | Hot Path | Index Needed |
|
||||
|-------|-----------|----------|--------------|
|
||||
| `write_record` from every component | up to ~100 Hz aggregate | Yes | n/a |
|
||||
| Post-flight read (operator retrieval) | once per flight | No | filesystem layout per `(flight_id, segment)` |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
| Data | Cache Type | TTL | Invalidation |
|
||||
|------|-----------|-----|-------------|
|
||||
| In-process queue from each producer | bounded ring (drop-oldest with rollover log) | flight lifetime | per-record write |
|
||||
| Writer-thread buffer | sized for ≥1 s of typical write load | flight lifetime | flush on segment rollover |
|
||||
|
||||
### Storage Estimates
|
||||
|
||||
| Table/Collection | Est. Row Count (1yr) | Row Size | Total Size | Growth Rate |
|
||||
|-----------------|---------------------|----------|------------|-------------|
|
||||
| Per-flight record file (segmented, oldest-segment-dropped policy) | bounded by 64 GB per AC-NEW-3 | varies per payload class | ≤ 64 GB / flight | bounded by AC-NEW-3 |
|
||||
| Per-flight tile snapshots (mid-flight tiles) | ~few hundred / flight | 50–200 KB each | up to ~50 MB / flight | bounded by F4 mid-flight gen |
|
||||
| Per-flight failed-tile thumbnails (AC-8.5 forensic exception) | ≤ 0.1 Hz × 8 h = ≤ 2880 thumbnails / flight | small JPEG | <50 MB | bounded by ≤ 0.1 Hz cap |
|
||||
|
||||
### Data Management
|
||||
|
||||
**Seed data**: none.
|
||||
|
||||
**Rollback**: per-segment file layout makes per-segment deletion safe. The writer never overwrites a closed segment; it only appends to the current open segment, then opens a new segment when the previous reaches a configurable size cap.
|
||||
|
||||
## 5. Implementation Details
|
||||
|
||||
**Algorithmic Complexity**: per-record cost is `O(record_size)` for serialisation + write. Aggregate throughput sized for the worst-case AC-NEW-3 cap.
|
||||
|
||||
**State Management**:
|
||||
- Owns the open per-flight segment file handle.
|
||||
- Owns the writer thread and the in-process producer queues.
|
||||
- Owns the rollover policy (oldest-segment-dropped first when total reaches 64 GB).
|
||||
|
||||
**Key Dependencies**:
|
||||
|
||||
| Library | Version | Purpose |
|
||||
|---------|---------|---------|
|
||||
| orjson / msgpack | per project pin | Record serialisation (serialised format choice during decompose phase) |
|
||||
| atomicwrites | latest | Segment file rotation (atomic open of new segment + close of previous) |
|
||||
| filelock | per project pin | Cross-process safety for the FDR root (operator-tool reads while companion writes — companion-only access during flight) |
|
||||
|
||||
**Error Handling Strategy**:
|
||||
- `FdrOpenError` at takeoff: refuse takeoff (per AC-NEW-3 every payload class must be present from t=0).
|
||||
- `FdrQueueOverrunError`: per-producer drop-oldest, but the rollover event itself is ALWAYS logged (a separate "overrun" record in the FDR records the dropped count and producer-id). Never silent.
|
||||
- Filesystem write failure mid-flight: log to stdout/stderr (since we can't log to FDR at this point) + STATUSTEXT to GCS; the system continues to emit external positions because losing the audit log doesn't compromise navigation, but the operator must be alerted.
|
||||
|
||||
## 6. Extensions and Helpers
|
||||
|
||||
| Helper | Purpose | Used By |
|
||||
|--------|---------|---------|
|
||||
| `RecordSchema` | versioned record schema for cross-version FDR compatibility | C13 only — this is internal |
|
||||
|
||||
## 7. Caveats & Edge Cases
|
||||
|
||||
**Known limitations**:
|
||||
- 64 GB cap is per AC-NEW-3. If payload-class throughput grows beyond what the cap supports for an 8 h flight, the producers MUST throttle or accept oldest-dropped — the FDR will not silently exceed the cap.
|
||||
- Failed-tile thumbnail forensic exception is the ONLY raw-imagery-adjacent persistence; AC-8.5 must be re-asserted if any new payload class is added.
|
||||
|
||||
**Potential race conditions**:
|
||||
- The writer thread is the single writer; producers enqueue lock-free. No filesystem contention from within the companion. Operator-tool reads happen post-landing only.
|
||||
|
||||
**Performance bottlenecks**:
|
||||
- Writer-thread serialisation throughput must exceed peak producer throughput. NFT-LIM-02 (8 h synthetic AC-NEW-3) validates.
|
||||
|
||||
## 8. Dependency Graph
|
||||
|
||||
**Must be implemented after**: nothing internal — C13 is foundational along with C7.
|
||||
|
||||
**Can be implemented in parallel with**: every other component.
|
||||
|
||||
**Blocks**: every component (every component logs to C13).
|
||||
|
||||
## 9. Logging Strategy
|
||||
|
||||
| Log Level | When | Example |
|
||||
|-----------|------|---------|
|
||||
| ERROR | `FdrOpenError`, mid-flight filesystem write failure | `C13 segment write failure: errno=ENOSPC; STATUSTEXT to GCS` |
|
||||
| WARN | queue overrun (any producer) | `C13 queue overrun: producer=c5_state; dropped_count=23` |
|
||||
| INFO | open/close flight; segment rollover | `C13 flight opened: flight_id=…; segment=0` |
|
||||
| DEBUG | per-write timing (only in dev tier) | `C13 record written: kind=estimate; bytes=412; took=0.1ms` |
|
||||
|
||||
**Log format**: structured JSON to stdout/journald.
|
||||
**Log storage**: stdout / journald — but not C13 itself for ERROR (we'd be writing to the broken thing). FDR records are the project-level "logs" for everything except C13's own operational status.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Test Specification — C13 Flight Data Recorder
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-1.4 | 95% covariance + source label (FDR record class) | FT-P-03, **C13-IT-01** | Covered |
|
||||
| AC-4.5 (revised) | Internal smoothing past keyframes (FDR-only path) | FT-P-10, **C13-IT-02** | Covered |
|
||||
| AC-8.5 | No raw nav/AI-cam frame retention except thumbnail log | FT-P-18, **C13-IT-03** | Covered |
|
||||
| AC-NEW-3 | FDR ≤64 GB / flight, no silent drops | NFT-LIM-02, **C13-IT-04** | Covered |
|
||||
| RESTRICT-UAV-4 | No raw-photo storage; tile cache + FDR only | FT-P-18, NFT-LIM-03 | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C13-IT-01: every payload class produces an FDR record
|
||||
|
||||
**Summary**: every component documented as an FDR producer publishes records of the right `kind`; missing producer = test failure.
|
||||
|
||||
**Traces to**: AC-1.4, AC-NEW-3 (every-payload-class-from-t=0 invariant)
|
||||
|
||||
**Description**: spin up a minimal in-process test harness wiring all 14 components (mocked where heavy, real where light); drive 10 s of synthetic flight; assert the FDR contains records of every documented `kind` (estimate, vio_health, vpr_health, match_health, pose_estimate, source_label_change, fc_emit, fc_inbound, signing_key_event, spoof_promotion_block, mid_flight_tile, failed_tile_thumbnail, system_health, segment_rollover).
|
||||
|
||||
**Input data**: synthetic 10 s flight.
|
||||
|
||||
**Expected result**: every kind present with at least one record.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C13-IT-02: smoothed past-keyframe records land in FDR (NOT in FC stream)
|
||||
|
||||
**Summary**: when C5 publishes a smoothed past-keyframe revision, it lands in FDR but is NOT forwarded to C8's emission path.
|
||||
|
||||
**Traces to**: AC-4.5 (revised)
|
||||
|
||||
**Description**: per C5-IT-04 — re-run that scenario; assert (a) FDR contains the smoothed-history record class, (b) the C8 emission stream contains the original (unshifted) value.
|
||||
|
||||
**Input data**: shared with C5-IT-04.
|
||||
|
||||
**Expected result**: per assertion.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C13-IT-03: AC-8.5 forensic-thumbnail-only enforcement
|
||||
|
||||
**Summary**: C13 refuses to write a raw nav-cam or AI-cam frame; the only allowed exception is the failed-tile thumbnail at ≤0.1 Hz cap.
|
||||
|
||||
**Traces to**: AC-8.5
|
||||
|
||||
**Description**: attempt to write an `FdrRecord` of `kind = "raw_nav_frame"`; assert C13 raises `RawFrameWriteForbiddenError`. Attempt `kind = "failed_tile_thumbnail"` at 0.05 Hz; assert accepted. Attempt the same kind at 0.5 Hz (above cap); assert rate-limited (drop with cap log).
|
||||
|
||||
**Input data**: scripted producer.
|
||||
|
||||
**Expected result**: per assertion.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C13-IT-04: 64 GB cap holds without silent drop
|
||||
|
||||
**Summary**: under synthetic 8 h replay producing > 64 GB worth of records, C13 enforces the cap via oldest-segment-dropped + always logs the rollover event.
|
||||
|
||||
**Traces to**: AC-NEW-3
|
||||
|
||||
**Description**: scripted producer that emits at peak rates known to cross 64 GB in 8 h; replay; assert (a) total disk usage stays ≤ 64 GB, (b) every segment-rollover-with-drop is recorded with producer-id + dropped count, (c) the FlightFooter shows non-zero `records_dropped_overrun` matching the rollover events.
|
||||
|
||||
**Input data**: synthetic high-rate producer.
|
||||
|
||||
**Expected result**: cap held; every drop visible.
|
||||
|
||||
**Max execution time**: 8 h on a Tier-1 runner (NFT-LIM-02 budget).
|
||||
|
||||
---
|
||||
|
||||
### C13-IT-05: queue overrun produces a structured drop record (never silent)
|
||||
|
||||
**Summary**: when a producer overruns its in-process queue, C13 writes a structured "overrun" record naming the producer and the dropped count.
|
||||
|
||||
**Traces to**: AC-NEW-3 (no-silent-drop)
|
||||
|
||||
**Description**: artificially throttle the writer thread; flood one producer's queue; assert (a) the first dropped record triggers an "overrun" record, (b) the overrun record includes producer-id + dropped count, (c) when throttling is removed, normal writes resume.
|
||||
|
||||
**Input data**: scripted producer + writer-thread throttle harness.
|
||||
|
||||
**Expected result**: overrun record present + accurate count.
|
||||
|
||||
**Max execution time**: 60 s.
|
||||
|
||||
---
|
||||
|
||||
### C13-IT-06: refuse takeoff if `open_flight` fails
|
||||
|
||||
**Summary**: per AC-NEW-3, every payload class must be present from t=0 — if C13 cannot open the segment file, takeoff is aborted.
|
||||
|
||||
**Traces to**: AC-NEW-3
|
||||
|
||||
**Description**: configure `flight_root` to a directory the process cannot write to; call `open_flight`; assert `FdrOpenError` raised; assert the calling code (compositional root) refuses to open the FC adapter.
|
||||
|
||||
**Input data**: read-only `flight_root` directory.
|
||||
|
||||
**Expected result**: takeoff aborts; error logged.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C13-PT-01: writer-thread throughput vs peak producer rate
|
||||
|
||||
**Traces to**: AC-NEW-3
|
||||
|
||||
**Load scenario**: aggregated peak producer rate (~100 Hz combined records).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| Writer throughput | ≥ 200 Hz sustained | < 100 Hz |
|
||||
| Per-record serialise + write p95 | ≤ 5 ms | 20 ms |
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C13-ST-01: FDR record cannot be silenced via config
|
||||
|
||||
**Summary**: there is no config flag that disables the spoofing-promotion-block, signing-key-rotation, or rollover-drop record kinds — they are mandatory per AC-NEW-3 + ADR-008.
|
||||
|
||||
**Traces to**: defensive (AC-NEW-3, ADR-008)
|
||||
|
||||
**Test procedure**:
|
||||
1. Search the config schema for any flag that could disable any of those record kinds.
|
||||
2. Assert no such flag exists.
|
||||
3. Inject events of each kind under every documented config preset; assert all land in FDR.
|
||||
|
||||
**Pass criteria**: no disabling flag found; all events land.
|
||||
**Fail criteria**: any disabling flag exists or any event suppressed.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
Covered transitively via FT-P-03 / FT-P-10 / FT-P-18 / NFT-LIM-02.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Synthetic 10 s flight harness | scripted | <10 MB |
|
||||
| 8 h synthetic high-rate producer | scripted | runtime-generated |
|
||||
| Read-only `flight_root` fixture | scripted | n/a (fs perms) |
|
||||
|
||||
**Setup**: per-test `flight_root` under `tests/tmp/c13/<test-id>/`.
|
||||
**Teardown**: drop tmp directory.
|
||||
**Data isolation**: per-test `flight_root`.
|
||||
Reference in New Issue
Block a user