azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 11:11:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refactor documentation to replace the Validation Harness with a separate E2E Test Suite, updating references throughout various documents. Adjust the autodev state to reflect the transition from the Decompose phase to the Implement phase, and revise the architecture documentation to clarify system boundaries and component relationships. Enhance risk mitigation documentation to specify affected components and update the component overview diagram accordingly.

Browse Source
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-03 11:50:39 +03:00
parent 5bf2dbd85f
commit dd9afe2797
31 changed files with 1479 additions and 396 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_document/FINAL_report.md
+5 -5
View File
@@ -2,7 +2,7 @@
## Executive Summary
The solution planning phase decomposed the GPS-denied onboard localization service into 9 implementation components, 2 cross-cutting foundation epics, a bootstrap epic, and a blackbox-test epic. The architecture centers on a Jetson-hosted hot path using camera ingest, BASALT VIO, and a project-owned safety/anchor wrapper, with triggered Satellite Service candidate retrieval and ALIKED/DISK-LightGlue anchor verification against an offline PostgreSQL/PostGIS-backed cache.
The solution planning phase decomposed the GPS-denied onboard localization service into 8 runtime implementation components, 2 cross-cutting foundation epics, a bootstrap epic, and separate e2e/blackbox test epics. The architecture centers on a Jetson-hosted hot path using camera ingest, BASALT VIO, and a project-owned safety/anchor wrapper, with triggered Satellite Service candidate retrieval and ALIKED/DISK-LightGlue anchor verification against an offline PostgreSQL/PostGIS-backed cache.
Jira epics were created in project `AZ` from AZ-206 through AZ-218. Total estimated effort across epics is approximately 87-141 story points, with large work intentionally decomposed into child tasks of 2, 3, or 5 points where possible.
@@ -30,14 +30,14 @@ The system is a trigger-based hybrid estimator. Normal flight uses camera ingest
| 06 | Tile Manager | Manage COGs, PostGIS manifests, sidecars, freshness, and orthorectified generated tiles | Bootstrap, shared helpers, config/errors | AZ-211 |
| 07 | MAVLink And GCS Integration | Consume FC telemetry and emit v1 `GPS_INPUT`/QGC status | Bootstrap, config/errors | AZ-210 |
| 08 | FDR And Observability | Record bounded replayable evidence and status | Bootstrap, config/errors, runtime DTOs | AZ-212 |
| 09 | Validation Harness | Drive blackbox, public replay, SITL, Jetson, and release evidence tests | All runtime components | AZ-217 |
| Test | E2E Test Suite | Separate black-box replay, SITL, Jetson, and release evidence tests; not onboard runtime | All runtime components | AZ-217 |
**Implementation order**:
1. Bootstrap and cross-cutting foundations: AZ-206, AZ-207, AZ-208.
2. Independent adapters/stores: AZ-209, AZ-210, AZ-211, AZ-212.
3. Estimation/relocalization: AZ-213, AZ-214, AZ-215.
4. Safety orchestration: AZ-216.
5. Validation and blackbox test implementation: AZ-217, AZ-218.
5. Separate e2e/blackbox test implementation: AZ-217, AZ-218.
## System Flows
@@ -79,7 +79,7 @@ See `risk_mitigations.md` for the full register.
| Tile Manager | 4 | 1 | 3 | 1 | 10 ACs |
| MAVLink And GCS Integration | 6 | 2 | 1 | 1 | 10 ACs |
| FDR And Observability | 6 | 1 | 1 | 1 | 11 ACs |
| Validation Harness | 9 | 2 | 1 | 2 | All AC groups |
| E2E Test Suite | 9 | 2 | 1 | 2 | All AC groups |
**Overall acceptance criteria coverage**: 39 / 39 acceptance criteria covered (100%).
**Restrictions coverage**: 10 / 10 restriction groups covered (100%).
@@ -99,7 +99,7 @@ See `risk_mitigations.md` for the full register.
| 9 | AZ-214: Satellite Service | Component 04 | L / 8-13 pts | AZ-209, AZ-211 |
| 10 | AZ-215: Anchor Verification | Component 05 | L / 8-13 pts | AZ-214, AZ-209, AZ-211 |
| 11 | AZ-216: Safety And Anchor Wrapper | Component 03 | XL / 13-21 pts | AZ-209, AZ-210, AZ-213, AZ-215 |
| 12 | AZ-217: Validation Harness | Component 09 | L / 8-13 pts | Component epics |
| 12 | AZ-217: E2E Test Suite | Separate test support | L / 8-13 pts | Component epics |
| 13 | AZ-218: Blackbox Tests | System tests | L / 8-13 pts | AZ-217, component epics |
**Total estimated effort**: 87-141 story points.
_docs/02_document/architecture.md
+1 -1
View File
@@ -33,7 +33,7 @@ Build a Jetson-hosted onboard localization pipeline for fixed-wing GPS-denied fl
**System boundaries**:
- In scope: onboard localization runtime, offline cache consumption, BASALT VIO integration, satellite anchor verification, MAVLink output, QGC status, FDR, generated tile metadata, validation harness.
- In scope: onboard localization runtime, offline cache consumption, BASALT VIO integration, satellite anchor verification, MAVLink output, QGC status, FDR, generated tile metadata, and a separate e2e/black-box test suite.
- Out of scope: upstream commercial satellite-provider sourcing, Satellite Service ingest implementation, AI mission-camera detection itself, PX4 support, raw-frame retention as a normal operating mode.
**External systems**:
_docs/02_document/components/02_basalt_vio_adapter/description.md
+1 -1
View File
@@ -8,7 +8,7 @@
**Upstream dependencies**: Camera ingest/calibration, MAVLink telemetry stream.
**Downstream consumers**: Safety/anchor wrapper, validation harness, FDR.
**Downstream consumers**: Safety/anchor wrapper, FDR, separate e2e test suite.
## 2. Internal Interfaces
_docs/02_document/components/03_safety_anchor_wrapper/description.md
+1 -1
View File
@@ -8,7 +8,7 @@
**Upstream dependencies**: BASALT VIO adapter, anchor verification, MAVLink telemetry, camera quality reports.
**Downstream consumers**: MAVLink/GCS integration, FDR, Tile Manager, validation harness.
**Downstream consumers**: MAVLink/GCS integration, FDR, Tile Manager, separate e2e test suite.
## 2. Internal Interfaces
_docs/02_document/components/09_validation_harness/description.md
-86
View File
@@ -1,86 +0,0 @@
# Validation Harness
## 1. High-Level Overview
**Purpose**: Drive black-box replay, public dataset, SITL, Jetson, and representative validation through the runtime's public interfaces.
**Architectural Pattern**: Test harness / scenario runner.
**Upstream dependencies**: Test data fixtures, public datasets, SITL, Jetson environment.
**Downstream consumers**: CI/CD pipeline, release evidence review.
## 2. Internal Interfaces
### Interface: `ScenarioRunner`
| Method | Input | Output | Async | Error Types |
|--------|-------|--------|-------|-------------|
| `run_scenario` | `ScenarioRequest` | `ScenarioReport` | Yes | `FixtureInvalid`, `RuntimeFailed`, `ThresholdFailed` |
| `validate_fixture` | `FixtureRequest` | `FixtureValidationReport` | No | `FixtureInvalid` |
**Input DTOs**:
```yaml
ScenarioRequest:
scenario_id: string
execution_environment: enum(replay, sitl, jetson, representative)
fixture_paths: list[string]
```
**Output DTOs**:
```yaml
ScenarioReport:
scenario_id: string
result: enum(pass, fail, blocked)
metrics: object
artifacts: list[path]
failure_reason: string optional
```
## 3. Data Access Patterns
Reads versioned fixtures and writes reports. Does not import runtime internals.
## 4. Implementation Details
**State Management**: Per-run temporary directories and report aggregation.
**Key Dependencies**:
| Library | Purpose |
|---------|---------|
| pytest or equivalent | Test orchestration |
| pymavlink/log parser | SITL and output validation |
| Docker/compose runner | Replay/SITL environment |
**Error Handling Strategy**:
- Fixture gaps are reported as blocked, not passed.
- Threshold failures include metrics and artifacts.
## 5. Caveats & Edge Cases
**Known limitations**:
- Public datasets are not final acceptance evidence unless representative and license-compatible.
- Missing synchronized target data remains a final acceptance blocker.
## 6. Dependency Graph
**Must be implemented after**: public interfaces are defined.
**Can be implemented in parallel with**: runtime components using mocks/fixtures only after interfaces are stable.
**Blocks**: CI/release gates.
## 7. Logging Strategy
| Log Level | When | Example |
|-----------|------|---------|
| ERROR | Runtime/test process fails | `scenario_failed id=... reason=...` |
| WARN | Fixture blocked | `fixture_blocked missing=...` |
| INFO | Scenario complete | `scenario_complete id=... result=pass` |
**Log format**: Test report CSV/Markdown plus structured runner logs.
**Log storage**: `test-results/`.
_docs/02_document/components/09_validation_harness/tests.md
-232
View File
@@ -1,232 +0,0 @@
# Test Specification — Validation Harness
## Acceptance Criteria Traceability
| AC ID | Acceptance Criterion | Test IDs | Coverage |
|-------|---------------------|----------|----------|
| AC-1.1 through AC-1.4 | Position accuracy, drift, confidence | IT-01, AT-01 | Covered |
| AC-2.1a/b, AC-2.2 | VO and satellite registration | IT-02, IT-03 | Covered |
| AC-3.1 through AC-3.5 | Resilience edge cases | IT-04, IT-05 | Covered |
| AC-4.1 through AC-4.5 | Latency, memory, MAVLink streaming | PT-01, IT-06 | Covered |
| AC-5.1 through AC-5.3 | Startup/failsafe/reboot | IT-07 | Covered |
| AC-6.1 through AC-6.3 | QGC/GCS/WGS84 | IT-06 | Covered |
| AC-7.1, AC-7.2 | Object coordinate contract | IT-08 | Covered at system boundary |
| AC-8.1 through AC-8.6 | Offline cache, freshness, tiles, VPR | IT-03, IT-09, ST-01 | Covered |
| AC-NEW-1 through AC-NEW-8 | Cold start, spoofing, FDR, false-position, thermal, freshness, poisoning, blackout | IT-05, IT-07, PT-02, ST-01, AT-02 | Covered |
## Blackbox Tests
### IT-01: Still-Image Accuracy Runner
**Summary**: Verify project still-image replay reports frame-center accuracy.
**Traces to**: AC-1.1, AC-1.2, AC-1.4
**Input data**: Project mapped images and `expected_results/results_report.md`.
**Expected result**: Report includes per-image error, aggregate 50 m/20 m pass rates, covariance, source label, and anchor age.
**Max execution time**: 15 minutes.
---
### IT-02: Synchronized VIO Replay Runner
**Summary**: Verify Derkachi and public/representative synchronized data can drive BASALT/wrapper tests.
**Traces to**: AC-1.3, AC-2.1a, AC-2.2
**Input data**: Derkachi cropped nadir video + telemetry fixture, MUN-FRL preferred slice, or representative synchronized dataset.
**Expected result**: Runner validates fixture alignment, trajectory comparison, VIO registration, latency, and covariance calibration where calibration data supports it.
**Max execution time**: Dataset-dependent.
---
### IT-03: Satellite Anchor Replay Runner
**Summary**: Verify VPR and anchor verification test scenarios are executable.
**Traces to**: AC-2.1b, AC-2.2, AC-8.1, AC-8.2, AC-8.6
**Input data**: ALTO/AerialVL/representative aerial localization fixture plus cache.
**Expected result**: Runner reports retrieval recall, MRE, accepted/rejected anchors, and freshness behavior.
**Max execution time**: Dataset-dependent.
---
### IT-04: Outlier/Sharp-Turn/Disconnected Runner
**Summary**: Verify resilience scenarios are executable and reported.
**Traces to**: AC-3.1, AC-3.2, AC-3.3, AC-3.4
**Input data**: Synthetic and public disconnected-segment fixtures.
**Expected result**: Runner validates relocalization and records degraded-mode timelines.
**Max execution time**: 30 minutes.
---
### IT-05: Blackout And Spoofing Runner
**Summary**: Verify total blackout plus spoofing scenarios can be driven through SITL/replay.
**Traces to**: AC-3.5, AC-NEW-2, AC-NEW-8
**Input data**: Plane SITL spoofing scenario with 5 s, 15 s, and 35 s blackout windows.
**Expected result**: Runner measures <=400 ms mode switch, <3 s promotion, monotonic covariance, and failsafe thresholds.
**Max execution time**: 30 minutes.
---
### IT-06: MAVLink/QGC Contract Runner
**Summary**: Verify MAVLink output and GCS status assertions are automated.
**Traces to**: AC-4.3, AC-4.4, AC-4.5, AC-6.1, AC-6.2, AC-6.3
**Input data**: Plane SITL, QGC observer/log parser, position fixtures.
**Expected result**: Runner validates v1 GPS_INPUT-only output, WGS84 coordinates, status rate, and command ingress.
**Max execution time**: 60 minutes.
---
### IT-07: Startup/Reboot Runner
**Summary**: Verify cold-start and reboot scenarios are measurable.
**Traces to**: AC-5.1, AC-5.2, AC-5.3, AC-NEW-1
**Input data**: 50 cold-start runs and companion reboot trace.
**Expected result**: First valid `GPS_INPUT` <30 s p95; reboot reinitializes from FC state.
**Max execution time**: Runset-dependent.
---
### IT-08: Object Coordinate Contract Runner
**Summary**: Verify AI-camera object coordinate request contract at system boundary.
**Traces to**: AC-7.1, AC-7.2
**Input data**: Frame-center estimate, object pixel/angle fixture, gimbal angle, altitude.
**Expected result**: Output coordinate includes frame-center-consistent accuracy and maneuvering-flight projection error bound.
**Max execution time**: 5 minutes.
---
### IT-09: Tile Manager Runner
**Summary**: Verify cache, generated tiles, and storage tests are executable.
**Traces to**: AC-8.3, AC-8.4, AC-8.5, AC-NEW-6, AC-NEW-7
**Input data**: Cache integrity fixtures, generated tile scenarios, PostGIS manifest.
**Expected result**: Runner validates cache load, tile write gates, no raw-frame retention, stale rejection, and poisoning budget evidence.
**Max execution time**: Dataset-dependent.
## Performance Tests
### PT-01: End-To-End Release Gate Runner
**Summary**: Verify performance and resource tests can run in the proper environment.
**Traces to**: AC-4.1, AC-4.2, AC-NEW-5
**Load scenario**:
- Environments: replay, Jetson hardware, SITL.
- Duration: smoke, nightly, and release-gate profiles.
| Metric | Target | Failure Threshold |
|--------|--------|-------------------|
| End-to-end p95 | <400 ms | >=400 ms |
| Memory | <8 GB | >=8 GB |
| Thermal throttle | 0 events in release gate | Any throttle event |
---
### PT-02: FDR/Storage Runner
**Summary**: Verify 8-hour storage/endurance test orchestration.
**Traces to**: AC-NEW-3
| Metric | Target | Failure Threshold |
|--------|--------|-------------------|
| FDR cap | <=64 GB | >64 GB |
| Rollover logging | Complete | Missing rollover event |
## Security Tests
### ST-01: Security Fixture Runner
**Summary**: Verify stale/tampered cache, spoofed MAVLink, and false-anchor scenarios are automated.
**Traces to**: AC-NEW-4, AC-NEW-6, AC-NEW-7
**Attack vector**: Cache tampering, stale imagery, spoofed GPS, impossible anchors.
**Test procedure**:
1. Load each security fixture.
2. Run scenario through public runtime interfaces.
3. Validate output labels, FDR, and rejection reasons.
**Expected behavior**: No tampered/stale/spoofed input produces a trusted false fix.
**Pass criteria**: 0 accepted unsafe anchors or spoofed GPS promotions outside gates.
## Acceptance Tests
### AT-01: Traceability Completeness Report
**Summary**: Verify every AC has executable or explicitly blocked test coverage.
**Traces to**: All ACs
| Step | Action | Expected Result |
|------|--------|-----------------|
| 1 | Read traceability matrix | All ACs mapped to tests |
| 2 | Run fixture validation | Missing public/representative data is reported as blocked, not passed |
---
### AT-02: Release Evidence Bundle
**Summary**: Verify release evidence can be assembled.
**Traces to**: AC-NEW-1 through AC-NEW-8
| Step | Action | Expected Result |
|------|--------|-----------------|
| 1 | Run release profile | Reports, tlogs, FDR summaries, cache reports are produced |
| 2 | Collate artifacts | Bundle contains pass/fail status and residual blockers |
## Test Data Management
| Data Set | Description | Source | Size |
|----------|-------------|--------|------|
| `project_60_still_images` | Frame-center geolocation smoke | Project data | Project size |
| `public_dataset_slices` | MUN-FRL/ALTO/Kagaru/EPFL/AerialVL as licensed | Public pinned fixtures | Dataset-dependent |
| `sitl_scenarios` | Plane spoofing/failsafe traces | Generated | Small |
| `security_fixtures` | Stale/tampered/cache poisoning cases | Generated | Small |
**Setup procedure**: Create isolated run directory, restore PostgreSQL schema, mount fixtures read-only, and start requested environment.
**Teardown procedure**: Stop environments, archive reports, drop run schema, and delete temp volumes.
**Data isolation strategy**: Unique run ID, schema, ports, cache staging directory, and FDR directory per scenario.
_docs/02_document/contracts/shared/config_errors_telemetry.md
+51
View File
@@ -0,0 +1,51 @@
# Contract: Config Errors Telemetry
**Component**: shared/config, shared/errors, shared/telemetry
**Producer task**: AZ-222 — AZ-222_runtime_config_errors_telemetry.md
**Consumer tasks**: AZ-223, AZ-224, AZ-225, AZ-226, AZ-227, AZ-228, AZ-229, AZ-230, AZ-231, AZ-232
**Version**: 1.0.0
**Status**: draft
**Last Updated**: 2026-05-03
## Purpose
Defines shared runtime configuration, error/result envelope, health, and telemetry metadata behavior consumed by all runtime components.
## Shape
| Contract | Required Behavior |
|----------|-------------------|
| Runtime profile | environment-specific settings loaded and validated before use |
| Error envelope | component, category, message, cause, retryability, severity |
| Health event | liveness/readiness status, dependency state, timestamp, component |
| Metrics labels | bounded component/action/status labels suitable for runtime reports |
## Invariants
- Missing required production settings fail startup or readiness loudly.
- Errors are returned or logged with component and category; no silent suppression.
- Secrets are referenced, not serialized into FDR, logs, or metrics.
## Non-Goals
- Does not define component-specific business errors.
- Does not replace FDR payload schemas.
## Versioning Rules
- Removing required config keys or error categories requires a major version bump.
- Adding optional health fields or metrics labels requires a minor version bump.
## Test Cases
| Case | Input | Expected | Notes |
|------|-------|----------|-------|
| missing-required-prod | production profile missing cache dir | readiness/startup failure | Clear error category |
| secret-value | signing key ref present | only key ref logged | No secret leakage |
| component-error | component reports dependency failure | structured envelope emitted | FDR-safe |
## Change Log
| Version | Date | Change | Author |
|---------|------|--------|--------|
| 1.0.0 | 2026-05-03 | Initial contract | autodev |
_docs/02_document/contracts/shared/geometry_time_sync.md
+52
View File
@@ -0,0 +1,52 @@
# Contract: Geometry And Time Sync Helpers
**Component**: shared/geo_geometry, shared/time_sync
**Producer task**: AZ-221 — AZ-221_shared_geometry_time_sync.md
**Consumer tasks**: AZ-223, AZ-225, AZ-226, AZ-228, AZ-230, AZ-231, AZ-232
**Version**: 1.0.0
**Status**: draft
**Last Updated**: 2026-05-03
## Purpose
Defines shared geospatial and timestamp helper behavior used by runtime components to avoid duplicated math and inconsistent frame/IMU alignment.
## Shape
| API Area | Shape | Errors |
|----------|-------|--------|
| Coordinate conversion | WGS84/local tangent conversions and distance calculations | invalid CRS, missing origin |
| Camera footprint | intrinsics/extrinsics/attitude/altitude to footprint and GSD | invalid calibration, missing altitude |
| Homography metrics | homography/covariance conversions and MRE support | invalid geometry |
| Time sync | monotonic checks, frame-to-IMU window selection, replay ordering | timestamp mismatch, gap/jitter exceeded |
## Invariants
- Helpers are deterministic for the same calibration, pose, and timestamp inputs.
- Time helpers report gaps/jitter instead of silently dropping samples.
- Geometry helpers do not decide safety policy; callers decide degrade/reject behavior.
## Non-Goals
- No VIO state estimation.
- No MAVLink parsing beyond normalized timestamp fields.
- No tile freshness or cache policy decisions.
## Versioning Rules
- Breaking changes to units, coordinate frames, or timestamp semantics require a major version bump.
- New helper outputs may be added as optional fields in minor versions.
## Test Cases
| Case | Input | Expected | Notes |
|------|-------|----------|-------|
| valid-wgs84-local | known WGS84 point and origin | round-trip within tolerance | Uses representative coordinates |
| frame-imu-window | frame timestamp plus IMU samples | correct aligned window | Includes gap metrics |
| invalid-calibration | missing intrinsics/extrinsics | explicit error | No silent fallback |
## Change Log
| Version | Date | Change | Author |
|---------|------|--------|--------|
| 1.0.0 | 2026-05-03 | Initial contract | autodev |
_docs/02_document/contracts/shared/runtime_contracts.md
+56
View File
@@ -0,0 +1,56 @@
# Contract: Runtime Shared Contracts
**Component**: shared/contracts
**Producer task**: AZ-220 — AZ-220_shared_runtime_contracts.md
**Consumer tasks**: AZ-223, AZ-224, AZ-225, AZ-226, AZ-227, AZ-228, AZ-229, AZ-230, AZ-231, AZ-232
**Version**: 1.0.0
**Status**: draft
**Last Updated**: 2026-05-03
## Purpose
Defines the shared runtime DTO/event contract surface that component implementations consume instead of inventing local shapes.
## Shape
| Contract | Required Fields / Methods | Consumers |
|----------|---------------------------|-----------|
| `FramePacket` | frame ID, timestamp, image reference, calibration ID, occlusion, quality, normalization hint | camera, VIO, Satellite Service, Anchor Verification, Tile Manager, FDR |
| `TelemetrySample` | timestamp, IMU, attitude, altitude, airspeed, GPS health | MAVLink, VIO, safety wrapper, FDR |
| `VioStatePacket` | timestamp, relative pose, velocity, bias, tracking quality, covariance hint | VIO, safety wrapper, FDR |
| `PositionEstimate` | WGS84 coordinates, covariance, source label, fix type, horizontal accuracy, anchor age | safety wrapper, MAVLink, Tile Manager, FDR |
| `VprCandidate` | chunk ID, tile ID, score, footprint, freshness status | Satellite Service, Anchor Verification, FDR |
| `AnchorDecision` | candidate ID, acceptance result, estimated pose, inliers, MRE, rejection reason | Anchor Verification, safety wrapper, FDR |
| `CacheTileRecord` | tile ID, CRS, meters per pixel, capture date, signature/hash, trust level | Tile Manager, Satellite Service, Anchor Verification |
| `FdrEvent` | event type, timestamp, component, severity, payload reference, mission/run ID | all runtime components |
## Invariants
- Timestamps are normalized to a shared monotonic nanosecond representation before cross-component use.
- Confidence fields must not under-report known uncertainty.
- Raw frame payloads are referenced, not persisted in shared DTOs.
- Generated tile and anchor records must carry provenance/freshness metadata.
## Non-Goals
- Does not prescribe internal classes or storage implementation.
- Does not define e2e test runner-only report schemas.
## Versioning Rules
- Removing or renaming a field requires a major version bump.
- Adding optional telemetry or diagnostic fields requires a minor version bump.
## Test Cases
| Case | Input | Expected | Notes |
|------|-------|----------|-------|
| valid-frame | frame with timestamp, calibration, quality | accepted by consumers | Includes normalization hint |
| invalid-time | non-monotonic timestamp | rejected or marked invalid | Time-sync contract decides details |
| stale-anchor | anchor decision with stale freshness | rejected/down-confidenced | Safety wrapper must not accept blindly |
## Change Log
| Version | Date | Change | Author |
|---------|------|--------|--------|
| 1.0.0 | 2026-05-03 | Initial contract | autodev |
_docs/02_document/diagrams/component_overview.md
+1 -1
View File
@@ -10,7 +10,7 @@ flowchart LR
cache[06 Tile Manager]
mav[07 MAVLink And GCS Integration]
fdr[08 FDR And Observability]
tests[09 Validation Harness]
tests[[Separate E2E Test Suite]]
navCam[[Nav Camera]] --> camera
fc[[ArduPilot Plane FC]] --> mav
_docs/02_document/epics.md
+11 -6
View File
@@ -21,8 +21,8 @@
| 9 | AZ-214 | Satellite Service | component | AZ-206, AZ-207, AZ-208, AZ-209, AZ-211 | L / 8-13 pts |
| 10 | AZ-215 | Anchor Verification | component | AZ-206, AZ-207, AZ-208, AZ-209, AZ-211, AZ-214 | L / 8-13 pts |
| 11 | AZ-216 | Safety And Anchor Wrapper | component | AZ-206, AZ-207, AZ-208, AZ-209, AZ-210, AZ-213, AZ-215 | XL / 13-21 pts |
| 12 | AZ-217 | Validation Harness | component | AZ-206, AZ-207, AZ-208, component epics | L / 8-13 pts |
| 13 | AZ-218 | Blackbox Tests | blackbox-tests | AZ-206, AZ-217, component epics | L / 8-13 pts |
| 12 | AZ-217 | E2E Test Suite | test-support | component epics | L / 8-13 pts |
| 13 | AZ-218 | Blackbox Tests | blackbox-tests | AZ-217, component epics | L / 8-13 pts |
## Component Mapping
@@ -40,7 +40,7 @@
| `components/06_cache_tile_lifecycle/` | AZ-211 |
| `components/07_mavlink_gcs_integration/` | AZ-210 |
| `components/08_fdr_observability/` | AZ-212 |
| `components/09_validation_harness/` | AZ-217 |
| `tests/e2e-test-suite.md`, `tests/blackbox-tests.md`, `tests/environment.md` | AZ-217 |
| System blackbox/performance/resilience/security/resource tests | AZ-218 |
## Epic Relationship Diagram
@@ -58,7 +58,7 @@ flowchart TD
retrieval[AZ-214 Satellite Service]
anchor[AZ-215 Anchor Verification]
safety[AZ-216 Safety And Anchor Wrapper]
validation[AZ-217 Validation Harness]
validation[AZ-217 E2E Test Suite]
blackbox[AZ-218 Blackbox Tests]
bootstrap --> geo
@@ -96,9 +96,14 @@ flowchart TD
safety --> fdr
camera --> fdr
cache --> fdr
validation --> blackbox
safety --> validation
fdr --> validation
camera --> validation
mavlink --> validation
retrieval --> validation
anchor --> validation
cache --> validation
validation --> blackbox
```
## Cross-Cutting Ownership
@@ -123,7 +128,7 @@ flowchart TD
- AZ-214 — Satellite Service
- AZ-215 — Anchor Verification
- AZ-216 — Safety And Anchor Wrapper
- AZ-217 — Validation Harness
- AZ-217 — E2E Test Suite
- AZ-218 — Blackbox Tests
## Tracker Notes
_docs/02_document/module-layout.md
+32 -31
View File
@@ -19,6 +19,7 @@
- **Epic**: AZ-209
- **Directory**: `src/camera_ingest_calibration/`
- **Technologies**: Python, OpenCV 4.x, camera SDK/V4L2/GigE adapter boundary, calibration files, shared geometry/time helpers
- **Public API**:
- `src/camera_ingest_calibration/__init__.py`
- `src/camera_ingest_calibration/types.py`
@@ -28,13 +29,14 @@
- `src/camera_ingest_calibration/_*.py`
- **Owns (exclusive write during implementation)**: `src/camera_ingest_calibration/**`
- **Imports from**: shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: BASALT VIO Adapter, Satellite Service, Anchor Verification, Tile Manager, FDR And Observability, Validation Harness
- **Consumed by**: BASALT VIO Adapter, Satellite Service, Anchor Verification, Tile Manager, FDR And Observability
### Component: BASALT VIO Adapter
- **Epic**: AZ-213
- **Directory**: `src/basalt_vio_adapter/`
- **Native Directory**: `src/native/basalt_bridge/`
- **Technologies**: Python adapter, C++ native bridge, BASALT, Eigen/Sophus or BASALT math stack, OpenCV 4.x, shared time-sync contracts
- **Public API**:
- `src/basalt_vio_adapter/__init__.py`
- `src/basalt_vio_adapter/types.py`
@@ -47,12 +49,13 @@
- `src/basalt_vio_adapter/**`
- `src/native/basalt_bridge/**`
- **Imports from**: Camera Ingest And Calibration, MAVLink And GCS Integration, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: Safety And Anchor Wrapper, FDR And Observability, Validation Harness
- **Consumed by**: Safety And Anchor Wrapper, FDR And Observability
### Component: Safety And Anchor Wrapper
- **Epic**: AZ-216
- **Directory**: `src/safety_anchor_wrapper/`
- **Technologies**: Python state machine, OpenCV geometry helpers, covariance/gating logic, shared DTO contracts, MAVLink output DTOs
- **Public API**:
- `src/safety_anchor_wrapper/__init__.py`
- `src/safety_anchor_wrapper/types.py`
@@ -62,13 +65,14 @@
- `src/safety_anchor_wrapper/_*.py`
- **Owns (exclusive write during implementation)**: `src/safety_anchor_wrapper/**`
- **Imports from**: BASALT VIO Adapter, Anchor Verification, MAVLink And GCS Integration, Camera Ingest And Calibration, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: MAVLink And GCS Integration, Tile Manager, FDR And Observability, Validation Harness
- **Consumed by**: MAVLink And GCS Integration, Tile Manager, FDR And Observability
### Component: Satellite Service
- **Epic**: AZ-214
- **Directory**: `src/satellite_service/`
- **Native Directory**: `src/native/tensor_rt/`
- **Technologies**: Python service adapter, DINOv2-VLAD descriptors, ONNX/TensorRT candidate path, CPU FAISS, offline package sync client
- **Public API**:
- `src/satellite_service/__init__.py`
- `src/satellite_service/types.py`
@@ -81,7 +85,7 @@
- `src/satellite_service/**`
- `src/native/tensor_rt/**`
- **Imports from**: Camera Ingest And Calibration, Tile Manager, Safety And Anchor Wrapper, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: Anchor Verification, FDR And Observability, Validation Harness
- **Consumed by**: Anchor Verification, FDR And Observability
- **Network invariant**: external Satellite Service sync is allowed only pre-flight or post-flight; no mid-flight satellite-provider or suite-service calls.
### Component: Anchor Verification
@@ -89,6 +93,7 @@
- **Epic**: AZ-215
- **Directory**: `src/anchor_verification/`
- **Native Directory**: `src/native/feature_matching/`
- **Technologies**: Python validation pipeline, ALIKED/DISK + LightGlue, OpenCV RANSAC/USAC, SIFT/ORB baseline, native feature-matching bridge
- **Public API**:
- `src/anchor_verification/__init__.py`
- `src/anchor_verification/types.py`
@@ -101,12 +106,13 @@
- `src/anchor_verification/**`
- `src/native/feature_matching/**`
- **Imports from**: Satellite Service, Camera Ingest And Calibration, Tile Manager, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: Safety And Anchor Wrapper, FDR And Observability, Validation Harness
- **Consumed by**: Safety And Anchor Wrapper, FDR And Observability
### Component: Tile Manager
- **Epic**: AZ-211
- **Directory**: `src/tile_manager/`
- **Technologies**: Python repository/policy layer, PostgreSQL/PostGIS, GDAL/rasterio COG handling, signed JSON sidecars, OpenCV/GDAL orthorectification, hash/signature validation
- **Public API**:
- `src/tile_manager/__init__.py`
- `src/tile_manager/types.py`
@@ -119,12 +125,13 @@
- `migrations/postgresql/cache_*.sql`
- `migrations/seed/cache_*`
- **Imports from**: Camera Ingest And Calibration, Safety And Anchor Wrapper, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: Satellite Service, Anchor Verification, FDR And Observability, Validation Harness
- **Consumed by**: Satellite Service, Anchor Verification, FDR And Observability
### Component: MAVLink And GCS Integration
- **Epic**: AZ-210
- **Directory**: `src/mavlink_gcs_integration/`
- **Technologies**: Python, MAVSDK telemetry subscriptions, pymavlink `GPS_INPUT` emission, MAVLink/QGC status messages
- **Public API**:
- `src/mavlink_gcs_integration/__init__.py`
- `src/mavlink_gcs_integration/types.py`
@@ -134,12 +141,13 @@
- `src/mavlink_gcs_integration/_*.py`
- **Owns (exclusive write during implementation)**: `src/mavlink_gcs_integration/**`
- **Imports from**: Safety And Anchor Wrapper, shared/contracts, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: BASALT VIO Adapter, Safety And Anchor Wrapper, FDR And Observability, Validation Harness
- **Consumed by**: BASALT VIO Adapter, Safety And Anchor Wrapper, FDR And Observability
### Component: FDR And Observability
- **Epic**: AZ-212
- **Directory**: `src/fdr_observability/`
- **Technologies**: Python append/export layer, PostgreSQL event index, CBOR segment payloads, optional Parquet export, structured logging/health events
- **Public API**:
- `src/fdr_observability/__init__.py`
- `src/fdr_observability/types.py`
@@ -152,25 +160,7 @@
- `migrations/postgresql/fdr_*.sql`
- `migrations/seed/fdr_*`
- **Imports from**: shared/contracts, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: all runtime components, Validation Harness
### Component: Validation Harness
- **Epic**: AZ-217
- **Directory**: `src/validation_harness/`
- **Public API**:
- `src/validation_harness/__init__.py`
- `src/validation_harness/types.py`
- `src/validation_harness/interfaces.py`
- **Internal (do NOT import from other components)**:
- `src/validation_harness/internal/*`
- `src/validation_harness/_*.py`
- **Owns (exclusive write during implementation)**:
- `src/validation_harness/**`
- `e2e/replay/**`
- `e2e/reports/**`
- **Imports from**: public runtime interfaces only, shared/contracts, shared/geo_geometry, shared/time_sync, shared/config, shared/errors, shared/telemetry
- **Consumed by**: CI/CD pipeline and release evidence review
- **Consumed by**: all runtime components
## Shared / Cross-Cutting
@@ -178,6 +168,7 @@
- **Epic**: AZ-206
- **Directory**: `src/shared/contracts/`
- **Technologies**: Python typed DTOs, schema/contract definitions, Markdown API-contract documents
- **Purpose**: Shared DTOs, protocol shapes, schemas, and public contract exports.
- **Owned by**: initial structure and shared-contract tasks under AZ-206.
- **Consumed by**: all components.
@@ -186,30 +177,34 @@
- **Epic**: AZ-207
- **Directory**: `src/shared/geo_geometry/`
- **Technologies**: Python geometry utilities, OpenCV 4.x, WGS84/local-frame math, homography/covariance conversions
- **Purpose**: WGS84/local conversions, GSD, camera footprint projection, homography/covariance unit conversion, and distance calculations.
- **Owned by**: shared geometry task under AZ-207.
- **Consumed by**: Camera Ingest And Calibration, Safety And Anchor Wrapper, Anchor Verification, Tile Manager, Validation Harness.
- **Consumed by**: Camera Ingest And Calibration, Safety And Anchor Wrapper, Anchor Verification, Tile Manager.
### shared/time_sync
- **Epic**: AZ-207
- **Directory**: `src/shared/time_sync/`
- **Technologies**: Python timestamp utilities, monotonic-clock validation, MAVLink/camera timestamp normalization, replay ordering checks
- **Purpose**: Monotonic timestamp checks, frame-to-IMU alignment, clock-domain metadata, replay ordering, and gap/jitter metrics.
- **Owned by**: time-sync task under AZ-207.
- **Consumed by**: Camera Ingest And Calibration, BASALT VIO Adapter, MAVLink And GCS Integration, FDR And Observability, Validation Harness.
- **Consumed by**: Camera Ingest And Calibration, BASALT VIO Adapter, MAVLink And GCS Integration, FDR And Observability.
### shared/config
- **Epic**: AZ-208
- **Directory**: `src/shared/config/`
- **Technologies**: Python configuration loader, environment variables, `.env.example`, startup readiness validation
- **Purpose**: Runtime profile loading, environment validation, typed settings, and startup readiness inputs.
- **Owned by**: runtime configuration task under AZ-208.
- **Consumed by**: all runtime components and Validation Harness.
- **Consumed by**: all runtime components.
### shared/errors
- **Epic**: AZ-208
- **Directory**: `src/shared/errors/`
- **Technologies**: Python exception/result envelope types, shared error categories, fail-fast helpers
- **Purpose**: Error categories, result envelopes, fail-fast helpers, and non-silent exception contracts.
- **Owned by**: runtime error contract task under AZ-208.
- **Consumed by**: all components.
@@ -218,6 +213,7 @@
- **Epic**: AZ-208
- **Directory**: `src/shared/telemetry/`
- **Technologies**: Python structured logging, metrics labels, health event DTOs, FDR-safe telemetry metadata
- **Purpose**: Structured logging, metrics labels, health event shapes, and FDR-safe event metadata helpers.
- **Owned by**: observability/config contract task under AZ-208.
- **Consumed by**: all components.
@@ -228,7 +224,6 @@ Read top-to-bottom; an upper layer may import from a lower layer but never the r
| Layer | Components | May import from |
|-------|------------|-----------------|
| 5. Validation / Evidence | Validation Harness | 1, 2, 3, 4 public interfaces |
| 4. Runtime Output / Coordination | Safety And Anchor Wrapper, MAVLink And GCS Integration, FDR And Observability | 1, 2, 3 public interfaces |
| 3. Perception / Satellite Anchor | BASALT VIO Adapter, Satellite Service, Anchor Verification | 1, 2 public interfaces |
| 2. Data Ingest / Persistence | Camera Ingest And Calibration, Tile Manager | 1 |
@@ -236,9 +231,15 @@ Read top-to-bottom; an upper layer may import from a lower layer but never the r
Violations of this table are Architecture findings in code-review Phase 7 and are High severity.
## Out-of-Product E2E Test Suite
The e2e replay/SITL/Jetson validation suite is not a product component and must not receive Step 6 product implementation tasks. It owns test-support artifacts under `tests/blackbox/**`, `tests/e2e/**`, `e2e/replay/**`, and `e2e/reports/**`, and it exercises the runtime only through public file, MAVLink, cache, status, and FDR interfaces.
- **Technologies**: Python, pytest-style runner, Docker/compose, pymavlink/log parser, ArduPilot Plane SITL, QGC observer/log parser, CSV/Markdown reports
## Self-Verification
- Every component under `_docs/02_document/components/` has a mapping entry.
- Every runtime component under `_docs/02_document/components/` has a mapping entry.
- Cross-cutting epics AZ-206, AZ-207, and AZ-208 have shared ownership entries.
- Layering covers all components and keeps shared code at the bottom.
- Component-owned paths do not overlap, except explicitly assigned native directories.
_docs/02_document/risk_mitigations.md
+4 -4
View File
@@ -55,7 +55,7 @@
**Trigger conditions**: Manufacturer documentation or hardware testing contradicts assumed FPS, interface, temperature, or lens characteristics.
**Affected components**: Camera ingest/calibration, BASALT VIO adapter, validation harness, deployment procedures.
**Affected components**: Camera ingest/calibration, BASALT VIO adapter, separate e2e test suite, deployment procedures.
**Mitigation strategy**:
1. Make camera specification verification a bootstrap task.
@@ -76,7 +76,7 @@
**Trigger conditions**: Public or representative replay shows high drift, frequent tracking loss, or poor initialization.
**Affected components**: BASALT VIO adapter, safety/anchor wrapper, validation harness.
**Affected components**: BASALT VIO adapter, safety/anchor wrapper, separate e2e test suite.
**Mitigation strategy**:
1. Run MUN-FRL first for synchronized nadir camera + IMU + ground truth.
@@ -160,7 +160,7 @@
**Trigger conditions**: Relocalization exceeds p95 latency, memory budget, or causes thermal throttling.
**Affected components**: Satellite Service, anchor verification, validation harness.
**Affected components**: Satellite Service, anchor verification, separate e2e test suite.
**Mitigation strategy**:
1. Keep VPR/local matching trigger-based.
@@ -244,7 +244,7 @@
**Trigger conditions**: Plane SITL rejects or mishandles emitted `GPS_INPUT`, or QGC status is insufficient.
**Affected components**: MAVLink/GCS integration, safety/anchor wrapper, validation harness.
**Affected components**: MAVLink/GCS integration, safety/anchor wrapper, separate e2e test suite.
**Mitigation strategy**:
1. Use pymavlink for exact `GPS_INPUT` field control.
_docs/02_document/system-flows.md
+5 -5
View File
@@ -10,7 +10,7 @@
| F4 | Visual blackout / spoofing degraded mode | Image-quality failure and GPS health failure | Camera ingest, MAVLink telemetry, safety/anchor wrapper, QGC, FDR | Critical |
| F5 | Generated tile lifecycle | High-confidence pose + usable frame | Camera ingest, safety/anchor wrapper, Tile Manager, FDR | Medium |
| F6 | Post-flight sync and audit | Landing / operator offload | Tile Manager, Satellite Service, FDR | Medium |
| F7 | Validation replay | Test harness invocation | Validation harness, system runtime, public datasets, SITL | High |
| F7 | E2E validation replay | Test-suite invocation | Separate e2e test suite, system runtime, public datasets, SITL | High |
## Flow Dependencies
@@ -298,7 +298,7 @@ After landing, generated tiles and FDR evidence are exported through Satellite S
### Description
The validation harness runs deterministic still-image, public dataset, SITL, Jetson, and representative replay scenarios against public interfaces.
The separate e2e test suite runs deterministic still-image, public dataset, SITL, Jetson, and representative replay scenarios against public interfaces.
### Preconditions
@@ -309,9 +309,9 @@ The validation harness runs deterministic still-image, public dataset, SITL, Jet
| Step | From | To | Data | Format |
|------|------|----|------|--------|
| 1 | Validation harness | Runtime | Images/telemetry/cache fixtures | File/stream/MAVLink |
| 2 | Runtime | Validation harness | GPS_INPUT/FDR/status | MAVLink/log files |
| 3 | Validation harness | Reports | Pass/fail metrics | CSV/Markdown |
| 1 | E2E test suite | Runtime | Images/telemetry/cache fixtures | File/stream/MAVLink |
| 2 | Runtime | E2E test suite | GPS_INPUT/FDR/status | MAVLink/log files |
| 3 | E2E test suite | Reports | Pass/fail metrics | CSV/Markdown |
### Performance Expectations
_docs/02_document/tests/e2e-test-suite.md
+81
View File
@@ -0,0 +1,81 @@
# E2E Test Suite
## Scope
The e2e test suite is separate test tooling, not part of the onboard runtime. It drives black-box replay, public dataset, SITL, Jetson, and representative validation through public runtime interfaces only.
## Purpose
- Feed navigation frames, telemetry traces, cache manifests, and fault triggers into the system under test.
- Validate emitted coordinates, confidence fields, MAVLink `GPS_INPUT`, QGC status, FDR, and generated-tile evidence.
- Produce release evidence without importing runtime internals.
## Ownership
- **Epic**: AZ-217 (E2E Test Suite / test-support work, not product runtime)
- **Owns**:
- `tests/blackbox/**`
- `tests/e2e/**`
- `e2e/replay/**`
- `e2e/reports/**`
- **Does not own**:
- `src/**`
- runtime component internals
- production deployment code
## Public Interfaces Under Test
| Interface | Protocol / Contract |
|-----------|---------------------|
| Navigation frames | Ordered image/video replay with timestamps |
| FC telemetry | MAVLink replay or generated stream |
| Satellite cache | Local COG + manifest + descriptor fixtures |
| GPS output | MAVLink `GPS_INPUT` |
| Operator status | QGC-visible MAVLink status |
| FDR | Filesystem/database-backed evidence outputs |
## Runner Contract
| Method | Input | Output | Error Types |
|--------|-------|--------|-------------|
| `run_scenario` | `ScenarioRequest` | `ScenarioReport` | `FixtureInvalid`, `RuntimeFailed`, `ThresholdFailed` |
| `validate_fixture` | `FixtureRequest` | `FixtureValidationReport` | `FixtureInvalid` |
```yaml
ScenarioRequest:
scenario_id: string
execution_environment: enum(replay, sitl, jetson, representative)
fixture_paths: list[string]
ScenarioReport:
scenario_id: string
result: enum(pass, fail, blocked)
metrics: object
artifacts: list[path]
failure_reason: string optional
```
## Scenario Coverage
| Scenario | Purpose | Evidence |
|----------|---------|----------|
| Still-image accuracy runner | Verify project still-image replay reports frame-center accuracy | Per-image error, aggregate pass rates, covariance, source label, anchor age |
| Synchronized VIO replay runner | Verify Derkachi and public/representative synchronized data drive BASALT/wrapper tests | Fixture alignment, trajectory comparison, VIO registration, latency, covariance calibration |
| Satellite anchor replay runner | Verify VPR and anchor verification scenarios are executable | Retrieval recall, MRE, accepted/rejected anchors, freshness behavior |
| Outlier/sharp-turn/disconnected runner | Verify relocalization resilience scenarios are executable | Degraded-mode timelines and relocalization outcomes |
| Blackout and spoofing runner | Verify total blackout plus spoofing through SITL/replay | Mode-switch timing, covariance growth, failsafe thresholds |
| MAVLink/QGC contract runner | Verify MAVLink output and GCS status assertions | `GPS_INPUT`, WGS84 coordinates, status rate, command ingress |
| Startup/reboot runner | Verify cold-start and companion reboot scenarios | First valid `GPS_INPUT` p95 and FC-state reinitialization |
| Object coordinate contract runner | Verify AI-camera object coordinate request at system boundary | Frame-center-consistent coordinate accuracy and projection bound |
| Tile Manager runner | Verify cache, generated tiles, and storage tests | Cache load, tile write gates, no raw-frame retention, stale rejection, poisoning evidence |
## Release Evidence
The suite assembles CSV, Markdown, MAVLink tlogs, FDR summaries, cache validation reports, and pass/fail metadata into release evidence bundles. Missing public or representative data is reported as `blocked`, not `passed`.
## Non-Responsibilities
- No onboard flight logic.
- No direct estimator, BASALT, wrapper, or tile-manager imports.
- No mutation of runtime internal state.
- No production service APIs.