azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 01:21:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-263] [AZ-264] [AZ-265] Decompose: layout, helpers epic, replay epic

Browse Source 
Decompose Step 1 + Step 1.5 + new cycle-1 epics:

- Step 1 (Bootstrap): AZ-263 spec at _docs/02_tasks/todo/. Single
  top-level Python package src/gps_denied_onboard/ + nested
  components/ subpackage per user feedback (replaces earlier
  src/gps_denied/ + sibling src/components/ split).
- Step 1.5 (Module Layout): _docs/02_document/module-layout.md is
  the file-ownership map consumed by /implement Step 4. Covers all
  14 components + cross-cuttings (_types, config, logging,
  fdr_client, helpers x8, frame_source, clock, runtime_root,
  cli/replay, healthcheck), 5-layer layering, and the Build-Time
  Exclusion Map for all 4 binaries (airborne, research,
  operator-tooling, replay-cli).
- New epic AZ-264 (E-CC-HELPERS): re-homes the 8 shared helpers
  from per-component child-issues into a single cross-cutting
  epic per the decompose skill cross-cutting rule. R14
  (LightGlue circular dep) is structurally prevented because
  both C2.5 and C3 import gps_denied_onboard.helpers.lightglue_runtime.
- New epic AZ-265 (E-DEMO-REPLAY): offline replay mode (video +
  tlog -> per-tick coordinate stream). 8 child tasks, 27-32 pts.
  Reuses C8 FcAdapter via TlogReplayFcAdapter strategy + new
  VideoFileFrameSource + JsonlReplaySink + compose_replay
  composition root + gps-denied-replay CLI + auto-sync via IMU
  take-off detection (per how_to_test.md). NO ROS dependency.
- Plan Final report at FINAL_report.md.
- _autodev_state.md updated with handoff notes for Step 2
  execution in a fresh chat (~290 MCP calls expected; epic
  ordering documented).

Step 2 task PLAN approved (97 implementation tasks across 18
epics) but EXECUTION deferred per user choice to a fresh chat.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-10 03:14:42 +03:00
parent 64542d32fc
commit 8171fcb29e
6 changed files with 1287 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_tasks/todo/AZ-263_initial_structure.md
+377
View File
@@ -0,0 +1,377 @@
# Initial Project Structure
**Task**: AZ-263_initial_structure
**Name**: Initial Structure
**Description**: Scaffold the gps-denied-onboard companion project skeleton — Python+C++ monolith with per-binary CMake `BUILD_*` flags (ADR-002), interface-first components under `src/gps_denied_onboard/components/<component>/` (ADR-009), three Dockerfiles + two compose files for Tier-1, GitHub Actions workflow with dual-binary emit + SBOM diff, PostgreSQL 16 schema mirrored from `satellite-provider`, and the test structure that the per-component test specs already require.
**Complexity**: 5 points
**Dependencies**: None
**Component**: Bootstrap
**Tracker**: AZ-263
**Epic**: AZ-244 (E-BOOT)
## Project Folder Layout
```
gps-denied-onboard/
├── pyproject.toml # Python project (ruff, mypy --strict, pytest, coverage gates per ci_cd_pipeline.md)
├── CMakeLists.txt # Top-level CMake — drives BUILD_VINS_MONO / BUILD_SALAD / etc.
├── cmake/
│ ├── dependencies.cmake # Pinned commits/refs for OKVIS2, VINS-Mono, GTSAM, FAISS, OpenCV ≥4.12.0 (D-CROSS-CVE-1)
│ ├── build_options.cmake # The BUILD_* option() set — single source of truth for ADR-002
│ └── strategies.cmake # Helper to register a strategy implementation behind its BUILD_* flag
├── .clang-format
├── .clang-tidy
├── .cmake-format.yaml
├── .editorconfig
├── .env.example # All env vars from § Environment Variables below
├── .dockerignore # Verbatim from deployment/containerization.md § .dockerignore
├── .gitignore # Excludes build/, *.engine, *.calib, *.index, .venv/, __pycache__/, tests/fixtures/large_replays/
├── README.md # Pointers into _docs/02_document/
│
├── docker/
│ ├── companion-tier1.Dockerfile # Multi-stage: system-deps → python-deps → cpp-build → runtime
│ ├── operator-tooling.Dockerfile # python:3.10-slim; installs C11 + C12 + healthcheck
│ ├── mock-suite-sat-service.Dockerfile # Lives next to fixtures/mock-suite-sat-service/ but referenced from compose
│ └── db-init/
│ └── 01_seed.sql.example # Template — real seed lives under tests/fixtures/seed-db.sql
│
├── docker-compose.yml # Local dev: companion + mock-sat + db + operator-tooling (verbatim from containerization.md § Local Development)
├── docker-compose.test.yml # Tier-1 integration/blackbox: extends docker-compose.yml + e2e-runner sidecar
│
├── .github/
│ └── workflows/
│ ├── ci.yml # Tier-1 lint + unit + integration + dual-binary build + SBOM diff + security
│ ├── ci-tier2.yml # Self-hosted Jetson runner (labels: [self-hosted, jetson, orin-nano-super]); AC-bound NFTs
│ ├── release.yml # Tag-on-main: JetPack image build + operator tooling tarball
│ └── cve-rescan.yml # Monthly scheduled CVE re-scan (D-CROSS-CVE-1 + GTSAM)
│
├── ci/
│ ├── sbom_diff.py # ADR-002 enforcement — deployment SBOM ⊂ research SBOM, vins_mono/salad/etc. excluded
│ └── opencv_pin_gate.py # Asserts resolved OpenCV ≥ 4.12.0
│
├── src/
│ └── gps_denied_onboard/ # Top-level Python package (single package per src-layout convention)
│ ├── __init__.py
│ ├── runtime_root.py # Composition root — config → strategy resolution → graph wiring (ADR-009)
│ ├── healthcheck.py # Used by Dockerfile HEALTHCHECK + CI smoke
│ ├── config/
│ │ ├── loader.py # YAML config loader + validation (per-flight + camera calibration JSON)
│ │ └── schema.py # Config dataclasses
│ ├── logging/
│ │ └── structured.py # Structured JSON logging (E-CC-LOG / AZ-245); no narrative log lines
│ ├── fdr_client/ # E-CC-FDR-CLIENT / AZ-247: lock-free queue + record schema; producer-side API
│ │ ├── client.py
│ │ ├── records.py # FdrRecord schema (estimates / IMU / MAVLink / health / tile / thumbnail)
│ │ └── queue.py # Lock-free SPSC queue per producer, drop-oldest on overrun
│ ├── _types/ # Cross-component DTOs (Protocol/dataclass — no implementation here)
│ │ ├── nav.py # NavCameraFrame, ImuSample, ImuWindow, AttitudeWindow, FlightStateSignal, GpsHealth
│ │ ├── vio.py # VioOutput
│ │ ├── vpr.py # VprQuery, VprResult, RerankResult
│ │ ├── matching.py # MatchResult
│ │ ├── pose.py # PoseEstimate, EstimatorOutput, EstimatorHealth
│ │ ├── tile.py # Tile, TileQualityMetadata, TileRecord, SectorClassification
│ │ ├── calibration.py # CameraCalibration (intrinsics + distortion + body-to-camera + acquisition_method)
│ │ ├── emitted.py # EmittedExternalPosition (per-FC encoded)
│ │ └── manifests.py # Manifest, EngineCacheEntry
│ ├── helpers/ # Common helpers (shared utilities — no component owns them)
│ │ ├── imu_preintegrator.py # _docs/02_document/common-helpers/01_helper_imu_preintegrator.md
│ │ ├── se3_utils.py # 02_helper_se3_utils.md
│ │ ├── lightglue_runtime.py # 03_helper_lightglue_runtime.md (shared by C2.5 + C3 — owned by helper, NOT C3, R14 fix)
│ │ ├── wgs_converter.py # 04_helper_wgs_converter.md
│ │ ├── sha256_sidecar.py # 05_helper_sha256_sidecar.md (D-C10-3 content-hash gate)
│ │ ├── engine_filename_schema.py # 06_helper_engine_filename_schema.md (D-C10-7 self-describing engine names)
│ │ ├── ransac_filter.py # 07_helper_ransac_filter.md
│ │ └── descriptor_normaliser.py # 08_helper_descriptor_normaliser.md
│ └── components/ # Subpackage — every component has interface.py + default impl + optional alt impls + tests/
│ ├── c1_vio/ # AZ-254: VioStrategy (OKVIS2 default, VINS-Mono research-only, KltRansac mandatory baseline)
│ ├── c2_vpr/ # AZ-255: VprStrategy (UltraVPR primary; MegaLoc/MixVPR/SelaVPR/EigenPlaces/NetVLAD/SALAD)
│ ├── c2_5_rerank/ # AZ-256: InlierBasedReranker (single-pair LightGlue inlier count K=10→N=3)
│ ├── c3_matcher/ # AZ-257: CrossDomainMatcher (DISK+LightGlue / ALIKED+LightGlue / XFeat)
│ ├── c3_5_adhop/ # AZ-258: AdHoPRefinementStrategy
│ ├── c4_pose/ # AZ-259: PoseEstimator (OpenCV solvePnPRansac + GTSAM Marginals)
│ ├── c5_state/ # AZ-260: StateEstimator (GTSAM iSAM2 + IncrementalFixedLagSmoother + ESKF baseline)
│ ├── c6_tile_cache/ # AZ-250: TileStore (Postgres mirror + filesystem mmap + FAISS HNSW)
│ ├── c7_inference/ # AZ-249: InferenceRuntime (TensorRT 10.3 / ONNX Runtime+TRT EP / PyTorch FP16)
│ ├── c8_fc_adapter/ # AZ-261: FcAdapter (PymavlinkArdupilotAdapter + Msp2InavAdapter) + GcsAdapter
│ ├── c10_provisioning/ # AZ-252: CacheProvisioner (engine compile + descriptors + manifest + content-hash)
│ ├── c11_tile_manager/ # AZ-251: TileDownloader + TileUploader (operator-side ONLY — excluded from airborne via CMake)
│ ├── c12_operator_tooling/ # AZ-253: CacheBuildWorkflow + OperatorReLocService (CLI; GUI deferred)
│ └── c13_fdr/ # AZ-248: FdrWriter (writer thread + segment rotation + ≤64 GB cap)
│
├── cpp/ # Native libraries linked from src/gps_denied_onboard/components/* via pybind11
│ ├── okvis2/ # OKVIS2 wrapper, exposes Python C-extension; C1 production-default
│ ├── vins_mono/ # Built only when BUILD_VINS_MONO=ON (research binary)
│ ├── klt_ransac/ # Mandatory simple-baseline; always linked
│ ├── gtsam_bindings/ # Thin wrapper over GTSAM iSAM2 + IncrementalFixedLagSmoother + Marginals
│ ├── faiss_index/ # FAISS HNSW wrapper (mmap-backed)
│ └── pybind11/ # Submodule
│
├── tests/
│ ├── conftest.py # Pytest fixtures (Postgres bring-up, SITL containers, Derkachi corpus mount)
│ ├── unit/ # Per-component pytest tests (mirrors src/gps_denied_onboard/components/ structure)
│ │ ├── c1_vio/ ... # one folder per component
│ ├── integration/ # IT-* scenarios from _docs/02_document/tests/blackbox-tests.md (Tier-1 capable)
│ ├── e2e/ # End-to-end runner — entrypoint for docker-compose.test.yml
│ │ ├── Dockerfile # Slim pytest container
│ │ ├── conftest.py
│ │ └── scenarios/ # FT-P-* / FT-N-* / NFT-LIM (Tier-1) scenarios
│ ├── perf/ # NFT-PERF-* (Tier-2) scenarios
│ ├── security/ # NFT-SEC-* scenarios (incl. NFT-SEC-02 network egress, NFT-SEC-05 DNS blackhole)
│ ├── resilience/ # NFT-RES-* scenarios
│ └── fixtures/
│ ├── flight_derkachi/ # Multi-GB; gitignored; provisioned via DVC or out-of-band
│ ├── tiles_corpus/ # Curated tile snapshots for FT-P-15/16
│ ├── seed-db.sql # PostgreSQL seed mirroring satellite-provider's tile rows
│ ├── calibration/
│ │ ├── adti26.json # Test-fixture camera calibration
│ │ └── adti20.json # Production calibration (D-PROJ-1 hybrid output, when available)
│ ├── mock-suite-sat-service/ # .NET 8 minimal API implementing the planned D-PROJ-2 ingest contract (e2e fixture only)
│ └── ardupilot_sitl/ # Helper compose for AP SITL (used by IT-3, NFT-PERF-04 AP)
│
├── scripts/
│ ├── run-tests.sh # Tier-1 wrapper around docker-compose.test.yml (deferred from test-spec Phase 4)
│ └── run-performance-tests.sh # Tier-2 wrapper (deferred from test-spec Phase 4)
│
└── _docs/ # Already exists (problem, solution, document, tasks)
```
### Layout Rationale
- **`src/gps_denied_onboard/`** is the single top-level Python package (per Python `src/`-layout convention). All imports are rooted here: `from gps_denied_onboard.components.c5_state import StateEstimator`, `from gps_denied_onboard.helpers.lightglue_runtime import LightGlueRuntime`, `from gps_denied_onboard._types.pose import EstimatorOutput`. One package per repo keeps `pip install -e .`, `mypy --strict`, and coverage globs unambiguous.
- **`gps_denied_onboard/components/<component>/`** — every component is its own folder with `interface.py` (Protocol/ABC) + one or more concrete impls + a private `_internal/` if needed. The `components/` subpackage exists (rather than flattening) because (a) per-component tooling globs (`--cov-fail-under=90` on `src/gps_denied_onboard/components/c5_state/**` and `.../c8_fc_adapter/**` per `ci_cd_pipeline.md § Unit`) need a clean prefix, (b) it preserves the ADR-009 architectural boundary between "components" (strategy-pattern units) and cross-cutting infra (`config/`, `logging/`, `_types/`, `helpers/`, `fdr_client/`), and (c) it gives a 1:1 mirror with `_docs/02_document/components/`. No reaching across folders for siblings; collaborators are constructor-injected. Build-time exclusion is enforced by CMake reading `cmake/build_options.cmake` and by the composition root validator refusing to wire a strategy whose `BUILD_*` flag is OFF.
- **`cpp/` is parallel to `src/`** rather than nested under it because CMake is the source of truth for native builds; the Python package imports the resulting `.so` files via pybind11 wrappers stored in `src/gps_denied_onboard/components/<component>/_native/` (not shown above to keep the tree readable — added per component task in Step 2).
- **`gps_denied_onboard/helpers/`** is the home of every shared utility called out in `_docs/02_document/common-helpers/*.md`. R14 (LightGlue circular dep) is structurally prevented because both C2.5 and C3 import `gps_denied_onboard.helpers.lightglue_runtime`, never each other.
- **`tests/` mirrors the component graph 1:1** so a developer landing on `src/gps_denied_onboard/components/c5_state/` can find its tests at `tests/unit/c5_state/`. Cross-component scenarios live in `tests/integration/`, `tests/e2e/`, `tests/perf/`, `tests/security/`, `tests/resilience/` matching the system-level `_docs/02_document/tests/*.md` files.
- **`docker/` and `docker-compose*.yml`** correspond to `_docs/02_document/deployment/containerization.md`. The bootstrap implementer copies the YAML bodies from there verbatim (see § Acceptance Criteria below).
- **`.github/workflows/`** mirrors `_docs/02_document/deployment/ci_cd_pipeline.md`. Two workflows: Tier-1 (everything except AC-bound NFTs) and Tier-2 (AC-bound NFTs on self-hosted Jetson).
## DTOs and Interfaces
Detailed DTO field lists are already in `architecture.md § 4 Data Model Overview` and per-component `description.md § 2 Internal Interfaces`. The bootstrap task creates **type-only stubs** (no implementation logic) for the cross-component DTOs listed in `src/gps_denied_onboard/_types/` so that downstream component tasks can import and depend on them.
### Shared DTOs (cross-component — bootstrap creates type stubs)
| DTO | File | Used By | Source |
|-----|------|---------|--------|
| `NavCameraFrame` | `_types/nav.py` | C1, C2, C13 | architecture.md § 4 |
| `ImuSample`, `ImuWindow`, `AttitudeWindow` | `_types/nav.py` | C1, C5, C8 | C8 inbound spec |
| `FlightStateSignal`, `GpsHealth` | `_types/nav.py` | C5, C8, C11 | C8 inbound spec |
| `VioOutput` | `_types/vio.py` | C1 → C5; C13 (FDR) | C1 spec |
| `VprQuery`, `VprResult` | `_types/vpr.py` | C2 → C2.5 | C2 spec |
| `RerankResult` | `_types/vpr.py` | C2.5 → C3 | C2.5 spec |
| `MatchResult` | `_types/matching.py` | C3, C3.5 → C4 | C3 spec |
| `PoseEstimate` | `_types/pose.py` | C4 → C5; C13 | C4 spec |
| `EstimatorOutput`, `EstimatorHealth` | `_types/pose.py` | C5 → C8, C13 | C5 spec |
| `Tile`, `TileRecord`, `TileQualityMetadata`, `SectorClassification` | `_types/tile.py` | C6, C10, C11 | C6 / data_model.md § 2.1 |
| `CameraCalibration` | `_types/calibration.py` | C1, C3, C4 | architecture.md § 4 |
| `EmittedExternalPosition` | `_types/emitted.py` | C8 outbound | C8 spec |
| `Manifest`, `EngineCacheEntry` | `_types/manifests.py` | C10, C7 | C10 spec / data_model.md § 2.4–2.5 |
| `FdrRecord` (and per-record-type discriminator) | `gps_denied_onboard/fdr_client/records.py` | producer side: every component; consumer side: C13 | C13 spec |
### Component Interfaces (bootstrap creates empty `interface.py` per component)
For each `src/gps_denied_onboard/components/<component>/`, bootstrap creates:
- `interface.py` — the `Protocol` / `ABC` taken verbatim from the component spec's § 2 Internal Interfaces
- `__init__.py` exporting the interface symbol
- `_native/` placeholder if the component has a C++ binding (C1, C5, C6, C7)
- `tests/__init__.py` placeholder under `tests/unit/<component>/`
Concrete implementations are NOT created here — they are the subject of Step 2 component tasks.
| Component | Interface | Defined in (spec) |
|-----------|-----------|-------------------|
| C1 | `VioStrategy` | `components/01_c1_vio/description.md § 2` |
| C2 | `VprStrategy`, `VprPipeline` | `components/02_c2_vpr/description.md § 2` |
| C2.5 | `Reranker` | `components/03_c2_5_rerank/description.md § 2` |
| C3 | `CrossDomainMatcher` | `components/04_c3_matcher/description.md § 2` |
| C3.5 | `AdHoPRefinementStrategy` | `components/05_c3_5_adhop/description.md § 2` |
| C4 | `PoseEstimator` | `components/06_c4_pose/description.md § 2` |
| C5 | `StateEstimator` | `components/07_c5_state/description.md § 2` |
| C6 | `TileStore`, `DescriptorIndex` | `components/08_c6_tile_cache/description.md § 2` |
| C7 | `InferenceRuntime`, `EngineLoader` | `components/09_c7_inference/description.md § 2` |
| C8 | `FcAdapter`, `GcsAdapter` | `components/10_c8_fc_adapter/description.md § 2` |
| C10 | `CacheProvisioner` | `components/11_c10_provisioning/description.md § 2` |
| C11 | `TileDownloader`, `TileUploader` | `components/12_c11_tilemanager/description.md § 2` |
| C12 | `CacheBuildWorkflow`, `OperatorReLocService` | `components/13_c12_operator_tooling/description.md § 2` |
| C13 | `FdrWriter` (consumer side) | `components/14_c13_fdr/description.md § 2` |
## CI/CD Pipeline
Bootstrap creates the workflow files matching `_docs/02_document/deployment/ci_cd_pipeline.md`. Stages (verbatim mapping):
| Stage | File | Trigger | Quality Gate |
|-------|------|---------|--------------|
| Lint | `.github/workflows/ci.yml` | every push, every PR | `ruff` + `mypy --strict` (Python); `clang-format --dry-run` + `clang-tidy` (C++); `cmake-format --check` (CMake); `yamllint` + `markdownlint-cli` (YAML/MD) |
| Unit | `.github/workflows/ci.yml` | every push, every PR | `pytest --cov-fail-under=75` per component; `--cov-fail-under=90` on C5, C8; C++ `gtest + lcov` ≥75% (90% on `klt_ransac`) |
| Integration (Tier-1) | `.github/workflows/ci.yml` | every push, every PR | `docker compose -f docker-compose.test.yml up --abort-on-container-exit --exit-code-from e2e-runner --build` |
| Build (dual binary) | `.github/workflows/ci.yml` | every push, every PR | matrix: `[deployment (BUILD_VINS_MONO=OFF, BUILD_SALAD=OFF), research (all ON)]` — both must build green |
| SBOM diff | `.github/workflows/ci.yml` | after build | `ci/sbom_diff.py`: deployment SBOM excludes `vins_mono`, `salad`, `c11_tilemanager` (R02 enforcement); deployment ⊂ research |
| Security | `.github/workflows/ci.yml` | after build | `pip-audit`, `dotnet list package --vulnerable` for mock-sat, Trivy on images, `ci/opencv_pin_gate.py` ≥4.12.0 |
| Push images (Tier-1) | `.github/workflows/ci.yml` | merge to `dev`/`stage`/`main` | tag `${BRANCH}-${BUILD_KIND}-${SHORT_SHA}`, push to GHCR; PRs do NOT push |
| Build (Tier-2 deployment) | `.github/workflows/ci-tier2.yml` | merge to `dev`/`stage`/`main` | native build on `[self-hosted, jetson, orin-nano-super]`; SBOM byte-equal to Tier-1 deployment |
| AC-bound NFTs (Tier-2) | `.github/workflows/ci-tier2.yml` | merge events; manual on PR | NFT-PERF-01..04, NFT-LIM-01..04, NFT-RES-03/04, NFT-SEC-01/03/05, IT-12 — thresholds in `tests/traceability-matrix.md` |
| JetPack image build | `.github/workflows/release.yml` | tag on `main` | image built + signed + attested; checksum signed by Tier-1 secret manager |
| Operator tooling tarball | `.github/workflows/release.yml` | tag on `main` | bundle: `operator-tooling` image + `mock-suite-sat-service` image + compose + verification script + `_docs/02_document/` |
| CVE re-scan (monthly) | `.github/workflows/cve-rescan.yml` | scheduled cron | re-runs security stage against pinned versions; pages onboard team on new High/Critical |
### Pipeline Configuration Notes
- **Manual-trigger override**: AC-bound NFTs initially run on manual trigger only while Tier-2 runner is being provisioned (see `deployment_procedures.md § Tier-2 enablement`). The merge gate on `dev` excludes Tier-2; `stage`/`main` retain the full gate.
- **Caching keys** per `ci_cd_pipeline.md § Caching Strategy`: Python deps keyed by `pyproject.toml` hash; C++ build deps keyed by `cmake/dependencies.cmake` hash; Docker layers keyed by Dockerfile + dep-file hashes; TRT engine cache (Tier-2) keyed by `engine_cache_bundle_hash` from `data_model.md § 2.4`.
- **Notifications**: Slack `#gps-denied-ci` for build/Tier-2 failures + SBOM diff fail PR comments; email + page on JetPack signature mismatch.
## Environment Strategy
Per `architecture.md § 3 Deployment Model`. Bootstrap defines **dev-tier1** and **production-operator-workstation** for local execution; **staging-tier2** and **production** are bare JetPack and have no Dockerfile (the JetPack image is assembled by `release.yml`).
| Environment | Purpose | Configuration Notes |
|-------------|---------|---------------------|
| `dev-tier1` | Local dev, lint, unit, most integration | Workstation Docker; `LOG_LEVEL=DEBUG`; `mock-suite-sat-service` Docker; test-fixture calibration `adti26.json`; both BUILD_* tracks available |
| `staging-tier1` | CI runs that don't require Jetson | GitHub-hosted x86_64 runner; same Docker as dev-tier1 |
| `staging-tier2` | CI runs that require Jetson (AC-bound) | Self-hosted Jetson runner; bare JetPack 6.2; no Docker; both binary tracks built |
| `production` | Deployed companion image on a UAV | Jetson Orin Nano Super; bare JetPack; `LOG_LEVEL=INFO` to FDR; per-flight MAVLink signing key generated at takeoff load + zeroised at FDR rollover; **no inbound network listening** + DNS blackhole + `iptables OUTPUT REJECT` (NFT-SEC-05) |
| `production-operator-workstation` | Pre-flight tile download + cache build (C10) + post-landing tile upload (C11) + FDR retrieval | Operator's Linux workstation; Docker for `satellite-provider` mirror + `operator-tooling` |
### Environment Variables (`.env.example`)
| Variable | dev-tier1 | staging-tier2 | production | Description |
|----------|-----------|---------------|------------|-------------|
| `GPS_DENIED_FC_PROFILE` | `ardupilot_plane` | `ardupilot_plane` or `inav` (matrix) | exactly one of `ardupilot_plane` / `inav` per build | Selects FC adapter at composition root |
| `GPS_DENIED_TIER` | `1` | `2` | `2` | Read by composition root + tests to gate hardware-dependent paths |
| `DB_URL` | `postgresql://gps_denied:dev@db:5432/gps_denied` | `postgresql://gps_denied:dev@db:5432/gps_denied` | local Postgres on companion (NVM) | C6 PostgreSQL connection |
| `SATELLITE_PROVIDER_URL` | `http://mock-sat:5100` | mix: real `satellite-provider` (download) + `mock-sat:5100` (upload until D-PROJ-2 ships) | not set in flight (no egress); operator workstation only | Pre-flight tile fetch |
| `CAMERA_CALIBRATION_PATH` | `/fixtures/calibration/adti26.json` | `/fixtures/calibration/adti26.json` | `/etc/gps-denied/calibration/adti20.json` | Loaded once at startup |
| `LOG_LEVEL` | `DEBUG` | `INFO` | `INFO` | Structured JSON output |
| `LOG_SINK` | `console` | `journald` + `fdr` | `fdr` only (per-flight ≤64 GB) | E-CC-LOG sink selection |
| `MAVLINK_SIGNING_KEY` | dev key from `tests/fixtures/mavlink_signing/dev_key` | per-flight key from test config | generated at takeoff load; rotated per flight; logged to FDR | D-C8-9 / R03 |
| `BUILD_VINS_MONO` | both | both (matrix) | `OFF` (production-only) | CMake-time exclusion (ADR-002) |
| `BUILD_SALAD` | both | both | `OFF` | ADR-002 |
| `BUILD_C11_TILE_MANAGER` | `ON` (operator tooling image) | `OFF` (airborne) / `ON` (operator tooling) | `OFF` (R02 process isolation) | Excludes operator-only code path from airborne image |
| `INFERENCE_BACKEND` | `tensorrt` if GPU; else `pytorch_fp16` | `tensorrt` | `tensorrt` | C7 backend selection |
| `FDR_PATH` | `/var/lib/gps-denied/fdr` | `/var/lib/gps-denied/fdr` | host NVM mount (≤64 GB) | C13 FDR ring path |
| `TILE_CACHE_PATH` | `/var/lib/gps-denied/tiles` | `/var/lib/gps-denied/tiles` | host NVM mount | C6 filesystem tile body store |
## Database Migration Approach
**Migration tool**: `alembic` (Python; matches Python+C++ stack and integrates with `pyproject.toml`).
**Strategy**: versioned migration scripts under `db/migrations/`. **Additive-only by default per `data_model.md § 1 principle #5`** — migrations may add tables/columns/indexes/CHECK constraints, may NEVER rename or drop existing columns without an ADR-recorded deprecation window. The `tiles` schema specifically is **frozen on the canonical columns** (mirrored from `satellite-provider`) and only extensible via additive onboard-only columns.
### Initial Schema (migration `0001_initial.sql`)
Per `data_model.md § 2`:
- `tiles` — mirrored from `satellite-provider` (canonical columns) + onboard-only additive columns (`source` CHECK in `('googlemaps', 'onboard_ingest')`, `tile_quality_metadata` jsonb on `onboard_ingest` rows, `voting_status` enum)
- `flights` — per-flight metadata; FK target for `engine_cache_entries` and `manifests`
- `sector_classifications` — operator-set; drives freshness threshold via `apply_freshness_threshold`
- `manifests` — D-C10-1 idempotence (hash of model + calibration + corpus + sector classification)
- `engine_cache_entries` — TRT engine + INT8 calibration cache keyed by SM/JP/TRT/precision tuple (D-C10-7)
Seed data (`tests/fixtures/seed-db.sql`): a curated subset of `satellite-provider` tile rows for `tests/fixtures/tiles_corpus/` so Tier-1 integration tests have deterministic input.
## Test Structure
```
tests/
├── conftest.py # Postgres bring-up via docker-compose; AP/iNav SITL fixtures; Derkachi corpus mount
├── unit/<component>/ # Per-component pytest tests; coverage gate per ci_cd_pipeline.md § Unit
├── integration/ # IT-* scenarios (Tier-1 capable); pytest-driven; uses docker-compose.test.yml
├── e2e/ # FT-P-* / FT-N-* / NFT-LIM (Tier-1) scenarios — runs inside e2e-runner sidecar
│ ├── Dockerfile
│ ├── conftest.py
│ └── scenarios/
├── perf/ # NFT-PERF-* (Tier-2 only); pytest + jetson-stats integration
├── security/ # NFT-SEC-* incl. NFT-SEC-02 network egress + NFT-SEC-05 DNS blackhole + NFT-SEC-01 cache poisoning
├── resilience/ # NFT-RES-* — Monte-Carlo + scheduled-perturbation
└── fixtures/
├── flight_derkachi/ # Multi-GB; gitignored; provisioned via DVC or out-of-band
├── tiles_corpus/ # Curated tile snapshots for FT-P-15/16
├── seed-db.sql
├── calibration/{adti26,adti20}.json
├── mavlink_signing/dev_key
├── mock-suite-sat-service/ # .NET 8 minimal API; ASP.NET Core; implements POST /api/satellite/tiles/ingest
└── ardupilot_sitl/
```
### Test Configuration Notes
- **Test runner**: `pytest` for everything Python; `gtest` for C++ unit tests built under CMake `BUILD_TESTING=ON`. Both run from the same Tier-1 CI workflow.
- **Fixture lifecycle**: `conftest.py` brings Postgres + mock-sat + ArduPilot SITL up via `docker-compose.test.yml`; tests that need real Jetson hardware (NFT-PERF, NFT-LIM, NFT-RES on real fixtures, IT-12) live under `tests/perf/` + `tests/resilience/` and are guarded by `pytest.mark.tier2` so they're auto-skipped on Tier-1.
- **Test isolation**: Postgres uses transaction-rollback fixtures for unit/integration tests; e2e tests get a fresh DB volume per scenario. Tile cache and FDR mount tmpfs in CI.
- **Test data management**: per `tests/test-data.md`. Large fixtures (Derkachi, tile corpus) live under `tests/fixtures/large_replays/` which is gitignored + `.dockerignore`'d (mounted as volume in `docker-compose.test.yml`).
- **Source of truth for scenario specs**: `_docs/02_document/tests/*.md` — implementation tasks (Step 9 Decompose Tests) bind these specs to actual test files. Bootstrap only creates the directory scaffolding.
## Implementation Order
Driven by the epic dependency graph in `_docs/02_document/epics.md § Implementation order`:
| Order | Component | Reason |
|-------|-----------|--------|
| 1 | Bootstrap (this task) — repo skeleton, Docker, CI, DB schema | foundation; no dependencies |
| 2 | E-CC-LOG (AZ-245), E-CC-CONF (AZ-246) — cross-cutting | unblocks every component's structured logging and config loading |
| 3 | E-CC-FDR-CLIENT (AZ-247) — FDR producer client | every component's FDR producer depends on the lock-free queue + record schema |
| 4 | E-C13 (AZ-248) — FDR writer | consumer side; needs records.py from #3; also unblocks E-C7 instrumentation |
| 5 | E-C7 (AZ-249) — inference runtime + E-C6 (AZ-250) — tile cache | parallel; enable C2 / C3 / C3.5 development |
| 6 | E-C11 (AZ-251) — Tile Manager (operator-side) | unblocks F1 + F10 tooling |
| 7 | E-C10 (AZ-252) — pre-flight cache provisioning → E-C12 (AZ-253) — operator tooling | sequential; F1 chain |
| 8 | E-C1 (AZ-254) — VIO | independent of #5–#7; can start in parallel after #2 |
| 9 | E-C2 (AZ-255) → E-C2.5 (AZ-256) → E-C3 (AZ-257) → E-C3.5 (AZ-258) → E-C4 (AZ-259) | F3 hot path; sequential per pipeline order |
| 10 | E-C5 (AZ-260) — state estimator | needs C1 + C4; the largest single epic |
| 11 | E-C8 (AZ-261) — FC + GCS adapter | needs C5; gates IT-3 (R03 lock) |
| 12 | E-BBT (AZ-262) — system-level FT/NFT scenarios | hardens behind every other epic; per-component tests live in their own epics |
## Acceptance Criteria
**AC-1: Project scaffolded matching the layout above**
Given the structure plan above
When the implementer executes this task
Then every folder shown in `## Project Folder Layout` exists, every `__init__.py` / `interface.py` placeholder exists, and `pyproject.toml` + `CMakeLists.txt` + `cmake/dependencies.cmake` + `cmake/build_options.cmake` parse without error.
**AC-2: DTO type stubs importable**
Given the bootstrap output
When `python -c "from gps_denied_onboard._types import nav, vio, vpr, matching, pose, tile, calibration, emitted, manifests"` is run
Then no `ImportError` raises (the modules need not implement methods — only declare the classes/Protocols/dataclasses).
**AC-3: Three Dockerfiles + two compose files match `containerization.md`**
Given the bootstrap output
When `docker compose -f docker-compose.yml config --quiet` and `docker compose -f docker-compose.test.yml config --quiet` are run
Then both exit zero (compose files are syntactically valid + service references resolve). The Dockerfile bodies match the per-image specifications in `_docs/02_document/deployment/containerization.md` § Component Dockerfiles.
**AC-4: CI workflow stubs are syntactically valid**
Given the bootstrap output
When the GitHub Actions workflows under `.github/workflows/` are validated (`actionlint` or equivalent)
Then they parse without error AND the dual-binary build matrix (deployment + research) is present in `ci.yml` per ADR-002.
**AC-5: Database migration tool wired**
Given the bootstrap output
When `alembic check` is run against an empty database
Then it reports `0001_initial.sql` as the head revision and the schema matches the canonical `tiles` columns from `satellite-provider` (with the additive onboard-only columns documented in `data_model.md § 2.1.1`).
**AC-6: Health checks defined**
Given the bootstrap output
Then `src/gps_denied_onboard/healthcheck.py` exists and is importable; the operator-tooling Dockerfile + companion-tier1 Dockerfile both reference it via `HEALTHCHECK CMD`; mock-suite-sat-service Dockerfile defines `GET /healthz` returning 200 when storage backend is mounted.
**AC-7: Structured logging entrypoint exists**
Given the bootstrap output
Then `src/gps_denied_onboard/logging/structured.py` exists, exports `get_logger(name)`, and emits one JSON object per log line (no narrative log lines per E-CC-LOG / AZ-245 contract).
**AC-8: `.env.example` documents every variable used by the composition root**
Given the bootstrap output
Then `.env.example` matches the table in § Environment Strategy above; missing required variables cause `runtime_root.py` to refuse startup with a clear error pointing at the offending variable name.
**AC-9: Stub tests pass**
Given the scaffolded project
When `pytest -q` runs against `tests/unit/`
Then every component folder has at least one `test_smoke.py` whose only assertion is that the component's interface is importable; the run exits zero.
**AC-10: SBOM diff script + OpenCV pin gate exist and run on stub builds**
Given the bootstrap output
Then `ci/sbom_diff.py --deployment <stub> --research <stub>` runs without exception; `ci/opencv_pin_gate.py` exits zero when fed a `pyproject.toml` with `opencv-python ≥ 4.12.0`.
## Out of Scope (deferred to other tasks)
- Concrete implementations of any component interface — that's Step 2 component tasks (one task per child issue from `epics.md`).
- Module layout — that's Decompose Step 1.5 (`module-layout.md`).
- Test scenario implementation — that's Step 9 Decompose Tests + Step 10 Implement Tests.
- D-PROJ-2 ingest contract implementation in `mock-suite-sat-service` — initial Dockerfile + minimal API placeholder only; the actual contract is implemented as a scoped task once the parent-suite design lands.
- TensorRT engine compilation, GTSAM iSAM2 substrate wiring, FAISS HNSW index build — owned by C7, C5, C6 epics respectively.