gps-denied-onboard/_docs/02_tasks/todo/AZ-219_initial_structure.md

Initial Project Structure

Task: AZ-219_initial_structure Name: Initial Structure Description: Scaffold the project skeleton for the Jetson-hosted GPS-denied localization runtime, replay harness, local infrastructure, CI, and deployment evidence paths. Complexity: 5 points Dependencies: None Component: Bootstrap Tracker: AZ-219 Epic: AZ-206

Project Folder Layout

project-root/
├── src/
│   ├── __init__.py
│   ├── camera_ingest_calibration/
│   ├── basalt_vio_adapter/
│   ├── safety_anchor_wrapper/
│   ├── satellite_retrieval/
│   ├── anchor_verification/
│   ├── cache_tile_lifecycle/
│   ├── mavlink_gcs_integration/
│   ├── fdr_observability/
│   ├── validation_harness/
│   ├── shared/
│   │   ├── contracts/
│   │   ├── geo_geometry/
│   │   ├── time_sync/
│   │   ├── config/
│   │   ├── errors/
│   │   └── telemetry/
│   └── native/
│       ├── basalt_bridge/
│       ├── feature_matching/
│       └── tensor_rt/
├── migrations/
│   ├── postgresql/
│   └── seed/
├── tests/
│   ├── unit/
│   ├── integration/
│   ├── blackbox/
│   ├── fixtures/
│   └── sitl/
├── e2e/
│   ├── replay/
│   └── reports/
├── deployment/
│   ├── docker/
│   ├── compose/
│   ├── jetson/
│   └── scripts/
├── config/
│   ├── development/
│   ├── ci/
│   ├── jetson/
│   └── production/
├── data/
│   ├── input/
│   ├── expected/
│   ├── cache/
│   ├── fdr/
│   └── test-results/
├── .github/
│   └── workflows/
│       └── ci.yml
├── docker-compose.yml
├── docker-compose.test.yml
├── .dockerignore
└── .env.example

Layout Rationale

The runtime is organized directly under src/ because this repository already represents the GPS-denied onboard system. Component directories live at the source root, with native bridges isolated under src/native/ for BASALT, feature matching, and TensorRT-sensitive paths. Shared contracts, geometry, time-sync, configuration, error envelopes, and telemetry DTOs are centralized so component tasks consume a single public interface instead of duplicating cross-cutting logic.

The scaffold separates runtime source, migrations, tests, deployment assets, configuration, and mutable data. Production runs on Jetson hardware, while Docker/compose is used for replay, SITL, and deterministic CI environments.

DTOs and Interfaces

Shared DTOs

DTO Name	Used By Components	Fields Summary
FramePacket	Camera ingest, BASALT VIO, Satellite Service, anchor verification, Tile Manager, FDR	Frame ID, timestamp, image reference, camera calibration ID, occlusion status, quality metrics
TelemetrySample	MAVLink/GCS, BASALT VIO, safety wrapper, FDR, validation harness	Timestamp, IMU, attitude, airspeed, altitude, GPS health
VioStatePacket	BASALT VIO, safety wrapper, FDR, validation harness	Timestamp, relative pose, velocity, bias, tracking quality, covariance hint
PositionEstimate	Safety wrapper, MAVLink/GCS, Tile Manager, FDR, validation harness	WGS84 coordinates, covariance semi-major axis, source label, fix type, horizontal accuracy, anchor age
VprCandidate	Satellite Service, anchor verification, FDR	Chunk ID, tile ID, score, footprint, freshness status
AnchorDecision	Anchor verification, safety wrapper, FDR	Candidate ID, acceptance result, estimated pose, inliers, MRE, rejection reason
CacheTileRecord	Tile Manager, Satellite Service, anchor verification, FDR	Tile ID, type, CRS, meters per pixel, capture date, signature/hash status, trust level
FdrEvent	All runtime components, validation harness	Event type, timestamp, component, severity, payload reference, mission/run ID
ScenarioReport	Validation harness, CI/CD, release evidence	Scenario ID, result, metrics, artifacts, failure reason

Component Interfaces

Component	Interface	Methods	Exposed To
Camera ingest/calibration	FrameProvider	next_frame, detect_occlusion, classify_quality	BASALT VIO, Satellite Service, anchor verification, Tile Manager
BASALT VIO adapter	VioAdapter	initialize, process, health	Safety wrapper, validation harness
Safety/anchor wrapper	LocalizationStateMachine	update_vio, consider_anchor, degrade, propagate_imu_only, tile_write_eligibility	MAVLink/GCS, Tile Manager, FDR, validation harness
Satellite Service	SatelliteService	import_mission_cache, load_index, retrieve, upload_generated_tiles	Safety wrapper, anchor verification, Tile Manager
Anchor verification	AnchorVerifier	verify, benchmark_matcher	Safety wrapper, FDR
Tile Manager	TileManager	validate_cache, get_tile_window, orthorectify_frame, write_generated_tile, package_sync	Satellite Service, anchor verification, post-flight sync
MAVLink/GCS integration	MavlinkGateway	subscribe_telemetry, emit_gps_input, emit_status	BASALT VIO, safety wrapper, QGC, FDR
FDR/observability	FlightRecorder	append_event, rollover, export	All runtime components, validation harness
Validation harness	ScenarioRunner	validate_fixture, run_scenario	CI/CD, release evidence review

CI/CD Pipeline

Stage	Purpose	Trigger
Format / lint	Enforce code style and static quality	Every PR and push to dev
Unit tests	Validate component-local behavior and shared contracts	Every PR and push to dev
Replay black-box smoke	Run deterministic still-image/cache/SITL subsets	Every PR
Cache/security fixture tests	Validate signed manifests, stale-tile rejection, and no mid-flight provider/Satellite Service calls	Every PR
Plane SITL spoof/failsafe	Validate ArduPilot Plane GPS_INPUT, failsafe, spoofing promotion	Nightly and release candidate
Public dataset replay	Exercise VIO, retrieval, and anchor behavior against pinned public slices	Nightly and release candidate
Jetson latency/resource tests	Measure p95 latency, memory, cold start, TensorRT/ONNX fidelity	Release candidate
Thermal/FDR endurance	Prove 8-hour 25 W no-throttle and <=64 GB FDR rollover	Hardware qualification
Build / package	Produce replay-compatible and Jetson deployment artifacts	Release candidate
Deploy staging evidence	Publish reports, tlogs, FDR summaries, cache validation artifacts	Release candidate

Pipeline Configuration Notes

CI uses Docker/compose for replay and SITL gates. Jetson, thermal, camera, and representative replay gates run on dedicated hardware runners and block release rather than every PR. Dataset downloads are license-tagged and not baked into images. Secrets, signing keys, and Satellite Service credentials are never cached.

Environment Strategy

Environment	Purpose	Configuration Notes
Development replay	Fast local iteration	Small fixture cache, local PostgreSQL/PostGIS, replay frames, test keys only
CI replay	Deterministic PR checks	Docker services for runtime, replay consumer, cache stub, and reports
Public dataset replay	Algorithm de-risking	Pinned dataset slices with license metadata and ground-truth reports
Plane SITL	MAVLink/failsafe validation	ArduPilot Plane SITL, QGC/tlog observer, production-like GPS_INPUT parameters
Jetson hardware validation	Production path profiling	JetPack/CUDA/TensorRT, local camera/GPU/MAVLink access, resource monitoring
Representative flight/replay	Final acceptance evidence	Target-like UAV, FC, camera, synchronized telemetry, ground truth
Production	Onboard mission runtime	Preloaded signed cache, local PostGIS, per-flight FDR, no provider network calls

Environment Variables

Variable	Dev	Staging	Production	Description
GPSD_ENV	development	staging	production	Selects runtime profile
GPSD_CONFIG_DIR	local path	staged config path	onboard config path	Versioned configuration root
GPSD_CACHE_DIR	fixture cache	staged mission cache	onboard mission cache	COG, manifest, sidecar, descriptor root
GPSD_FDR_DIR	temp output	staged evidence path	per-flight NVMe path	Flight data recorder output
GPSD_DATABASE_URL	local PostGIS	staging PostGIS	onboard local PostGIS	Manifest, mission state, FDR event index
GPSD_MAVLINK_URL	SITL/replay	SITL/hardware rig	FC telemetry link	MAVLink connection endpoint
GPSD_CAMERA_SOURCE	fixture/replay	fixture or hardware	live camera	Navigation frame source
GPSD_SIGNING_KEY_REF	test key ref	staging secret ref	mission secret ref	Cache/sidecar signature verification
GPSD_MAX_FDR_BYTES	small fixture cap	release-like cap	68719476736	FDR rollover threshold
GPSD_LOG_LEVEL	debug/info	info	info/warn	Runtime logging level

Database Migration Approach

Migration tool: Versioned PostgreSQL migration scripts, with PostGIS extension setup and deterministic seed scripts. Strategy: Additive migrations by default. Database, table, and column renames require explicit approval before implementation. Runtime rejects unknown required schema versions loudly.

Initial Schema

• Mission profile tables for route geometry, sector classification, altitude band, and cache budget.
• Camera calibration tables for camera model, intrinsics, extrinsics, and verification status.
• Cache manifest tables for COG tiles, generated tiles, freshness, signatures, sidecar hashes, trust level, and PostGIS footprints.
• VPR chunk tables for retrieval footprints, descriptor metadata, multi-scale index references, and sector top-K policy.
• FDR event index tables for mission/run IDs, timestamps, event type, component, severity, and CBOR segment references.
• Validation run tables or report records for scenario IDs, metrics, artifacts, and release evidence pointers.

Test Structure

tests/
├── unit/
│   ├── shared/
│   ├── camera_ingest_calibration/
│   ├── basalt_vio_adapter/
│   ├── safety_anchor_wrapper/
│   ├── satellite_service/
│   ├── anchor_verification/
│   ├── tile_manager/
│   ├── mavlink_gcs_integration/
│   ├── fdr_observability/
│   └── validation_harness/
├── integration/
│   ├── contracts/
│   ├── cache_postgis/
│   ├── mavlink/
│   └── fdr/
├── blackbox/
│   ├── still_image_geolocation/
│   ├── satellite_anchor/
│   ├── visual_blackout_spoofing/
│   ├── cache_freshness/
│   └── resource_limits/
├── fixtures/
│   ├── project_60_images/
│   ├── expected_results/
│   ├── satellite_cache/
│   ├── telemetry/
│   └── public_dataset_slices/
└── sitl/
    ├── plane_gps_input/
    ├── spoofing_promotion/
    └── failsafe/

Test Configuration Notes

Unit tests validate contracts and component behavior without external hardware. Integration tests exercise PostGIS, MAVLink, FDR segments, and cache fixtures. Black-box tests interact only through public inputs and outputs: frames, telemetry, offline cache, GPS_INPUT, QGC status, and FDR. Release gates add Jetson hardware, thermal/endurance, and representative replay evidence.

Health Checks

Service / Component	Liveness	Readiness
GPS-denied service	Process event loop responsive	Config, PostGIS, cache, FDR path, MAVLink, and camera/replay source validated
Replay consumer	Runner process responsive	Fixtures, expected results, and output path available
Satellite cache stub	Fixture volume mounted	Manifests, sidecars, descriptors, and signatures validated
ArduPilot Plane SITL	SITL process responsive	MAVLink ports accepting telemetry and production parameters loaded
QGC observer/log parser	Parser process responsive	Tlog/status stream connected

Each deployable service exposes /health/live, /health/ready, and /metrics where HTTP is available. Non-HTTP hardware processes write equivalent structured health events to FDR and CI reports.

Docker And Compose Requirements

• Create a replay-compatible Dockerfile for the Python runtime and native optional stubs.
• Create a replay-consumer Dockerfile for black-box test execution.
• Create a satellite-cache-stub fixture image or volume contract.
• Create an ArduPilot Plane SITL service for integration tests.
• Use non-root users, pinned base images, multi-stage builds where native compilation is needed, and health checks for long-running services.
• Provide docker-compose.yml for local development/replay and docker-compose.test.yml for black-box/SITL test execution.
• Do not bake secrets, provider credentials, mission signing keys, or public dataset payloads into images.

Implementation Order

Order	Component	Reason
1	Bootstrap and shared contracts	Establish package, configuration, migrations, CI, DTOs, and error envelopes
2	Shared geometry/time helpers	Foundational math and timestamp contracts used by camera, VIO, cache, wrapper, and validation
3	Runtime configuration and error handling	Prevent duplicated config/error behavior across components
4	Camera ingest/calibration	Produces the frame and occlusion signals required by VIO, anchor, cache, and tests
5	MAVLink/GCS integration	Supplies FC telemetry DTOs and validates GPS_INPUT output contract early
6	Tile Manager	Owns PostGIS cache manifest, sidecars, COG access, freshness gates, and generated-tile orthorectification
7	FDR/observability	Provides audit path for all components and validation reports
8	BASALT VIO adapter	Depends on frame and telemetry contracts, blocks wrapper integration
9	Satellite Service	Depends on tile schema and frame DTOs, feeds anchor verification, and handles pre-flight/post-flight package sync
10	Anchor verification	Depends on retrieval candidates and cache tile access
11	Safety/anchor wrapper	Consumes VIO, anchor, camera degradation, MAVLink, and FDR contracts
12	Validation harness	Uses public interfaces once contracts and runtime components are stable
13	Black-box, SITL, Jetson, and endurance test tasks	Exercise release gates and acceptance criteria end to end

Acceptance Criteria

AC-1: Project scaffolded Given the structure plan above When the implementer executes this task Then source, test, migration, deployment, configuration, and data directories exist with placeholder files where needed so empty directories are retained.

AC-2: Shared contracts initialized Given the planned component interfaces When the scaffold is complete Then shared DTO, error, configuration, telemetry, geometry, and time-sync contract locations exist and are importable from component skeletons.

AC-3: Local infrastructure defined Given the system requires local PostGIS, replay, SITL, cache, and FDR paths When the scaffold is complete Then docker-compose.yml, docker-compose.test.yml, .env.example, and migration seed directories describe all required local services and volumes.

AC-4: CI/CD configured Given the pipeline stages in the planning artifacts When CI runs on a PR Then format/lint, unit tests, replay black-box smoke, cache/security fixture tests, and artifact collection stages are defined.

AC-5: Test harness skeleton ready Given black-box tests must use public interfaces only When the scaffold is complete Then unit, integration, black-box, fixtures, and SITL test locations exist with runner entry points ready for implementation tasks.

AC-6: Deployment evidence paths ready Given release requires Jetson, SITL, FDR, cache, and representative replay evidence When the scaffold is complete Then deployment scripts, report output paths, and documentation placeholders exist for those evidence artifacts.

AC-7: No production secrets or raw frames committed Given onboard runtime must not retain raw frames and must protect signing credentials When the scaffold is reviewed Then .gitignore, .dockerignore, and .env.example exclude secrets, generated FDR payloads, raw frame dumps, cache payloads, and test result artifacts unless explicitly fixture-scoped.