autopilot/_docs/02_tasks/done/AZ-640_initial_structure.md

Initial Project Structure

Task: AZ-640_initial_structure Name: Initial Structure Description: Scaffold the Rust cargo workspace — per-component crates, shared crate, runtime composition root, Dockerfile + docker-compose for dev/test, Woodpecker CI pipeline, observability scaffold, on-device state directory, env config, and replay-based integration test layout. Complexity: 5 points Dependencies: None Component: Bootstrap Tracker: AZ-640 Epic: AZ-626

Project Folder Layout

autopilot/
├── Cargo.toml                          # cargo workspace manifest
├── Cargo.lock
├── rust-toolchain.toml                 # pin stable channel + components
├── .cargo/
│   └── config.toml                     # cross-compile target = aarch64-unknown-linux-gnu
├── .woodpecker.yml                     # CI pipeline (per deployment/ci_cd_pipeline.md)
├── .dockerignore
├── Dockerfile                          # multi-stage; non-root; pinned l4t-base for prod, ubuntu:22.04 for emul
├── docker-compose.yml                  # dev: autopilot + mock detections + mock missions + mock ground-station
├── docker-compose.test.yml             # blackbox: autopilot + ArduPilot SITL + mock detections + replay sources
├── .env.example                        # documented environment variables
├── config/
│   ├── autopilot.dev.toml              # dev profile (mock endpoints)
│   ├── autopilot.staging.toml          # staging profile (real endpoints, non-flight)
│   └── autopilot.prod.toml             # prod template (Jetson on-airframe)
├── crates/
│   ├── autopilot/                      # binary crate — runtime composition root
│   │   ├── Cargo.toml                  # `[[bin]] name = "autopilot"`
│   │   ├── src/
│   │   │   ├── main.rs                 # CLI parse, config load, wire actors, run
│   │   │   ├── runtime.rs              # actor topology, health aggregator, shutdown
│   │   │   └── health_server.rs        # HTTP /health endpoint (port from config)
│   │   └── tests/                      # cross-crate integration tests (replay-based)
│   ├── shared/
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs                  # re-exports
│   │       ├── models/                 # canonical entities from data_model.md
│   │       │   ├── mod.rs
│   │       │   ├── frame.rs            # Frame, BoundingBox
│   │       │   ├── detection.rs        # Detection, DetectionBatch
│   │       │   ├── movement.rs         # MovementCandidate
│   │       │   ├── tier2.rs            # Tier2Evidence
│   │       │   ├── vlm.rs              # VlmAssessment
│   │       │   ├── poi.rs              # POI
│   │       │   ├── mapobject.rs        # MapObject, MapObjectObservation, MapObjectsBundle, IgnoredItem
│   │       │   ├── mission.rs          # MissionItem, MissionWaypoint, Geofence, Coordinate
│   │       │   ├── operator.rs         # OperatorCommand
│   │       │   └── gimbal.rs           # GimbalState
│   │       ├── config/                 # toml loader + typed config sections
│   │       ├── error.rs                # AutopilotError enum, Result alias
│   │       ├── health.rs               # ComponentHealth, AggregatedHealth
│   │       ├── observability/          # tracing-subscriber init + log field constants
│   │       └── clock.rs                # monotonic + wall-clock binding (GPS / NTP)
│   ├── frame_ingest/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs                  # public API trait + actor handle
│   │   ├── src/internal/               # decoder, RTSP client
│   │   └── tests/                      # replay-based unit tests against fixture RTSP clips
│   ├── detection_client/
│   │   ├── Cargo.toml
│   │   ├── build.rs                    # tonic-build for ../detections .proto
│   │   ├── proto/                      # copy of ../detections gRPC contract
│   │   ├── src/lib.rs
│   │   └── tests/
│   ├── movement_detector/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/               # ego-motion, optical-flow, per-zoom-band thresholds
│   │   └── tests/                      # replay fixtures, zoom-out + zoom-in
│   ├── semantic_analyzer/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/               # primitive graph, ROI CNN call
│   │   └── tests/
│   ├── vlm_client/
│   │   ├── Cargo.toml                  # feature = ["vlm"] — see autopilot/Cargo.toml
│   │   ├── src/lib.rs                  # default impl returns VlmAssessment{status=vlm_disabled}
│   │   ├── src/internal/               # UDS client + peer-cred check
│   │   └── tests/
│   ├── scan_controller/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/state_machine/          # ZoomedOut / ZoomedIn / TargetFollow types
│   │   ├── src/poi_queue/              # priority queue + ≤5 POIs/min cap
│   │   └── tests/                      # behaviour-tree scenarios from system-flows.md §F4
│   ├── mapobjects_store/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/h3_index/      # h3rs wrapper
│   │   ├── src/internal/engine/        # engine trait + in-memory+snapshot default impl (Q3)
│   │   └── tests/
│   ├── gimbal_controller/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/a40_protocol/  # ViewPro A40 UDP vendor protocol
│   │   └── tests/                      # mock A40 over UDP
│   ├── operator_bridge/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/auth/          # OperatorCommand envelope validation (Q9 — stubbed)
│   │   └── tests/
│   ├── mission_executor/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/multirotor/    # multirotor variant FSM
│   │   ├── src/internal/fixed_wing/    # fixed-wing variant FSM
│   │   ├── src/internal/geofence/      # INCLUSION + EXCLUSION enforcement
│   │   ├── src/internal/failsafe/      # lost-link ladder, battery thresholds
│   │   └── tests/                      # ArduPilot SITL fixtures
│   ├── mavlink_layer/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/codec/         # MAVLink v2 encode/decode (only §7.7 surface)
│   │   ├── src/internal/transport/     # UDP and serial connection abstraction
│   │   └── tests/                      # SITL conformance fixtures
│   ├── mission_client/
│   │   ├── Cargo.toml
│   │   ├── src/lib.rs
│   │   ├── src/internal/missions_api/  # HTTPS REST client; pull + middle-waypoint POST
│   │   ├── src/internal/mapobjects_sync/ # pre-flight GET + post-flight POST of /mapobjects bundles
│   │   └── tests/                      # mock missions API
│   └── telemetry_stream/
│       ├── Cargo.toml
│       ├── src/lib.rs
│       ├── src/internal/uplink/        # modem push of frames + telemetry + bbox overlay
│       └── tests/                      # mock Ground Station receiver
├── tests/
│   └── e2e/                            # cross-crate blackbox scenarios (used by docker-compose.test.yml)
├── benches/
│   ├── tier1_latency.rs                # benchmark-gate harness for §6 NFRs
│   ├── tier2_latency.rs
│   ├── gimbal_zoom.rs
│   └── movement_fpr.rs                 # per-zoom-band FPR replay benchmark
├── fixtures/
│   ├── rtsp/                           # pre-recorded RTSP clips
│   ├── mavlink/                        # ArduPilot SITL replay scripts
│   ├── missions/                       # mission JSON fixtures
│   └── detections/                     # deterministic Tier-1 response fixtures
├── deploy/
│   ├── systemd/
│   │   └── autopilot.service           # per deployment/containerization.md §3
│   └── jetson/
│       └── README.md                   # on-airframe install steps
└── README.md

Layout Rationale

• Cargo workspace with one crate per component matches the recommended Rust layout in _docs/02_document/decompose/templates/module-layout.md (crates/<component>/). It enforces module boundaries: a crate's internals (internal/, private modules) are unreachable from sibling components — only its lib.rs public surface is.
• Single binary crate crates/autopilot/ is the runtime composition root (per deployment/containerization.md — "single Rust binary"). It depends on every component crate and wires the actor topology in runtime.rs.
• crates/shared/ owns the canonical entity catalogue from data_model.md and cross-cutting concerns (config, error, health, observability, clock). All component crates may import from it; it imports from no one.
• fixtures/ separate from tests/ because the same fixtures feed unit tests, replay-based integration tests, blackbox tests, and benchmark gates.
• vlm_client crate exists unconditionally; the optional behaviour is implemented via a default VlmAssessment provider that returns status=vlm_disabled when the vlm feature is off (per architecture.md §7.6 "Optionality model").

DTOs and Interfaces

Shared DTOs (live in crates/shared/src/models/)

DTO	Source spec	Used by components
Frame, BoundingBox	data_model.md §2	frame_ingest, detection_client, movement_detector, semantic_analyzer, telemetry_stream
Detection, DetectionBatch	data_model.md §2	detection_client, scan_controller, telemetry_stream, operator_bridge
MovementCandidate	data_model.md §2	movement_detector, scan_controller
Tier2Evidence	data_model.md §2	semantic_analyzer, scan_controller
VlmAssessment	data_model.md §2	vlm_client, scan_controller
POI	data_model.md §3	scan_controller, operator_bridge, telemetry_stream
MapObject, MapObjectObservation, MapObjectsBundle, IgnoredItem	data_model.md §3	mapobjects_store, mission_client, scan_controller
Coordinate, Geofence, MissionItem	data_model.md §4	mission_client, mission_executor, operator_bridge
MissionWaypoint	data_model.md §4	mission_executor, mavlink_layer
OperatorCommand	data_model.md §4	operator_bridge, scan_controller, mission_executor
GimbalState	data_model.md §4	gimbal_controller, frame_ingest, movement_detector
AutopilotError, Result<T>	new	every crate
ComponentHealth, AggregatedHealth	new (per containerization.md §7)	every crate + autopilot/runtime.rs

Component Public APIs (live in each component's lib.rs)

Each component exposes an actor handle plus its narrow request/response trait. Inter-component communication is Tokio channels owned inside the component; consumers receive a typed handle, not the underlying tokio::sync::* types.

Component	Public surface (handle methods)	Exposed to
frame_ingest	FrameIngestHandle::subscribe() -> FrameStream, health()	detection_client, movement_detector, telemetry_stream
detection_client	DetectionClientHandle::request(Frame) -> Result<DetectionBatch>, health()	scan_controller, movement_detector, telemetry_stream
movement_detector	MovementDetectorHandle::candidates() -> CandidateStream, health()	scan_controller
semantic_analyzer	SemanticAnalyzerHandle::analyze(Roi) -> Result<Tier2Evidence>, health()	scan_controller
vlm_client (trait)	VlmProvider::assess(Roi) -> Result<VlmAssessment> (default impl returns vlm_disabled)	scan_controller
scan_controller	ScanControllerHandle::tick(), submit_operator_cmd(OperatorCommand), health()	autopilot::runtime
mapobjects_store	MapObjectsStoreHandle::classify(Detection) -> Classification, apply_decline(Poi), dump_pending() -> MapObjectsBundle, hydrate(MapObjectsBundle), health()	scan_controller, mission_client
gimbal_controller	GimbalControllerHandle::set_pose(GimbalCommand), zoom(level), state() -> GimbalState, health()	scan_controller
operator_bridge	OperatorBridgeHandle::surface_poi(POI) -> OperatorDecision, cmds() -> CommandStream, health()	scan_controller, mission_executor
mission_executor	MissionExecutorHandle::start(Mission), insert_middle_waypoint(Coordinate), failsafe_trigger(FailsafeKind), health()	scan_controller, operator_bridge
mavlink_layer	MavlinkHandle::send(Command), telemetry() -> TelemetryStream, health()	mission_executor, telemetry_stream
mission_client	MissionClientHandle::pull_mission() -> Mission, post_middle_waypoint(Coordinate), pull_mapobjects(MissionId) -> MapObjectsBundle, push_mapobjects(MapObjectsBundle), health()	mission_executor, mapobjects_store
telemetry_stream	TelemetryStreamHandle::push_frame(Frame, Overlay), push_telemetry(Sample), health()	frame_ingest, detection_client, mavlink_layer, operator_bridge

CI/CD Pipeline

Single Woodpecker pipeline (per deployment/ci_cd_pipeline.md §2). Stages run sequentially; a failed stage stops the run.

Stage	Purpose	Tool / Command
Fetch	Clone, restore Cargo cache	cargo fetch with remote cache key
Lint	cargo fmt --check; cargo clippy --all-targets --all-features -- -D warnings	Hard fail on any warning
Unit Tests	cargo test --workspace (host-arch)	Most logic is platform-independent
Build arm64	Cross-compile for aarch64-unknown-linux-gnu	cross or cargo zigbuild; produce binary + debug symbols
Build no-vlm	cargo build --workspace --no-default-features	Enforces VLM optionality contract
Integration Tests	Replay-based, no hardware	cargo test --test '*' -- --include-ignored=false; fixtures from fixtures/
SITL Conformance	ArduPilot SITL + autopilot binary in containers, fixed mission, asserts §7.7 surface + geofence	docker compose -f docker-compose.test.yml up --abort-on-container-exit
Security Scan	cargo audit + cargo deny check	Dependency CVE scan
Benchmark Gate (manual / nightly)	Tier 1 / 2 / VLM / gimbal latency on real Jetson	Runs on self-hosted Jetson Orin Nano runner
Package	Build container image	Multi-arch tag azaion/autopilot:<branch>-arm64
Sign	Cosign for image; OS signing flow for binary	Tagged builds only
Publish	Push image + binary to internal registry	Tagged builds only

Pipeline Configuration Notes

• Cache ~/.cargo/registry/, ~/.cargo/git/, and target/ between runs keyed on Cargo.lock hash.
• --features vlm and the no-feature path are both built and tested to enforce the optionality contract.
• dev and main branches are protected; force-push forbidden; merges require a green pipeline.
• Benchmark gate is opt-in (manual approval or nightly cron) because it requires a Jetson runner.

Environment Strategy

Environment	Purpose	Configuration Notes
Development (local)	Run autopilot locally against mock detections + mock missions + mock Ground Station; iterate on logic	docker compose -f docker-compose.yml up; config/autopilot.dev.toml; RUST_LOG=info,autopilot=debug
Staging	Pre-production: real ../detections, real missions API, real Ground Station, but no airframe MAVLink (SITL instead)	config/autopilot.staging.toml; secrets via EnvironmentFile=
Production (airframe)	Native systemd on Jetson Orin Nano per containerization.md §3	/etc/azaion/autopilot/config.toml; /etc/systemd/system/autopilot.service; /var/lib/autopilot/; /run/azaion/in-flight flight-gate marker
CI (Tier-1)	Lint + unit + replay-based integration on amd64	GitHub-hosted runner; no GPU
CI (Tier-2)	Benchmark gate on real Jetson	Self-hosted Jetson Orin Nano Super runner; pinned JetPack + power mode

Environment Variables

Variable	Dev	Staging	Production	Description
AUTOPILOT_CONFIG	./config/autopilot.dev.toml	/etc/azaion/autopilot/config.toml	/etc/azaion/autopilot/config.toml	Path to TOML config
RUST_LOG	info,autopilot=debug	info	info	tracing-subscriber filter
AUTOPILOT_MISSION_ID	(per-flight CLI arg)	(per-flight CLI arg)	(per-flight CLI arg)	Active mission UUID; CLI arg, not env
AUTOPILOT_HEALTH_BIND	127.0.0.1:8080	127.0.0.1:8080	127.0.0.1:8080	HTTP /health bind address
AUTOPILOT_VLM_ENABLED	false	false (until benchmark passes)	per benchmark	Runtime VLM flag; binary must also build with --features vlm
MISSIONS_API_TOKEN	(mock)	from EnvironmentFile=	from EnvironmentFile=	Bearer token; never in config.toml
GROUND_STATION_TOKEN	(mock)	from EnvironmentFile=	from EnvironmentFile=	Bearer / session token

All non-secret configuration lives in config.toml (per containerization.md §6). Secrets come from EnvironmentFile= on systemd, from compose secrets: in containers.

Database Migration Approach

Migration tool: none — autopilot has no traditional database.

Persistence strategy: the only persisted data is the on-device mapobjects_store. Its engine is open (architecture.md §8 Q3); the bootstrap default is in-memory + snapshot to /var/lib/autopilot/mapobjects/ (file-backed, no schema migrations). When Q3 resolves toward SQLite + H3 or another engine, the mapobjects_store crate's engine module is swapped without changing its public API. The central missions API owns its own Postgres schema (per architecture.md §7.13) — autopilot does NOT migrate central tables.

Initial Persisted Surface

Subsystem	What is persisted	Where	Format
mapobjects_store	current_state, pending_observations, pending_ignored, sync_state	/var/lib/autopilot/mapobjects/	engine-defined; default = JSON snapshots + append-only log
operator_bridge audit log	accepted/rejected OperatorCommand envelopes	/var/lib/autopilot/audit/	newline-delimited JSON
mission_client deferred uploads	post-flight push payload on push failure	/var/lib/autopilot/pending_pushes/	JSON files keyed by mission ID

Disk quota for /var/lib/autopilot/ is configured in config.toml; persistent-store-full at pre-flight BIT is a takeoff blocker (per architecture.md §5).

Test Structure

crates/<component>/
└── tests/                              # crate-level integration tests; per-crate
    └── <scenario>.rs

tests/
└── e2e/                                # workspace-level end-to-end (uses docker-compose.test.yml)
    ├── sitl_conformance.rs             # SITL gate per ci_cd_pipeline.md §5
    ├── geofence_inclusion.rs
    ├── geofence_exclusion.rs           # explicit regression vs earlier silent-ignore behaviour
    ├── lost_link_failsafe.rs
    └── operator_command_replay.rs

fixtures/
├── rtsp/<clip>.h264
├── mavlink/<replay>.tlog
├── missions/<mission>.json
└── detections/<deterministic>.json

benches/
├── tier1_latency.rs                    # benchmark-gate harness
├── tier2_latency.rs
├── gimbal_zoom.rs
└── movement_fpr.rs                     # per-zoom-band FPR replay

Test Configuration Notes

• Unit tests live alongside each component's source in #[cfg(test)] mod tests { ... } within src/ files. They MUST run in <5 s on developer workstation; no network, no Docker.
• Crate-level integration tests live in crates/<component>/tests/. They may use fixtures from fixtures/ but MUST NOT cross component boundaries — that's what workspace e2e is for.
• Workspace e2e in tests/e2e/ exercises the full binary against a docker-compose-managed stack (ArduPilot SITL, mock missions API, mock detections gRPC, replay RTSP).
• Replay-driven debugging: all non-trivial decisions are reconstructable from logs + size-capped raw inputs (per observability.md §6). Replay fixtures are the foundation of regression tests.
• Test runner: cargo test --workspace for unit + integration; docker compose -f docker-compose.test.yml up --abort-on-container-exit for e2e; cargo bench (or criterion) for benchmark-gate measurements.
• Mock-data discipline: mocks live in tests/ directories only — never in production crates (per coderule.mdc).

Implementation Order

Order	Component	Reason
1	shared (models + config + error + health + observability + clock)	Every other crate depends on it; nothing depends on it. Must land first.
2	mavlink_layer	Self-contained transport; required by mission_executor and telemetry_stream; SITL conformance lands the first hard gate early.
3	mission_client	Self-contained REST client; required by mission_executor and mapobjects_store sync.
4	mission_executor	Combines mavlink_layer + mission_client + geofence/failsafe logic; gates takeoff via BIT.
5	gimbal_controller	Self-contained A40 UDP driver; required by scan_controller.
6	frame_ingest	RTSP decoder; required by all perception crates.
7	detection_client	gRPC client to ../detections; required by scan_controller and telemetry_stream.
8	movement_detector	Depends on frame_ingest + GimbalState; standalone otherwise.
9	mapobjects_store	Engine choice may be deferred; default in-memory+snapshot unblocks scan_controller.
10	semantic_analyzer	Tier 2; depends on Frame + Detection.
11	vlm_client	Optional; default impl returns vlm_disabled. Real IPC implementation can land later.
12	telemetry_stream	Pure egress; ready once frame_ingest, detection_client, mavlink_layer exist.
13	operator_bridge	Depends on telemetry_stream + mapobjects_store; envelope auth scheme is Q9-stubbed.
14	scan_controller	Sits on top of everything in Perception + Action; lands last.
15	autopilot binary (composition root)	Wires every component handle; runs the actor topology.

Acceptance Criteria

AC-1: Workspace scaffolded Given the structure plan above When the implementer executes this task Then cargo metadata lists all 14 crates (shared, autopilot, and 12 components — vlm_client is the 13th component crate but listed under perception above) and cargo check --workspace succeeds with no compile errors.

AC-2: Stub tests runnable Given the scaffolded workspace When cargo test --workspace runs on a developer workstation (no Docker, no GPU) Then every crate's stub test (e.g. it_compiles()) passes within 5 seconds total.

AC-3: CI pipeline configured Given the scaffolded workspace When the Woodpecker pipeline runs on a feature branch push Then fetch → lint → unit-test → build-arm64 → build-no-vlm → integration-test → sitl-conformance all complete successfully on a known-good baseline commit.

AC-4: Dev compose boots Given docker-compose.yml When docker compose -f docker-compose.yml up -d runs on a fresh workstation Then the autopilot container starts, the /health endpoint returns HTTP 200 with status: green | yellow (red is acceptable here only for components without a mock target), and the mock detections + mock missions services are reachable.

AC-5: Blackbox compose boots with SITL Given docker-compose.test.yml When docker compose -f docker-compose.test.yml up --abort-on-container-exit runs Then ArduPilot SITL + autopilot + mock detections + replay RTSP all start, and the SITL conformance e2e test exits 0.

AC-6: Optionality contract enforced Given the scaffolded workspace When cargo build --workspace --no-default-features runs Then the binary builds and links without the vlm feature; cargo test --workspace --no-default-features passes; the VlmProvider default impl returns VlmAssessment{status=vlm_disabled}.

AC-7: Cross-compile target ready Given .cargo/config.toml configured for aarch64-unknown-linux-gnu When cross build --target aarch64-unknown-linux-gnu --release (or cargo zigbuild equivalent) runs in CI Then an aarch64 binary is produced and stored as an artifact.

AC-8: Flight-gate marker wiring exists Given deploy/systemd/autopilot.service When systemd parses the unit Then ExecStartPre asserts /run/azaion/in-flight is created and ExecStopPost removes it (per containerization.md §3 and the suite-level flight-gate convention).

AC-9: Observability scaffold initialised Given the autopilot binary When it starts Then tracing-subscriber emits JSON-formatted logs to stdout with the per-line fields enumerated in observability.md §2 (ts, ts_mono_ns, level, target, event), and the /health endpoint returns the per-component breakdown documented in containerization.md §7.

AC-10: Persistent state directory created Given /var/lib/autopilot/ (or its container-mounted equivalent) When autopilot starts in dev or prod Then the binary creates mapobjects/, audit/, and pending_pushes/ subdirectories with the owning user, fails closed if any directory cannot be created, and surfaces the failure to /health (red on mapobjects_store).