autopilot/_docs/02_document/deployment/observability.md

Observability

Status: forward-looking design (Rust). Treat the choices below as the intended approach; the exact tracing exporter / metrics scraper / log-shipping target depend on the suite's overall observability stack at deploy time.

1. Posture

• One binary, one process. Per-component instrumentation is structured (each component listed in architecture.md §3 is a tracing target).
• Structured logs are primary, metrics are derived from log spans and counters, traces are end-to-end on a frame's journey through the pipeline.
• No silent error swallowing. Every failure path increments a counter, emits a span event, or both.
• Health is aggregated, not derived from logs. The HTTP health endpoint (containerization.md §7) is the source of truth for live readiness.

2. Logs

Library: tracing + tracing-subscriber.

Format: JSON to stdout. Captured by the host's journald (Option A) or by the container runtime (Option B), then shipped to the suite's log aggregator.

Per-line fields:

Field	Source	Notes
ts	wall clock	ISO-8601 UTC.
ts_mono_ns	monotonic clock	For ordering across components without clock-skew artefacts.
level	tracing	error | warn | info | debug | trace.
target	component name	One of frame_ingest, detection_client, movement_detector, semantic_analyzer, vlm_client, scan_controller, mapobjects_store, gimbal_controller, operator_bridge, mission_executor, mavlink_layer, mission_client, telemetry_stream.
frame_seq	propagated context	Where applicable. Lets us reconstruct one frame's journey.
poi_id, roi_id, target_id, mission_id, command_id	propagated context	Where applicable.
event	message	Short, machine-friendly identifier (e.g., frame.dropped, vlm.timeout, mission.geofence_violation, bit.check_failed, failsafe.lost_link, mapobjects.push_failed, operator.auth_rejected).
model_version	propagated context	Version string for tier1_model_version and vlm_model_version. Required on every vlm.response and on every Tier-2 evidence span for forensic correlation.
wall_clock_source	telemetry frame	gnss | host | coast; emitted on every state-transition span and on every operator-command audit log line.
reason	message	Free-form for human readers.

Log level defaults:

• info: lifecycle (startup / shutdown / state transitions), all error and security events.
• warn: degraded-but-running events (yellow health, retries, drops).
• error: red health, hard failures, schema violations, security violations.
• debug / trace: off in production; enabled per-target via RUST_LOG.

Always logged at warn or higher (per coderule.mdc):

• Every exception path that the operator could care about.
• Authentication / authorisation failures (peer-cred check failures on VLM IPC, malformed Ground Station session, MAVLink-2 signing rejection).
• Geofence violations.
• Schema validation failures (Tier 1 response, VLM response, mission JSON).

3. Metrics

Derived from log spans + a small set of explicit counters. Exporter: Prometheus-compatible (per the suite's stack).

Per-component counters (illustrative — exact names finalised at implementation):

Component	Counter	Type
frame_ingest	frames_received_total, frames_dropped_total{reason}, decode_errors_total	counter
frame_ingest	decode_ms	histogram
detection_client	requests_total, errors_total{kind}, latency_ms	counter / histogram
movement_detector	candidates_total, telemetry_skew_drops_total	counter
semantic_analyzer	tier2_runs_total, tier2_latency_ms, tier2_oversize_total	counter / histogram
vlm_client	vlm_requests_total{status}, vlm_latency_ms	counter / histogram
scan_controller	state_transitions_total{from,to}, pois_in_queue, pois_per_min, tick_latency_ms	counter / gauge / histogram
mapobjects_store	classify_total{result}, ignored_items_total, removed_candidates_total	counter
gimbal_controller	commands_total, decision_to_movement_ms, zoom_transition_ms, vendor_faults_total	counter / histogram
mavlink_layer	messages_in_total{kind}, messages_out_total{kind}, command_acks_total{result}, parse_errors_total, link_state	counter / gauge
mission_executor	state_transitions_total{from,to}, mission_uploads_total{result}, geofence_violations_total{kind}	counter
mission_client	fetches_total{result}, middle_waypoint_posts_total{result}, mapobjects_pull_total{result}, mapobjects_push_total{result}, mapobjects_pull_bytes, mapobjects_push_bytes, mapobjects_sync_lag_s	counter / gauge
mission_executor (BIT)	bit_runs_total{result}, bit_check_failures_total{check}	counter
mission_executor (failsafe)	link_loss_events_total{trigger}, failsafe_action_total{action}	counter
operator_bridge	pois_surfaced_total, commands_received_total{kind,result}, decision_latency_ms, auth_rejections_total{reason}, command_e2e_ms	counter / histogram
telemetry_stream	bytes_out_total, frames_out_total, link_state, bandwidth_used_mbps	counter / gauge

Aggregated:

• health_state{component} — 0 (red) / 1 (yellow) / 2 (green); enables alerting per-component.
• process_uptime_seconds, process_resident_memory_bytes — standard.

4. Traces

tracing spans cover the path of a single frame and the path of a single POI.

Frame trace (per Frame):

frame_ingest.publish
  detection_client.request
    detection_client.response
  movement_detector.tick
    [movement_detector.emit_candidate]
  telemetry_stream.push

POI trace (per POI):

scan_controller.enqueue
  scan_controller.dequeue
    gimbal_controller.zoom
    semantic_analyzer.tier2
      [vlm_client.request -> vlm_client.response]
    operator_bridge.surface
      [operator_bridge.confirm | decline | timeout]
        mission_executor.middle_waypoint    # confirm path
        mapobjects_store.append_ignored     # decline path

Spans propagate via context across in-process channels. Trace export target depends on the suite's stack (OTLP / Jaeger / Tempo).

5. Health endpoint

See containerization.md §7. The endpoint is the operator-facing readiness API; metrics + logs are the engineer-facing investigation API.

A red health state for any of these components is unrecoverable for the current flight:

• frame_ingest red → no input → cannot operate.
• mavlink_layer red → no UAV control → trigger RTL via the autopilot's failsafe (the autopilot itself enforces this when MAVLink heartbeat stops).
• mission_executor red → mission lifecycle stuck → operator must take RC control.

A red health state for these components is degraded-but-survivable:

• detection_client → continue zoom-out sweep; lose Tier 1.
• movement_detector → continue; lose movement-candidate POI source.
• semantic_analyzer → continue; surface Tier-1-only POIs.
• vlm_client → fail-closed (POIs surfaced without VLM evidence).
• mapobjects_store → continue with in-memory state; persistent diff lost on restart. Sync state may transition to Stale (operator visible).
• mapobjects_sync (logical, owned by mission_client) → mission proceeds with stale snapshot; post-flight push retries via leftover spool. Operator sees mapobjects_sync = degraded.
• operator_bridge / telemetry_stream → continue zoom-out sweep; pause POI surfacing; resume on reconnect. F10 lost-link ladder owns the larger response.
• gimbal_controller → pause zoom-in / target-follow; zoom-out sweep stops.
• mission_client → continue current mission from in-memory copy.

6. Replay-driven debugging

All non-trivial decisions in scan_controller, movement_detector, semantic_analyzer, vlm_client, and mission_executor are reconstructable from logs + the (size-capped) raw inputs that drove them:

• Frame seq, gimbal state at decode, telemetry sample used, Tier-1 detections returned, Tier-2 score, VLM raw response (size-capped), operator command, resulting state transition.

This is the foundation of the replay-based integration tests in ci_cd_pipeline.md §2.

7. Out of scope here

• Suite-wide observability stack choice (OTLP vs Loki vs Tempo vs Promtail) — owned by suite ops.
• Persistent log retention policy — owned by suite ops.
• Alerting routing (Slack / PagerDuty / email) — owned by suite ops.