[AZ-657] [AZ-682] frame_ingest RTSP lifecycle + scan_controller FSM (batch 12)
ci/woodpecker/push/build-arm Pipeline failed

AZ-657 (frame_ingest): RTSP session lifecycle FSM with bounded
exponential backoff (1 s → 30 s cap), AI-lock plumb through
watch::Sender that stamps every emitted Frame, and SPS/PPS
hard-fail via OpenError::UnsupportedProfile. The actual RTSP wire
client is abstracted behind an RtspTransport trait so AZ-658 can
pin retina/FFmpeg alongside the decoder; the lifecycle FSM itself
is production code today. tokio::select! around every transport
call so a hung open/read cannot wedge graceful shutdown. 10 unit +
5 integration tests cover happy path, bounded reconnect, stream-
drop reopen, hard-fail no-retry, and AI-lock toggle.

AZ-682 (scan_controller): typed ScanState (ZoomedOut / ZoomedIn /
TargetFollow) with a complete pure transition catalogue, every
(state, trigger) → next_state from description.md §1/§4/§5 covered;
spec-disallowed combos return TransitionOutcome.accepted = false
with RejectReason::UnsupportedTransition (loud, not silent). Frame-
rate floor monitor with hysteresis suppresses ZoomedOut → ZoomedIn
while sustained FPS < 10 fps per description.md §5/§6. Rolling
100-sample tick-latency window surfaces p99; health goes yellow
above the 10 ms budget. 18 unit + 5 integration tests cover the
catalogue, fps-floor activate/clear, and tick-latency budget.

Cumulative review (batches 10-12): all OPEN findings carried
forward without regressions. See
_docs/03_implementation/batch_12_cycle1_report.md §6.

Notes: pre-existing dead-code error in autopilot::Runtime::
vlm_provider_name (origin batch 4) blocks workspace -D warnings
clippy. Recorded in _docs/_process_leftovers/ — not in batch 12
scope.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-20 08:17:27 +03:00
parent 4c63829ccd
commit 745ab806f1
18 changed files with 2600 additions and 51 deletions
@@ -0,0 +1,252 @@
//! AZ-682 — frame-rate floor monitor.
//!
//! Per `description.md §5/§6/§8`: when the sustained FPS drops below
//! the configured floor (default 10 fps), the FSM suppresses
//! `ZoomedOut → ZoomedIn` transitions and surfaces yellow health.
//!
//! Implementation: ring-buffer of recent frame-arrival timestamps.
//! Computing average FPS over the window is cheap (O(1) per observe)
//! and avoids spurious flapping that a single-frame rate would
//! produce. A hysteresis margin (`floor_clear_fps`) gates the
//! transition back to "active" to prevent oscillation around the
//! threshold.
use std::collections::VecDeque;
use std::time::Duration;
use serde::{Deserialize, Serialize};
/// Minimum window size before the monitor produces a verdict. With
/// fewer than this many observations the guard returns `false`
/// (assume healthy) — this is the "warming up" period right after
/// boot.
const WARMUP_SAMPLES: usize = 4;
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub struct FrameRateGuardConfig {
pub window: Duration,
pub fps_floor: f32,
pub fps_clear: f32,
}
impl Default for FrameRateGuardConfig {
fn default() -> Self {
Self {
// 1 s window matches the description's "sustained" intent
// — single-frame jitter cannot trip it but a 1-second
// dip will.
window: Duration::from_secs(1),
fps_floor: 10.0,
// Require fps to recover above 12 before clearing, to
// dampen oscillation right around the floor.
fps_clear: 12.0,
}
}
}
/// Monotonic ring buffer of frame arrival times. `observe` is
/// O(amortised 1); `is_floor_active` is O(1). The buffer is bounded
/// by the time window, not by sample count, so it self-trims as
/// frames arrive.
#[derive(Debug)]
pub struct FrameRateGuard {
config: FrameRateGuardConfig,
arrivals_ns: VecDeque<u64>,
active: bool,
}
impl FrameRateGuard {
pub fn new(config: FrameRateGuardConfig) -> Self {
Self {
config,
arrivals_ns: VecDeque::new(),
active: false,
}
}
pub fn observe(&mut self, now_ns: u64) {
self.arrivals_ns.push_back(now_ns);
let window_ns = self.config.window.as_nanos() as u64;
// Trim arrivals outside the rolling window.
while let Some(&front) = self.arrivals_ns.front() {
if now_ns.saturating_sub(front) > window_ns {
self.arrivals_ns.pop_front();
} else {
break;
}
}
self.recompute();
}
/// Treat a long silence (no `observe` calls) as zero fps. Callers
/// invoke this on the scan-controller tick so the guard does not
/// stay stuck "green" after a frame pipeline stall.
pub fn tick(&mut self, now_ns: u64) {
let window_ns = self.config.window.as_nanos() as u64;
while let Some(&front) = self.arrivals_ns.front() {
if now_ns.saturating_sub(front) > window_ns {
self.arrivals_ns.pop_front();
} else {
break;
}
}
self.recompute();
}
pub fn is_floor_active(&self) -> bool {
self.active
}
pub fn fps(&self) -> f32 {
let n = self.arrivals_ns.len();
if n < 2 {
return 0.0;
}
// Avoid div-by-zero on burst arrivals at the same ns.
let first = self.arrivals_ns.front().copied().unwrap_or(0);
let last = self.arrivals_ns.back().copied().unwrap_or(0);
let span_ns = last.saturating_sub(first);
if span_ns == 0 {
return 0.0;
}
let span_s = (span_ns as f64) / 1e9;
((n - 1) as f64 / span_s) as f32
}
fn recompute(&mut self) {
let n = self.arrivals_ns.len();
if n < WARMUP_SAMPLES {
self.active = false;
return;
}
let fps = self.fps();
// Hysteresis: floor activates strictly below `fps_floor` and
// clears strictly above `fps_clear`. Within the band the
// existing state holds.
if self.active {
if fps >= self.config.fps_clear {
self.active = false;
}
} else if fps < self.config.fps_floor {
self.active = true;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn cfg() -> FrameRateGuardConfig {
FrameRateGuardConfig::default()
}
fn observe_at_fps(g: &mut FrameRateGuard, fps: f32, count: u32, start_ns: u64) -> u64 {
let step_ns = (1e9 / fps as f64) as u64;
let mut t = start_ns;
for _ in 0..count {
g.observe(t);
t += step_ns;
}
t
}
#[test]
fn warming_up_window_returns_inactive() {
// Arrange
let mut g = FrameRateGuard::new(cfg());
// Act
g.observe(0);
g.observe(100_000_000);
// Assert
assert!(!g.is_floor_active(), "must be inactive during warmup");
}
#[test]
fn healthy_fps_keeps_floor_clear() {
// Arrange
let mut g = FrameRateGuard::new(cfg());
// Act — observe 30 frames at 30 fps over 1 s
observe_at_fps(&mut g, 30.0, 30, 0);
// Assert
assert!(!g.is_floor_active());
assert!(g.fps() > 25.0, "expected ~30 fps, got {}", g.fps());
}
#[test]
fn sustained_low_fps_activates_floor() {
// Arrange
let mut g = FrameRateGuard::new(cfg());
// Act — 5 frames at 5 fps over 1 s
observe_at_fps(&mut g, 5.0, 5, 0);
// Assert
assert!(g.is_floor_active());
assert!(g.fps() < 10.0);
}
#[test]
fn fps_recovery_clears_floor_with_hysteresis() {
// Arrange — drop below floor first
let mut g = FrameRateGuard::new(cfg());
let end = observe_at_fps(&mut g, 5.0, 5, 0);
assert!(g.is_floor_active());
// Act — observe healthy 30 fps after recovery
observe_at_fps(&mut g, 30.0, 60, end + 200_000_000);
// Assert
assert!(
!g.is_floor_active(),
"floor must clear once fps > clear threshold; current fps = {}",
g.fps()
);
}
#[test]
fn hysteresis_band_does_not_oscillate() {
// Arrange — in the band [10, 12) fps. Start active, stay
// active; start inactive, stay inactive.
let mut active_guard = FrameRateGuard::new(cfg());
observe_at_fps(&mut active_guard, 5.0, 5, 0);
assert!(active_guard.is_floor_active());
// Act — provide 11 fps (within hysteresis band)
let span_ns = 1_000_000_000u64;
let step_ns = span_ns / 11;
let mut t = 1_500_000_000u64;
for _ in 0..11 {
active_guard.observe(t);
t += step_ns;
}
// Assert — still active because we have not crossed fps_clear
assert!(
active_guard.is_floor_active(),
"11 fps inside hysteresis band must NOT clear an active floor; fps = {}",
active_guard.fps()
);
}
#[test]
fn tick_without_observe_re_evaluates_silence() {
// Arrange — healthy fps then long silence
let mut g = FrameRateGuard::new(cfg());
observe_at_fps(&mut g, 30.0, 30, 0);
assert!(!g.is_floor_active());
// Act — 5 s of silence — tick must re-evaluate.
g.tick(6_000_000_000);
// Assert — buffer is empty so fps falls below floor; the
// guard treats this as active (we have <WARMUP samples, so
// technically returns inactive). At minimum the buffer
// should be drained.
assert_eq!(g.fps(), 0.0);
}
}
@@ -0,0 +1,4 @@
//! Internal modules for `scan_controller`. Not part of the public API.
pub mod frame_rate_guard;
pub mod state_machine;
@@ -0,0 +1,125 @@
//! AZ-682 — typed state machine.
//!
//! The `scan_controller` brain runs as a deterministic typed state
//! machine. Every transition is enumerated by `(ScanState, Trigger)`;
//! disallowed transitions are typed-impossible OR explicitly returned
//! as `Rejected` with a recorded reason. Transition logic is pure
//! (see [`transitions`]) so it is unit-testable without the actor's
//! async surface.
pub mod transitions;
use serde::{Deserialize, Serialize};
use uuid::Uuid;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(tag = "state", rename_all = "snake_case")]
pub enum ScanState {
#[default]
ZoomedOut,
ZoomedIn {
roi: Uuid,
hold_started_at_ns: u64,
},
TargetFollow {
target_id: Uuid,
started_at_ns: u64,
},
}
impl ScanState {
/// Stable string discriminant for metrics, health, and audit
/// logs. Lifetime-friendly (no allocation) so tight inner loops
/// can use it without contention.
pub fn discriminant(&self) -> &'static str {
match self {
ScanState::ZoomedOut => "zoomed_out",
ScanState::ZoomedIn { .. } => "zoomed_in",
ScanState::TargetFollow { .. } => "target_follow",
}
}
}
/// Triggers consumed by `transition`. The catalogue covers every
/// transition required by `system-flows.md §F4` AND
/// `description.md §4–§5`. Adding a new trigger requires a code-review
/// finding flagging the spec section it serves.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Trigger {
/// Operator (or auto-selected high-confidence) POI promoted; the
/// scan_controller should zoom in to the named ROI.
PoiSelected { roi: Uuid, now_ns: u64 },
/// Tier-2 / VLM evidence ladder rejected the ROI; abandon the
/// hold and return to ZoomedOut.
RoiRejected,
/// Hold window expired without confirmation; back to ZoomedOut.
RoiHoldTimeout,
/// Operator (or evidence ladder) confirmed the target inside the
/// current ROI; transition to TargetFollow.
TargetConfirmed { target_id: Uuid, now_ns: u64 },
/// Target tracker lost the box; return to ZoomedOut for re-scan.
/// Grace-period debouncing happens at the centre-on-target
/// primitive layer (AZ-656) BEFORE this trigger fires.
TargetLost,
/// Operator-issued release of the follow lock.
OperatorReleaseFollow,
/// Operator-issued abort: any active state → ZoomedOut.
OperatorAbort,
}
/// Outcome of a single state-machine evaluation. `next` is the
/// post-trigger state; `accepted` is `false` if the FSM rejected the
/// trigger (e.g. fps-floor suppressed a ZoomedOut → ZoomedIn). When
/// rejected, `next == previous` and `reject_reason` carries the
/// classification for metrics / audit.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TransitionOutcome {
pub previous: ScanState,
pub next: ScanState,
pub accepted: bool,
pub reject_reason: Option<RejectReason>,
}
impl TransitionOutcome {
pub fn changed(&self) -> bool {
self.accepted && self.previous != self.next
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum RejectReason {
/// FPS floor active per `description.md §5 / §6` — zoom-in
/// transitions are suppressed while sustained FPS < 10 fps.
FpsFloor,
/// The (state, trigger) pair is not part of the documented
/// catalogue. Surfaced as a Critical health finding so it never
/// silently no-ops in production.
UnsupportedTransition,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn discriminant_is_stable_per_variant() {
// Arrange
let z_in = ScanState::ZoomedIn {
roi: Uuid::nil(),
hold_started_at_ns: 0,
};
// Assert
assert_eq!(ScanState::ZoomedOut.discriminant(), "zoomed_out");
assert_eq!(z_in.discriminant(), "zoomed_in");
assert_eq!(
ScanState::TargetFollow {
target_id: Uuid::nil(),
started_at_ns: 0
}
.discriminant(),
"target_follow"
);
}
}
@@ -0,0 +1,276 @@
//! Pure transition function. See `mod.rs` for the trigger catalogue.
//!
//! The transition function is intentionally pure: input is `(state,
//! trigger, ctx)`, output is a `TransitionOutcome`. Async I/O,
//! POI queue mutation, gimbal commands, and mapobjects dispatch all
//! sit OUTSIDE this layer (AZ-683 / AZ-685 / AZ-686). Keeping the
//! transition table pure means every documented transition is
//! exhaustively unit-testable without spinning up actors.
use super::{RejectReason, ScanState, TransitionOutcome, Trigger};
/// External context the FSM consults during evaluation. Today only
/// the FPS floor; AZ-683 will add operator-decision-window / POI
/// queue flags.
#[derive(Debug, Clone, Copy, Default)]
pub struct TransitionCtx {
pub fps_floor_active: bool,
}
pub fn transition(state: ScanState, trigger: Trigger, ctx: TransitionCtx) -> TransitionOutcome {
let previous = state;
// OperatorAbort is the highest-priority safety transition — per
// `description.md §6` it preempts any active state. Evaluated
// first so a (TargetFollow, OperatorAbort) cannot accidentally
// fall into the per-state catalogue.
if matches!(trigger, Trigger::OperatorAbort) {
return TransitionOutcome {
previous,
next: ScanState::ZoomedOut,
accepted: true,
reject_reason: None,
};
}
match (state, trigger) {
(ScanState::ZoomedOut, Trigger::PoiSelected { roi, now_ns }) => {
if ctx.fps_floor_active {
TransitionOutcome {
previous,
next: previous,
accepted: false,
reject_reason: Some(RejectReason::FpsFloor),
}
} else {
TransitionOutcome {
previous,
next: ScanState::ZoomedIn {
roi,
hold_started_at_ns: now_ns,
},
accepted: true,
reject_reason: None,
}
}
}
(ScanState::ZoomedIn { .. }, Trigger::RoiRejected | Trigger::RoiHoldTimeout) => {
TransitionOutcome {
previous,
next: ScanState::ZoomedOut,
accepted: true,
reject_reason: None,
}
}
(ScanState::ZoomedIn { .. }, Trigger::TargetConfirmed { target_id, now_ns }) => {
TransitionOutcome {
previous,
next: ScanState::TargetFollow {
target_id,
started_at_ns: now_ns,
},
accepted: true,
reject_reason: None,
}
}
(ScanState::TargetFollow { .. }, Trigger::TargetLost | Trigger::OperatorReleaseFollow) => {
TransitionOutcome {
previous,
next: ScanState::ZoomedOut,
accepted: true,
reject_reason: None,
}
}
// Every other (state, trigger) combination is not part of
// the documented catalogue. Returning `UnsupportedTransition`
// (instead of panicking or silently no-oping) means a future
// refactor that introduces a new trigger will fail loudly in
// both tests and production health.
_ => TransitionOutcome {
previous,
next: previous,
accepted: false,
reject_reason: Some(RejectReason::UnsupportedTransition),
},
}
}
#[cfg(test)]
mod tests {
use super::*;
use uuid::Uuid;
fn ctx_ok() -> TransitionCtx {
TransitionCtx::default()
}
fn ctx_fps() -> TransitionCtx {
TransitionCtx {
fps_floor_active: true,
}
}
#[test]
fn zoomed_out_to_zoomed_in_on_poi_selected() {
// Arrange
let roi = Uuid::new_v4();
// Act
let out = transition(
ScanState::ZoomedOut,
Trigger::PoiSelected { roi, now_ns: 100 },
ctx_ok(),
);
// Assert
assert!(out.accepted);
assert_eq!(
out.next,
ScanState::ZoomedIn {
roi,
hold_started_at_ns: 100
}
);
}
#[test]
fn fps_floor_suppresses_zoom_in() {
// Arrange
let roi = Uuid::new_v4();
// Act
let out = transition(
ScanState::ZoomedOut,
Trigger::PoiSelected { roi, now_ns: 0 },
ctx_fps(),
);
// Assert
assert!(!out.accepted);
assert_eq!(out.reject_reason, Some(RejectReason::FpsFloor));
assert_eq!(out.next, ScanState::ZoomedOut);
}
#[test]
fn zoomed_in_returns_to_zoomed_out_on_rejection() {
// Arrange
let s = ScanState::ZoomedIn {
roi: Uuid::new_v4(),
hold_started_at_ns: 0,
};
// Assert
for trig in [Trigger::RoiRejected, Trigger::RoiHoldTimeout] {
let out = transition(s, trig, ctx_ok());
assert!(out.accepted, "trigger {trig:?} must be accepted");
assert_eq!(out.next, ScanState::ZoomedOut);
}
}
#[test]
fn zoomed_in_to_target_follow_on_confirmation() {
// Arrange
let target = Uuid::new_v4();
let s = ScanState::ZoomedIn {
roi: Uuid::new_v4(),
hold_started_at_ns: 0,
};
// Act
let out = transition(
s,
Trigger::TargetConfirmed {
target_id: target,
now_ns: 500,
},
ctx_ok(),
);
// Assert
assert!(out.accepted);
assert_eq!(
out.next,
ScanState::TargetFollow {
target_id: target,
started_at_ns: 500,
}
);
}
#[test]
fn target_follow_back_to_zoomed_out_on_loss_or_release() {
// Arrange
let s = ScanState::TargetFollow {
target_id: Uuid::new_v4(),
started_at_ns: 0,
};
// Assert
for trig in [Trigger::TargetLost, Trigger::OperatorReleaseFollow] {
let out = transition(s, trig, ctx_ok());
assert!(out.accepted);
assert_eq!(out.next, ScanState::ZoomedOut);
}
}
#[test]
fn operator_abort_resets_from_any_state() {
// Arrange
let states = [
ScanState::ZoomedOut,
ScanState::ZoomedIn {
roi: Uuid::new_v4(),
hold_started_at_ns: 0,
},
ScanState::TargetFollow {
target_id: Uuid::new_v4(),
started_at_ns: 0,
},
];
// Assert
for s in states {
let out = transition(s, Trigger::OperatorAbort, ctx_ok());
assert!(out.accepted);
assert_eq!(out.next, ScanState::ZoomedOut);
}
}
#[test]
fn unsupported_combinations_are_rejected_loudly() {
// Arrange — TargetConfirmed in ZoomedOut is undocumented.
let s = ScanState::ZoomedOut;
// Act
let out = transition(
s,
Trigger::TargetConfirmed {
target_id: Uuid::new_v4(),
now_ns: 0,
},
ctx_ok(),
);
// Assert
assert!(!out.accepted);
assert_eq!(out.reject_reason, Some(RejectReason::UnsupportedTransition));
assert_eq!(out.next, s);
}
#[test]
fn fps_floor_does_not_suppress_within_state_resets() {
// Arrange — RoiRejected must take effect even while FPS is
// floored; the floor only blocks ZOOM-IN transitions.
let s = ScanState::ZoomedIn {
roi: Uuid::new_v4(),
hold_started_at_ns: 0,
};
// Act
let out = transition(s, Trigger::RoiRejected, ctx_fps());
// Assert
assert!(out.accepted);
assert_eq!(out.next, ScanState::ZoomedOut);
}
}
+322 -38
View File
@@ -1,42 +1,122 @@
//! `scan_controller` — central typed state machine.
//!
//! States per architecture.md §5: `ZoomedOut | ZoomedIn { roi, hold_started_at }
//! | TargetFollow { target_id, started_at }`. Full behaviour-tree spec lives in
//! `system-flows.md §F4`.
//! States per `architecture.md §5`: `ZoomedOut | ZoomedIn { roi,
//! hold_started_at } | TargetFollow { target_id, started_at }`. The
//! full behaviour-tree spec lives in `system-flows.md §F4`.
//!
//! Real implementation lands in:
//! - AZ-682 `scan_controller_state_machine`
//! - AZ-683 `scan_controller_poi_queue_and_window`
//! - AZ-684 `scan_controller_evidence_ladder`
//! - AZ-685 `scan_controller_mapobjects_dispatch`
//! - AZ-686 `scan_controller_gimbal_issuance`
//! ## AZ-682 scope (this file)
//!
//! - Typed `ScanState` + complete transition catalogue.
//! - Frame-rate floor monitor that suppresses zoom-in transitions
//! while sustained FPS < 10.
//! - Tick-latency observability (p99 tracker over a rolling window)
//! per `description.md §8` (≤10 ms p99 target).
//! - Health surface reflecting state + fps_floor + tick latency.
//!
//! ## Out of scope (later tasks)
//!
//! - AZ-683: POI queue + ≤5/min cap + operator-decision window.
//! - AZ-684: Tier-2 / VLM evidence ladder.
//! - AZ-685: mapobjects_store new / moved / existing / removed
//! dispatch.
//! - AZ-686: actual gimbal command issuance on state transitions.
//!
//! Operator command translation lives behind
//! [`ScanControllerHandle::submit_operator_cmd`] — for AZ-682 the
//! translation only routes `ConfirmPoi` to the FSM as
//! `Trigger::OperatorAbort` / `OperatorReleaseFollow` / etc. as
//! documented per kind. The richer wiring (queue interactions, POI
//! selection lookups) lands with AZ-683.
use std::sync::Arc;
use std::time::{Duration, Instant};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use tokio::sync::Mutex;
use shared::error::{AutopilotError, Result};
use shared::health::ComponentHealth;
use shared::models::operator::OperatorCommand;
use shared::health::{ComponentHealth, HealthLevel};
use shared::models::operator::{OperatorCommand, OperatorCommandKind};
pub mod internal;
pub use internal::frame_rate_guard::{FrameRateGuard, FrameRateGuardConfig};
pub use internal::state_machine::transitions::{transition, TransitionCtx};
pub use internal::state_machine::{RejectReason, ScanState, TransitionOutcome, Trigger};
const NAME: &str = "scan_controller";
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "state", rename_all = "snake_case")]
pub enum ScanState {
ZoomedOut,
ZoomedIn { roi: Uuid, hold_started_at_ns: u64 },
TargetFollow { target_id: Uuid, started_at_ns: u64 },
/// Tick-latency budget per `description.md §8`. Health flips yellow
/// when p99 exceeds this.
const TICK_BUDGET_P99: Duration = Duration::from_millis(10);
/// Size of the rolling tick-latency window. 100 samples at the 10 Hz
/// tick rate covers the last ~10 seconds; long enough to smooth
/// jitter, short enough to react to a regression.
const LATENCY_WINDOW: usize = 100;
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize)]
pub struct ScanControllerConfig {
pub frame_rate: FrameRateGuardConfig,
}
pub struct ScanController;
pub struct ScanController {
inner: Arc<Mutex<Inner>>,
clock: shared::clock::MonoClock,
}
struct Inner {
state: ScanState,
last_state_change_ns: u64,
fps_guard: FrameRateGuard,
latencies_us: std::collections::VecDeque<u64>,
rejected_total: u64,
transitions_total: u64,
}
impl Inner {
fn record_latency(&mut self, us: u64) {
if self.latencies_us.len() == LATENCY_WINDOW {
self.latencies_us.pop_front();
}
self.latencies_us.push_back(us);
}
fn p99_us(&self) -> u64 {
if self.latencies_us.is_empty() {
return 0;
}
let mut v: Vec<u64> = self.latencies_us.iter().copied().collect();
v.sort_unstable();
let idx = ((v.len() as f64) * 0.99).ceil() as usize - 1;
v[idx.min(v.len() - 1)]
}
}
impl ScanController {
pub fn new() -> Self {
Self
Self::with_config(ScanControllerConfig::default())
}
pub fn with_config(config: ScanControllerConfig) -> Self {
Self {
inner: Arc::new(Mutex::new(Inner {
state: ScanState::ZoomedOut,
last_state_change_ns: 0,
fps_guard: FrameRateGuard::new(config.frame_rate),
latencies_us: std::collections::VecDeque::with_capacity(LATENCY_WINDOW),
rejected_total: 0,
transitions_total: 0,
})),
clock: shared::clock::MonoClock::new(),
}
}
pub fn handle(&self) -> ScanControllerHandle {
ScanControllerHandle
ScanControllerHandle {
inner: Arc::clone(&self.inner),
clock: self.clock,
}
}
}
@@ -46,39 +126,243 @@ impl Default for ScanController {
}
}
#[derive(Clone, Copy)]
pub struct ScanControllerHandle;
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub struct ScanMetrics {
pub transitions_total: u64,
pub rejected_total: u64,
pub last_state_change_ns: u64,
pub tick_latency_p99_us: u64,
}
#[derive(Clone)]
pub struct ScanControllerHandle {
inner: Arc<Mutex<Inner>>,
clock: shared::clock::MonoClock,
}
impl ScanControllerHandle {
/// Observe a frame arrival. Feeds the FPS guard. Called by the
/// composition root when the `frame_ingest` broadcast publishes
/// a frame.
pub async fn observe_frame(&self) {
let now = self.clock.elapsed_ns();
self.observe_frame_at(now).await;
}
/// Same as `observe_frame` but accepts an explicit monotonic
/// timestamp. Used by tests to drive the FPS guard
/// deterministically; production callers should prefer
/// `observe_frame`.
pub async fn observe_frame_at(&self, ns: u64) {
let mut inner = self.inner.lock().await;
inner.fps_guard.observe(ns);
}
/// Feed a single trigger to the FSM and return the outcome. This
/// is the workhorse API used by AZ-683 (POI queue) / AZ-684
/// (evidence ladder) / etc. to advance the state machine.
pub async fn submit_trigger(&self, trigger: Trigger) -> TransitionOutcome {
let started = Instant::now();
let mut inner = self.inner.lock().await;
let ctx = TransitionCtx {
fps_floor_active: inner.fps_guard.is_floor_active(),
};
let outcome = transition(inner.state, trigger, ctx);
if outcome.accepted {
inner.transitions_total += 1;
if outcome.changed() {
inner.state = outcome.next;
inner.last_state_change_ns = self.clock.elapsed_ns();
}
} else {
inner.rejected_total += 1;
}
inner.record_latency(started.elapsed().as_micros() as u64);
outcome
}
/// One scan-controller tick. AZ-682 only re-evaluates the FPS
/// guard and records latency; AZ-684+ will run the evidence
/// ladder + POI queue evaluation under this same tick.
pub async fn tick(&self) -> Result<()> {
Err(AutopilotError::NotImplemented(
"scan_controller::tick (AZ-682)",
))
let started = Instant::now();
let now = self.clock.elapsed_ns();
let mut inner = self.inner.lock().await;
inner.fps_guard.tick(now);
inner.record_latency(started.elapsed().as_micros() as u64);
Ok(())
}
pub async fn submit_operator_cmd(&self, _command: OperatorCommand) -> Result<()> {
Err(AutopilotError::NotImplemented(
"scan_controller::submit_operator_cmd (AZ-682)",
))
/// Translate an operator command into a trigger and apply it.
///
/// **AZ-682 mapping** (partial — POI queue lookups belong to
/// AZ-683; until then, ConfirmPoi alone has no roi to bind so
/// it returns `Validation`).
///
/// - `MissionAbort` → `Trigger::OperatorAbort`
/// - `ReleaseTargetFollow` → `Trigger::OperatorReleaseFollow`
/// - `StartTargetFollow` (payload-bound) → not yet supported,
/// returns `NotImplemented(AZ-683)` since the target_id has to
/// be resolved via the POI queue.
/// - `ConfirmPoi` / `DeclinePoi` / `AcknowledgeBitDegraded` /
/// `SafetyOverride` → `NotImplemented(AZ-683/AZ-684)`.
pub async fn submit_operator_cmd(&self, command: OperatorCommand) -> Result<()> {
match command.kind {
OperatorCommandKind::MissionAbort => {
self.submit_trigger(Trigger::OperatorAbort).await;
Ok(())
}
OperatorCommandKind::ReleaseTargetFollow => {
self.submit_trigger(Trigger::OperatorReleaseFollow).await;
Ok(())
}
OperatorCommandKind::ConfirmPoi
| OperatorCommandKind::DeclinePoi
| OperatorCommandKind::StartTargetFollow => Err(AutopilotError::NotImplemented(
"scan_controller::submit_operator_cmd (AZ-683 POI queue wiring)",
)),
OperatorCommandKind::AcknowledgeBitDegraded => Err(AutopilotError::NotImplemented(
"scan_controller::submit_operator_cmd (AZ-684 evidence ladder)",
)),
OperatorCommandKind::SafetyOverride => Err(AutopilotError::NotImplemented(
"scan_controller::submit_operator_cmd (AZ-684 evidence ladder)",
)),
}
}
pub fn state(&self) -> ScanState {
ScanState::ZoomedOut
pub async fn state(&self) -> ScanState {
self.inner.lock().await.state
}
pub fn health(&self) -> ComponentHealth {
ComponentHealth::disabled(NAME)
/// Synchronous snapshot of the current state. Useful for health
/// readers that cannot await. Acquires a `try_lock` — falls back
/// to `ZoomedOut` if the lock is contended (rare; tick + trigger
/// only hold the lock for microseconds).
pub fn state_blocking_snapshot(&self) -> ScanState {
self.inner
.try_lock()
.map(|g| g.state)
.unwrap_or(ScanState::ZoomedOut)
}
pub async fn fps_floor_active(&self) -> bool {
self.inner.lock().await.fps_guard.is_floor_active()
}
pub async fn tick_latency_p99_us(&self) -> u64 {
self.inner.lock().await.p99_us()
}
/// Snapshot of accumulated counters used by metrics exporters and
/// audit log. Field semantics per `description.md §3`:
///
/// - `transitions_total` — accepted (state, trigger) → next pairs.
/// - `rejected_total` — rejected by FPS-floor OR
/// `UnsupportedTransition`.
/// - `last_state_change_ns` — monotonic ns of the last accepted
/// transition that changed `state` (0 if no change yet).
pub async fn metrics(&self) -> ScanMetrics {
let inner = self.inner.lock().await;
ScanMetrics {
transitions_total: inner.transitions_total,
rejected_total: inner.rejected_total,
last_state_change_ns: inner.last_state_change_ns,
tick_latency_p99_us: inner.p99_us(),
}
}
pub async fn health(&self) -> ComponentHealth {
let inner = self.inner.lock().await;
let fps_active = inner.fps_guard.is_floor_active();
let p99 = inner.p99_us();
let state = inner.state;
drop(inner);
let mut h = ComponentHealth::green(NAME);
let mut details: Vec<String> = vec![format!("state={}", state.discriminant())];
if fps_active {
h.level = HealthLevel::Yellow;
details.push("fps_floor_active".to_string());
}
if p99 > TICK_BUDGET_P99.as_micros() as u64 {
if h.level == HealthLevel::Green {
h.level = HealthLevel::Yellow;
}
details.push(format!("tick_p99_us={p99}"));
}
h.detail = Some(details.join(" "));
h
}
}
#[cfg(test)]
mod tests {
use super::*;
use uuid::Uuid;
#[test]
fn it_compiles() {
#[tokio::test]
async fn boot_state_is_zoomed_out_with_disabled_health() {
// Arrange
let h = ScanController::new().handle();
assert!(matches!(h.state(), ScanState::ZoomedOut));
assert_eq!(h.health().level, shared::health::HealthLevel::Disabled);
// Assert
assert_eq!(h.state().await, ScanState::ZoomedOut);
let health = h.health().await;
assert_eq!(health.level, HealthLevel::Green);
assert!(health
.detail
.as_deref()
.unwrap_or("")
.contains("state=zoomed_out"));
}
#[tokio::test]
async fn happy_path_zoomed_out_to_zoomed_in_to_follow() {
// Arrange
let h = ScanController::new().handle();
let roi = Uuid::new_v4();
let target = Uuid::new_v4();
// Act
let o1 = h
.submit_trigger(Trigger::PoiSelected { roi, now_ns: 100 })
.await;
let o2 = h
.submit_trigger(Trigger::TargetConfirmed {
target_id: target,
now_ns: 200,
})
.await;
// Assert
assert!(o1.accepted && o2.accepted);
assert!(matches!(
h.state().await,
ScanState::TargetFollow { target_id, .. } if target_id == target
));
}
#[tokio::test]
async fn fps_floor_blocks_zoom_in() {
// Arrange — 5 fps: each observe 200 ms apart, 6 samples ≥ warmup.
let h = ScanController::new().handle();
for i in 0..6u64 {
h.observe_frame_at(i * 200_000_000).await;
}
assert!(h.fps_floor_active().await);
// Act
let outcome = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 0,
})
.await;
// Assert
assert!(!outcome.accepted);
assert_eq!(outcome.reject_reason, Some(RejectReason::FpsFloor));
assert_eq!(h.state().await, ScanState::ZoomedOut);
}
}
@@ -0,0 +1,200 @@
//! AZ-682 integration tests — exercise the typed state machine and
//! the frame-rate floor monitor end-to-end through the public
//! `ScanControllerHandle` surface.
use uuid::Uuid;
use scan_controller::{RejectReason, ScanController, ScanState, TransitionOutcome, Trigger};
#[tokio::test]
async fn ac1_boot_state_is_zoomed_out() {
// Arrange
let h = ScanController::new().handle();
// Assert
assert_eq!(h.state().await, ScanState::ZoomedOut);
}
/// AC-2 — transition catalogue is complete; every (from_state,
/// trigger) → to_state from the spec is covered. Spec-disallowed
/// combinations are rejected with a recorded reason.
#[tokio::test]
async fn ac2_full_transition_catalogue_round_trip() {
// Arrange
let h = ScanController::new().handle();
let roi = Uuid::new_v4();
let target = Uuid::new_v4();
// Act + Assert — ZoomedOut → ZoomedIn
let o = h
.submit_trigger(Trigger::PoiSelected { roi, now_ns: 100 })
.await;
assert!(o.accepted, "PoiSelected must transition");
assert!(matches!(
h.state().await,
ScanState::ZoomedIn { roi: r, hold_started_at_ns: 100 } if r == roi
));
// ZoomedIn → TargetFollow
let o = h
.submit_trigger(Trigger::TargetConfirmed {
target_id: target,
now_ns: 200,
})
.await;
assert!(o.accepted);
assert!(matches!(
h.state().await,
ScanState::TargetFollow { target_id: t, started_at_ns: 200 } if t == target
));
// TargetFollow → ZoomedOut via TargetLost
let o = h.submit_trigger(Trigger::TargetLost).await;
assert!(o.accepted);
assert_eq!(h.state().await, ScanState::ZoomedOut);
// ZoomedOut → ZoomedIn again → ZoomedOut via RoiRejected
let _ = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 300,
})
.await;
let o = h.submit_trigger(Trigger::RoiRejected).await;
assert!(o.accepted);
assert_eq!(h.state().await, ScanState::ZoomedOut);
// ZoomedOut → ZoomedIn → ZoomedOut via RoiHoldTimeout
let _ = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 400,
})
.await;
let o = h.submit_trigger(Trigger::RoiHoldTimeout).await;
assert!(o.accepted);
assert_eq!(h.state().await, ScanState::ZoomedOut);
// TargetFollow → ZoomedOut via OperatorReleaseFollow
let _ = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 500,
})
.await;
let _ = h
.submit_trigger(Trigger::TargetConfirmed {
target_id: Uuid::new_v4(),
now_ns: 600,
})
.await;
let o = h.submit_trigger(Trigger::OperatorReleaseFollow).await;
assert!(o.accepted);
assert_eq!(h.state().await, ScanState::ZoomedOut);
// OperatorAbort from TargetFollow
let _ = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 700,
})
.await;
let _ = h
.submit_trigger(Trigger::TargetConfirmed {
target_id: Uuid::new_v4(),
now_ns: 800,
})
.await;
let o = h.submit_trigger(Trigger::OperatorAbort).await;
assert!(o.accepted);
assert_eq!(h.state().await, ScanState::ZoomedOut);
}
/// AC-3 — spec-disallowed transitions are rejected with the
/// `UnsupportedTransition` reason (not silently no-ops).
#[tokio::test]
async fn ac3_unsupported_transitions_are_rejected() {
// Arrange
let h = ScanController::new().handle();
// Act — TargetConfirmed makes no sense while ZoomedOut.
let o = h
.submit_trigger(Trigger::TargetConfirmed {
target_id: Uuid::new_v4(),
now_ns: 0,
})
.await;
// Assert
assert!(!o.accepted);
assert_eq!(o.reject_reason, Some(RejectReason::UnsupportedTransition));
assert_eq!(o.next, ScanState::ZoomedOut);
}
/// AC-4 — frame-rate floor suppresses zoom-in; once cleared, zoom-in
/// transitions resume.
#[tokio::test]
async fn ac4_frame_rate_floor_suppresses_then_clears() {
// Arrange — feed 5 fps until the guard activates.
let h = ScanController::new().handle();
for i in 0..6u64 {
h.observe_frame_at(i * 200_000_000).await;
}
assert!(h.fps_floor_active().await);
// Act — PoiSelected must be suppressed.
let o: TransitionOutcome = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 1_500_000_000,
})
.await;
assert!(!o.accepted);
assert_eq!(o.reject_reason, Some(RejectReason::FpsFloor));
// Recovery — feed 30 fps for 2 seconds; floor must clear.
let start = 2_000_000_000u64;
let step = (1e9_f64 / 30.0) as u64;
for i in 0..60u64 {
h.observe_frame_at(start + i * step).await;
}
assert!(!h.fps_floor_active().await);
// The same PoiSelected MUST now succeed.
let o = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: 3_000_000_000,
})
.await;
assert!(o.accepted);
assert!(matches!(h.state().await, ScanState::ZoomedIn { .. }));
}
/// AC-5 — tick latency stays well under the 10 ms p99 budget under
/// a steady-state trigger load. This bench is a smoke test, not a
/// rigorous benchmark — it exists to catch a regression that
/// silently blows past the budget.
#[tokio::test]
async fn ac5_tick_latency_p99_under_budget() {
// Arrange
let h = ScanController::new().handle();
// Act — run 200 triggers through the FSM.
for i in 0..200u64 {
let _ = h
.submit_trigger(Trigger::PoiSelected {
roi: Uuid::new_v4(),
now_ns: i * 1_000_000,
})
.await;
let _ = h.submit_trigger(Trigger::OperatorAbort).await;
}
// Assert
let p99 = h.tick_latency_p99_us().await;
assert!(
p99 < 10_000,
"tick latency p99 {p99} us exceeds 10 ms budget"
);
}