# ScanController ## 1. High-Level Overview **Purpose**: Central orchestrator that drives the scan behavior tree — ticks the tree each cycle, which coordinates frame capture, inference dispatch, POI management, gimbal control, health monitoring, and L1/L2 scan transitions. Search behavior is data-driven via configurable **Search Scenarios**. **Architectural Pattern**: Behavior Tree (py_trees) for high-level scan orchestration. Leaf nodes contain simple procedural logic that calls into other components. Search scenarios loaded from YAML config define what to look for and how to investigate. **Upstream dependencies**: Tier1Detector, Tier2SpatialAnalyzer, VLMClient (optional), GimbalDriver, OutputManager, Config helper, Types helper **Downstream consumers**: None — this is the top-level orchestrator. Exposes health API endpoint. ## 2. Search Scenarios (data-driven) A **SearchScenario** defines what triggers a Level 2 investigation and how to investigate it. Multiple scenarios can be active simultaneously. Defined in YAML config: ```yaml search_scenarios: - name: winter_concealment enabled: true trigger: classes: [footpath_winter, branch_pile, dark_entrance] min_confidence: 0.5 investigation: type: path_follow follow_class: footpath_winter target_classes: [concealed_position, branch_pile, dark_entrance, trash] use_vlm: true priority_boost: 1.0 - name: autumn_concealment enabled: true trigger: classes: [footpath_autumn, branch_pile, dark_entrance] min_confidence: 0.5 investigation: type: path_follow follow_class: footpath_autumn target_classes: [concealed_position, branch_pile, dark_entrance] use_vlm: true priority_boost: 1.0 - name: building_area_search enabled: true trigger: classes: [building_block, road_with_traces, house_with_vehicle] min_confidence: 0.6 investigation: type: area_sweep target_classes: [vehicle, military_vehicle, traces, dark_entrance] use_vlm: false priority_boost: 0.8 - name: aa_defense_network enabled: false trigger: classes: [radar_dish, aa_launcher, military_truck] min_confidence: 0.4 min_cluster_size: 2 investigation: type: cluster_follow target_classes: [radar_dish, aa_launcher, military_truck, command_vehicle] cluster_radius_px: 300 use_vlm: true priority_boost: 1.5 ``` ### Investigation Types | Type | Description | Subtree Used | When | |------|-------------|-------------|------| | `path_follow` | Skeletonize footpath → PID follow → endpoint analysis | PathFollowSubtree | Footpath-based scenarios | | `area_sweep` | Slow pan across POI area at high zoom, Tier 1 continuously | AreaSweepSubtree | Building blocks, tree rows, clearings | | `zoom_classify` | Zoom to POI → run Tier 1 at high zoom → report | ZoomClassifySubtree | Single long-range targets | | `cluster_follow` | Cluster nearby detections → visit each in order → classify per point | ClusterFollowSubtree | AA defense networks, radar clusters, vehicle groups | Adding a new investigation type requires a new BT subtree. Adding a new scenario that uses existing investigation types requires only YAML config changes. ## 3. Behavior Tree Structure ``` Root (Selector — try highest priority first) │ ├── [1] HealthGuard (Decorator: checks capability flags) │ └── FallbackBehavior │ ├── If semantic_available=false → run existing YOLO only │ └── If gimbal_available=false → fixed camera, Tier 1 detect only │ ├── [2] L2Investigation (Sequence — runs if POI queue non-empty) │ ├── CheckPOIQueue (Condition: queue non-empty?) │ ├── PickHighestPOI (Action: pop from priority queue) │ ├── ZoomToPOI (Action: gimbal zoom + wait) │ ├── L2DetectLoop (Repeat until timeout) │ │ ├── CaptureFrame (Action) │ │ ├── RunTier1 (Action: YOLOE on zoomed frame) │ │ ├── RecordFrame (Action: L2 rate) │ │ └── InvestigateByScenario (Selector — picks subtree based on POI's scenario) │ │ ├── PathFollowSubtree (Sequence — if scenario.type == path_follow) │ │ │ ├── TraceMask (Action: Tier2.trace_mask → SpatialAnalysisResult) │ │ │ ├── PIDFollow (Action: gimbal PID along trajectory) │ │ │ └── WaypointAnalysis (Selector — for each waypoint) │ │ │ ├── HighConfidence (Condition: heuristic > threshold) │ │ │ │ └── LogDetection (Action: tier=2) │ │ │ └── AmbiguousWithVLM (Sequence — if scenario.use_vlm) │ │ │ ├── CheckVLMAvailable (Condition) │ │ │ ├── RunVLM (Action: VLMClient.analyze) │ │ │ └── LogDetection (Action: tier=3) │ │ ├── ClusterFollowSubtree (Sequence — if scenario.type == cluster_follow) │ │ │ ├── TraceCluster (Action: Tier2.trace_cluster → SpatialAnalysisResult) │ │ │ ├── VisitLoop (Repeat over waypoints) │ │ │ │ ├── MoveToWaypoint (Action: gimbal to next waypoint position) │ │ │ │ ├── CaptureFrame (Action) │ │ │ │ ├── RunTier1 (Action: YOLOE at high zoom) │ │ │ │ ├── ClassifyWaypoint (Selector — heuristic or VLM) │ │ │ │ │ ├── HighConfidence (Condition) │ │ │ │ │ │ └── LogDetection (Action: tier=2) │ │ │ │ │ └── AmbiguousWithVLM (Sequence) │ │ │ │ │ ├── CheckVLMAvailable (Condition) │ │ │ │ │ ├── RunVLM (Action) │ │ │ │ │ └── LogDetection (Action: tier=3) │ │ │ │ └── RecordFrame (Action) │ │ │ └── LogClusterSummary (Action: report cluster as a whole) │ │ ├── AreaSweepSubtree (Sequence — if scenario.type == area_sweep) │ │ │ ├── ComputeSweepPattern (Action: bounding box → pan/tilt waypoints) │ │ │ ├── SweepLoop (Repeat over waypoints) │ │ │ │ ├── SendGimbalCommand (Action) │ │ │ │ ├── CaptureFrame (Action) │ │ │ │ ├── RunTier1 (Action) │ │ │ │ └── CheckTargets (Action: match against scenario.target_classes) │ │ │ └── LogDetections (Action: all found targets) │ │ └── ZoomClassifySubtree (Sequence — if scenario.type == zoom_classify) │ │ ├── HoldZoom (Action: maintain zoom on POI) │ │ ├── CaptureMultipleFrames (Action: 3-5 frames for confidence) │ │ ├── RunTier1 (Action: on each frame) │ │ ├── AggregateResults (Action: majority vote on target_classes) │ │ └── LogDetection (Action) │ ├── ReportToOperator (Action) │ └── ReturnToSweep (Action: gimbal zoom out) │ ├── [3] L1Sweep (Sequence — default behavior) │ ├── HealthCheck (Action: read tegrastats, update capability flags) │ ├── AdvanceSweep (Action: compute next pan angle) │ ├── SendGimbalCommand (Action: set sweep target) │ ├── CaptureFrame (Action) │ ├── QualityGate (Condition: Laplacian variance > threshold) │ ├── RunTier1 (Action: YOLOE inference) │ ├── EvaluatePOI (Action: match detections against ALL active scenarios' trigger_classes) │ ├── RecordFrame (Action: L1 rate) │ └── LogDetections (Action) │ └── [4] Idle (AlwaysSucceeds — fallback) ``` ### EvaluatePOI Logic (scenario-aware) ``` for each detection in detections: for each scenario in active_scenarios: if detection.label in scenario.trigger.classes AND detection.confidence >= scenario.trigger.min_confidence: if scenario.investigation.type == "cluster_follow": # Aggregate nearby detections into a single cluster POI matching = [d for d in detections if d.label in scenario.trigger.classes and d.confidence >= scenario.trigger.min_confidence] clusters = spatial_cluster(matching, scenario.cluster_radius_px) for cluster in clusters: if len(cluster) >= scenario.min_cluster_size: create POI with: trigger_class = cluster[0].label scenario_name = scenario.name investigation_type = "cluster_follow" cluster_detections = cluster priority = mean(d.confidence for d in cluster) * scenario.priority_boost add to POI queue (deduplicate by cluster overlap) break # scenario fully evaluated, don't create individual POIs else: create POI with: trigger_class = detection.label scenario_name = scenario.name investigation_type = scenario.investigation.type priority = detection.confidence * scenario.priority_boost * recency_factor add to POI queue (deduplicate by bbox overlap) ``` ### Blackboard Variables (py_trees shared state) | Variable | Type | Written by | Read by | |----------|------|-----------|---------| | `frame` | FrameContext | CaptureFrame | RunTier1, RecordFrame, QualityGate | | `detections` | list[Detection] | RunTier1 | EvaluatePOI, InvestigateByScenario, LogDetections | | `poi_queue` | list[POI] | EvaluatePOI | CheckPOIQueue, PickHighestPOI | | `current_poi` | POI | PickHighestPOI | ZoomToPOI, InvestigateByScenario | | `active_scenarios` | list[SearchScenario] | Config load | EvaluatePOI, InvestigateByScenario | | `spatial_result` | SpatialAnalysisResult | TraceMask, TraceCluster | PIDFollow, WaypointAnalysis, VisitLoop | | `capability_flags` | CapabilityFlags | HealthCheck | HealthGuard, CheckVLMAvailable | | `scan_angle` | float | AdvanceSweep | SendGimbalCommand | ## 4. External API Specification | Endpoint | Method | Auth | Rate Limit | Description | |----------|--------|------|------------|-------------| | `/api/v1/health` | GET | None | — | Returns health status + metrics | | `/api/v1/detect` | POST | None | Frame rate | Submit single frame for processing (dev/test mode) | **Health response**: ```json { "status": "ok", "tier1_ready": true, "gimbal_alive": true, "vlm_alive": false, "t_junction_c": 68.5, "capabilities": {"vlm_available": true, "gimbal_available": true, "semantic_available": true}, "active_behavior": "L2Investigation.PathFollowSubtree.PIDFollow", "active_scenarios": ["winter_concealment", "building_area_search"], "frames_processed": 12345, "detections_total": 89, "poi_queue_depth": 2 } ``` ## 5. Data Access Patterns No database. State lives on the py_trees Blackboard: - POI queue: priority list on blackboard, max size from config (default 10) - Capability flags: 3 booleans on blackboard - Active scenarios: loaded from config at startup, stored on blackboard - Current frame: single FrameContext on blackboard (overwritten each tick) - Scan angle: float on blackboard (incremented each L1 tick) ## 6. Implementation Details **Main Loop**: ```python scenarios = config.get("search_scenarios") tree = create_scan_tree(config, components, scenarios) while running: tree.tick() ``` Each `tick()` traverses the tree from root. The Selector tries HealthGuard first (preempts if degraded), then L2 (if POI queued), then L1 (default). Leaf nodes call into Tier1Detector, Tier2SpatialAnalyzer, VLMClient, GimbalDriver, OutputManager. **InvestigateByScenario** dispatching: reads `current_poi.investigation_type` from blackboard, routes to the matching subtree (PathFollow / ClusterFollow / AreaSweep / ZoomClassify). Each subtree reads `current_poi.scenario` for target classes and VLM usage. **Leaf Node Pattern**: Each leaf node is a simple py_trees.behaviour.Behaviour subclass. `setup()` gets component references. `update()` calls the component method and returns SUCCESS/FAILURE/RUNNING. **Key Dependencies**: | Library | Version | Purpose | |---------|---------|---------| | py_trees | 2.4.0 | Behavior tree framework | | OpenCV | 4.x | Frame capture, Laplacian variance | | FastAPI | existing | Health + detect endpoints | **Error Handling Strategy**: - Leaf node exceptions → catch, log, return FAILURE → tree falls through to next Selector child - Component unavailable → Condition nodes gate access (CheckVLMAvailable, QualityGate) - Health degradation → HealthGuard decorator at root preempts all other behaviors - Invalid scenario config → log error at startup, skip invalid scenario, continue with valid ones **Frame Quality Gate**: QualityGate is a Condition node in L1Sweep. If it returns FAILURE (blurry frame), the Sequence aborts and the tree ticks L1Sweep again next cycle (new frame). ## 7. Extensions and Helpers | Helper | Purpose | Used By | |--------|---------|---------| | config | YAML config loading + validation | All components | | types | Shared structs (FrameContext, POI, SearchScenario, etc.) | All components | ### Adding a New Search Scenario (config only) 1. Define new detection classes in YOLOE (retrain if needed) 2. Add YAML block under `search_scenarios` with trigger classes, investigation type, targets 3. Restart service — new scenario is active ### Adding a New Investigation Type (code + config) 1. Create new BT subtree (e.g., `SpiralSearchSubtree`) 2. Register it in InvestigateByScenario dispatcher 3. Use `type: spiral_search` in scenario config ## 8. Caveats & Edge Cases **Known limitations**: - Single-threaded tree ticking — throughput capped by slowest leaf per tick - py_trees Blackboard is not thread-safe (fine — single-threaded design) - POI queue doesn't persist across restarts - Scenario changes require service restart (no hot-reload) **Performance bottlenecks**: - Tier 1 inference leaf is the bottleneck (~7-100ms) - Tree traversal overhead is negligible (<1ms) ## 9. Dependency Graph **Must be implemented after**: Config helper, Types helper, ALL other components (02-06) **Can be implemented in parallel with**: None (top-level orchestrator) **Blocks**: Nothing (implemented last) ## 10. Logging Strategy | Log Level | When | Example | |-----------|------|---------| | ERROR | Component crash, 3x retry exhausted, invalid scenario | `Gimbal UART failed 3 times, disabling gimbal` | | WARN | Frame skipped, VLM timeout, leaf FAILURE | `QualityGate FAILURE: Laplacian 12.3 < 50.0` | | INFO | State transitions, POI created, scenario match | `POI queued: winter_concealment triggered by footpath_winter (conf=0.72)` | **py_trees built-in logging**: Tree can render active path as ASCII for debugging: ``` [o] Root [-] HealthGuard [o] L2Investigation [o] L2DetectLoop [o] InvestigateByScenario [o] PathFollowSubtree [*] PIDFollow (RUNNING) ```