detections-semantic/_docs/02_plans/system-flows.md

# Semantic Detection System — System Flows

## Flow Inventory

| # | Flow Name | Trigger | Primary Components | Criticality |
|---|-----------|---------|-------------------|-------------|
| F1 | Level 1 Wide-Area Scan | System startup / return from L2 | ScanController, GimbalDriver, Tier1Detector | High |
| F2 | Level 2 Detailed Investigation | POI queued for investigation | ScanController, GimbalDriver, Tier1Detector, Tier2SpatialAnalyzer, VLMProcess | High |
| F3 | Path / Cluster Following | Spatial pattern detected at L2 zoom | Tier2SpatialAnalyzer, GimbalDriver, ScanController | High |
| F4 | Health & Degradation | Continuous monitoring | HealthChecks (inline), ScanController | High |
| F5 | System Startup | Power on | All components | Medium |

## Flow Dependencies

| Flow | Depends On | Shares Data With |
|------|-----------|-----------------|
| F1 | F5 (startup complete) | F2 (POI queue) |
| F2 | F1 (POI available) | F3 (spatial analysis result) |
| F3 | F2 (spatial pattern detected at L2) | F2 (gimbal position, waypoint detections) |
| F4 | — (inline in main loop) | F1, F2 (capability flags) |
| F5 | — | All flows |

---

## Flow F1: Level 1 Wide-Area Scan

### Description

The scan controller drives the gimbal in a left-right sweep perpendicular to the UAV flight path at medium zoom. Each frame is processed by Tier 1 (YOLOE). When a POI-class detection exceeds the confidence threshold, it is queued for Level 2 investigation. Frames are recorded at configurable rate. Detections are logged and reported to operator.

### Preconditions

- System startup complete (F5)
- Gimbal responding
- YOLOE TRT engine loaded

### Sequence Diagram

```mermaid
sequenceDiagram
    participant SC as ScanController
    participant GD as GimbalDriver
    participant T1 as Tier1Detector
    participant Log as Logger/Recorder

    loop Every sweep position
        SC->>SC: health_check() — read T_junction, check gimbal, check VLM
        SC->>GD: set_sweep_target(pan_angle)
        GD->>GD: send ViewLink command
        Note over SC: capture frame from camera
        SC->>T1: process_frame(frame)
        T1-->>SC: detections[] (classes, masks, confidences)
        SC->>Log: record_frame(frame, level=1) + log_detections(detections)

        alt POI-class detected above threshold
            SC->>SC: queue_poi(detection, priority)
            alt High-priority POI ready
                Note over SC: Transition to F2
            end
        end

        SC->>SC: advance sweep angle
    end
```

### POI Queueing (inline in F1)

When Tier 1 detects any class, EvaluatePOI checks it against ALL active search scenarios:

1. For each detection, match against each active scenario's trigger_classes and min_confidence
2. Check if duplicate of existing queued POI (bbox overlap > 0.5) → update confidence
3. Otherwise create new POI entry with scenario_name and investigation_type, compute priority (confidence × scenario.priority_boost × recency)
4. Insert into priority queue (max size configurable, default 10)
5. If queue full, drop lowest-priority entry
6. Transition to L2 when: current sweep position allows (not mid-transition) AND queue is non-empty

### Data Flow

| Step | From | To | Data | Format |
|------|------|----|------|--------|
| 1 | ScanController | GimbalDriver | target pan/tilt/zoom | GimbalCommand |
| 2 | Camera | ScanController | raw frame | 1920x1080 |
| 3 | ScanController | Tier1Detector | frame buffer | numpy array (HWC) |
| 4 | Tier1Detector | ScanController | detection array | list of dicts |
| 5 | ScanController | Logger/Recorder | frame + detections | JPEG + JSON-lines |

### Error Scenarios

| Error | Where | Detection | Recovery |
|-------|-------|-----------|----------|
| Gimbal timeout | GimbalDriver | No response within 2s | Retry 3x, then set gimbal_available=false, continue with fixed camera |
| YOLOE inference failure | Tier1Detector | Exception / timeout | Skip frame, log error, continue |
| Frame quality too low | ScanController | Laplacian variance < threshold | Skip frame, continue to next |

### Performance Expectations

| Metric | Target | Notes |
|--------|--------|-------|
| Sweep cycle time | 100-200ms per position | Tier 1 inference + gimbal command |
| Full sweep coverage | ≤10s per left-right cycle | Depends on sweep angle range and step size |

---

## Flow F2: Level 2 Detailed Investigation

### Description

Camera zooms into the highest-priority POI. The investigation type is determined by the POI's search scenario (path_follow, cluster_follow, area_sweep, or zoom_classify). For path_follow: F3 activates with mask tracing. For cluster_follow: F3 activates with cluster tracing (visits each member in order). For area_sweep: slow pan at high zoom. For zoom_classify: hold zoom and classify. Tier 2/3 analysis as needed. After analysis or timeout, returns to Level 1.

### Preconditions

- POI queue has at least 1 entry
- gimbal_available == true
- Tier 1 engine loaded

### Sequence Diagram

```mermaid
sequenceDiagram
    participant SC as ScanController
    participant GD as GimbalDriver
    participant T1 as Tier1Detector
    participant T2 as Tier2SpatialAnalyzer
    participant VLM as VLMProcess
    participant Log as Logger/Recorder

    SC->>GD: zoom_to_poi(poi.coords, zoom=high)
    Note over GD: 1-2s zoom transition
    GD-->>SC: zoom_complete

    loop Until timeout or analysis complete
        SC->>SC: health_check()
        Note over SC: capture zoomed frame
        SC->>T1: process_frame(zoomed_frame)
        SC->>Log: record_frame(frame, level=2)
        T1-->>SC: detections[]

        alt Footpath detected
            SC->>T2: trace_path(footpath_mask)
            T2->>T2: skeletonize + find endpoints
            T2-->>SC: endpoints[], skeleton

            SC->>SC: evaluate endpoints (V1 heuristic: darkness + contrast)
            alt Endpoint is dark mass → HIGH confidence
                SC->>Log: log_detection(tier=2, class=concealed_position)
            else Ambiguous endpoint AND vlm_available
                SC->>VLM: analyze_roi(endpoint_crop, prompt)
                VLM-->>SC: vlm_response
                SC->>Log: log_detection(tier=3, vlm_result)
            else Ambiguous endpoint AND NOT vlm_available
                SC->>Log: log_detection(tier=2, class=uncertain)
            end

            SC->>GD: follow_path(skeleton.direction)
            Note over SC: Activates F3
        else No footpath, other POI type
            SC->>T2: analyze_roi(poi_region)
            T2-->>SC: classification
            SC->>Log: log_detection(tier=2)
        end
    end

    SC->>Log: report_detections_to_operator()
    SC->>GD: return_to_sweep(zoom=medium)
    Note over SC: Back to F1
```

### Error Scenarios

| Error | Where | Detection | Recovery |
|-------|-------|-----------|----------|
| Zoom transition timeout | GimbalDriver | No confirmation within 3s | Proceed with current zoom |
| VLM timeout | VLMProcess | No response within 5s | Skip Tier 3, report Tier 2 result only |
| VLM crash | VLMProcess | IPC connection refused | Set vlm_available=false, continue Tier 1+2 |
| Investigation timeout | ScanController | Timer exceeds limit (default 10s) | Return to L1, mark POI as "timeout" |

### Performance Expectations

| Metric | Target | Notes |
|--------|--------|-------|
| L1→L2 transition | ≤2s | Including zoom |
| Per-POI investigation | ≤10s (configurable) | Including VLM if triggered |
| Return to L1 | ≤2s | Zoom-out + first sweep position |

---

## Flow F3: Path / Cluster Following

### Description

Activated from F2 when the investigation type is `path_follow` or `cluster_follow`. The Tier2SpatialAnalyzer produces a `SpatialAnalysisResult` with ordered waypoints and a trajectory. The gimbal follows the trajectory, visiting each waypoint for analysis.

**Mask trace mode** (path_follow): footpath skeleton provides a continuous trajectory. PID control keeps the path centered. At each waypoint (endpoint), camera holds for analysis.

**Cluster trace mode** (cluster_follow): discrete detections provide point-to-point waypoints. Gimbal moves between points in nearest-neighbor order. At each waypoint, camera zooms in for detailed Tier 1 + heuristic/VLM analysis.

### Preconditions

- Level 2 active (F2)
- SpatialAnalysisResult available with >= 1 waypoint

### Flowchart (mask trace)

```mermaid
flowchart TD
    Start([SpatialAnalysisResult available]) --> CheckType{pattern_type?}
    CheckType -->|mask_trace| ComputeDir[Compute path direction from trajectory]
    ComputeDir --> SetTarget[Set gimbal PID target: path direction]
    SetTarget --> PanLoop{Path still visible in frame?}
    PanLoop -->|Yes| UpdatePID[PID update: adjust pan/tilt to center path]
    UpdatePID --> SendCmd[Send gimbal command]
    SendCmd --> CaptureFrame[Capture next frame]
    CaptureFrame --> RunT1[Tier 1 on new frame]
    RunT1 --> UpdateSkeleton[Update skeleton from new mask]
    UpdateSkeleton --> CheckWaypoint{Reached next waypoint?}
    CheckWaypoint -->|No| PanLoop
    CheckWaypoint -->|Yes| HoldCamera[Hold camera on waypoint]
    HoldCamera --> AnalyzeWaypoint([Tier 2/3 waypoint analysis in F2])
    PanLoop -->|No, path lost| FallbackCentroid[Use last known direction]
    FallbackCentroid --> RetryDetect{Re-detect within 3 frames?}
    RetryDetect -->|Yes| UpdateSkeleton
    RetryDetect -->|No| AbortFollow([Return to F2 main loop])
    CheckType -->|cluster_trace| InitVisit[Get first waypoint from visit order]
    InitVisit --> MoveToPoint[Move gimbal to waypoint position]
    MoveToPoint --> CaptureZoomed[Capture zoomed frame]
    CaptureZoomed --> RunT1Zoomed[Tier 1 on zoomed frame]
    RunT1Zoomed --> ClassifyPoint[Heuristic / VLM classify]
    ClassifyPoint --> LogPoint[Log detection for this waypoint]
    LogPoint --> NextPoint{More waypoints?}
    NextPoint -->|Yes| MoveToPoint
    NextPoint -->|No| ClusterDone([Log cluster summary, return to F2])
```

### Error Scenarios

| Error | Where | Detection | Recovery |
|-------|-------|-----------|----------|
| Path lost from frame | Tier1Detector | No footpath mask in 3 consecutive frames | Abort follow, return to F2 |
| Cluster member not visible at zoom | Tier1Detector | No target_class at waypoint position | Log as unconfirmed, proceed to next waypoint |
| PID oscillation | GimbalDriver | Direction changes >5 times in 1s | Hold position, re-acquire |
| Gimbal at physical limit | GimbalDriver | Pan/tilt at max angle | Analyze what's visible, return to F2 |

### Performance Expectations

| Metric | Target | Notes |
|--------|--------|-------|
| PID update rate | 10 Hz | Gimbal command every 100ms (mask trace) |
| Path centering | Path within center 50% of frame | AC requirement (mask trace) |
| Follow duration | ≤5s per path segment | Part of total POI budget |
| Cluster visit | ≤3s per waypoint | Zoom + capture + classify (cluster trace) |

---

## Flow F4: Health & Degradation

### Description

Health checks are performed inline at the top of each main-loop iteration (not a separate thread). The scan controller reads sensor values and sets capability flags that control what features are available.

### Capability Flags

| Flag | Default | Set to false when | Effect |
|------|---------|-------------------|--------|
| vlm_available | true | VLM process crashed 3x, or T_junction > 75°C, or power > 80% budget | Tier 3 skipped; Tier 1+2 continue |
| gimbal_available | true | Gimbal UART failed 3x | Fixed camera; L2 zoom disabled; L1 sweep disabled |
| semantic_available | true | Semantic process crashed 3x, or T_junction > 80°C | Existing YOLO only |

### Inline Health Check (runs each iteration)

```
1. Read T_junction from tegrastats
2. If T_junction > 80°C → semantic_available = false
3. If T_junction > 75°C → vlm_available = false
4. If last gimbal response > 4s ago → gimbal_available = false
5. If VLM IPC failed last 3 attempts → vlm_available = false
6. If all clear and T_junction < 70°C → restore flags to true
```

No separate monitoring thread. No formal state machine. Flags are checked wherever decisions depend on them.

---

## Flow F5: System Startup

### Description

Power-on sequence: load models, initialize gimbal, begin Level 1 scan.

### Sequence Diagram

```mermaid
sequenceDiagram
    participant Boot as JetPack Boot
    participant Main as MainProcess
    participant T1 as Tier1Detector
    participant GD as GimbalDriver
    participant SC as ScanController

    Boot->>Main: OS ready
    Main->>T1: load YOLOE TRT engine
    T1-->>Main: engine ready (~10-20s)
    Main->>GD: initialize gimbal (UART open, handshake)
    GD-->>Main: gimbal ready (or gimbal_available=false)
    Main->>Main: initialize logger, recorder (NVMe check)
    Main->>SC: start Level 1 scan (F1)
```

### Performance Expectations

| Metric | Target | Notes |
|--------|--------|-------|
| Total startup | ≤60s | Power-on to first detection |
| TRT engine load | ≤20s | Model size + NVMe speed |
| Gimbal handshake | ≤2s | UART open + version check |