# LiveKit Stream Detection

**Task**: AZ-150_livekit_stream_detection
**Name**: LiveKit Stream Detection Integration
**Description**: Enable real-time object detection on 5-10 simultaneous LiveKit WebRTC streams. Two-app architecture: a Playwright companion app for authentication and stream discovery, plus LiveKit SDK integration in the detection service for frame capture and inference.
**Complexity**: 5 points
**Dependencies**: None (extends existing detection service)
**Component**: Feature
**Jira**: AZ-150
**Epic**: AZ-149

## Problem

The platform streams live video via LiveKit WebRTC. The detection service currently only processes pre-recorded video files and static images via `cv2.VideoCapture`. There is no way to run real-time object detection on live streams. The user needs to detect objects on 5-10 out of 50+ simultaneous streams shown on the platform's web page.

Key constraints:
- No LiveKit API key/secret available (only browser-level access)
- LiveKit WebRTC streams cannot be consumed by `cv2.VideoCapture`
- Tokens are issued by the platform's backend and expire periodically
- Must handle 5-10 concurrent streams without overwhelming the GPU inference engine

## Outcome

- User can open the platform's stream page in a Playwright-controlled browser, log in, and see all available streams
- The system automatically discovers stream IDs, LiveKit room names, tokens, and server URL from network traffic
- User selects which streams to run detection on via an injected UI overlay
- Detection runs continuously on selected streams with results flowing through the existing SSE endpoint
- Tokens are automatically refreshed as the page renews them

## Architecture

### Two-App Design

```
App 1: stream_discover.py (Playwright companion)
  - Launches real Chromium browser (separate process)
  - Python controls it via Chrome DevTools Protocol (CDP) over local WebSocket
  - User interacts with browser normally (login, navigation)
  - Python silently intercepts network traffic and reads the DOM
  - Injects a floating selection UI onto the page
  - Sends selected stream configs to the detection service API

App 2: Detection Service (existing FastAPI in main.py)
  - New /detect/livekit/* endpoints receive stream configs from companion app
  - livekit_source.py connects to LiveKit rooms via livekit.rtc SDK
  - livekit_detector.py orchestrates multi-stream frame capture and batched inference
  - inference.pyx gains a new detect_frames() method for raw numpy frame batches
  - Results flow through existing SSE /detect/stream endpoint
```

### How Playwright Works (NOT a Webview)

Playwright is a browser automation library by Microsoft. It does NOT embed a browser inside a Python window. Instead:

1. `playwright.chromium.launch(headless=False)` starts a **real standalone Chromium process** -- identical to opening Chrome
2. Python communicates with this browser via CDP (Chrome DevTools Protocol) over a local WebSocket
3. The user sees a normal browser window and interacts with it normally (login, clicking, scrolling)
4. Python silently observes all network traffic, reads the DOM, and can inject HTML/JavaScript
5. There is no Python GUI -- the browser window IS the entire interface

```
Python Process                    Chromium Process (separate)
+--------------------------+      +---------------------------+
| stream_discover.py       |      | Normal browser window     |
|                          |      |                           |
| - Playwright library     | CDP  | - User logs in normally   |
| - Token interceptor      |<====>| - DevTools Protocol       |
| - DOM parser             | WS   | - Full web app rendering  |
| - Selection UI injector  |      | - LiveKit video playback  |
+--------------------------+      +---------------------------+
```

Advantages over a webview:
- No GUI code to write -- browser IS the UI
- User sees the exact same web app they normally use
- Full access to network requests, cookies, localStorage
- Playwright handles CDP complexity

### Data Flow

```
1. User logs in via browser
2. User navigates to streams page
3. Python intercepts HTTP responses containing LiveKit JWT tokens
4. Python parses DOM for data-testid="mission-video-*" elements
5. Python decodes JWTs to extract room names
6. Python injects floating panel with stream checkboxes onto the page
7. User selects streams, clicks "Start Detection"
8. Python POSTs {livekit_url, rooms[{name, token, stream_id}]} to detection service
9. Detection service connects to LiveKit rooms via livekit.rtc
10. Frames are sampled, batched, and run through inference engine
11. DetectionEvents emitted via existing SSE /detect/stream
12. Python companion stays open, intercepts token refreshes, pushes to detection service
```

### Multi-Stream Frame Processing

```
Stream 1 (async task) ─── sample every Nth frame ──┐
Stream 2 (async task) ─── sample every Nth frame ──├─► Shared Frame Queue ─► Detection Worker Thread
Stream N (async task) ─── sample every Nth frame ──┘         │                    │
                                                              │                    ▼
                                                     backpressure:          inference.detect_frames()
                                                     keep only latest           │
                                                     frame per stream           ▼
                                                                          DetectionEvent → SSE
```

- At 30fps input with frame_period_recognition=4: ~7.5 fps per stream
- 10 streams = ~75 frames/sec into the queue
- Engine batch size determines how many frames are processed at once
- Backpressure: each stream keeps only its latest unprocessed frame; stale frames dropped

## Scope

### Included

**Companion App (stream_discover.py)**
- Playwright browser launch and lifecycle management
- Network response interception for LiveKit JWT token capture
- WebSocket URL interception for LiveKit server URL discovery
- DOM parsing for stream ID and display name extraction
- JWT decoding to map stream_id -> room_name
- Injected floating UI panel with stream checkboxes and "Start Detection" button
- HTTP POST to detection service with selected stream configs
- Token refresh monitoring and forwarding

**Detection Service**
- `livekit_source.py`: LiveKit room connection, video track subscription, VideoFrame -> BGR numpy conversion
- `livekit_detector.py`: multi-stream task orchestration, frame sampling, shared queue, batched detection loop, SSE event emission
- `inference.pyx`/`.pxd`: new `detect_frames(frames, config)` cpdef method for raw numpy frame batches
- `main.py`: new endpoints POST /detect/livekit/start, POST /detect/livekit/refresh-tokens, DELETE /detect/livekit/stop, GET /detect/livekit/status
- `requirements.txt`: add `livekit` and `playwright` dependencies

### Excluded
- LiveKit API key/secret based token generation (no access)
- Publishing video back to LiveKit
- Recording or saving stream frames to disk
- Modifying existing /detect or /detect/{media_id} endpoints
- UI beyond the injected browser overlay

## Acceptance Criteria

**AC-1: Stream Discovery**
Given the user opens the platform's stream page in the Playwright browser
When the page loads and streams are rendered
Then the companion app discovers all stream IDs, display names, LiveKit tokens, room names, and server URL from network traffic and DOM

**AC-2: Stream Selection UI**
Given streams are discovered
When the companion app injects the selection panel
Then the user sees a floating panel listing all streams with checkboxes and a "Start Detection" button

**AC-3: Start Detection**
Given the user selects N streams and clicks "Start Detection"
When the companion app sends the config to the detection service
Then the detection service connects to N LiveKit rooms and begins receiving video frames

**AC-4: Real-Time Inference**
Given the detection service is receiving frames from LiveKit streams
When frames are sampled and batched through the inference engine
Then DetectionEvents with annotations are emitted via the existing SSE /detect/stream endpoint

**AC-5: Multi-Stream Handling**
Given 5-10 streams are active simultaneously
When inference runs continuously
Then all streams are processed fairly (round-robin or queue-based) without any stream being starved

**AC-6: Token Refresh**
Given the platform's frontend refreshes LiveKit tokens periodically
When the companion app detects a token renewal in network traffic
Then the new token is forwarded to the detection service and the LiveKit connection continues without interruption

**AC-7: Stop Detection**
Given detection is running on N streams
When the user calls DELETE /detect/livekit/stop
Then all LiveKit connections are cleanly closed and detection tasks cancelled

## File Changes

| File | Action | Description |
|------|--------|-------------|
| `stream_discover.py` | New | Playwright companion app |
| `livekit_source.py` | New | LiveKit room connection and frame capture |
| `livekit_detector.py` | New | Multi-stream detection orchestration |
| `inference.pyx` | Modified | Add `detect_frames` cpdef method |
| `inference.pxd` | Modified | Declare `detect_frames` method |
| `main.py` | Modified | Add /detect/livekit/* endpoints |
| `requirements.txt` | Modified | Add `livekit`, `playwright` |

## Non-Functional Requirements

**Performance**
- Frame-to-detection latency < 500ms per batch (excluding network latency)
- 10 concurrent streams without OOM or queue overflow

**Reliability**
- Graceful handling of LiveKit disconnections (auto-reconnect or clean stop)
- Token expiry handled without crash

## Risks & Mitigation

**Risk 1: LiveKit Python SDK frame format compatibility**
- *Risk*: VideoFrame format (RGBA/I420/NV12) may vary by codec and platform
- *Mitigation*: Use `frame.convert(VideoBufferType.RGBA)` to normalize, then convert to BGR

**Risk 2: Token expiration before refresh is captured**
- *Risk*: If the companion app misses a token refresh, the LiveKit connection drops
- *Mitigation*: Implement reconnection logic in livekit_source.py; companion app can re-request tokens

**Risk 3: Inference engine bottleneck with 10 streams**
- *Risk*: GPU/CPU inference cannot keep up with frame arrival rate
- *Mitigation*: Backpressure design (drop stale frames); configurable frame_period_recognition to reduce load

**Risk 4: Playwright browser stability**
- *Risk*: Long-running browser session may leak memory or crash
- *Mitigation*: Monitor browser process health; provide manual restart capability

**Risk 5: LiveKit room structure unknown**
- *Risk*: Rooms may be structured differently than expected (multi-track, SFU routing)
- *Mitigation*: Start with single-track subscription per room; adapt after initial testing