mirror of
https://github.com/azaion/detections.git
synced 2026-04-22 22:06:32 +00:00
Oleksandr Bezdieniezhnykh
be4cab4fcb
- Added `/detect/video` endpoint for true streaming video detection, allowing inference to start as upload bytes arrive. - Introduced `run_detect_video_stream` method in the inference module to handle video processing from a file-like object. - Updated media hashing to include a new function for computing hashes directly from files with minimal I/O. - Enhanced documentation to reflect changes in video processing and API behavior. Made-with: Cursor
83 lines
3.7 KiB
Markdown
83 lines
3.7 KiB
Markdown
# Module: streaming_buffer
|
|
|
|
## Purpose
|
|
|
|
File-like object backed by a temp file that supports concurrent append (write) and read+seek (read) from separate threads. Designed for true streaming video detection: the HTTP handler appends incoming chunks while the inference thread reads and decodes frames via PyAV — simultaneously, without buffering the entire file in memory.
|
|
|
|
## Public Interface
|
|
|
|
### Class: StreamingBuffer
|
|
|
|
| Method | Signature | Description |
|
|
|--------|-----------|-------------|
|
|
| `__init__` | `(temp_dir: str \| None = None)` | Creates a temp file in `temp_dir`; opens separate write and read handles |
|
|
| `append` | `(data: bytes) -> None` | Writes data to temp file, flushes, notifies waiting readers |
|
|
| `close_writer` | `() -> None` | Signals EOF — wakes all blocked readers |
|
|
| `read` | `(size: int = -1) -> bytes` | Reads up to `size` bytes; blocks if data not yet available; returns `b""` on EOF |
|
|
| `seek` | `(offset: int, whence: int = 0) -> int` | Seeks reader position; SEEK_END blocks until EOF is signaled |
|
|
| `tell` | `() -> int` | Returns current reader position |
|
|
| `readable` | `() -> bool` | Always returns `True` |
|
|
| `seekable` | `() -> bool` | Always returns `True` |
|
|
| `writable` | `() -> bool` | Always returns `False` |
|
|
| `close` | `() -> None` | Closes both file handles |
|
|
|
|
### Properties
|
|
|
|
| Property | Type | Description |
|
|
|----------|------|-------------|
|
|
| `path` | `str` | Absolute path to the backing temp file |
|
|
| `written` | `int` | Total bytes appended so far |
|
|
|
|
## Internal Logic
|
|
|
|
### Thread Coordination
|
|
|
|
Uses `threading.Condition` to synchronize one writer (HTTP handler) and one reader (PyAV/inference thread):
|
|
|
|
- **append()**: acquires lock → writes to file → flushes → increments `_written` → `notify_all()` → releases lock
|
|
- **read(size)**: acquires lock → checks if data available → if not and not EOF, calls `wait()` (releases lock, sleeps) → woken by `notify_all()` → calculates bytes to read → releases lock → reads from file (outside lock)
|
|
- **seek(0, 2)** (SEEK_END): acquires lock → if EOF not signaled, calls `wait()` in loop → once EOF, delegates to `_reader.seek(offset, 2)`
|
|
|
|
The file read itself happens **outside** the lock to avoid holding the lock during I/O.
|
|
|
|
### File Handle Separation
|
|
|
|
Two independent file descriptors on the same temp file:
|
|
- `_writer` opened with `"wb"` — append-only, used by the HTTP handler
|
|
- `_reader` opened with `"rb"` — seekable, used by PyAV
|
|
|
|
On POSIX systems, writes flushed by one fd are immediately visible to reads on another fd of the same inode (shared kernel page cache). `os.rename()` on the path while the reader fd is open is safe — the fd retains access to the underlying inode.
|
|
|
|
### SEEK_END Behavior
|
|
|
|
When PyAV tries to seek to the end of the file (e.g. to find MP4 moov atom), `seek(0, 2)` blocks until `close_writer()` is called. This provides graceful degradation for non-faststart MP4 files: the decoder waits for the full upload, then processes normally. For faststart MP4/MKV/WebM, SEEK_END is never called and frames are decoded immediately.
|
|
|
|
## Dependencies
|
|
|
|
- **External**: `os`, `tempfile`, `threading`
|
|
- **Internal**: none (leaf module)
|
|
|
|
## Consumers
|
|
|
|
- `main` — creates `StreamingBuffer` in `POST /detect/video`, feeds chunks via `append()`, passes buffer to inference
|
|
|
|
## Data Models
|
|
|
|
None.
|
|
|
|
## Configuration
|
|
|
|
None.
|
|
|
|
## External Integrations
|
|
|
|
None.
|
|
|
|
## Security
|
|
|
|
None. Temp file permissions follow OS defaults (`tempfile.mkstemp`).
|
|
|
|
## Tests
|
|
|
|
- `tests/test_az178_streaming_video.py::TestStreamingBuffer` — sequential write/read, blocking read, EOF, concurrent chunked read/write, seek set, seek end blocking, tell, file persistence, written property, seekable/readable flags
|