Oleksandr Bezdieniezhnykh
be4cab4fcb
- Added `/detect/video` endpoint for true streaming video detection, allowing inference to start as upload bytes arrive. - Introduced `run_detect_video_stream` method in the inference module to handle video processing from a file-like object. - Updated media hashing to include a new function for computing hashes directly from files with minimal I/O. - Enhanced documentation to reflect changes in video processing and API behavior. Made-with: Cursor
3.7 KiB
Module: streaming_buffer
Purpose
File-like object backed by a temp file that supports concurrent append (write) and read+seek (read) from separate threads. Designed for true streaming video detection: the HTTP handler appends incoming chunks while the inference thread reads and decodes frames via PyAV — simultaneously, without buffering the entire file in memory.
Public Interface
Class: StreamingBuffer
| Method | Signature | Description |
|---|---|---|
__init__ |
(temp_dir: str | None = None) |
Creates a temp file in temp_dir; opens separate write and read handles |
append |
(data: bytes) -> None |
Writes data to temp file, flushes, notifies waiting readers |
close_writer |
() -> None |
Signals EOF — wakes all blocked readers |
read |
(size: int = -1) -> bytes |
Reads up to size bytes; blocks if data not yet available; returns b"" on EOF |
seek |
(offset: int, whence: int = 0) -> int |
Seeks reader position; SEEK_END blocks until EOF is signaled |
tell |
() -> int |
Returns current reader position |
readable |
() -> bool |
Always returns True |
seekable |
() -> bool |
Always returns True |
writable |
() -> bool |
Always returns False |
close |
() -> None |
Closes both file handles |
Properties
| Property | Type | Description |
|---|---|---|
path |
str |
Absolute path to the backing temp file |
written |
int |
Total bytes appended so far |
Internal Logic
Thread Coordination
Uses threading.Condition to synchronize one writer (HTTP handler) and one reader (PyAV/inference thread):
- append(): acquires lock → writes to file → flushes → increments
_written→notify_all()→ releases lock - read(size): acquires lock → checks if data available → if not and not EOF, calls
wait()(releases lock, sleeps) → woken bynotify_all()→ calculates bytes to read → releases lock → reads from file (outside lock) - seek(0, 2) (SEEK_END): acquires lock → if EOF not signaled, calls
wait()in loop → once EOF, delegates to_reader.seek(offset, 2)
The file read itself happens outside the lock to avoid holding the lock during I/O.
File Handle Separation
Two independent file descriptors on the same temp file:
_writeropened with"wb"— append-only, used by the HTTP handler_readeropened with"rb"— seekable, used by PyAV
On POSIX systems, writes flushed by one fd are immediately visible to reads on another fd of the same inode (shared kernel page cache). os.rename() on the path while the reader fd is open is safe — the fd retains access to the underlying inode.
SEEK_END Behavior
When PyAV tries to seek to the end of the file (e.g. to find MP4 moov atom), seek(0, 2) blocks until close_writer() is called. This provides graceful degradation for non-faststart MP4 files: the decoder waits for the full upload, then processes normally. For faststart MP4/MKV/WebM, SEEK_END is never called and frames are decoded immediately.
Dependencies
- External:
os,tempfile,threading - Internal: none (leaf module)
Consumers
main— createsStreamingBufferinPOST /detect/video, feeds chunks viaappend(), passes buffer to inference
Data Models
None.
Configuration
None.
External Integrations
None.
Security
None. Temp file permissions follow OS defaults (tempfile.mkstemp).
Tests
tests/test_az178_streaming_video.py::TestStreamingBuffer— sequential write/read, blocking read, EOF, concurrent chunked read/write, seek set, seek end blocking, tell, file persistence, written property, seekable/readable flags