add chunking

This commit is contained in:
Oleksandr Bezdieniezhnykh
2025-11-27 03:43:19 +02:00
parent 4f8c18a066
commit 2037870f67
43 changed files with 7041 additions and 4135 deletions
@@ -0,0 +1,649 @@
# Flight API
## Interface Definition
**Interface Name**: `IFlightAPI`
### Interface Methods
```python
class IFlightAPI(ABC):
@abstractmethod
def create_flight(self, flight_data: FlightCreateRequest) -> FlightResponse:
pass
@abstractmethod
def get_flight(self, flight_id: str) -> FlightDetailResponse:
pass
@abstractmethod
def delete_flight(self, flight_id: str) -> DeleteResponse:
pass
@abstractmethod
def update_waypoint(self, flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse:
pass
@abstractmethod
def batch_update_waypoints(self, flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse:
pass
@abstractmethod
def upload_image_batch(self, flight_id: str, batch: ImageBatch) -> BatchResponse:
pass
@abstractmethod
def submit_user_fix(self, flight_id: str, fix_data: UserFixRequest) -> UserFixResponse:
pass
@abstractmethod
def get_flight_status(self, flight_id: str) -> FlightStatusResponse:
pass
@abstractmethod
def create_sse_stream(self, flight_id: str) -> SSEStream:
pass
```
## Component Description
### Responsibilities
- Expose REST API endpoints for complete flight lifecycle management
- Handle flight CRUD operations (create, read, update, delete)
- Manage waypoints and geofences within flights
- Handle satellite data prefetching on flight creation
- Accept batch image uploads (10-50 images per request)
- Accept user-provided GPS fixes for blocked flights
- Provide real-time status updates
- Stream results via Server-Sent Events (SSE)
### Scope
- FastAPI-based REST endpoints
- Request/response validation
- Coordinate with Flight Processor for all operations
- Multipart form data handling for image uploads
- SSE connection management
- Authentication and rate limiting
---
## Flight Management Endpoints
### `create_flight(flight_data: FlightCreateRequest) -> FlightResponse`
**REST Endpoint**: `POST /flights`
**Description**: Creates a new flight with initial waypoints, geofences, camera parameters, and triggers satellite data prefetching.
**Called By**:
- Client applications (Flight UI, Mission Planner UI)
**Input**:
```python
FlightCreateRequest:
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
**Output**:
```python
FlightResponse:
flight_id: str
status: str # "prefetching", "ready", "error"
message: Optional[str]
created_at: datetime
```
**Processing Flow**:
1. Validate request data
2. Call F02 Flight Processor → create_flight()
3. Flight Processor triggers satellite prefetch
4. Return flight_id immediately (prefetch is async)
**Error Conditions**:
- `400 Bad Request`: Invalid input data (missing required fields, invalid GPS coordinates)
- `409 Conflict`: Flight with same ID already exists
- `500 Internal Server Error`: Database or internal error
**Test Cases**:
1. **Valid flight creation**: Provide valid flight data → returns 201 with flight_id
2. **Missing required field**: Omit name → returns 400 with error message
3. **Invalid GPS coordinates**: Provide lat > 90 → returns 400
4. **Concurrent flight creation**: Multiple flights → all succeed
---
### `get_flight(flight_id: str) -> FlightDetailResponse`
**REST Endpoint**: `GET /flights/{flightId}`
**Description**: Retrieves complete flight information including all waypoints, geofences, and processing status.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightDetailResponse:
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: Flight ID does not exist
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Existing flight**: Valid flightId → returns 200 with complete flight data
2. **Non-existent flight**: Invalid flightId → returns 404
3. **Flight with many waypoints**: Flight with 2000+ waypoints → returns 200 with all data
---
### `delete_flight(flight_id: str) -> DeleteResponse`
**REST Endpoint**: `DELETE /flights/{flightId}`
**Description**: Deletes a flight and all associated waypoints, images, and processing data.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
DeleteResponse:
deleted: bool
flight_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight does not exist
- `409 Conflict`: Flight is currently being processed
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Delete existing flight**: Valid flightId → returns 200
2. **Delete non-existent flight**: Invalid flightId → returns 404
3. **Delete processing flight**: Active processing → returns 409
---
### `update_waypoint(flight_id: str, waypoint_id: str, waypoint: Waypoint) -> UpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/{waypointId}`
**Description**: Updates a specific waypoint within a flight. Used for per-frame GPS refinement.
**Called By**:
- Internal (F13 Result Manager for per-frame updates)
- Client applications (manual corrections)
**Input**:
```python
flight_id: str
waypoint_id: str
waypoint: Waypoint:
lat: float
lon: float
altitude: Optional[float]
confidence: float
timestamp: datetime
refined: bool
```
**Output**:
```python
UpdateResponse:
updated: bool
waypoint_id: str
```
**Error Conditions**:
- `404 Not Found`: Flight or waypoint not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Update existing waypoint**: Valid data → returns 200
2. **Refinement update**: Refined coordinates → updates successfully
3. **Invalid coordinates**: lat > 90 → returns 400
4. **Non-existent waypoint**: Invalid waypoint_id → returns 404
---
### `batch_update_waypoints(flight_id: str, waypoints: List[Waypoint]) -> BatchUpdateResponse`
**REST Endpoint**: `PUT /flights/{flightId}/waypoints/batch`
**Description**: Updates multiple waypoints in a single request. Used for trajectory refinements.
**Called By**:
- Internal (F13 Result Manager for asynchronous refinement updates)
**Input**:
```python
flight_id: str
waypoints: List[Waypoint]
```
**Output**:
```python
BatchUpdateResponse:
success: bool
updated_count: int
failed_ids: List[str]
```
**Error Conditions**:
- `404 Not Found`: Flight not found
- `400 Bad Request`: Invalid waypoint data
- `500 Internal Server Error`: Database error
**Test Cases**:
1. **Batch update 100 waypoints**: All succeed
2. **Partial failure**: 5 waypoints fail → returns failed_ids
3. **Empty batch**: Returns success=True, updated_count=0
4. **Large batch**: 500 waypoints → succeeds
---
## Image Processing Endpoints
### `upload_image_batch(flight_id: str, batch: ImageBatch) -> BatchResponse`
**REST Endpoint**: `POST /flights/{flightId}/images/batch`
**Description**: Uploads a batch of 10-50 UAV images for processing.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
ImageBatch: multipart/form-data
images: List[UploadFile]
metadata: BatchMetadata
start_sequence: int
end_sequence: int
```
**Output**:
```python
BatchResponse:
accepted: bool
sequences: List[int]
next_expected: int
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists
2. Validate batch size (10-50 images)
3. Validate sequence numbers (strict sequential)
4. Pass to F05 Image Input Pipeline
5. Return immediately (processing is async)
**Error Conditions**:
- `400 Bad Request`: Invalid batch size, out-of-sequence images
- `404 Not Found`: flight_id doesn't exist
- `413 Payload Too Large`: Batch exceeds size limit
- `429 Too Many Requests`: Rate limit exceeded
**Test Cases**:
1. **Valid batch upload**: 20 images → returns 202 Accepted
2. **Out-of-sequence batch**: Sequence gap detected → returns 400
3. **Too many images**: 60 images → returns 400
4. **Large images**: 50 × 8MB images → successfully uploads
---
### `submit_user_fix(flight_id: str, fix_data: UserFixRequest) -> UserFixResponse`
**REST Endpoint**: `POST /flights/{flightId}/user-fix`
**Description**: Submits user-provided GPS anchor point to unblock failed localization.
**Called By**:
- Client applications (when user responds to `user_input_needed` event)
**Input**:
```python
UserFixRequest:
frame_id: int
uav_pixel: Tuple[float, float]
satellite_gps: GPSPoint
```
**Output**:
```python
UserFixResponse:
accepted: bool
processing_resumed: bool
message: Optional[str]
```
**Processing Flow**:
1. Validate flight_id exists and is blocked
2. Pass to F11 Failure Recovery Coordinator
3. Coordinator applies anchor to Factor Graph
4. Resume processing pipeline
**Error Conditions**:
- `400 Bad Request`: Invalid fix data
- `404 Not Found`: flight_id or frame_id not found
- `409 Conflict`: Flight not in blocked state
**Test Cases**:
1. **Valid user fix**: Blocked flight → returns 200, processing resumes
2. **Fix for non-blocked flight**: Returns 409
3. **Invalid GPS coordinates**: Returns 400
---
### `get_flight_status(flight_id: str) -> FlightStatusResponse`
**REST Endpoint**: `GET /flights/{flightId}/status`
**Description**: Retrieves current processing status of a flight.
**Called By**:
- Client applications (polling for status)
**Input**:
```python
flight_id: str
```
**Output**:
```python
FlightStatusResponse:
status: str # "prefetching", "ready", "processing", "blocked", "completed", "failed"
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
**Test Cases**:
1. **Processing flight**: Returns current progress
2. **Blocked flight**: Returns blocked=true with search_grid_size
3. **Completed flight**: Returns status="completed" with final counts
---
### `create_sse_stream(flight_id: str) -> SSEStream`
**REST Endpoint**: `GET /flights/{flightId}/stream`
**Description**: Opens Server-Sent Events connection for real-time result streaming.
**Called By**:
- Client applications
**Input**:
```python
flight_id: str
```
**Output**:
```python
SSE Stream with events:
- frame_processed
- frame_refined
- search_expanded
- user_input_needed
- processing_blocked
- flight_completed
```
**Event Format**:
```json
{
"event": "frame_processed",
"data": {
"frame_id": 237,
"gps": {"lat": 48.123, "lon": 37.456},
"altitude": 800.0,
"confidence": 0.95,
"heading": 87.3,
"timestamp": "2025-11-24T10:30:00Z"
}
}
```
**Error Conditions**:
- `404 Not Found`: flight_id doesn't exist
- Connection closed on client disconnect
**Test Cases**:
1. **Connect to stream**: Opens SSE connection successfully
2. **Receive frame events**: Process 100 frames → receive 100 events
3. **Receive user_input_needed**: Blocked frame → event sent
4. **Client reconnect**: Replay missed events from last_event_id
---
## Integration Tests
### Test 1: Complete Flight Lifecycle
1. POST /flights with valid data
2. GET /flights/{flightId} → verify data
3. GET /flights/{flightId}/stream (open SSE)
4. POST /flights/{flightId}/images/batch × 40
5. Receive frame_processed events via SSE
6. Receive flight_completed event
7. GET /flights/{flightId} → verify waypoints updated
8. DELETE /flights/{flightId}
### Test 2: User Fix Flow
1. Create flight and process images
2. Receive user_input_needed event
3. POST /flights/{flightId}/user-fix
4. Receive processing_resumed event
5. Continue receiving frame_processed events
### Test 3: Concurrent Flights
1. Create 10 flights concurrently
2. Upload batches to all flights in parallel
3. Stream results from all flights simultaneously
4. Verify no cross-contamination
### Test 4: Waypoint Updates
1. Create flight
2. Simulate per-frame updates via PUT /flights/{flightId}/waypoints/{waypointId} × 100
3. GET flight and verify all waypoints updated
4. Verify refined=true flag set
---
## Non-Functional Requirements
### Performance
- **create_flight**: < 500ms response (prefetch is async)
- **get_flight**: < 200ms for flights with < 2000 waypoints
- **update_waypoint**: < 100ms (critical for real-time updates)
- **upload_image_batch**: < 2 seconds for 50 × 2MB images
- **submit_user_fix**: < 200ms response
- **get_flight_status**: < 100ms
- **SSE latency**: < 500ms from event generation to client receipt
### Scalability
- Support 100 concurrent flight processing sessions
- Handle 1000+ concurrent SSE connections
- Handle flights with up to 3000 waypoints
- Support 10,000 requests per minute
### Reliability
- Request timeout: 30 seconds for batch uploads
- SSE keepalive: Ping every 30 seconds
- Automatic SSE reconnection with event replay
- Graceful handling of client disconnects
### Security
- API key authentication
- Rate limiting: 100 requests/minute per client
- Max upload size: 500MB per batch
- CORS configuration for web clients
- Input validation on all endpoints
- SQL injection prevention
---
## Dependencies
### Internal Components
- **F02 Flight Processor**: For all flight operations and processing orchestration
- **F05 Image Input Pipeline**: For batch processing
- **F11 Failure Recovery Coordinator**: For user fixes
- **F15 SSE Event Streamer**: For real-time streaming
- **F03 Flight Database**: For persistence (via F02)
### External Dependencies
- **FastAPI**: Web framework
- **Uvicorn**: ASGI server
- **Pydantic**: Validation
- **python-multipart**: Multipart form handling
---
## Data Models
### GPSPoint
```python
class GPSPoint(BaseModel):
lat: float # Latitude -90 to 90
lon: float # Longitude -180 to 180
```
### CameraParameters
```python
class CameraParameters(BaseModel):
focal_length: float # mm
sensor_width: float # mm
sensor_height: float # mm
resolution_width: int # pixels
resolution_height: int # pixels
distortion_coefficients: Optional[List[float]] = None
```
### Polygon
```python
class Polygon(BaseModel):
north_west: GPSPoint
south_east: GPSPoint
```
### Geofences
```python
class Geofences(BaseModel):
polygons: List[Polygon]
```
### FlightCreateRequest
```python
class FlightCreateRequest(BaseModel):
name: str
description: str
start_gps: GPSPoint
rough_waypoints: List[GPSPoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
```
### Waypoint
```python
class Waypoint(BaseModel):
id: str
lat: float
lon: float
altitude: Optional[float] = None
confidence: float
timestamp: datetime
refined: bool = False
```
### FlightDetailResponse
```python
class FlightDetailResponse(BaseModel):
flight_id: str
name: str
description: str
start_gps: GPSPoint
waypoints: List[Waypoint]
geofences: Geofences
camera_params: CameraParameters
altitude: float
status: str
frames_processed: int
frames_total: int
created_at: datetime
updated_at: datetime
```
### FlightStatusResponse
```python
class FlightStatusResponse(BaseModel):
status: str
frames_processed: int
frames_total: int
current_frame: Optional[int]
current_heading: Optional[float]
blocked: bool
search_grid_size: Optional[int]
message: Optional[str]
created_at: datetime
updated_at: datetime
```
### BatchMetadata
```python
class BatchMetadata(BaseModel):
start_sequence: int
end_sequence: int
batch_number: int
```
### BatchUpdateResponse
```python
class BatchUpdateResponse(BaseModel):
success: bool
updated_count: int
failed_ids: List[str]
errors: Optional[Dict[str, str]]
```