# Failure Recovery Coordinator

## Interface Definition

**Interface Name**: `IFailureRecoveryCoordinator`

### Interface Methods

```python
class IFailureRecoveryCoordinator(ABC):
    # Status Checks
    @abstractmethod
    def check_confidence(self, vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment:
        pass
    
    @abstractmethod
    def detect_tracking_loss(self, confidence: ConfidenceAssessment) -> bool:
        pass
    
    # Search & Recovery (Synchronous returns, NO EVENTS)
    @abstractmethod
    def start_search(self, flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession:
        pass
    
    @abstractmethod
    def expand_search_radius(self, session: SearchSession) -> List[TileCoords]:
        pass
    
    @abstractmethod
    def try_current_grid(self, session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]:
        pass
    
    @abstractmethod
    def mark_found(self, session: SearchSession, result: AlignmentResult) -> bool:
        pass
    
    @abstractmethod
    def get_search_status(self, session: SearchSession) -> SearchStatus:
        pass
    
    @abstractmethod
    def create_user_input_request(self, flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest:
        """Returns request object. Caller (F02.2) is responsible for sending to F15."""
        pass
    
    @abstractmethod
    def apply_user_anchor(self, flight_id: str, frame_id: int, anchor: UserAnchor) -> bool:
        pass
    
    # Chunk Recovery
    @abstractmethod
    def create_chunk_on_tracking_loss(self, flight_id: str, frame_id: int) -> ChunkHandle:
        pass
    
    @abstractmethod
    def try_chunk_semantic_matching(self, chunk_id: str) -> Optional[List[TileCandidate]]:
        pass
    
    @abstractmethod
    def try_chunk_litesam_matching(self, chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]:
        pass
    
    @abstractmethod
    def merge_chunk_to_trajectory(self, flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool:
        pass
    
    @abstractmethod
    def process_unanchored_chunks(self, flight_id: str) -> None:
        """Background task logic. Should be invoked/managed by F02.2."""
        pass
```

## Component Description

### Responsibilities
- **Pure Logic Component**: Coordinates recovery strategies but delegates execution and communication to F02.2.
- **No Events**: Returns status objects or booleans. F02.2 decides state transitions based on these returns.
- **Chunk Orchestration**: Coordinates F12 and F10 operations during recovery but does not own the thread/task.
- Monitor confidence metrics (inlier count, MRE, covariance)
- Detect tracking loss and trigger recovery
- Coordinate progressive tile search (1→4→9→16→25)
- Handle human-in-the-loop when all strategies exhausted
- Apply user-provided anchors to Factor Graph
- **Proactive chunk creation on tracking loss** (via F12)
- **Chunk LiteSAM matching with rotation sweeps**
- **Chunk merging orchestration**
- **Background chunk matching processing**

### Interaction Pattern (Direct Calls)
**Caller**: F02.2 Flight Processing Engine

1.  **Tracking Loss**:
    *   F02.2 calls `start_search()`.
    *   F02.2 calls `try_current_grid()` iteratively.
    *   If found, F02.2 calls `mark_found()`.
    *   If exhausted, F02.2 calls `create_user_input_request()` -> gets `Request` object -> F02.2 calls F15.

2.  **Chunk Matching**:
    *   F02.2 calls `create_chunk_on_tracking_loss()`.
    *   F02.2 (background task) calls `try_chunk_semantic_matching()`.
    *   F02.2 calls `try_chunk_litesam_matching()`.
    *   F02.2 calls `merge_chunk_to_trajectory()`.

### External SSE Events (to Clients)

F11 does NOT directly send events to clients. F02.2 orchestrates client communication based on F11's return values:

- When F11.create_user_input_request() returns `UserInputRequest` → F02.2 calls F15.send_user_input_request()
- When F11.merge_chunk_to_trajectory() returns True → F02.2 calls F14.update_results_after_chunk_merge()
- When recovery fails (all F11 methods return None/False) → F02.2 calls F15.send_processing_blocked()

This separation ensures:
1. F11 is a pure business logic component (no I/O dependencies)
2. F02.2 controls all coordination and state management
3. F14/F15 handle all client-facing communication

### Scope
- Confidence monitoring
- Progressive search coordination
- User input request/response handling
- Recovery strategy orchestration
- Integration point for F04, F06, F08, F09, F10
- **Chunk lifecycle and matching coordination**
- **Multi-chunk simultaneous processing**

### Architecture Note: Single Responsibility Consideration

F11 has extensive responsibilities including progressive search, chunk creation coordination, chunk matching coordination (semantic + LiteSAM), chunk merging coordination, and user input handling. While this represents significant scope, these remain together in a single component because:

1. **Tight Coupling**: All recovery strategies share state (SearchSession, chunk status) and need coordinated fallback logic (progressive search → chunk matching → user input).

2. **Sequential Dependency**: Chunk semantic matching feeds into chunk LiteSAM matching, which feeds into chunk merging. Splitting these would create circular dependencies or complex event choreography.

3. **Pure Logic Pattern**: F11 remains a pure logic component (no I/O, no threads). All execution is delegated to F02.2, keeping F11 focused on decision-making rather than coordination.

4. **Single Point of Recovery Policy**: Having all recovery strategies in one component makes the fallback logic explicit and maintainable. The alternative—multiple coordinators—would require a meta-coordinator.

**Future Consideration**: If F11 grows beyond ~1000 lines or if specific recovery strategies need independent evolution, consider splitting into:
- `F11a_search_coordinator` - Progressive search, user input handling
- `F11b_chunk_recovery_coordinator` - Chunk semantic matching, LiteSAM matching, merging

For now, the component remains unified with clear internal organization (search methods, chunk methods).

## API Methods

### `check_confidence(vo_result: RelativePose, litesam_result: Optional[AlignmentResult]) -> ConfidenceAssessment`

**Description**: Assesses tracking confidence from VO and LiteSAM results.

**Called By**: Main processing loop (per frame)

**Input**:
```python
vo_result: RelativePose
litesam_result: Optional[AlignmentResult]
```

**Output**:
```python
ConfidenceAssessment:
    overall_confidence: float  # 0-1
    vo_confidence: float
    litesam_confidence: float
    inlier_count: int
    tracking_status: str  # "good", "degraded", "lost"
```

**Confidence Metrics**:
- VO inlier count and ratio
- LiteSAM match confidence
- Factor graph marginal covariance
- Reprojection error

**Thresholds**:
- **Good**: VO inliers > 50, LiteSAM confidence > 0.7
- **Degraded**: VO inliers 20-50
- **Lost**: VO inliers < 20

**Test Cases**:
1. Good tracking → "good" status
2. Low overlap → "degraded"
3. Sharp turn → "lost"

---

### `detect_tracking_loss(confidence: ConfidenceAssessment) -> bool`

**Description**: Determines if tracking is lost.

**Called By**: Main processing loop

**Input**: `ConfidenceAssessment`

**Output**: `bool` - True if tracking lost

**Test Cases**:
1. Confidence good → False
2. Confidence lost → True

---

### `start_search(flight_id: str, frame_id: int, estimated_gps: GPSPoint) -> SearchSession`

**Description**: Initiates progressive search session.

**Called By**: Main processing loop (when tracking lost)

**Input**:
```python
flight_id: str
frame_id: int
estimated_gps: GPSPoint  # Dead-reckoning estimate
```

**Output**:
```python
SearchSession:
    session_id: str
    flight_id: str
    frame_id: int
    center_gps: GPSPoint
    current_grid_size: int  # Starts at 1
    max_grid_size: int  # 25
    found: bool
```

**Processing Flow**:
1. Create search session
2. Set center from estimated_gps
3. Set current_grid_size = 1
4. Return session

**Test Cases**:
1. Start search → session created with grid_size=1

---

### `expand_search_radius(session: SearchSession) -> List[TileCoords]`

**Description**: Expands search grid to next size (1→4→9→16→25).

**Called By**: Internal (after try_current_grid fails)

**Input**: `SearchSession`

**Output**: `List[TileCoords]` - Tiles for next grid size

**Processing Flow**:
1. Increment current_grid_size (1→4→9→16→25)
2. Call F04.expand_search_grid() to get new tiles only
3. Return new tile coordinates

**Test Cases**:
1. Expand 1→4 → returns 3 new tiles
2. Expand 4→9 → returns 5 new tiles
3. At grid_size=25 → no more expansion

---

### `try_current_grid(session: SearchSession, tiles: Dict[str, np.ndarray]) -> Optional[AlignmentResult]`

**Description**: Tries LiteSAM matching on current tile grid.

**Called By**: Internal (progressive search loop)

**Input**:
```python
session: SearchSession
tiles: Dict[str, np.ndarray]  # From F04
```

**Output**: `Optional[AlignmentResult]` - Match result or None

**Processing Flow**:
1. Get UAV image for frame_id
2. For each tile in grid:
   - Get tile_bounds via F04.compute_tile_bounds(tile_coords)
   - Call F09.align_to_satellite(uav_image, tile, tile_bounds)
   - If match found with confidence > threshold:
     - mark_found(session, result)
     - Return result
3. Return None if no match

**Test Cases**:
1. Match on 3rd tile → returns result
2. No match in grid → returns None

---

### `mark_found(session: SearchSession, result: AlignmentResult) -> bool`

**Description**: Marks search session as successful.

**Called By**: Internal

**Input**:
```python
session: SearchSession
result: AlignmentResult
```

**Output**: `bool` - True

**Processing Flow**:
1. Set session.found = True
2. Log success (grid_size where found)
3. Resume processing

---

### `get_search_status(session: SearchSession) -> SearchStatus`

**Description**: Gets current search status.

**Called By**: F01 REST API (for status endpoint)

**Output**:
```python
SearchStatus:
    current_grid_size: int
    found: bool
    exhausted: bool  # Reached grid_size=25 without match
```

---

### `create_user_input_request(flight_id: str, frame_id: int, candidate_tiles: List[TileCandidate]) -> UserInputRequest`

**Description**: Creates user input request when all search strategies exhausted.

**Called By**: Internal (when grid_size=25 and no match)

**Input**:
```python
flight_id: str
frame_id: int
candidate_tiles: List[TileCandidate]  # Top-5 from G08
```

**Output**:
```python
UserInputRequest:
    request_id: str
    flight_id: str
    frame_id: int
    uav_image: np.ndarray
    candidate_tiles: List[TileCandidate]
    message: str
```

**Processing Flow**:
1. Get UAV image for frame_id
2. Get top-5 candidates from F08
3. Create request
4. Return Request object (Caller F02.2 handles sending)

**Test Cases**:
1. All search failed → creates request
2. Request returned to caller

---

### `apply_user_anchor(flight_id: str, frame_id: int, anchor: UserAnchor) -> bool`

**Description**: Applies user-provided GPS anchor.

**Called By**: F01 REST API (user-fix endpoint)

**Input**:
```python
flight_id: str
frame_id: int
anchor: UserAnchor:
    uav_pixel: Tuple[float, float]
    satellite_gps: GPSPoint
```

**Output**: `bool` - True if applied

**Processing Flow**:
1. Validate anchor data
2. Call F10.add_absolute_factor(frame_id, gps, is_user_anchor=True)
3. F10.optimize() → refines trajectory
4. Return True (Caller F02.2 updates status)

**Test Cases**:
1. Valid anchor → applied, processing resumes
2. Invalid anchor → rejected

---

### `create_chunk_on_tracking_loss(flight_id: str, frame_id: int) -> ChunkHandle`

**Description**: Creates a new chunk proactively when tracking is lost.

**Called By**:
- F02.2 Flight Processing Engine (when tracking lost detected)

**Input**:
```python
flight_id: str
frame_id: int  # First frame in new chunk
```

**Output**:
```python
ChunkHandle:
    chunk_id: str
    flight_id: str
    start_frame_id: int
    end_frame_id: Optional[int]
    frames: List[int]
    is_active: bool
    has_anchor: bool
    anchor_frame_id: Optional[int]
    anchor_gps: Optional[GPSPoint]
    matching_status: str
```

**Processing Flow**:
1. Call F12 Route Chunk Manager.create_chunk()
2. F12 creates chunk in F10 Factor Graph Optimizer
3. Mark chunk as active
4. Return ChunkHandle

**Proactive Behavior**:
- Chunk created immediately (not waiting for matching to fail)
- Processing continues in new chunk
- Matching attempted asynchronously

**Test Cases**:
1. **Create chunk**: Chunk created successfully
2. **Proactive creation**: Chunk created before matching attempts
3. **Continue processing**: Processing continues in new chunk

---

### `try_chunk_semantic_matching(chunk_id: str) -> Optional[List[TileCandidate]]`

**Description**: Attempts semantic matching for a chunk using aggregate DINOv2 descriptor.

**Called By**:
- Internal (when chunk ready for matching)
- process_unanchored_chunks() (background task)

**Input**:
```python
chunk_id: str
```

**Output**:
```python
Optional[List[TileCandidate]]: Top-k candidate tiles or None if matching failed
```

**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. Call F08 Global Place Recognition.retrieve_candidate_tiles_for_chunk()
3. F08 computes chunk descriptor and queries Faiss
4. Return candidate tiles if found, None otherwise

**Test Cases**:
1. **Chunk matching**: Returns candidate tiles
2. **Featureless terrain**: Succeeds where single-image fails
3. **No match**: Returns None

---

### `try_chunk_litesam_matching(chunk_id: str, candidate_tiles: List[TileCandidate]) -> Optional[ChunkAlignmentResult]`

**Description**: Attempts LiteSAM matching for chunk with rotation sweeps.

**Called By**:
- Internal (after chunk semantic matching succeeds)
- process_unanchored_chunks() (background task)

**Input**:
```python
chunk_id: str
candidate_tiles: List[TileCandidate]  # From chunk semantic matching
```

**Output**:
```python
Optional[ChunkAlignmentResult]: Match result or None
```

**Processing Flow**:
1. Get chunk images via F12.get_chunk_images()
2. For each candidate tile:
   - Get tile from F04 Satellite Data Manager
   - Call F06.try_chunk_rotation_steps() (12 rotations: 0°, 30°, ..., 330°)
   - F06 calls F09.align_chunk_to_satellite() for each rotation
   - If match found with confidence > threshold:
     - Return ChunkAlignmentResult
3. Return None if no match found

**Rotation Sweeps**:
- Critical for chunks from sharp turns (unknown orientation)
- Tries all 12 rotation angles
- Returns best matching rotation

**Test Cases**:
1. **Match on first tile**: Returns alignment result
2. **Match on 3rd tile**: Returns alignment result
3. **Match at 120° rotation**: Returns result with correct rotation angle
4. **No match**: Returns None

---

### `merge_chunk_to_trajectory(flight_id: str, chunk_id: str, alignment_result: ChunkAlignmentResult) -> bool`

**Description**: Merges chunk into main trajectory after successful matching.

**Called By**:
- Internal (after chunk LiteSAM matching succeeds)
- process_unanchored_chunks() (which has flight_id from chunk handle)

**Input**:
```python
flight_id: str  # Flight identifier (available from chunk handle)
chunk_id: str
alignment_result: ChunkAlignmentResult:
    chunk_center_gps: GPSPoint
    transform: Sim3Transform
    confidence: float
```

**Output**:
```python
bool: True if merge successful
```

**Processing Flow**:
1. Get chunk frames via F12.get_chunk_frames(chunk_id) → merged_frames
2. Get chunk anchor frame (middle frame or best frame)
3. Call F12.mark_chunk_anchored() with GPS (F12 coordinates with F10)
4. **Determine merge target**:
   - Main chunk is typically the temporal predecessor (previous chunk by frame_id order)
   - If no predecessor: merge to main trajectory (main_chunk_id="main")
   - F11 determines main_chunk based on chunk frame_id ordering
5. Call F12.merge_chunks(main_chunk_id, chunk_id, transform)
   - Note: `merge_chunks(main_chunk, new_chunk)` merges new_chunk INTO main_chunk
   - chunk_id (source) is merged into target_chunk_id
6. F12 handles chunk state updates (deactivation, status updates)
7. F10 optimizes merged graph globally (via F12.merge_chunks())
8. Return True (Caller F02.2 coordinates result updates)

**Sim(3) Transform**:
- Translation: GPS offset
- Rotation: From alignment result
- Scale: Resolved from altitude and GSD

**Test Cases**:
1. **Merge successful**: Chunks merged successfully
2. **Global consistency**: Merged trajectory globally consistent
3. **Multiple chunks**: Can merge multiple chunks sequentially

---

### `process_unanchored_chunks(flight_id: str) -> None`

**Description**: Background task that periodically attempts matching for unanchored chunks.

**Called By**:
- Background thread (periodic, e.g., every 5 seconds)
- F02.2 Flight Processing Engine (after frame processing)

**Input**:
```python
flight_id: str
```

**Output**: None (runs asynchronously)

**Processing Flow**:
```
while flight_active:
    unanchored_chunks = F12.get_chunks_for_matching(flight_id)
    for chunk in unanchored_chunks:
        if F12.is_chunk_ready_for_matching(chunk.chunk_id):
            F12.mark_chunk_matching(chunk.chunk_id)
            candidates = try_chunk_semantic_matching(chunk.chunk_id)
            if candidates:
                alignment = try_chunk_litesam_matching(chunk.chunk_id, candidates)
                if alignment:
                    merge_chunk_to_trajectory(chunk.flight_id, chunk.chunk_id, alignment)
    sleep(5 seconds)
```

**Background Processing**:
- **Trigger**: Started by F02.2 Flight Processing Engine after flight creation
- Runs asynchronously, doesn't block frame processing
- Periodically checks for ready chunks (every 5 seconds)
- Attempts matching and merging
- Reduces user input requests
- **Lifecycle**: Starts when flight becomes active, stops when flight completed

**Chunk Failure Flow**:
When chunk matching fails after trying all candidates:
1. Emit `ChunkMatchingFailed` event with chunk_id and flight_id (to F02.2 internal queue)
2. F02.2 receives event and decides next action:
   - **Option A**: Wait for more frames and retry later (chunk may gain more distinctive features)
   - **Option B**: Create user input request for the chunk
3. If user input requested:
   - F02.2 calls F15.send_user_input_request() with chunk context
   - Client receives via SSE
   - User provides GPS anchor for one frame in chunk
   - F02.2 receives user fix via handle_user_fix()
   - F11.apply_user_anchor() anchors the chunk
   - Processing resumes

**Test Cases**:
1. **Background matching**: Unanchored chunks matched asynchronously
2. **Chunk merging**: Chunks merged when matches found
3. **Non-blocking**: Frame processing continues during matching
4. **Chunk matching fails**: Failure handled, user input requested if needed

## Integration Tests

### Test 1: Progressive Search Flow
1. Tracking lost detected
2. start_search() → grid_size=1
3. try_current_grid(1 tile) → no match
4. expand_search_radius() → grid_size=4
5. try_current_grid(4 tiles) → match found
6. mark_found() → success

### Test 2: Full Search Exhaustion
1. start_search()
2. try grids: 1→4→9→16→25, all fail
3. create_user_input_request()
4. User provides anchor
5. apply_user_anchor() → processing resumes

### Test 3: Confidence Monitoring
1. Normal frames → confidence good
2. Low overlap frame → confidence degraded
3. Sharp turn → tracking lost, trigger search

### Test 4: Proactive Chunk Creation
1. Tracking lost detected
2. create_chunk_on_tracking_loss() → chunk created immediately
3. Processing continues in new chunk
4. Verify chunk matching attempted asynchronously

### Test 5: Chunk Semantic Matching
1. Build chunk with 10 images (plain field)
2. try_chunk_semantic_matching() → returns candidate tiles
3. Verify chunk matching succeeds where single-image fails

### Test 6: Chunk LiteSAM Matching with Rotation
1. Build chunk with unknown orientation
2. try_chunk_litesam_matching() with candidate tiles
3. Rotation sweep finds match at 120°
4. Returns ChunkAlignmentResult with correct GPS

### Test 7: Chunk Merging
1. Create chunk_1 (frames 1-10), chunk_2 (frames 20-30)
2. Anchor chunk_2 via chunk matching
3. merge_chunk_to_trajectory() → chunks merged
4. Verify global trajectory consistent

### Test 8: Background Chunk Processing
1. Create 3 unanchored chunks
2. process_unanchored_chunks() runs in background
3. Chunks matched and merged asynchronously
4. Frame processing continues uninterrupted

## Non-Functional Requirements

### Performance
- **check_confidence**: < 10ms
- **Progressive search (25 tiles)**: < 1.5s total
- **User input latency**: < 500ms from creation to SSE event

### Reliability
- Always exhausts all search strategies before requesting user input
- Guarantees processing block when awaiting user input
- Graceful recovery from all failure modes

## Dependencies

### Internal Components
- **F12 Route Chunk Manager**: Chunk operations.
- **F10 Factor Graph Optimizer**: Anchor application.
- **F04 Satellite Data Manager**: Search grids.
- **F08 Global Place Recognition**: Candidate retrieval.
- **F09 Metric Refinement**: Alignment.
- **F06 Image Rotation Manager**: Rotation sweeps.

### External Dependencies
- None

## Data Models

### ConfidenceAssessment
```python
class ConfidenceAssessment(BaseModel):
    overall_confidence: float
    vo_confidence: float
    litesam_confidence: float
    inlier_count: int
    tracking_status: str
```

### SearchSession
```python
class SearchSession(BaseModel):
    session_id: str
    flight_id: str
    frame_id: int
    center_gps: GPSPoint
    current_grid_size: int
    max_grid_size: int
    found: bool
    exhausted: bool
```

### UserInputRequest
```python
class UserInputRequest(BaseModel):
    request_id: str
    flight_id: str
    frame_id: int
    uav_image: np.ndarray
    candidate_tiles: List[TileCandidate]
    message: str
    created_at: datetime
```

### UserAnchor
```python
class UserAnchor(BaseModel):
    uav_pixel: Tuple[float, float]
    satellite_gps: GPSPoint
    confidence: float = 1.0
```

### ChunkHandle
```python
class ChunkHandle(BaseModel):
    chunk_id: str
    flight_id: str
    start_frame_id: int
    end_frame_id: Optional[int]
    frames: List[int]
    is_active: bool
    has_anchor: bool
    anchor_frame_id: Optional[int]
    anchor_gps: Optional[GPSPoint]
    matching_status: str  # "unanchored", "matching", "anchored", "merged"
```

### ChunkAlignmentResult
```python
class ChunkAlignmentResult(BaseModel):
    matched: bool
    chunk_id: str
    chunk_center_gps: GPSPoint
    rotation_angle: float
    confidence: float
    inlier_count: int
    transform: Sim3Transform
```

### Sim3Transform
```python
class Sim3Transform(BaseModel):
    translation: np.ndarray  # (3,)
    rotation: np.ndarray  # (3, 3) or (4,) quaternion
    scale: float
```

### RecoveryStatus (Returned by recovery methods)
```python
class RecoveryStatus(BaseModel):
    """Status returned by F11 methods, used by F02 to determine next action."""
    success: bool
    method: str  # "rotation_sweep", "progressive_search", "chunk_matching", "user_input"
    gps: Optional[GPSPoint]
    chunk_id: Optional[str]
    message: Optional[str]
```
