initial structure implemented

docs -> _docs
This commit is contained in:
Oleksandr Bezdieniezhnykh
2025-12-01 14:20:56 +02:00
parent 9134c5db06
commit abc26d5c20
360 changed files with 3881 additions and 101 deletions
@@ -0,0 +1,41 @@
# Feature: Tile Cache Management
## Description
Manages persistent disk-based caching of satellite tiles with flight-specific organization. Provides storage, retrieval, and cleanup of cached tiles to minimize redundant API calls and enable offline access to prefetched data.
## Component APIs Implemented
- `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
- `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
- `clear_flight_cache(flight_id: str) -> bool`
## External Tools and Services
- **diskcache**: Persistent cache library for disk storage management
- **opencv-python**: Image serialization (PNG encoding/decoding)
- **numpy**: Image array handling
## Internal Methods
- `_generate_cache_path(flight_id: str, tile_coords: TileCoords) -> Path`: Generates cache file path following pattern `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
- `_ensure_cache_directory(flight_id: str, zoom: int) -> bool`: Creates cache directory structure if not exists
- `_serialize_tile(tile_data: np.ndarray) -> bytes`: Encodes tile array to PNG bytes
- `_deserialize_tile(data: bytes) -> Optional[np.ndarray]`: Decodes PNG bytes to tile array
- `_update_cache_index(flight_id: str, tile_coords: TileCoords, action: str) -> None`: Updates cache index for tracking
- `_check_global_cache(tile_coords: TileCoords) -> Optional[np.ndarray]`: Fallback lookup in shared cache
## Unit Tests
1. **cache_tile_success**: Cache new tile → file created at correct path
2. **cache_tile_overwrite**: Cache existing tile → file updated
3. **cache_tile_disk_error**: Simulate disk full → returns False
4. **get_cached_tile_hit**: Tile exists → returns np.ndarray
5. **get_cached_tile_miss**: Tile not exists → returns None
6. **get_cached_tile_corrupted**: Invalid file → returns None, logs warning
7. **get_cached_tile_global_fallback**: Not in flight cache, found in global → returns tile
8. **clear_flight_cache_success**: Flight with tiles → all files removed
9. **clear_flight_cache_nonexistent**: No such flight → returns True (no-op)
10. **cache_path_generation**: Various tile coords → correct paths generated
## Integration Tests
1. **cache_round_trip**: cache_tile() then get_cached_tile() → returns identical data
2. **multi_flight_isolation**: Cache tiles for flight A and B → each retrieves only own tiles
3. **clear_does_not_affect_others**: Clear flight A → flight B cache intact
4. **large_cache_handling**: Cache 1000 tiles → all retrievable
@@ -0,0 +1,44 @@
# Feature: Tile Coordinate Operations
## Description
Handles all tile coordinate calculations including GPS-to-tile conversion, tile grid computation, and grid expansion for progressive search. Delegates core Web Mercator projection math to H06 Web Mercator Utils to maintain single source of truth.
## Component APIs Implemented
- `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
- `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
- `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
- `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
## External Tools and Services
None (pure computation, delegates to H06)
## Internal Dependencies
- **H06 Web Mercator Utils**: Core projection calculations
- `H06.latlon_to_tile()` for coordinate conversion
- `H06.compute_tile_bounds()` for bounding box calculation
## Internal Methods
- `_compute_grid_offset(grid_size: int) -> int`: Calculates offset from center for symmetric grid (e.g., 3×3 → offset 1)
- `_grid_size_to_dimensions(grid_size: int) -> Tuple[int, int]`: Maps grid_size (1,4,9,16,25) to (rows, cols)
- `_generate_grid_tiles(center: TileCoords, rows: int, cols: int) -> List[TileCoords]`: Generates all tile coords in grid
## Unit Tests
1. **compute_tile_coords_ukraine**: Ukraine GPS coords at zoom 19 → valid tile coords
2. **compute_tile_coords_origin**: lat=0, lon=0 → correct center tile
3. **compute_tile_coords_edge_cases**: lat=90, lon=180, lon=-180 → handled correctly
4. **compute_tile_bounds_zoom19**: Zoom 19 tile → GSD ≈ 0.3 m/pixel
5. **compute_tile_bounds_corners**: Returns valid GPS for all 4 corners
6. **get_tile_grid_1**: grid_size=1 → returns [center]
7. **get_tile_grid_4**: grid_size=4 → returns 4 tiles (2×2)
8. **get_tile_grid_9**: grid_size=9 → returns 9 tiles (3×3) centered
9. **get_tile_grid_25**: grid_size=25 → returns 25 tiles (5×5)
10. **expand_search_grid_1_to_4**: Returns 3 new tiles only
11. **expand_search_grid_4_to_9**: Returns 5 new tiles only
12. **expand_search_grid_9_to_16**: Returns 7 new tiles only
13. **expand_search_grid_no_duplicates**: Expanded tiles not in original set
## Integration Tests
1. **h06_delegation_verify**: compute_tile_coords() result matches direct H06.latlon_to_tile()
2. **grid_bounds_coverage**: get_tile_grid(9) → all 9 tile bounds form contiguous area
3. **expand_completes_grid**: get_tile_grid(4) + expand_search_grid(4,9) == get_tile_grid(9)
@@ -0,0 +1,51 @@
# Feature: Tile Fetching
## Description
Handles HTTP-based satellite tile retrieval from external provider API with multiple fetching patterns: single tile, grid, progressive expansion, and route corridor prefetching. Integrates with cache for performance optimization and supports parallel fetching for throughput.
## Component APIs Implemented
- `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
- `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
- `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
- `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
## External Tools and Services
- **Satellite Provider API**: HTTP tile source (`GET /api/satellite/tiles/latlon`)
- **httpx** or **requests**: HTTP client with async support
- **numpy**: Image array handling
## Internal Dependencies
- **01_feature_tile_cache_management**: cache_tile, get_cached_tile
- **02_feature_tile_coordinate_operations**: compute_tile_coords, get_tile_grid
## Internal Methods
- `_fetch_from_api(tile_coords: TileCoords) -> Optional[np.ndarray]`: HTTP GET to satellite provider, handles response parsing
- `_fetch_with_retry(tile_coords: TileCoords, max_retries: int = 3) -> Optional[np.ndarray]`: Wraps _fetch_from_api with retry logic
- `_fetch_tiles_parallel(tiles: List[TileCoords], max_concurrent: int = 20) -> Dict[str, np.ndarray]`: Parallel fetching with connection pooling
- `_compute_corridor_tiles(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> List[TileCoords]`: Calculates tiles covering route corridor polygon
- `_generate_tile_id(tile_coords: TileCoords) -> str`: Creates unique tile identifier string
## Unit Tests
1. **fetch_tile_cache_hit**: Tile in cache → returns immediately, no HTTP call
2. **fetch_tile_cache_miss**: Not cached → HTTP fetch, cache, return
3. **fetch_tile_api_error**: HTTP 500 → returns None
4. **fetch_tile_invalid_coords**: Invalid GPS → returns None
5. **fetch_tile_retry_success**: First attempt fails, second succeeds → returns tile
6. **fetch_tile_retry_exhausted**: All 3 attempts fail → returns None
7. **fetch_tile_grid_2x2**: grid_size=4 → returns dict with 4 tiles
8. **fetch_tile_grid_3x3**: grid_size=9 → returns dict with 9 tiles
9. **fetch_tile_grid_partial_failure**: 2 of 9 tiles fail → returns 7 tiles
10. **fetch_tile_grid_all_cached**: All tiles cached → no HTTP calls
11. **prefetch_route_corridor_success**: 10 waypoints → prefetches tiles, returns True
12. **prefetch_route_corridor_partial_failure**: Some tiles fail → continues, returns True
13. **prefetch_route_corridor_complete_failure**: All tiles fail → returns False
14. **progressive_fetch_yields_sequence**: [1,4,9] → yields 3 dicts in order
15. **progressive_fetch_early_termination**: Break after 4 → doesn't fetch 9,16,25
## Integration Tests
1. **fetch_and_cache_verify**: fetch_tile() → get_cached_tile() returns same data
2. **progressive_search_simulation**: progressive_fetch with simulated match on grid 9
3. **grid_expansion_no_refetch**: fetch_tile_grid(4) then expand → no duplicate fetches
4. **corridor_prefetch_coverage**: prefetch_route_corridor → all corridor tiles cached
5. **concurrent_fetch_stress**: Fetch 100 tiles in parallel → all complete within timeout
@@ -0,0 +1,562 @@
# Satellite Data Manager
## Interface Definition
**Interface Name**: `ISatelliteDataManager`
### Interface Methods
```python
class ISatelliteDataManager(ABC):
@abstractmethod
def fetch_tile(self, lat: float, lon: float, zoom: int) -> Optional[np.ndarray]:
pass
@abstractmethod
def fetch_tile_grid(self, center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]:
pass
@abstractmethod
def prefetch_route_corridor(self, waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool:
pass
@abstractmethod
def progressive_fetch(self, center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]:
pass
@abstractmethod
def cache_tile(self, flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool:
pass
@abstractmethod
def get_cached_tile(self, flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]:
pass
@abstractmethod
def get_tile_grid(self, center: TileCoords, grid_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_coords(self, lat: float, lon: float, zoom: int) -> TileCoords:
pass
@abstractmethod
def expand_search_grid(self, center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]:
pass
@abstractmethod
def compute_tile_bounds(self, tile_coords: TileCoords) -> TileBounds:
pass
@abstractmethod
def clear_flight_cache(self, flight_id: str) -> bool:
pass
```
## Component Description
### Responsibilities
- Fetch satellite tiles from external provider API
- Manage local tile cache per flight
- Calculate tile coordinates and grid layouts
- Support progressive tile grid expansion (1→4→9→16→25)
- Handle Web Mercator projection calculations
- Coordinate corridor prefetching for flight routes
### Scope
- **HTTP client** for Satellite Provider API
- **Local caching** with disk storage
- **Grid calculations** for search patterns
- **Tile coordinate transformations** (GPS↔Tile coordinates)
- **Progressive retrieval** for "kidnapped robot" recovery
## API Methods
### `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
**Description**: Fetches a single satellite tile by GPS coordinates.
**Called By**:
- F09 Metric Refinement (single tile for drift correction)
- Internal (during prefetching)
**Input**:
```python
lat: float # Latitude
lon: float # Longitude
zoom: int # Zoom level (19 for 0.3m/pixel at Ukraine latitude)
```
**Output**:
```python
np.ndarray: Tile image (H×W×3 RGB) or None if failed
```
**HTTP Request**:
```
GET /api/satellite/tiles/latlon?lat={lat}&lon={lon}&zoom={zoom}
```
**Processing Flow**:
1. Convert GPS to tile coordinates
2. Check cache
3. If not cached, fetch from satellite provider
4. Cache tile
5. Return tile image
**Error Conditions**:
- Returns `None`: Tile unavailable, HTTP error
- Logs errors for monitoring
**Test Cases**:
1. **Cache hit**: Tile in cache → returns immediately
2. **Cache miss**: Fetches from API → caches → returns
3. **API error**: Returns None
4. **Invalid coordinates**: Returns None
---
### `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
**Description**: Fetches NxN grid of tiles centered on GPS coordinates.
**Called By**:
- F09 Metric Refinement (for progressive search)
- F11 Failure Recovery Coordinator
**Input**:
```python
center_lat: float
center_lon: float
grid_size: int # 1, 4 (2×2), 9 (3×3), 16 (4×4), or 25 (5×5)
zoom: int
```
**Output**:
```python
Dict[str, np.ndarray] # tile_id -> tile_image
```
**Processing Flow**:
1. Compute tile grid centered on coordinates
2. For each tile in grid:
- Check cache
- If not cached, fetch from API
3. Return dict of tiles
**HTTP Request** (if using batch endpoint):
```
GET /api/satellite/tiles/batch?tiles=[...]
```
**Error Conditions**:
- Returns partial dict if some tiles fail
- Empty dict if all tiles fail
**Test Cases**:
1. **2×2 grid**: Returns 4 tiles
2. **3×3 grid**: Returns 9 tiles
3. **5×5 grid**: Returns 25 tiles
4. **Partial failure**: Some tiles unavailable → returns available tiles
5. **All cached**: Fast retrieval without HTTP requests
---
### `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
**Description**: Prefetches satellite tiles along route corridor for a flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (during flight creation)
**Input**:
```python
waypoints: List[GPSPoint] # Rough route waypoints
corridor_width_m: float # Corridor width in meters (e.g., 1000m)
zoom: int
```
**Output**:
```python
bool: True if prefetch completed, False on error
```
**Processing Flow**:
1. For each waypoint pair:
- Calculate corridor polygon
- Determine tiles covering corridor
2. Fetch tiles (async, parallel)
3. Cache all tiles with flight_id reference
**Algorithm**:
- Use H06 Web Mercator Utils for tile calculations
- Parallel fetching (10-20 concurrent requests)
- Progress tracking for monitoring
**Error Conditions**:
- Returns `False`: Major error preventing prefetch
- Logs warnings for individual tile failures
**Test Cases**:
1. **Simple route**: 10 waypoints → prefetches 50-100 tiles
2. **Long route**: 50 waypoints → prefetches 200-500 tiles
3. **Partial failure**: Some tiles fail → continues, returns True
4. **Complete failure**: All tiles fail → returns False
---
### `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
**Description**: Progressively fetches expanding tile grids for "kidnapped robot" recovery.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search)
**Input**:
```python
center_lat: float
center_lon: float
grid_sizes: List[int] # e.g., [1, 4, 9, 16, 25]
zoom: int
```
**Output**:
```python
Iterator yielding Dict[str, np.ndarray] for each grid size
```
**Processing Flow**:
1. For each grid_size in sequence:
- Fetch tile grid
- Yield tiles
- If match found by caller, iterator can be stopped
**Usage Pattern**:
```python
for tiles in progressive_fetch(lat, lon, [1, 4, 9, 16, 25], 19):
if litesam_match_found(tiles):
break # Stop expanding search
```
**Test Cases**:
1. **Progressive search**: Yields 1, then 4, then 9 tiles
2. **Early termination**: Match on 4 tiles → doesn't fetch 9, 16, 25
3. **Full search**: No match → fetches all grid sizes
---
### `cache_tile(flight_id: str, tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
**Description**: Caches a satellite tile to disk with flight_id association.
**Called By**:
- Internal (after fetching tiles)
**Input**:
```python
flight_id: str # Flight this tile belongs to
tile_coords: TileCoords:
x: int
y: int
zoom: int
tile_data: np.ndarray
```
**Output**:
```python
bool: True if cached successfully
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Create flight cache directory if not exists
3. Serialize tile_data (PNG format)
4. Write to disk cache directory
5. Update cache index with flight_id association
**Error Conditions**:
- Returns `False`: Disk write error, space full
**Test Cases**:
1. **Cache new tile**: Writes successfully
2. **Overwrite existing**: Updates tile
3. **Disk full**: Returns False
---
### `get_cached_tile(flight_id: str, tile_coords: TileCoords) -> Optional[np.ndarray]`
**Description**: Retrieves a cached tile from disk, checking flight-specific cache first.
**Called By**:
- Internal (before fetching from API)
- F09 Metric Refinement (direct cache lookup)
**Input**:
```python
flight_id: str # Flight to check cache for
tile_coords: TileCoords
```
**Output**:
```python
Optional[np.ndarray]: Tile image or None if not cached
```
**Processing Flow**:
1. Generate cache path: `/satellite_cache/{flight_id}/{zoom}/{tile_x}_{tile_y}.png`
2. Check flight-specific cache first
3. If not found, check global cache (shared tiles)
4. If file exists, load and deserialize
5. Return tile_data or None
**Error Conditions**:
- Returns `None`: Not cached, corrupted file
**Test Cases**:
1. **Cache hit**: Returns tile quickly
2. **Cache miss**: Returns None
3. **Corrupted cache**: Returns None, logs warning
---
### `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
**Description**: Calculates tile coordinates for NxN grid centered on a tile.
**Called By**:
- Internal (for grid fetching)
- F11 Failure Recovery Coordinator
**Input**:
```python
center: TileCoords
grid_size: int # 1, 4, 9, 16, 25
```
**Output**:
```python
List[TileCoords] # List of tile coordinates in grid
```
**Algorithm**:
- For grid_size=9 (3×3): tiles from center-1 to center+1 in both x and y
- For grid_size=16 (4×4): asymmetric grid with center slightly off-center
**Test Cases**:
1. **1-tile grid**: Returns [center]
2. **4-tile grid (2×2)**: Returns 4 tiles
3. **9-tile grid (3×3)**: Returns 9 tiles centered
4. **25-tile grid (5×5)**: Returns 25 tiles
---
### `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
**Description**: Converts GPS coordinates to tile coordinates.
**Called By**:
- All methods that need tile coordinates from GPS
**Input**:
```python
lat: float
lon: float
zoom: int
```
**Output**:
```python
TileCoords:
x: int
y: int
zoom: int
```
**Algorithm** (Web Mercator):
```python
n = 2 ** zoom
x = int((lon + 180) / 360 * n)
lat_rad = lat * π / 180
y = int((1 - log(tan(lat_rad) + sec(lat_rad)) / π) / 2 * n)
```
**Test Cases**:
1. **Ukraine coordinates**: Produces valid tile coords
2. **Edge cases**: lat=0, lon=0, lat=90, lon=180
---
### `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
**Description**: Returns only NEW tiles when expanding from current grid to larger grid.
**Called By**:
- F11 Failure Recovery Coordinator (progressive search optimization)
**Input**:
```python
center: TileCoords
current_size: int # e.g., 4
new_size: int # e.g., 9
```
**Output**:
```python
List[TileCoords] # Only tiles not in current_size grid
```
**Purpose**: Avoid re-fetching tiles already tried in smaller grid.
**Test Cases**:
1. **4→9 expansion**: Returns 5 new tiles (9-4)
2. **9→16 expansion**: Returns 7 new tiles
3. **1→4 expansion**: Returns 3 new tiles
---
### `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
**Description**: Computes GPS bounding box of a tile.
**Called By**:
- F09 Metric Refinement (for homography calculations)
- H06 Web Mercator Utils (shared calculation)
**Input**:
```python
tile_coords: TileCoords
```
**Output**:
```python
TileBounds:
nw: GPSPoint # North-West corner
ne: GPSPoint # North-East corner
sw: GPSPoint # South-West corner
se: GPSPoint # South-East corner
center: GPSPoint
gsd: float # Ground Sampling Distance (meters/pixel)
```
**Algorithm**:
- Inverse Web Mercator projection
- GSD calculation: `156543.03392 * cos(lat * π/180) / 2^zoom`
**Test Cases**:
1. **Zoom 19 at Ukraine**: GSD ≈ 0.3 m/pixel
2. **Tile bounds**: Valid GPS coordinates
---
### `clear_flight_cache(flight_id: str) -> bool`
**Description**: Clears cached tiles for a completed flight.
**Called By**:
- F02.1 Flight Lifecycle Manager (cleanup after flight completion)
**Input**:
```python
flight_id: str
```
**Output**:
```python
bool: True if cleared successfully
```
**Processing Flow**:
1. Find all tiles associated with flight_id
2. Delete tile files
3. Update cache index
**Test Cases**:
1. **Clear flight cache**: Removes all associated tiles
2. **Non-existent flight**: Returns True (no-op)
## Integration Tests
### Test 1: Prefetch and Retrieval
1. prefetch_route_corridor() with 20 waypoints
2. Verify tiles cached
3. get_cached_tile() for each tile → all hit cache
4. clear_flight_cache() → cache cleared
### Test 2: Progressive Search Simulation
1. progressive_fetch() with [1, 4, 9, 16, 25]
2. Simulate match on 9 tiles
3. Verify only 1, 4, 9 fetched (not 16, 25)
### Test 3: Grid Expansion
1. fetch_tile_grid(4) → 4 tiles
2. expand_search_grid(4, 9) → 5 new tiles
3. Verify no duplicate fetches
## Non-Functional Requirements
### Performance
- **fetch_tile**: < 200ms (cached: < 10ms)
- **fetch_tile_grid(9)**: < 1 second
- **prefetch_route_corridor**: < 30 seconds for 500 tiles
- **Cache lookup**: < 5ms
### Scalability
- Cache 10,000+ tiles per flight
- Support 100 concurrent tile fetches
- Handle 10GB+ cache size
### Reliability
- Retry failed HTTP requests (3 attempts)
- Graceful degradation on partial failures
- Cache corruption recovery
## Dependencies
### Internal Components
- **H06 Web Mercator Utils**: Tile coordinate calculations
**Note on Tile Coordinate Calculations**: F04 delegates ALL tile coordinate calculations to H06 Web Mercator Utils:
- `compute_tile_coords()` → internally calls `H06.latlon_to_tile()`
- `compute_tile_bounds()` → internally calls `H06.compute_tile_bounds()`
- `get_tile_grid()` → uses H06 for coordinate math
This ensures single source of truth for Web Mercator projection logic and avoids duplication with H06.
### External Dependencies
- **Satellite Provider API**: HTTP tile source
- **requests** or **httpx**: HTTP client
- **numpy**: Image handling
- **opencv-python**: Image I/O
- **diskcache**: Persistent cache
## Data Models
### TileCoords
```python
class TileCoords(BaseModel):
x: int
y: int
zoom: int
```
### TileBounds
```python
class TileBounds(BaseModel):
nw: GPSPoint
ne: GPSPoint
sw: GPSPoint
se: GPSPoint
center: GPSPoint
gsd: float # meters/pixel
```
### CacheConfig
```python
class CacheConfig(BaseModel):
cache_dir: str = "./satellite_cache"
max_size_gb: int = 50
eviction_policy: str = "lru"
ttl_days: int = 30
```