mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-04-22 23:56:36 +00:00
add chunking
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,551 @@
|
||||
# Satellite Data Manager
|
||||
|
||||
## Interface Definition
|
||||
|
||||
**Interface Name**: `ISatelliteDataManager`
|
||||
|
||||
### Interface Methods
|
||||
|
||||
```python
|
||||
class ISatelliteDataManager(ABC):
|
||||
@abstractmethod
|
||||
def fetch_tile(self, lat: float, lon: float, zoom: int) -> Optional[np.ndarray]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def fetch_tile_grid(self, center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def prefetch_route_corridor(self, waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def progressive_fetch(self, center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def cache_tile(self, tile_coords: TileCoords, tile_data: np.ndarray) -> bool:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_cached_tile(self, tile_coords: TileCoords) -> Optional[np.ndarray]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_tile_grid(self, center: TileCoords, grid_size: int) -> List[TileCoords]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def compute_tile_coords(self, lat: float, lon: float, zoom: int) -> TileCoords:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def expand_search_grid(self, center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def compute_tile_bounds(self, tile_coords: TileCoords) -> TileBounds:
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def clear_flight_cache(self, flight_id: str) -> bool:
|
||||
pass
|
||||
```
|
||||
|
||||
## Component Description
|
||||
|
||||
### Responsibilities
|
||||
- Fetch satellite tiles from external provider API
|
||||
- Manage local tile cache per flight
|
||||
- Calculate tile coordinates and grid layouts
|
||||
- Support progressive tile grid expansion (1→4→9→16→25)
|
||||
- Handle Web Mercator projection calculations
|
||||
- Coordinate corridor prefetching for flight routes
|
||||
|
||||
### Scope
|
||||
- **HTTP client** for Satellite Provider API
|
||||
- **Local caching** with disk storage
|
||||
- **Grid calculations** for search patterns
|
||||
- **Tile coordinate transformations** (GPS↔Tile coordinates)
|
||||
- **Progressive retrieval** for "kidnapped robot" recovery
|
||||
|
||||
## API Methods
|
||||
|
||||
### `fetch_tile(lat: float, lon: float, zoom: int) -> Optional[np.ndarray]`
|
||||
|
||||
**Description**: Fetches a single satellite tile by GPS coordinates.
|
||||
|
||||
**Called By**:
|
||||
- F09 Metric Refinement (single tile for drift correction)
|
||||
- Internal (during prefetching)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
lat: float # Latitude
|
||||
lon: float # Longitude
|
||||
zoom: int # Zoom level (19 for 0.3m/pixel at Ukraine latitude)
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
np.ndarray: Tile image (H×W×3 RGB) or None if failed
|
||||
```
|
||||
|
||||
**HTTP Request**:
|
||||
```
|
||||
GET /api/satellite/tiles/latlon?lat={lat}&lon={lon}&zoom={zoom}
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. Convert GPS to tile coordinates
|
||||
2. Check cache
|
||||
3. If not cached, fetch from satellite provider
|
||||
4. Cache tile
|
||||
5. Return tile image
|
||||
|
||||
**Error Conditions**:
|
||||
- Returns `None`: Tile unavailable, HTTP error
|
||||
- Logs errors for monitoring
|
||||
|
||||
**Test Cases**:
|
||||
1. **Cache hit**: Tile in cache → returns immediately
|
||||
2. **Cache miss**: Fetches from API → caches → returns
|
||||
3. **API error**: Returns None
|
||||
4. **Invalid coordinates**: Returns None
|
||||
|
||||
---
|
||||
|
||||
### `fetch_tile_grid(center_lat: float, center_lon: float, grid_size: int, zoom: int) -> Dict[str, np.ndarray]`
|
||||
|
||||
**Description**: Fetches NxN grid of tiles centered on GPS coordinates.
|
||||
|
||||
**Called By**:
|
||||
- F09 Metric Refinement (for progressive search)
|
||||
- F11 Failure Recovery Coordinator
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
center_lat: float
|
||||
center_lon: float
|
||||
grid_size: int # 1, 4 (2×2), 9 (3×3), 16 (4×4), or 25 (5×5)
|
||||
zoom: int
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
Dict[str, np.ndarray] # tile_id -> tile_image
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. Compute tile grid centered on coordinates
|
||||
2. For each tile in grid:
|
||||
- Check cache
|
||||
- If not cached, fetch from API
|
||||
3. Return dict of tiles
|
||||
|
||||
**HTTP Request** (if using batch endpoint):
|
||||
```
|
||||
GET /api/satellite/tiles/batch?tiles=[...]
|
||||
```
|
||||
|
||||
**Error Conditions**:
|
||||
- Returns partial dict if some tiles fail
|
||||
- Empty dict if all tiles fail
|
||||
|
||||
**Test Cases**:
|
||||
1. **2×2 grid**: Returns 4 tiles
|
||||
2. **3×3 grid**: Returns 9 tiles
|
||||
3. **5×5 grid**: Returns 25 tiles
|
||||
4. **Partial failure**: Some tiles unavailable → returns available tiles
|
||||
5. **All cached**: Fast retrieval without HTTP requests
|
||||
|
||||
---
|
||||
|
||||
### `prefetch_route_corridor(waypoints: List[GPSPoint], corridor_width_m: float, zoom: int) -> bool`
|
||||
|
||||
**Description**: Prefetches satellite tiles along route corridor for a flight.
|
||||
|
||||
**Called By**:
|
||||
- F02 Flight Manager (during flight creation)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
waypoints: List[GPSPoint] # Rough route waypoints
|
||||
corridor_width_m: float # Corridor width in meters (e.g., 1000m)
|
||||
zoom: int
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
bool: True if prefetch completed, False on error
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. For each waypoint pair:
|
||||
- Calculate corridor polygon
|
||||
- Determine tiles covering corridor
|
||||
2. Fetch tiles (async, parallel)
|
||||
3. Cache all tiles with flight_id reference
|
||||
|
||||
**Algorithm**:
|
||||
- Use H06 Web Mercator Utils for tile calculations
|
||||
- Parallel fetching (10-20 concurrent requests)
|
||||
- Progress tracking for monitoring
|
||||
|
||||
**Error Conditions**:
|
||||
- Returns `False`: Major error preventing prefetch
|
||||
- Logs warnings for individual tile failures
|
||||
|
||||
**Test Cases**:
|
||||
1. **Simple route**: 10 waypoints → prefetches 50-100 tiles
|
||||
2. **Long route**: 50 waypoints → prefetches 200-500 tiles
|
||||
3. **Partial failure**: Some tiles fail → continues, returns True
|
||||
4. **Complete failure**: All tiles fail → returns False
|
||||
|
||||
---
|
||||
|
||||
### `progressive_fetch(center_lat: float, center_lon: float, grid_sizes: List[int], zoom: int) -> Iterator[Dict[str, np.ndarray]]`
|
||||
|
||||
**Description**: Progressively fetches expanding tile grids for "kidnapped robot" recovery.
|
||||
|
||||
**Called By**:
|
||||
- F11 Failure Recovery Coordinator (progressive search)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
center_lat: float
|
||||
center_lon: float
|
||||
grid_sizes: List[int] # e.g., [1, 4, 9, 16, 25]
|
||||
zoom: int
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
Iterator yielding Dict[str, np.ndarray] for each grid size
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. For each grid_size in sequence:
|
||||
- Fetch tile grid
|
||||
- Yield tiles
|
||||
- If match found by caller, iterator can be stopped
|
||||
|
||||
**Usage Pattern**:
|
||||
```python
|
||||
for tiles in progressive_fetch(lat, lon, [1, 4, 9, 16, 25], 19):
|
||||
if litesam_match_found(tiles):
|
||||
break # Stop expanding search
|
||||
```
|
||||
|
||||
**Test Cases**:
|
||||
1. **Progressive search**: Yields 1, then 4, then 9 tiles
|
||||
2. **Early termination**: Match on 4 tiles → doesn't fetch 9, 16, 25
|
||||
3. **Full search**: No match → fetches all grid sizes
|
||||
|
||||
---
|
||||
|
||||
### `cache_tile(tile_coords: TileCoords, tile_data: np.ndarray) -> bool`
|
||||
|
||||
**Description**: Caches a satellite tile to disk.
|
||||
|
||||
**Called By**:
|
||||
- Internal (after fetching tiles)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
tile_coords: TileCoords:
|
||||
x: int
|
||||
y: int
|
||||
zoom: int
|
||||
tile_data: np.ndarray
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
bool: True if cached successfully
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. Generate cache key from tile_coords
|
||||
2. Serialize tile_data (PNG format)
|
||||
3. Write to disk cache directory
|
||||
4. Update cache index
|
||||
|
||||
**Error Conditions**:
|
||||
- Returns `False`: Disk write error, space full
|
||||
|
||||
**Test Cases**:
|
||||
1. **Cache new tile**: Writes successfully
|
||||
2. **Overwrite existing**: Updates tile
|
||||
3. **Disk full**: Returns False
|
||||
|
||||
---
|
||||
|
||||
### `get_cached_tile(tile_coords: TileCoords) -> Optional[np.ndarray]`
|
||||
|
||||
**Description**: Retrieves a cached tile from disk.
|
||||
|
||||
**Called By**:
|
||||
- Internal (before fetching from API)
|
||||
- F09 Metric Refinement (direct cache lookup)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
tile_coords: TileCoords
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
Optional[np.ndarray]: Tile image or None if not cached
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. Generate cache key
|
||||
2. Check if file exists
|
||||
3. Load and deserialize
|
||||
4. Return tile_data
|
||||
|
||||
**Error Conditions**:
|
||||
- Returns `None`: Not cached, corrupted file
|
||||
|
||||
**Test Cases**:
|
||||
1. **Cache hit**: Returns tile quickly
|
||||
2. **Cache miss**: Returns None
|
||||
3. **Corrupted cache**: Returns None, logs warning
|
||||
|
||||
---
|
||||
|
||||
### `get_tile_grid(center: TileCoords, grid_size: int) -> List[TileCoords]`
|
||||
|
||||
**Description**: Calculates tile coordinates for NxN grid centered on a tile.
|
||||
|
||||
**Called By**:
|
||||
- Internal (for grid fetching)
|
||||
- F11 Failure Recovery Coordinator
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
center: TileCoords
|
||||
grid_size: int # 1, 4, 9, 16, 25
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
List[TileCoords] # List of tile coordinates in grid
|
||||
```
|
||||
|
||||
**Algorithm**:
|
||||
- For grid_size=9 (3×3): tiles from center-1 to center+1 in both x and y
|
||||
- For grid_size=16 (4×4): asymmetric grid with center slightly off-center
|
||||
|
||||
**Test Cases**:
|
||||
1. **1-tile grid**: Returns [center]
|
||||
2. **4-tile grid (2×2)**: Returns 4 tiles
|
||||
3. **9-tile grid (3×3)**: Returns 9 tiles centered
|
||||
4. **25-tile grid (5×5)**: Returns 25 tiles
|
||||
|
||||
---
|
||||
|
||||
### `compute_tile_coords(lat: float, lon: float, zoom: int) -> TileCoords`
|
||||
|
||||
**Description**: Converts GPS coordinates to tile coordinates.
|
||||
|
||||
**Called By**:
|
||||
- All methods that need tile coordinates from GPS
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
lat: float
|
||||
lon: float
|
||||
zoom: int
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
TileCoords:
|
||||
x: int
|
||||
y: int
|
||||
zoom: int
|
||||
```
|
||||
|
||||
**Algorithm** (Web Mercator):
|
||||
```python
|
||||
n = 2 ** zoom
|
||||
x = int((lon + 180) / 360 * n)
|
||||
lat_rad = lat * π / 180
|
||||
y = int((1 - log(tan(lat_rad) + sec(lat_rad)) / π) / 2 * n)
|
||||
```
|
||||
|
||||
**Test Cases**:
|
||||
1. **Ukraine coordinates**: Produces valid tile coords
|
||||
2. **Edge cases**: lat=0, lon=0, lat=90, lon=180
|
||||
|
||||
---
|
||||
|
||||
### `expand_search_grid(center: TileCoords, current_size: int, new_size: int) -> List[TileCoords]`
|
||||
|
||||
**Description**: Returns only NEW tiles when expanding from current grid to larger grid.
|
||||
|
||||
**Called By**:
|
||||
- F11 Failure Recovery Coordinator (progressive search optimization)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
center: TileCoords
|
||||
current_size: int # e.g., 4
|
||||
new_size: int # e.g., 9
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
List[TileCoords] # Only tiles not in current_size grid
|
||||
```
|
||||
|
||||
**Purpose**: Avoid re-fetching tiles already tried in smaller grid.
|
||||
|
||||
**Test Cases**:
|
||||
1. **4→9 expansion**: Returns 5 new tiles (9-4)
|
||||
2. **9→16 expansion**: Returns 7 new tiles
|
||||
3. **1→4 expansion**: Returns 3 new tiles
|
||||
|
||||
---
|
||||
|
||||
### `compute_tile_bounds(tile_coords: TileCoords) -> TileBounds`
|
||||
|
||||
**Description**: Computes GPS bounding box of a tile.
|
||||
|
||||
**Called By**:
|
||||
- F09 Metric Refinement (for homography calculations)
|
||||
- H06 Web Mercator Utils (shared calculation)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
tile_coords: TileCoords
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
TileBounds:
|
||||
nw: GPSPoint # North-West corner
|
||||
ne: GPSPoint # North-East corner
|
||||
sw: GPSPoint # South-West corner
|
||||
se: GPSPoint # South-East corner
|
||||
center: GPSPoint
|
||||
gsd: float # Ground Sampling Distance (meters/pixel)
|
||||
```
|
||||
|
||||
**Algorithm**:
|
||||
- Inverse Web Mercator projection
|
||||
- GSD calculation: `156543.03392 * cos(lat * π/180) / 2^zoom`
|
||||
|
||||
**Test Cases**:
|
||||
1. **Zoom 19 at Ukraine**: GSD ≈ 0.3 m/pixel
|
||||
2. **Tile bounds**: Valid GPS coordinates
|
||||
|
||||
---
|
||||
|
||||
### `clear_flight_cache(flight_id: str) -> bool`
|
||||
|
||||
**Description**: Clears cached tiles for a completed flight.
|
||||
|
||||
**Called By**:
|
||||
- F02 Flight Manager (cleanup after flight completion)
|
||||
|
||||
**Input**:
|
||||
```python
|
||||
flight_id: str
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```python
|
||||
bool: True if cleared successfully
|
||||
```
|
||||
|
||||
**Processing Flow**:
|
||||
1. Find all tiles associated with flight_id
|
||||
2. Delete tile files
|
||||
3. Update cache index
|
||||
|
||||
**Test Cases**:
|
||||
1. **Clear flight cache**: Removes all associated tiles
|
||||
2. **Non-existent flight**: Returns True (no-op)
|
||||
|
||||
## Integration Tests
|
||||
|
||||
### Test 1: Prefetch and Retrieval
|
||||
1. prefetch_route_corridor() with 20 waypoints
|
||||
2. Verify tiles cached
|
||||
3. get_cached_tile() for each tile → all hit cache
|
||||
4. clear_flight_cache() → cache cleared
|
||||
|
||||
### Test 2: Progressive Search Simulation
|
||||
1. progressive_fetch() with [1, 4, 9, 16, 25]
|
||||
2. Simulate match on 9 tiles
|
||||
3. Verify only 1, 4, 9 fetched (not 16, 25)
|
||||
|
||||
### Test 3: Grid Expansion
|
||||
1. fetch_tile_grid(4) → 4 tiles
|
||||
2. expand_search_grid(4, 9) → 5 new tiles
|
||||
3. Verify no duplicate fetches
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
### Performance
|
||||
- **fetch_tile**: < 200ms (cached: < 10ms)
|
||||
- **fetch_tile_grid(9)**: < 1 second
|
||||
- **prefetch_route_corridor**: < 30 seconds for 500 tiles
|
||||
- **Cache lookup**: < 5ms
|
||||
|
||||
### Scalability
|
||||
- Cache 10,000+ tiles per flight
|
||||
- Support 100 concurrent tile fetches
|
||||
- Handle 10GB+ cache size
|
||||
|
||||
### Reliability
|
||||
- Retry failed HTTP requests (3 attempts)
|
||||
- Graceful degradation on partial failures
|
||||
- Cache corruption recovery
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Internal Components
|
||||
- **H06 Web Mercator Utils**: Tile coordinate calculations
|
||||
|
||||
### External Dependencies
|
||||
- **Satellite Provider API**: HTTP tile source
|
||||
- **requests** or **httpx**: HTTP client
|
||||
- **numpy**: Image handling
|
||||
- **opencv-python**: Image I/O
|
||||
- **diskcache**: Persistent cache
|
||||
|
||||
## Data Models
|
||||
|
||||
### TileCoords
|
||||
```python
|
||||
class TileCoords(BaseModel):
|
||||
x: int
|
||||
y: int
|
||||
zoom: int
|
||||
```
|
||||
|
||||
### TileBounds
|
||||
```python
|
||||
class TileBounds(BaseModel):
|
||||
nw: GPSPoint
|
||||
ne: GPSPoint
|
||||
sw: GPSPoint
|
||||
se: GPSPoint
|
||||
center: GPSPoint
|
||||
gsd: float # meters/pixel
|
||||
```
|
||||
|
||||
### CacheConfig
|
||||
```python
|
||||
class CacheConfig(BaseModel):
|
||||
cache_dir: str = "./satellite_cache"
|
||||
max_size_gb: int = 50
|
||||
eviction_policy: str = "lru"
|
||||
ttl_days: int = 30
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user