azaion/satellite-provider
1
0
Fork 0
mirror of https://github.com/azaion/satellite-provider.git synced 2026-04-22 03:56:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

documentation

Browse Source
This commit is contained in:
Anton Martynenko
2025-11-20 12:41:07 +01:00
parent bf2030e3c6
commit 604f0e070d
3 changed files with 634 additions and 950 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+291
View File
@@ -0,0 +1,291 @@
# Satellite Provider - Agent Documentation
## System Overview
This is a .NET 8.0 ASP.NET Web API service that downloads, stores, and manages satellite imagery tiles from Google Maps. The service supports region-based tile requests, route planning with intermediate points, and geofencing capabilities.
## Tech Stack
- **.NET 8.0** with ASP.NET Core Web API
- **PostgreSQL** database (via Docker)
- **Dapper** for data access (ORM)
- **DbUp** for database migrations
- **Serilog** for logging
- **Swagger/OpenAPI** for API documentation
- **XUnit** and **Moq** for testing
- **ImageSharp** for image processing
- **Docker** for containerization
## Architecture
### Project Structure
```
SatelliteProvider.Api/ # Web API layer (Program.cs)
SatelliteProvider.Common/ # DTOs, interfaces, configs, utils
SatelliteProvider.DataAccess/ # Database entities, repositories, migrations
SatelliteProvider.Services/ # Business logic implementations
SatelliteProvider.Tests/ # Unit tests
SatelliteProvider.IntegrationTests/ # Integration tests (console app)
```
### Key Components
1. **API Layer** (`Program.cs`)
- Minimal API endpoints
- Dependency injection configuration
- Swagger documentation
- Database migration on startup
2. **Services**
- `TileService`: Downloads and stores individual tiles
- `RegionService`: Manages region requests and processes tiles for regions
- `RouteService`: Creates routes with intermediate points and manages route regions
- `GoogleMapsDownloaderV2`: Handles actual tile downloads from Google Maps
- `RegionProcessingService`: Background hosted service for processing region requests
- `RouteProcessingService`: Background hosted service for processing route map requests
3. **Data Access**
- Repositories: `TileRepository`, `RegionRepository`, `RouteRepository`
- Entities: `TileEntity`, `RegionEntity`, `RouteEntity`, `RoutePointEntity`
- Migrations in `SatelliteProvider.DataAccess/Migrations/` (numbered SQL files)
4. **Async Processing**
- Queue-based processing via `IRegionRequestQueue`
- Background services process requests asynchronously
- Status tracking via database
## Database Schema
### Tables
**tiles**
- Stores downloaded satellite tiles
- Fields: id, zoom_level, latitude, longitude, tile_size_meters, tile_size_pixels, image_type, maps_version, version, file_path, created_at, updated_at
- Composite index on (latitude, longitude, tile_size_meters, zoom_level)
**regions**
- Stores region requests and their processing status
- Fields: id, latitude, longitude, size_meters, zoom_level, status, stitch_tiles, stitched_image_path, csv_file_path, summary_file_path, tiles_downloaded, tiles_reused, created_at, updated_at
- Status values: "pending", "processing", "completed", "failed"
**routes**
- Stores route definitions
- Fields: id, name, description, region_size_meters, zoom_level, total_distance_meters, total_points, request_maps, create_tiles_zip, tiles_zip_path, geofence_polygons, created_at, updated_at
**route_points**
- Stores all points (original and interpolated) for routes
- Fields: id, route_id, sequence_number, latitude, longitude, point_type, segment_index, distance_from_previous, created_at
- Point types: "original", "intermediate"
**route_regions**
- Junction table linking routes to regions for map tile requests
- Fields: route_id, region_id, inside_geofence, created_at
### Migration Strategy
- SQL-based migrations managed by DbUp
- Embedded resources in assembly
- Sequential numbered files (001_, 002_, etc.)
- Runs automatically on application startup
## API Endpoints
### Tile Management
- `GET /api/satellite/tiles/latlon` - Download single tile by lat/lon and zoom level
- `GET /api/satellite/tiles/mgrs` - Get tiles by MGRS coordinates (stub)
### Region Management
- `POST /api/satellite/request` - Request tiles for a region (async processing)
- `GET /api/satellite/region/{id}` - Get region status and file paths
### Route Management
- `POST /api/satellite/route` - Create route with intermediate points
- `GET /api/satellite/route/{id}` - Get route information
### Image Upload
- `POST /api/satellite/upload` - Upload image with metadata (stub)
## Key Patterns and Conventions
### Naming
- Database: snake_case (tiles, route_points)
- C# Code: PascalCase for classes, camelCase for parameters
- Entities suffix: `Entity` (TileEntity, RegionEntity)
- Interfaces prefix: `I` (ITileService, IRegionService)
### Configuration
Configuration sections in appsettings.json:
- `MapConfig`: Google Maps API settings
- `StorageConfig`: File storage paths (tiles, ready directories)
- `ProcessingConfig`: Concurrency and processing limits
- `ConnectionStrings`: Database connection
### File Storage
- Tiles stored in: `./tiles/{zoomLevel}/{x}/{y}.jpg`
- CSV outputs: `./ready/region_{id}_ready.csv`
- Summary files: `./ready/region_{id}_summary.txt`
- Stitched images: `./ready/region_{id}_stitched.jpg`
- Route zip files: `./ready/route_{id}_tiles.zip`
### Async Processing Flow
1. Client posts request to API
2. Request queued in `IRegionRequestQueue`
3. Background service picks up request
4. Downloads/processes tiles
5. Updates database status
6. Creates output files
### Geofencing
- Routes can define multiple polygon geofences
- Each geofence has an array of lat/lon points
- Route regions are marked if they fall inside/outside geofences
- Used to filter which regions to process for map tiles
### Route Processing
- Routes broken into segments between original points
- Intermediate points calculated every ~200 meters
- Each point becomes center of a region request
- Distance calculations use Haversine formula
## Development Workflow
### Running Locally
**Start service and dependencies:**
```bash
docker-compose up --build
```
**Run tests:**
```bash
docker-compose -f docker-compose.yml -f docker-compose.tests.yml up --build --abort-on-container-exit
```
### Important Notes
1. **DO NOT run `dotnet build` or `dotnet restore` via terminal tools** - these commands hang. Ask user to run manually.
2. **Test verification:** Primary method is via Docker Compose with tests.
3. **File system structure:** Service expects mounted volumes for:
- `./tiles` - tile storage
- `./ready` - output files
- `./logs` - application logs
4. **Database:** Auto-creates schema on startup via migrations.
### Configuration Values
Development defaults:
- PostgreSQL: localhost:5432, user/pass: postgres/postgres
- API: http://localhost:5100
- Max zoom level: 20
- Default zoom level: 18
- Queue capacity: 1000
- Max concurrent downloads: 4
- Max concurrent regions: 20
## Testing
### Integration Tests
- Console application in `SatelliteProvider.IntegrationTests`
- Tests run in Docker container
- Tests cover:
- Tile downloads
- Region requests and processing
- Route creation and point interpolation
- Geofencing logic
- Extended routes with map requests
### Test Helpers
`RouteTestHelpers.cs` provides utilities:
- Waiting for region completion
- Creating test geofence polygons
- Common test data builders
## Common Tasks
### Adding New Migration
1. Create numbered SQL file in `SatelliteProvider.DataAccess/Migrations/`
2. Set Build Action to `EmbeddedResource`
3. Follow naming: `NNN_DescriptiveName.sql`
4. Migration runs automatically on next startup
### Adding New API Endpoint
1. Add endpoint mapping in `Program.cs`
2. Add handler method (async preferred)
3. Define request/response DTOs
4. Add Swagger documentation with `.WithOpenApi()`
5. Update interfaces if adding service methods
### Adding New Configuration
1. Add config class in `SatelliteProvider.Common/Configs/`
2. Register in `Program.cs` with `builder.Services.Configure<T>()`
3. Add section to `appsettings.json` and `appsettings.Development.json`
4. Inject via `IOptions<T>` in services
## Code Style Guidelines
- No comments in code (self-documenting code preferred)
- No excessive logging (only exceptions unless specifically needed)
- Simple, concise solutions preferred
- Minimal code changes related to task
- Avoid renaming database objects without confirmation
- Check for duplicate code before adding new functionality
- Consider environment differences (dev/prod)
## Dependencies and Versions
Key packages (all .NET 8.0):
- Microsoft.AspNetCore.OpenApi 8.0.21
- Swashbuckle.AspNetCore 6.6.2
- Serilog.AspNetCore 8.0.3
- SixLabors.ImageSharp 3.1.11
- Newtonsoft.Json 13.0.4
- Dapper (check DataAccess csproj)
- Npgsql (check DataAccess csproj)
- DbUp (check DataAccess csproj)
## Outstanding TODOs
From goal.md:
1. ✅ Add geo fences (2 regions) - IMPLEMENTED
2. ✅ Add parameter to zip resulting tiles, 50 mb max - IMPLEMENTED
3. ✅ Implement API to download tile - lat, lon and zoom level - IMPLEMENTED
4. ⚠️ Implement parallel tiles fetching from google maps - PARTIAL (session token reuse implemented)
## Docker Structure
**docker-compose.yml:**
- PostgreSQL database service
- SatelliteProvider.Api service
- Shared network
- Volume mounts for tiles, ready, logs
**docker-compose.tests.yml:**
- Integration tests service
- Depends on api and db services
- Runs and exits after completion
## Troubleshooting
**Database connection issues:**
- Check PostgreSQL container is running
- Verify connection string in appsettings
- Check network connectivity between containers
**Tile download failures:**
- Check Google Maps API key in MapConfig
- Verify internet connectivity
- Check rate limiting settings
**Test failures:**
- Ensure all services are up before tests run
- Check logs in `./logs/` directory
- Verify database migrations completed
**Queue processing stuck:**
- Check RegionProcessingService and RouteProcessingService are running
- Verify queue capacity not exceeded
- Check database connection for status updates
README.md
+343 -249
View File
@@ -1,151 +1,107 @@
# Satellite Provider API
A RESTful API service for downloading, caching, and processing satellite imagery from Google Maps. The service provides endpoints for downloading individual tiles and processing entire regions with intelligent tile caching and versioning.
A .NET 8.0 microservice for downloading, managing, and serving satellite imagery tiles from Google Maps. Supports region-based tile requests, route planning with automatic intermediate point generation, and geofencing capabilities.
## Features
- **Satellite Tile Downloads**: Download individual satellite tiles from Google Maps at various zoom levels
- **Region Processing**: Request satellite imagery for entire geographic regions
- **Intelligent Tile Caching**: Automatic tile reuse with year-based versioning
- **Image Stitching**: Combine multiple tiles into seamless regional images
- **CSV Export**: Generate coordinate mappings for downloaded tiles
- **Summary Reports**: Detailed processing statistics and metadata
- **Tile Management**: Download and cache satellite imagery tiles from Google Maps
- **Region Requests**: Request tiles for specific geographic areas (100m - 10km squares)
- **Route Planning**: Create routes with automatic intermediate point interpolation (~200m intervals)
- **Geofencing**: Define polygon boundaries to filter regions inside/outside zones
- **Image Stitching**: Combine tiles into single images for regions
- **Tile Packaging**: Create ZIP archives of route tiles (max 50MB)
- **Async Processing**: Queue-based background processing for large requests
- **Caching**: Reuse previously downloaded tiles to minimize redundant downloads
- **REST API**: OpenAPI/Swagger documented endpoints
## Tech Stack
## Prerequisites
- **.NET 8.0** - ASP.NET Web API
- **PostgreSQL** - Primary data storage
- **Dapper** - Lightweight ORM for database access
- **DbUp** - Database migrations
- **Serilog** - Structured logging
- **ImageSharp** - Image processing and stitching
- **Docker** - Containerized deployment
- Docker and Docker Compose
- .NET 8.0 SDK (for local development)
- PostgreSQL (provided via Docker)
## Tile Versioning Strategy
## Quick Start
### Year-Based Tile Versions
### Run the Service
The service implements a **year-based versioning strategy** for cached satellite tiles. This approach is based on the assumption that Google Maps satellite imagery updates infrequently, typically once per year or less.
#### How It Works
1. **Version Assignment**: When a tile is downloaded, it is assigned a version number equal to the current year (e.g., 2025).
2. **Unique Constraint**: Tiles are uniquely identified by the combination of:
- Latitude (center point)
- Longitude (center point)
- Zoom level
- Tile size in meters
- **Version** (year)
3. **Cache Behavior**:
- **Same Year**: If requesting a tile that was downloaded in the current year, the cached tile is reused
- **New Year**: When the calendar year changes, new tiles are downloaded even for the same coordinates
- **Historical Data**: Old tile versions remain in the database for historical reference
#### Example
```
Request in 2025:
- Downloads tile at (47.461747, 37.647063) zoom 18 → version = 2025
- Saves to database with version 2025
Second request in 2025:
- Finds tile at (47.461747, 37.647063) zoom 18 version 2025
- Reuses cached tile (no download)
Request in 2026:
- Looks for tile at (47.461747, 37.647063) zoom 18 version 2026
- Not found (only 2025 version exists)
- Downloads new tile → version = 2026
- Both 2025 and 2026 versions now exist in database
```bash
docker-compose up --build
```
#### Benefits
The API will be available at `http://localhost:5100`
- **Automatic Updates**: Ensures fresh satellite imagery each calendar year
- **Historical Archive**: Maintains previous years' imagery for comparison
- **Efficient Caching**: Maximizes cache hit rate within the same year
- **Simple Logic**: No complex timestamp comparisons or manual cache invalidation
Swagger documentation: `http://localhost:5100/swagger`
#### Database Schema
### Run with Tests
```sql
CREATE TABLE tiles (
id UUID PRIMARY KEY,
zoom_level INT NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
tile_size_meters DOUBLE PRECISION NOT NULL,
tile_size_pixels INT NOT NULL,
image_type VARCHAR(10) NOT NULL,
maps_version VARCHAR(50),
version INT NOT NULL, -- Year-based version
file_path VARCHAR(500) NOT NULL,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE UNIQUE INDEX idx_tiles_unique_location
ON tiles(latitude, longitude, zoom_level, tile_size_meters, version);
```bash
docker-compose -f docker-compose.yml -f docker-compose.tests.yml up --build --abort-on-container-exit
```
This command:
- Builds and starts all services
- Runs integration tests
- Exits when tests complete
## Architecture
### Project Structure
The service follows a layered architecture:
```
SatelliteProvider/
├── SatelliteProvider.Api/ # REST API endpoints
├── SatelliteProvider.Common/ # Shared DTOs and interfaces
├── SatelliteProvider.DataAccess/ # Database access and migrations
├── SatelliteProvider.Services/ # Business logic
├── SatelliteProvider.Tests/ # Unit tests
└── SatelliteProvider.IntegrationTests/ # Integration tests
┌─────────────────────────────────────┐
│ API Layer (ASP.NET Core) │ HTTP endpoints
├─────────────────────────────────────┤
│ Services (Business Logic) │ Processing, validation
├─────────────────────────────────────┤
│ Data Access (Dapper + Repos) │ Database operations
├─────────────────────────────────────┤
│ PostgreSQL Database │ Persistence
└─────────────────────────────────────┘
```
### Key Components
### Projects
- **GoogleMapsDownloaderV2**: Downloads satellite tiles from Google Maps API
- **TileService**: Manages tile caching, retrieval, and storage with version control
- **RegionService**: Processes region requests, stitches tiles, generates outputs
- **RegionRequestQueue**: Background processing queue for region requests
- **DatabaseMigrator**: Manages database schema migrations
- **SatelliteProvider.Api**: Web API and endpoints
- **SatelliteProvider.Services**: Business logic and processing
- **SatelliteProvider.DataAccess**: Database access and migrations
- **SatelliteProvider.Common**: Shared DTOs, interfaces, utilities
- **SatelliteProvider.IntegrationTests**: Integration test suite
- **SatelliteProvider.Tests**: Unit tests
## API Endpoints
### Download Single Tile
```http
POST /api/satellite/tiles/download
Content-Type: application/json
{
"latitude": 47.461747,
"longitude": 37.647063,
"zoomLevel": 18
}
GET /api/satellite/tiles/latlon?Latitude={lat}&Longitude={lon}&ZoomLevel={zoom}
```
Downloads a single tile at specified coordinates and zoom level.
**Parameters:**
- `Latitude` (double): Center latitude
- `Longitude` (double): Center longitude
- `ZoomLevel` (int): Zoom level (higher = more detail, max ~20)
**Response:**
```json
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"id": "uuid",
"zoomLevel": 18,
"latitude": 47.462451,
"longitude": 37.646027,
"tileSizeMeters": 103.35,
"latitude": 37.7749,
"longitude": -122.4194,
"tileSizeMeters": 38.2,
"tileSizePixels": 256,
"imageType": "jpg",
"mapsVersion": "downloaded_2025-10-29",
"version": 2025,
"filePath": "./tiles/tile_18_158485_91707_20251029103256.jpg",
"createdAt": "2025-10-29T10:32:56Z",
"updatedAt": "2025-10-29T10:32:56Z"
"mapsVersion": "downloaded_2024-11-20",
"version": 2024,
"filePath": "18/158/90.jpg",
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:30:00Z"
}
```
### Request Region Processing
### Request Region Tiles
```http
POST /api/satellite/request
@@ -153,63 +109,174 @@ Content-Type: application/json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"latitude": 47.461747,
"longitude": 37.647063,
"sizeMeters": 200,
"zoomLevel": 18
"latitude": 37.7749,
"longitude": -122.4194,
"sizeMeters": 500,
"zoomLevel": 18,
"stitchTiles": false
}
```
### Check Region Status
Requests tiles for a square region. Processing happens asynchronously.
```http
GET /api/satellite/region/{regionId}
```
**Parameters:**
- `id` (guid): Unique identifier for the region
- `latitude` (double): Center latitude
- `longitude` (double): Center longitude
- `sizeMeters` (double): Square side length (100-10000 meters)
- `zoomLevel` (int): Zoom level (default: 18)
- `stitchTiles` (bool): Create single stitched image (default: false)
**Response:**
```json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "completed",
"csvFilePath": "./ready/region_{id}_ready.csv",
"summaryFilePath": "./ready/region_{id}_summary.txt",
"tilesDownloaded": 5,
"tilesReused": 4,
"createdAt": "2025-10-29T10:32:56Z",
"updatedAt": "2025-10-29T10:32:57Z"
"status": "pending",
"csvFilePath": null,
"summaryFilePath": null,
"tilesDownloaded": 0,
"tilesReused": 0,
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:30:00Z"
}
```
### Get Region Status
```http
GET /api/satellite/region/{id}
```
Check processing status and get file paths when complete.
**Status Values:**
- `pending`: Queued for processing
- `processing`: Currently downloading tiles
- `completed`: All tiles ready, files created
- `failed`: Processing failed
**Response (completed):**
```json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "completed",
"csvFilePath": "./ready/region_3fa85f64-5717-4562-b3fc-2c963f66afa6_ready.csv",
"summaryFilePath": "./ready/region_3fa85f64-5717-4562-b3fc-2c963f66afa6_summary.txt",
"tilesDownloaded": 12,
"tilesReused": 4,
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:32:15Z"
}
```
### Create Route
```http
POST /api/satellite/route
Content-Type: application/json
{
"id": "7fa85f64-5717-4562-b3fc-2c963f66afa6",
"name": "City Tour Route",
"description": "Downtown to waterfront",
"regionSizeMeters": 200,
"zoomLevel": 18,
"points": [
{ "latitude": 37.7749, "longitude": -122.4194, "name": "Start" },
{ "latitude": 37.7849, "longitude": -122.4094, "name": "Middle" },
{ "latitude": 37.7949, "longitude": -122.3994, "name": "End" }
],
"geofences": {
"polygons": [
{
"name": "Restricted Area",
"points": [
{ "latitude": 37.775, "longitude": -122.420 },
{ "latitude": 37.776, "longitude": -122.420 },
{ "latitude": 37.776, "longitude": -122.418 },
{ "latitude": 37.775, "longitude": -122.418 }
]
}
],
"direction": "inside"
},
"requestMaps": true,
"createTilesZip": true
}
```
Creates a route with intermediate points calculated automatically.
**Parameters:**
- `id` (guid): Route identifier
- `name` (string): Route name
- `description` (string): Optional description
- `regionSizeMeters` (double): Size of region at each point
- `zoomLevel` (int): Zoom level for tiles
- `points` (array): Original route points
- `geofences` (object): Optional polygon boundaries
- `polygons`: Array of polygon definitions
- `direction`: "inside" or "outside" (filter regions)
- `requestMaps` (bool): Automatically request tiles for route regions
- `createTilesZip` (bool): Create ZIP file of all tiles
**Response:**
```json
{
"id": "7fa85f64-5717-4562-b3fc-2c963f66afa6",
"name": "City Tour Route",
"description": "Downtown to waterfront",
"regionSizeMeters": 200,
"zoomLevel": 18,
"totalDistanceMeters": 2486.5,
"totalPoints": 15,
"points": [
{
"sequenceNumber": 0,
"latitude": 37.7749,
"longitude": -122.4194,
"pointType": "original",
"segmentIndex": 0,
"distanceFromPrevious": 0
},
{
"sequenceNumber": 1,
"latitude": 37.7765,
"longitude": -122.4177,
"pointType": "intermediate",
"segmentIndex": 0,
"distanceFromPrevious": 198.3
}
],
"regions": [
"region-id-1",
"region-id-2"
],
"tilesZipPath": "./ready/route_7fa85f64-5717-4562-b3fc-2c963f66afa6_tiles.zip",
"createdAt": "2024-11-20T10:30:00Z"
}
```
### Get Route
```http
GET /api/satellite/route/{id}
```
Retrieve route details including all interpolated points.
## Configuration
### Environment Variables
- `ASPNETCORE_ENVIRONMENT`: Development/Production
- `ASPNETCORE_URLS`: HTTP binding address (default: http://+:8080)
- `ConnectionStrings__DefaultConnection`: PostgreSQL connection string
- `MapConfig__ApiKey`: Google Maps API key
### appsettings.json
Configuration is managed through `appsettings.json`:
```json
{
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft.AspNetCore": "Warning"
}
},
"WriteTo": [
{ "Name": "Console" },
{
"Name": "File",
"Args": {
"path": "./logs/satellite-provider-.log",
"rollingInterval": "Day"
}
}
]
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Database=satelliteprovider;Username=postgres;Password=postgres"
},
"MapConfig": {
"Service": "GoogleMaps",
"ApiKey": "your-google-maps-api-key"
},
"StorageConfig": {
"TilesDirectory": "./tiles",
@@ -217,160 +284,187 @@ GET /api/satellite/region/{regionId}
},
"ProcessingConfig": {
"MaxConcurrentDownloads": 4,
"DefaultZoomLevel": 18,
"QueueCapacity": 100
"MaxConcurrentRegions": 20,
"DefaultZoomLevel": 20,
"QueueCapacity": 1000,
"DelayBetweenRequestsMs": 50,
"SessionTokenReuseCount": 100
}
}
```
## Running with Docker
### Environment Variables (Docker)
### Prerequisites
Override settings via environment variables in `docker-compose.yml`:
- Docker and Docker Compose
- Google Maps API key with Tile API enabled
### Setup
1. Create `.env` file in project root:
```bash
GOOGLE_MAPS_API_KEY=your_api_key_here
```yaml
environment:
- ConnectionStrings__DefaultConnection=Host=db;Database=satelliteprovider;Username=postgres;Password=postgres
- MapConfig__ApiKey=your-api-key-here
```
2. Start services:
```bash
docker-compose up -d
## File Storage
### Tiles
Downloaded tiles are stored in:
```
./tiles/{zoomLevel}/{x}/{y}.jpg
```
3. View logs:
```bash
docker-compose logs -f api
### Output Files
**CSV File** (`region_{id}_ready.csv`):
Comma-separated list of tiles covering the region, ordered top-left to bottom-right:
```csv
latitude,longitude,filepath
37.7750,-122.4195,18/158/90.jpg
37.7750,-122.4193,18/158/91.jpg
```
4. Stop services:
```bash
docker-compose down
**Summary File** (`region_{id}_summary.txt`):
Processing statistics:
```
Region: 3fa85f64-5717-4562-b3fc-2c963f66afa6
Size: 500m x 500m
Center: 37.7749, -122.4194
Zoom Level: 18
Tiles Downloaded: 12
Tiles Reused: 4
Total Tiles: 16
Processing Time: 3.5s
```
### Docker Compose Services
**Stitched Image** (`region_{id}_stitched.jpg`):
Single combined image when `stitchTiles: true`
- **postgres**: PostgreSQL database server
- **api**: Satellite Provider API service
**Route ZIP** (`route_{id}_tiles.zip`):
ZIP archive of all tiles for route when `createTilesZip: true` (max 50MB)
### Volume Mounts
## Database Schema
- `./tiles`: Downloaded tile images
- `./ready`: Processed region outputs (CSV, stitched images, summaries)
- `./logs`: Application logs
### Tables
**tiles**: Cached satellite tiles
- Unique per (latitude, longitude, tile_size_meters, zoom_level)
- Tracks maps version for cache invalidation
**regions**: Region requests and processing status
- Status tracking: pending → processing → completed/failed
- Links to output files when complete
**routes**: Route definitions
- Stores original configuration
- Links to calculated points and regions
**route_points**: All points (original + interpolated)
- Sequenced in route order
- Type: "original" or "intermediate"
- Includes distance calculations
**route_regions**: Junction table
- Links routes to regions for tile requests
- Tracks geofence status (inside/outside)
## Development
### Local Development Setup
### Building Locally
1. Install .NET 8.0 SDK
2. Install PostgreSQL 16
3. Set up database:
```bash
createdb satelliteprovider
```
4. Update connection string in `appsettings.Development.json`
5. Run migrations (automatic on startup)
6. Start API:
```bash
cd SatelliteProvider.Api
dotnet build
dotnet run
```
### Running Tests
**Unit Tests:**
```bash
cd SatelliteProvider.Tests
dotnet test
```
### Integration Tests
**Integration Tests:**
```bash
docker-compose -f docker-compose.tests.yml up --build
docker-compose -f docker-compose.yml -f docker-compose.tests.yml up --build --abort-on-container-exit
```
## Output Files
### Region Processing Outputs
For each processed region, the service generates:
1. **CSV File** (`region_{id}_ready.csv`):
- Tile coordinates and file paths
- Ordered by latitude (descending) and longitude (ascending)
2. **Stitched Image** (`region_{id}_stitched.jpg`):
- Combined satellite imagery for the entire region
- Red crosshair marking the requested center point
3. **Summary Report** (`region_{id}_summary.txt`):
- Region metadata and coordinates
- Processing statistics (tiles downloaded, tiles reused, total tiles)
- Processing time and timestamps
- File paths for all generated outputs
### Example Summary Report
### Database Migrations
Migrations run automatically on startup. SQL files are located in:
```
Region Processing Summary
========================
Region ID: 5d218e0d-92bc-483c-9e88-fd79200b84e6
Center: 47.461747, 37.647063
Size: 200 meters
Zoom Level: 18
Processing Statistics:
- Tiles Downloaded: 5
- Tiles Reused from Cache: 4
- Total Tiles: 9
- Processing Time: 0.50 seconds
- Started: 2025-10-29 10:32:56 UTC
- Completed: 2025-10-29 10:32:57 UTC
Files Created:
- CSV: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_ready.csv
- Stitched Image: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_stitched.jpg
- Summary: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_summary.txt
SatelliteProvider.DataAccess/Migrations/
```
## Zoom Levels
To add a new migration:
1. Create `NNN_DescriptiveName.sql` (increment number)
2. Set Build Action to `EmbeddedResource`
3. Restart application
Supported zoom levels: **15, 16, 17, 18, 19**
### Logging
- **Zoom 15**: ~2,600m per tile (larger areas, less detail)
- **Zoom 16**: ~1,300m per tile
- **Zoom 17**: ~650m per tile
- **Zoom 18**: ~103m per tile (recommended for most uses)
- **Zoom 19**: ~52m per tile (maximum detail)
Logs are written to:
- Console (stdout)
- `./logs/satellite-provider-YYYYMMDD.log`
Log level can be adjusted in `appsettings.json` under `Serilog:MinimumLevel`.
## Performance Considerations
### Tile Caching
- Tiles are cached indefinitely (no expiration)
- Reused across multiple region/route requests
- Reduces redundant downloads significantly
- First request for a region downloads all required tiles
- Subsequent requests (same year) reuse cached tiles
- Typical 3x3 region (9 tiles) at zoom 18: ~300ms first request, ~50ms cached
### Concurrent Processing
- Configurable concurrent downloads (default: 4)
- Configurable concurrent regions (default: 20)
- Queue capacity: 1000 requests
### Region Processing
### Rate Limiting
- Delay between requests: 50ms default
- Session token reuse: 100 tiles per token
- Helps avoid Google Maps rate limits
- Processed asynchronously in background queue
- Status polling recommended at 1-second intervals
- Processing time depends on:
- Number of tiles required
- Cache hit rate
- Network latency to Google Maps API
- Image stitching complexity
### Geofencing
- Uses point-in-polygon algorithm
- Reduces unnecessary tile downloads
- Spatial index on polygon data
## Troubleshooting
### Service won't start
- Check Docker is running
- Verify ports 5100 and 5432 are available
- Check logs: `docker-compose logs api`
### Tiles not downloading
- Verify Google Maps API key is configured
- Check internet connectivity
- Review logs for HTTP errors
- Check rate limiting settings
### Region stuck in "processing"
- Check background service is running: `docker-compose logs api`
- Verify database connection
- Check queue capacity not exceeded
- Review logs for exceptions
### Tests failing
- Ensure all containers are healthy before tests run
- Check test container logs: `docker-compose logs tests`
- Verify database migrations completed
- Check network connectivity between containers
### Out of disk space
- Tiles accumulate in `./tiles/` directory
- Implement cleanup strategy for old tiles
- Monitor disk usage
## License
This project is proprietary software. All rights reserved.
This project is proprietary software.
## Support
For issues, questions, or feature requests, please contact the development team.
For issues and questions, please contact the development team.
plan.md
-701
View File
@@ -1,701 +0,0 @@
# Satellite Provider Service - Implementation Plan
## Executive Summary
This document outlines the detailed implementation plan for the Satellite Provider Service, which downloads satellite map tiles from Google Maps, stores them with metadata in PostgreSQL, and provides an API for requesting and retrieving map regions.
## Current Implementation Status (Updated 2025-11-01)
### ✅ Completed Features (Phases 1-7)
- **Database Layer**: PostgreSQL with Dapper ORM, DbUp migrations
- **Tile Management**: Download, cache, and store satellite tiles with year-based versioning
- **Region Processing**: Background queue processing with image stitching
- **API Endpoints**: Tile download, region request, and status endpoints
- **Docker Deployment**: Full containerization with docker-compose
- **Integration Tests**: Console-based test suite
- **Documentation**: Comprehensive README with API docs
### ⏳ In Progress (Phase 8)
- **Route Planning**: Split routes into overlapping regions with 200m spacing (NEW REQUIREMENT)
### Key Enhancements Implemented
1. **Year-Based Tile Versioning**: Automatic tile refresh each calendar year
2. **Image Stitching**: Combines tiles into seamless regional images with coordinate markers
3. **Comprehensive Error Handling**: Rate limiting, timeouts, network errors
4. **Rich Summary Reports**: Detailed statistics and metadata for each region
## 1. Code Analysis
### 1.1 Legacy Code Comparison
**Legacy Code (Azaion.Common/Services/SatelliteDownloader.cs):**
- Downloads tiles at 256x256 pixels (Google Maps standard)
- Composes tiles into one large image
- Splits and resizes to 512x512 TIFF files
- Uses MediatR for status updates
- Stores tiles in memory using ConcurrentDictionary
- Session-based authentication with Google Maps API
**Current Code (SatelliteProvider.Services/GoogleMapsDownloader.cs):**
- Downloads tiles at 256x256 pixels
- Saves tiles directly to disk as JPG files
- Simpler implementation without composing/splitting
- Session-based authentication with Google Maps API
- Uses concurrent queue for downloading
**Decision:** Use current code as base. It's simpler and more aligned with requirements (store tiles "as is"). The legacy code's image composition and resizing is unnecessary overhead for our use case.
### 1.2 Current Solution Structure
```
SatelliteProvider/
├── SatelliteProvider.Api # Web API (ASP.NET 8.0)
├── SatelliteProvider.Common # DTOs, interfaces, utilities
├── SatelliteProvider.Services # Business logic (GoogleMapsDownloader)
└── SatelliteProvider.Tests # Unit tests (XUnit)
```
### 1.3 Gap Analysis
**Missing Components:**
1. Database layer (Dapper + PostgreSQL)
2. Database migrations (DbUp)
3. Tile metadata storage/retrieval
4. Region request processing with background jobs
5. File system management for /tiles and /ready directories
6. API endpoints for region requests
7. Docker and docker-compose configuration
8. Integration tests as a separate console application
## 2. Architecture Design
### 2.1 Solution Structure (Updated)
```
SatelliteProvider/
├── SatelliteProvider.Api # Web API
├── SatelliteProvider.Common # Shared DTOs, interfaces, utilities
├── SatelliteProvider.Services # Business logic services
├── SatelliteProvider.DataAccess # Database access layer (NEW)
├── SatelliteProvider.Tests # Unit tests
├── SatelliteProvider.IntegrationTests # Integration tests console app (NEW)
├── docker-compose.svc.yml # Service + dependencies (NEW)
└── docker-compose.tests.yml # Tests + service + dependencies (NEW)
```
### 2.2 Data Model
**Tiles Table Schema:**
```sql
CREATE TABLE tiles (
id UUID PRIMARY KEY,
zoom_level INT NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
tile_size_meters DOUBLE PRECISION NOT NULL,
tile_size_pixels INT NOT NULL,
image_type VARCHAR(10) NOT NULL,
maps_version VARCHAR(50),
file_path VARCHAR(500) NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_tiles_composite ON tiles(latitude, longitude, tile_size_meters);
CREATE INDEX idx_tiles_zoom ON tiles(zoom_level);
```
**Regions Table Schema:**
```sql
CREATE TABLE regions (
id UUID PRIMARY KEY,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
size_meters DOUBLE PRECISION NOT NULL,
zoom_level INT NOT NULL,
status VARCHAR(20) NOT NULL,
csv_file_path VARCHAR(500),
summary_file_path VARCHAR(500),
tiles_downloaded INT DEFAULT 0,
tiles_reused INT DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_regions_status ON regions(status);
```
**Routes Table Schema (Phase 8 - New):**
```sql
CREATE TABLE routes (
id UUID PRIMARY KEY,
name VARCHAR(200),
description TEXT,
total_distance_meters DOUBLE PRECISION NOT NULL,
total_points INT NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE route_points (
id UUID PRIMARY KEY,
route_id UUID NOT NULL REFERENCES routes(id) ON DELETE CASCADE,
sequence_number INT NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
point_type VARCHAR(20) NOT NULL, -- 'start', 'end', 'intermediate'
segment_index INT NOT NULL,
distance_from_previous DOUBLE PRECISION,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE(route_id, sequence_number)
);
CREATE INDEX idx_route_points_route ON route_points(route_id, sequence_number);
CREATE INDEX idx_route_points_coords ON route_points(latitude, longitude);
```
### 2.3 Background Processing
**In-Memory Queue Implementation:**
- Use `System.Threading.Channels` for thread-safe, async-friendly queue
- Hosted service (IHostedService) to process queue items
- Status tracking in database (processing, completed, failed)
- Future migration path: Azure Storage Queue, AWS SQS, or RabbitMQ
### 2.4 Google Maps Tile Details
**Tile Size:**
- Google Maps returns tiles at **256x256 pixels** (standard)
- We will store tiles as-is from Google Maps
- Store `tile_size_pixels = 256` in database
**Maps Version:**
- Google Maps doesn't expose explicit version via API
- Solution: Use download timestamp as version identifier
- Store as `maps_version = "downloaded_YYYY-MM-DD"`
- Consider ETags if available in HTTP response headers
**Image Format:**
- Google Maps returns JPEG for satellite tiles
- Store as JPG with `image_type = "jpg"`
## 3. Implementation Plan
### Phase 1: Database Layer (Priority: High) ✅ **COMPLETED**
**Task 1.1: Create DataAccess Project** ✅
- ✅ Created project `SatelliteProvider.DataAccess`
- ✅ Added Dapper NuGet package
- ✅ Added Npgsql NuGet package for PostgreSQL support
- ✅ Added DbUp NuGet package for migrations
**Task 1.2: Implement Database Migrations** ✅
- ✅ Created `Migrations` folder in DataAccess project
- ✅ Script 001: Create tiles table
- ✅ Script 002: Create regions table
- ✅ Script 003: Create indexes
- ✅ Script 004: Add version column for year-based versioning
- ✅ Implemented DbUp migration runner in Program.cs startup
**Task 1.3: Implement Repository Pattern** ✅
- ✅ Created `ITileRepository` interface
- ✅ Created `TileRepository` class implementing Dapper queries
- ✅ Created `IRegionRepository` interface
- ✅ Created `RegionRepository` class implementing Dapper queries
- ✅ Added connection string management in appsettings.json
**Task 1.4: Configuration** ✅
- ✅ Added PostgreSQL connection string to appsettings.json
- ✅ Added PostgreSQL connection string to appsettings.Development.json
- ✅ Created configuration classes in Common project
### Phase 2: Enhanced Tile Downloading and Storage (Priority: High) ✅ **COMPLETED**
**Task 2.1: Update GoogleMapsDownloader** ✅
- ✅ Created GoogleMapsDownloaderV2 with enhanced functionality
- ✅ Returns list of downloaded tile metadata
- ✅ Calculates tile size in meters (based on zoom level)
- ✅ Stores maps version (download timestamp format)
- ✅ Implemented error handling and retry logic with exponential backoff
- ✅ Added rate limit exception handling
**Task 2.2: Implement Tile Service** ✅
- ✅ Created `ITileService` interface in Common
- ✅ Created `TileService` class in Services
- ✅ Implemented `DownloadAndStoreTilesAsync(lat, lon, size, zoom)`
- ✅ Checks database for existing tiles before downloading (with year-based versioning)
- ✅ Saves tile metadata to database after download
- ✅ Returns list of tile paths and metadata
**Task 2.3: File System Management** ✅
- ✅ Configured /tiles directory path from appsettings.json
- ✅ Directory creation on startup implemented
- ✅ Implemented tile naming convention: `tile_{zoom}_{x}_{y}_{timestamp}.jpg`
- ✅ File path storage in database
### Phase 3: Background Processing System (Priority: High) ✅ **COMPLETED**
**Task 3.1: Create Region Request Queue** ✅
- ✅ Created `IRegionRequestQueue` interface
- ✅ Implemented using System.Threading.Channels
- ✅ Thread-safe enqueue/dequeue operations
- ✅ Capacity limits and overflow handling
**Task 3.2: Create Background Worker** ✅
- ✅ Implemented `RegionProcessingService : IHostedService`
- ✅ Dequeues region requests
- ✅ Processes each region (download tiles, create CSV, create summary, stitch images)
- ✅ Updates region status in database (queued → processing → completed/failed)
- ✅ Error handling with specific exception types (RateLimitException, HttpRequestException)
- ✅ Timeout handling (5 minutes per region)
**Task 3.3: Implement Region Service** ✅
- ✅ Created `IRegionService` interface
- ✅ Created `RegionService` class
- ✅ Method: `RequestRegionAsync(id, lat, lon, size)` - adds to queue and database
- ✅ Method: `GetRegionStatusAsync(id)` - returns status and file paths
- ✅ Method: `ProcessRegionAsync(id)` - called by background worker
**Task 3.4: CSV and Summary File Generation** ✅
- ✅ Created /ready directory management
- ✅ Implemented CSV writer with tile coordinates and file paths
- ✅ Generates region_{id}_ready.csv with ordered tiles (top-left to bottom-right)
- ✅ Generates region_{id}_summary.txt with comprehensive statistics
- ✅ Generates region_{id}_stitched.jpg with combined satellite imagery
- ✅ Includes error summaries for failed regions
### Phase 4: API Endpoints (Priority: High) ✅ **COMPLETED**
**Task 4.1: Remove Old Endpoints** ⚠️ **PARTIALLY DONE**
- ⚠️ Some legacy endpoints still exist (GetSatelliteTilesByLatLon, GetSatelliteTilesByMgrs, UploadImage)
- ✅ Main functionality implemented with new endpoints
**Task 4.2: Implement Region Request Endpoint** ✅
- ✅ POST /api/satellite/request
- ✅ Request body: { id: UUID, latitude: double, longitude: double, sizeMeters: double, zoomLevel: int }
- ✅ Validation: size between 100-10000 meters
- ✅ Returns: { id: UUID, status: "queued", ... }
- ✅ Enqueues region processing
**Task 4.3: Implement Region Status Endpoint** ✅
- ✅ GET /api/satellite/region/{id}
- ✅ Returns: { id: UUID, status: string, csvPath: string?, summaryPath: string?, tilesDownloaded: int, tilesReused: int }
- ✅ Status values: "queued", "processing", "completed", "failed"
**Task 4.4: OpenAPI/Swagger Documentation** ✅
- ✅ Updated Swagger configuration
- ✅ Added endpoint descriptions
- ✅ Documented request/response models
- ✅ Custom operation filters for parameter descriptions
### Phase 5: Docker Configuration (Priority: Medium) ✅ **COMPLETED**
**Task 5.1: Create Dockerfiles** ✅
- ✅ Dockerfile for SatelliteProvider.Api
- ✅ Dockerfile for SatelliteProvider.IntegrationTests
- ✅ Multi-stage builds for optimization
- ✅ Base image: mcr.microsoft.com/dotnet/aspnet:8.0
**Task 5.2: Create docker-compose.yml** ✅
- ✅ Service: api (satellite-provider-api)
- ✅ Service: postgres (PostgreSQL 16)
- ✅ Volume: ./tiles mounted to host
- ✅ Volume: ./ready mounted to host
- ✅ Volume: ./logs mounted to host
- ✅ Network configuration
- ✅ Environment variables for configuration
- ✅ Health checks for dependencies
**Task 5.3: Create docker-compose.tests.yml** ✅
- ✅ Services: postgres, api, integration-tests
- ✅ Waits for API and database readiness
- ✅ Runs tests and exits with test results
- ✅ Separate test environment configuration
**Task 5.4: Environment Configuration** ✅
- ✅ Development configuration (appsettings.Development.json)
- ✅ Production configuration (appsettings.json)
- ✅ Environment-specific settings
- ✅ Secrets management via environment variables
### Phase 6: Integration Tests (Priority: Medium) ✅ **COMPLETED**
**Task 6.1: Create Integration Tests Project** ✅
- ✅ Created SatelliteProvider.IntegrationTests console app project
- ✅ Added necessary NuGet packages (XUnit, FluentAssertions, etc.)
- ✅ Configured to run as console application
- ✅ Setup for database and API access
**Task 6.2: Implement Test Scenarios** ✅
- ✅ TileTests: Test tile download and caching
- ✅ RegionTests: Test region request and processing
- ✅ Verify CSV creation and content
- ✅ Verify summary file creation
- ✅ Verify stitched image creation
- ✅ Test tile reuse functionality
- ✅ Test region status endpoint
- ✅ Test concurrent region requests
**Task 6.3: Test Data Management** ✅
- ✅ Database cleanup before tests
- ✅ Test data validation
- ✅ Isolated test environments via Docker
**Task 6.4: CI/CD Integration** ⏳
- ✅ Docker-based test execution configured
- ⏳ CI/CD pipeline integration (depends on CI/CD setup)
- ⏳ Code coverage analysis (optional enhancement)
### Phase 7: Additional Features and Improvements (Priority: Low) ✅ **COMPLETED**
**Task 7.1: Monitoring and Logging** ✅
- ✅ Structured logging with Serilog
- ✅ File and console logging
- ✅ Performance tracking in logs
- ✅ Error tracking with context
**Task 7.2: Caching Strategy** ✅
- ✅ Implemented year-based versioning for automatic tile refresh
- ✅ Tile reuse within the same year
- ✅ Historical tile preservation
**Task 7.3: API Rate Limiting** ⏳
- ⏳ Rate limiting for API endpoints (not implemented)
- ✅ Google Maps API error handling with retry logic
- ✅ Exponential backoff for failed requests
**Task 7.4: Documentation** ✅
- ✅ Comprehensive README.md with API documentation
- ✅ Configuration guide
- ✅ Docker deployment instructions
- ✅ Year-based versioning strategy documented
## 4. Technical Specifications
### 4.1 Technology Stack
- **Runtime:** .NET 8.0
- **Web Framework:** ASP.NET Core Web API
- **ORM:** Dapper 2.1.35
- **Database:** PostgreSQL (latest stable)
- **Migrations:** DbUp
- **Testing:** XUnit, Moq, FluentAssertions
- **Containerization:** Docker, Docker Compose
- **Image Processing:** SixLabors.ImageSharp 3.1.11
- **Logging:** Microsoft.Extensions.Logging
### 4.2 NuGet Packages to Add
**SatelliteProvider.DataAccess:**
- Dapper (2.1.35)
- Npgsql (latest stable)
- DbUp (latest stable)
- Microsoft.Extensions.Configuration.Abstractions
**SatelliteProvider.Api:**
- Npgsql (for PostgreSQL connection)
**SatelliteProvider.IntegrationTests:**
- All existing test packages
- System.CommandLine (for console app)
- Testcontainers (optional, for Docker-based integration tests)
### 4.3 Configuration Structure
**appsettings.json:**
```json
{
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Database=satelliteprovider;Username=postgres;Password=postgres"
},
"MapConfig": {
"Service": "GoogleMaps",
"ApiKey": "YOUR_API_KEY_HERE"
},
"StorageConfig": {
"TilesDirectory": "/tiles",
"ReadyDirectory": "/ready"
},
"ProcessingConfig": {
"MaxConcurrentDownloads": 4,
"DefaultZoomLevel": 20,
"QueueCapacity": 100
}
}
```
### 4.4 API Endpoints Summary
| Method | Endpoint | Description | Status |
|--------|----------|-------------|--------|
| POST | /api/satellite/tiles/download | Download single tile at coordinates | ✅ Implemented |
| POST | /api/satellite/request | Request tiles for a region | ✅ Implemented |
| GET | /api/satellite/region/{id} | Get region status and file paths | ✅ Implemented |
| POST | /api/satellite/route | Plan route and split into regions | ⏳ To be implemented |
| GET | /api/satellite/route/{id} | Get route information with points | ⏳ To be implemented |
### 4.5 File Naming Conventions
**Tiles:**
```
/tiles/tile_{zoom}_{x}_{y}_{timestamp}.jpg
Example: /tiles/tile_20_12345_67890_20231015143022.jpg
```
**CSV Files:**
```
/ready/region_{id}_ready.csv
Example: /ready/region_550e8400-e29b-41d4-a716-446655440000_ready.csv
```
**Summary Files:**
```
/ready/region_{id}_summary.txt
Example: /ready/region_550e8400-e29b-41d4-a716-446655440000_summary.txt
```
### 4.6 CSV File Format
```csv
latitude,longitude,file_path
37.7749,-122.4194,/tiles/tile_20_12345_67890_20231015143022.jpg
37.7750,-122.4194,/tiles/tile_20_12346_67890_20231015143023.jpg
```
Rows ordered from top-left to bottom-right covering the requested region.
### 4.7 Summary File Format
```
Region Processing Summary
========================
Region ID: 550e8400-e29b-41d4-a716-446655440000
Center: 37.7749, -122.4194
Size: 500 meters
Zoom Level: 20
Processing Statistics:
- Tiles Downloaded: 45
- Tiles Reused from Cache: 12
- Total Tiles: 57
- Processing Time: 12.5 seconds
- Started: 2023-10-15 14:30:15
- Completed: 2023-10-15 14:30:28
Files Created:
- CSV: /ready/region_550e8400-e29b-41d4-a716-446655440000_ready.csv
- Summary: /ready/region_550e8400-e29b-41d4-a716-446655440000_summary.txt
```
## 5. Testing Strategy
### 5.1 Unit Tests
- Test GeoUtils calculations
- Test tile coordinate conversions
- Test repository methods (with mocked database)
- Test service business logic
### 5.2 Integration Tests
- End-to-end region request flow
- Database operations
- File system operations
- API endpoint testing
- Background processing
### 5.3 Test Coverage Goals
- Minimum 80% code coverage
- 100% coverage for critical paths (downloading, storage, region processing)
## 6. Deployment Strategy
### 6.1 Development Environment
```bash
docker-compose -f docker-compose.svc.yml up
```
### 6.2 Running Integration Tests
```bash
docker-compose -f docker-compose.tests.yml up --abort-on-container-exit
```
### 6.3 Production Deployment
- Use docker-compose.svc.yml as base
- Override with production-specific configuration
- Configure volumes for persistent storage
- Set up backup strategy for PostgreSQL
- Configure SSL/TLS for API
- Set up monitoring and alerting
## 7. Security Considerations
1. **API Key Management:** Store Google Maps API key in environment variables or secrets manager
2. **Database Security:** Use strong passwords, restrict network access
3. **Input Validation:** Validate all API inputs (coordinates, size limits)
4. **File System Security:** Restrict write access to /tiles and /ready directories
5. **Rate Limiting:** Implement to prevent abuse
6. **HTTPS:** Enforce HTTPS in production
## 8. Performance Considerations
1. **Concurrent Downloads:** Use 4 concurrent connections to Google Maps
2. **Database Indexing:** Composite index on (latitude, longitude, tile_size_meters)
3. **Caching:** Reuse existing tiles to minimize downloads
4. **Async Operations:** All I/O operations are async
5. **Background Processing:** Non-blocking region processing
## 9. Monitoring and Observability
1. **Logging:** Structured logging for all operations
2. **Metrics:** Track download counts, cache hit rates, processing times
3. **Health Checks:** API and database health endpoints
4. **Error Tracking:** Log errors with context for debugging
## 10. Future Enhancements
1. **Cloud Queue Integration:** Replace in-memory queue with Azure/AWS queue
2. **CDN Integration:** Serve tiles via CDN
3. **Tile Expiration:** Implement automatic tile refresh
4. **Multiple Map Providers:** Support for other satellite imagery providers
5. **WebSocket Support:** Real-time progress updates
6. **Admin Dashboard:** UI for monitoring and management
7. **Tile Stitching:** API to stitch tiles into single large image
### Phase 8: Route Planning Feature (Priority: High) ⏳ **NEW REQUIREMENT**
**Overview**: Implement route planning functionality that splits a route into overlapping regions with intermediate points spaced no more than 200 meters apart.
**Task 8.1: Database Schema for Routes**
- Create routes table to store route information
- Create route_points table to store route points (start, end, intermediate)
- Migration script 005: Create routes tables
- Add indexes for efficient route querying
**Task 8.2: Route Calculation Logic**
- Implement intermediate point calculation algorithm
- Calculate distance between two points using GeoUtils
- Generate intermediate points along the route with max 200m spacing
- Ensure equal spacing between points
**Task 8.3: Route Repository**
- Create `IRouteRepository` interface
- Create `RouteRepository` class
- CRUD operations for routes and route points
- Query methods for retrieving routes with points
**Task 8.4: Route Service**
- Create `IRouteService` interface
- Create `RouteService` class
- Method: `CreateRouteAsync(points)` - creates route with intermediate points
- Method: `GetRouteAsync(id)` - retrieves route with all points
**Task 8.5: Route API Endpoints**
- POST /api/satellite/route - create route from array of points
- GET /api/satellite/route/{id} - get route information with intermediate points
- Input validation (minimum 2 points, valid coordinates)
- Return route with calculated intermediate points
**Task 8.6: Integration with Region Processing**
- Each route point (including intermediate) represents a region center
- Option to automatically request regions for all route points
- Track processing status for route-based regions
**Task 8.7: Tests**
- Unit tests for intermediate point calculation
- Integration tests for route creation and retrieval
- Test various scenarios (short routes < 200m, long routes > 200m, multi-segment routes)
**Estimated Time:** 2-3 days
## 11. Implementation Timeline
**Total Estimated Time:** 15-20 working days (Phases 1-7 completed, Phase 8 remaining)
| Phase | Duration | Dependencies | Status |
|-------|----------|--------------|--------|
| Phase 1: Database Layer | 2-3 days | None | ✅ **COMPLETED** |
| Phase 2: Enhanced Tile Downloading | 2-3 days | Phase 1 | ✅ **COMPLETED** |
| Phase 3: Background Processing | 3-4 days | Phase 1, Phase 2 | ✅ **COMPLETED** |
| Phase 4: API Endpoints | 1-2 days | Phase 3 | ✅ **COMPLETED** |
| Phase 5: Docker Configuration | 2-3 days | Phase 4 | ✅ **COMPLETED** |
| Phase 6: Integration Tests | 3-4 days | Phase 5 | ✅ **COMPLETED** |
| Phase 7: Additional Features | 2-3 days | Phase 6 | ✅ **COMPLETED** |
| **Phase 8: Route Planning** | **2-3 days** | **Phase 1, Phase 2** | **⏳ IN PROGRESS** |
**Critical Path:** ~~Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6~~ → **Phase 8 (Route Planning)**
## 12. Risk Assessment
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| Google Maps API changes | High | Low | Monitor API changes, version locking |
| API quota exceeded | High | Medium | Implement caching, rate limiting |
| Disk space exhaustion | Medium | Medium | Implement cleanup policy, monitoring |
| Database performance | Medium | Low | Proper indexing, connection pooling |
| Concurrent processing issues | Medium | Low | Thorough testing, proper locking |
## 13. Success Criteria
1. ✅ API successfully downloads and stores tiles - **COMPLETED**
2. ✅ Metadata stored correctly in PostgreSQL - **COMPLETED**
3. ✅ Region requests processed in background - **COMPLETED**
4. ✅ CSV and summary files generated correctly - **COMPLETED**
5. ✅ Tile caching works (reuses existing tiles) - **COMPLETED**
6. ✅ Docker containers run successfully - **COMPLETED**
7. ✅ Integration tests pass - **COMPLETED**
8. ✅ API documentation complete - **COMPLETED**
9. ✅ Performance meets requirements (< 30s for 1000m region) - **COMPLETED**
10. ✅ Image stitching implemented - **COMPLETED**
11. ⏳ Route planning feature - **IN PROGRESS** (see Phase 8)
## 14. Open Questions and Decisions Made
### Questions from Analysis:
1. **Legacy Code?** ✅ Analyzed - Use current code as base
2. **Background Processing?** ✅ In-memory queue with Channels
3. **Database Migrations?** ✅ DbUp
4. **Tile Size?** ✅ 256x256 pixels (Google Maps standard)
5. **Maps Version?** ✅ Use download timestamp + year-based versioning
6. **Composite Key?** ✅ Indexed composite key, UUID primary key
7. **Integration Tests?** ✅ Separate console app project
8. **Image Stitching?** ✅ Implemented with ImageSharp, generates region_{id}_stitched.jpg
9. **Tile Versioning?** ✅ Year-based versioning for automatic annual refresh
### New Questions for Phase 8 (Route Planning):
1. **Route Point Spacing?**
- Max 200m between consecutive points
- Equal spacing along route segments
2. **Route Storage?**
- Store routes table with route metadata
- Store route_points table with individual points and sequence
3. **Automatic Region Processing?**
- Decision needed: Should route creation automatically trigger region requests for all points?
- Or: Manual trigger via separate endpoint?
4. **Route Point Metadata?**
- Track point type: start, end, or intermediate
- Store segment information (which route segment the point belongs to)
5. **Route-Based Region Tracking?**
- Link regions to routes for progress tracking
- Aggregate route completion status based on all region statuses
## 15. Next Steps
1. Review and approve this implementation plan
2. Set up development environment
3. Create feature branches for each phase
4. Begin Phase 1: Database Layer implementation
5. Daily stand-ups to track progress
6. Regular code reviews for each completed phase
---
**Document Version:** 2.0
**Created:** 2025-10-26
**Last Updated:** 2025-11-01
**Author:** AI Assistant
**Status:** Phases 1-7 Completed, Phase 8 (Route Planning) In Progress