azaion/satellite-provider
1
0
Fork 0
mirror of https://github.com/azaion/satellite-provider.git synced 2026-04-22 08:56:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

documentation

Browse Source
This commit is contained in:
Anton Martynenko
2025-11-20 12:41:07 +01:00
parent bf2030e3c6
commit 604f0e070d
3 changed files with 634 additions and 950 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+343 -249
View File
@@ -1,151 +1,107 @@
# Satellite Provider API
A RESTful API service for downloading, caching, and processing satellite imagery from Google Maps. The service provides endpoints for downloading individual tiles and processing entire regions with intelligent tile caching and versioning.
A .NET 8.0 microservice for downloading, managing, and serving satellite imagery tiles from Google Maps. Supports region-based tile requests, route planning with automatic intermediate point generation, and geofencing capabilities.
## Features
- **Satellite Tile Downloads**: Download individual satellite tiles from Google Maps at various zoom levels
- **Region Processing**: Request satellite imagery for entire geographic regions
- **Intelligent Tile Caching**: Automatic tile reuse with year-based versioning
- **Image Stitching**: Combine multiple tiles into seamless regional images
- **CSV Export**: Generate coordinate mappings for downloaded tiles
- **Summary Reports**: Detailed processing statistics and metadata
- **Tile Management**: Download and cache satellite imagery tiles from Google Maps
- **Region Requests**: Request tiles for specific geographic areas (100m - 10km squares)
- **Route Planning**: Create routes with automatic intermediate point interpolation (~200m intervals)
- **Geofencing**: Define polygon boundaries to filter regions inside/outside zones
- **Image Stitching**: Combine tiles into single images for regions
- **Tile Packaging**: Create ZIP archives of route tiles (max 50MB)
- **Async Processing**: Queue-based background processing for large requests
- **Caching**: Reuse previously downloaded tiles to minimize redundant downloads
- **REST API**: OpenAPI/Swagger documented endpoints
## Tech Stack
## Prerequisites
- **.NET 8.0** - ASP.NET Web API
- **PostgreSQL** - Primary data storage
- **Dapper** - Lightweight ORM for database access
- **DbUp** - Database migrations
- **Serilog** - Structured logging
- **ImageSharp** - Image processing and stitching
- **Docker** - Containerized deployment
- Docker and Docker Compose
- .NET 8.0 SDK (for local development)
- PostgreSQL (provided via Docker)
## Tile Versioning Strategy
## Quick Start
### Year-Based Tile Versions
### Run the Service
The service implements a **year-based versioning strategy** for cached satellite tiles. This approach is based on the assumption that Google Maps satellite imagery updates infrequently, typically once per year or less.
#### How It Works
1. **Version Assignment**: When a tile is downloaded, it is assigned a version number equal to the current year (e.g., 2025).
2. **Unique Constraint**: Tiles are uniquely identified by the combination of:
- Latitude (center point)
- Longitude (center point)
- Zoom level
- Tile size in meters
- **Version** (year)
3. **Cache Behavior**:
- **Same Year**: If requesting a tile that was downloaded in the current year, the cached tile is reused
- **New Year**: When the calendar year changes, new tiles are downloaded even for the same coordinates
- **Historical Data**: Old tile versions remain in the database for historical reference
#### Example
```
Request in 2025:
- Downloads tile at (47.461747, 37.647063) zoom 18 → version = 2025
- Saves to database with version 2025
Second request in 2025:
- Finds tile at (47.461747, 37.647063) zoom 18 version 2025
- Reuses cached tile (no download)
Request in 2026:
- Looks for tile at (47.461747, 37.647063) zoom 18 version 2026
- Not found (only 2025 version exists)
- Downloads new tile → version = 2026
- Both 2025 and 2026 versions now exist in database
```bash
docker-compose up --build
```
#### Benefits
The API will be available at `http://localhost:5100`
- **Automatic Updates**: Ensures fresh satellite imagery each calendar year
- **Historical Archive**: Maintains previous years' imagery for comparison
- **Efficient Caching**: Maximizes cache hit rate within the same year
- **Simple Logic**: No complex timestamp comparisons or manual cache invalidation
Swagger documentation: `http://localhost:5100/swagger`
#### Database Schema
### Run with Tests
```sql
CREATE TABLE tiles (
id UUID PRIMARY KEY,
zoom_level INT NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
tile_size_meters DOUBLE PRECISION NOT NULL,
tile_size_pixels INT NOT NULL,
image_type VARCHAR(10) NOT NULL,
maps_version VARCHAR(50),
version INT NOT NULL, -- Year-based version
file_path VARCHAR(500) NOT NULL,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE UNIQUE INDEX idx_tiles_unique_location
ON tiles(latitude, longitude, zoom_level, tile_size_meters, version);
```bash
docker-compose -f docker-compose.yml -f docker-compose.tests.yml up --build --abort-on-container-exit
```
This command:
- Builds and starts all services
- Runs integration tests
- Exits when tests complete
## Architecture
### Project Structure
The service follows a layered architecture:
```
SatelliteProvider/
├── SatelliteProvider.Api/ # REST API endpoints
├── SatelliteProvider.Common/ # Shared DTOs and interfaces
├── SatelliteProvider.DataAccess/ # Database access and migrations
├── SatelliteProvider.Services/ # Business logic
├── SatelliteProvider.Tests/ # Unit tests
└── SatelliteProvider.IntegrationTests/ # Integration tests
┌─────────────────────────────────────┐
│ API Layer (ASP.NET Core) │ HTTP endpoints
├─────────────────────────────────────┤
│ Services (Business Logic) │ Processing, validation
├─────────────────────────────────────┤
│ Data Access (Dapper + Repos) │ Database operations
├─────────────────────────────────────┤
│ PostgreSQL Database │ Persistence
└─────────────────────────────────────┘
```
### Key Components
### Projects
- **GoogleMapsDownloaderV2**: Downloads satellite tiles from Google Maps API
- **TileService**: Manages tile caching, retrieval, and storage with version control
- **RegionService**: Processes region requests, stitches tiles, generates outputs
- **RegionRequestQueue**: Background processing queue for region requests
- **DatabaseMigrator**: Manages database schema migrations
- **SatelliteProvider.Api**: Web API and endpoints
- **SatelliteProvider.Services**: Business logic and processing
- **SatelliteProvider.DataAccess**: Database access and migrations
- **SatelliteProvider.Common**: Shared DTOs, interfaces, utilities
- **SatelliteProvider.IntegrationTests**: Integration test suite
- **SatelliteProvider.Tests**: Unit tests
## API Endpoints
### Download Single Tile
```http
POST /api/satellite/tiles/download
Content-Type: application/json
{
"latitude": 47.461747,
"longitude": 37.647063,
"zoomLevel": 18
}
GET /api/satellite/tiles/latlon?Latitude={lat}&Longitude={lon}&ZoomLevel={zoom}
```
Downloads a single tile at specified coordinates and zoom level.
**Parameters:**
- `Latitude` (double): Center latitude
- `Longitude` (double): Center longitude
- `ZoomLevel` (int): Zoom level (higher = more detail, max ~20)
**Response:**
```json
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"id": "uuid",
"zoomLevel": 18,
"latitude": 47.462451,
"longitude": 37.646027,
"tileSizeMeters": 103.35,
"latitude": 37.7749,
"longitude": -122.4194,
"tileSizeMeters": 38.2,
"tileSizePixels": 256,
"imageType": "jpg",
"mapsVersion": "downloaded_2025-10-29",
"version": 2025,
"filePath": "./tiles/tile_18_158485_91707_20251029103256.jpg",
"createdAt": "2025-10-29T10:32:56Z",
"updatedAt": "2025-10-29T10:32:56Z"
"mapsVersion": "downloaded_2024-11-20",
"version": 2024,
"filePath": "18/158/90.jpg",
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:30:00Z"
}
```
### Request Region Processing
### Request Region Tiles
```http
POST /api/satellite/request
@@ -153,63 +109,174 @@ Content-Type: application/json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"latitude": 47.461747,
"longitude": 37.647063,
"sizeMeters": 200,
"zoomLevel": 18
"latitude": 37.7749,
"longitude": -122.4194,
"sizeMeters": 500,
"zoomLevel": 18,
"stitchTiles": false
}
```
### Check Region Status
Requests tiles for a square region. Processing happens asynchronously.
```http
GET /api/satellite/region/{regionId}
```
**Parameters:**
- `id` (guid): Unique identifier for the region
- `latitude` (double): Center latitude
- `longitude` (double): Center longitude
- `sizeMeters` (double): Square side length (100-10000 meters)
- `zoomLevel` (int): Zoom level (default: 18)
- `stitchTiles` (bool): Create single stitched image (default: false)
**Response:**
```json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "completed",
"csvFilePath": "./ready/region_{id}_ready.csv",
"summaryFilePath": "./ready/region_{id}_summary.txt",
"tilesDownloaded": 5,
"tilesReused": 4,
"createdAt": "2025-10-29T10:32:56Z",
"updatedAt": "2025-10-29T10:32:57Z"
"status": "pending",
"csvFilePath": null,
"summaryFilePath": null,
"tilesDownloaded": 0,
"tilesReused": 0,
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:30:00Z"
}
```
### Get Region Status
```http
GET /api/satellite/region/{id}
```
Check processing status and get file paths when complete.
**Status Values:**
- `pending`: Queued for processing
- `processing`: Currently downloading tiles
- `completed`: All tiles ready, files created
- `failed`: Processing failed
**Response (completed):**
```json
{
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"status": "completed",
"csvFilePath": "./ready/region_3fa85f64-5717-4562-b3fc-2c963f66afa6_ready.csv",
"summaryFilePath": "./ready/region_3fa85f64-5717-4562-b3fc-2c963f66afa6_summary.txt",
"tilesDownloaded": 12,
"tilesReused": 4,
"createdAt": "2024-11-20T10:30:00Z",
"updatedAt": "2024-11-20T10:32:15Z"
}
```
### Create Route
```http
POST /api/satellite/route
Content-Type: application/json
{
"id": "7fa85f64-5717-4562-b3fc-2c963f66afa6",
"name": "City Tour Route",
"description": "Downtown to waterfront",
"regionSizeMeters": 200,
"zoomLevel": 18,
"points": [
{ "latitude": 37.7749, "longitude": -122.4194, "name": "Start" },
{ "latitude": 37.7849, "longitude": -122.4094, "name": "Middle" },
{ "latitude": 37.7949, "longitude": -122.3994, "name": "End" }
],
"geofences": {
"polygons": [
{
"name": "Restricted Area",
"points": [
{ "latitude": 37.775, "longitude": -122.420 },
{ "latitude": 37.776, "longitude": -122.420 },
{ "latitude": 37.776, "longitude": -122.418 },
{ "latitude": 37.775, "longitude": -122.418 }
]
}
],
"direction": "inside"
},
"requestMaps": true,
"createTilesZip": true
}
```
Creates a route with intermediate points calculated automatically.
**Parameters:**
- `id` (guid): Route identifier
- `name` (string): Route name
- `description` (string): Optional description
- `regionSizeMeters` (double): Size of region at each point
- `zoomLevel` (int): Zoom level for tiles
- `points` (array): Original route points
- `geofences` (object): Optional polygon boundaries
- `polygons`: Array of polygon definitions
- `direction`: "inside" or "outside" (filter regions)
- `requestMaps` (bool): Automatically request tiles for route regions
- `createTilesZip` (bool): Create ZIP file of all tiles
**Response:**
```json
{
"id": "7fa85f64-5717-4562-b3fc-2c963f66afa6",
"name": "City Tour Route",
"description": "Downtown to waterfront",
"regionSizeMeters": 200,
"zoomLevel": 18,
"totalDistanceMeters": 2486.5,
"totalPoints": 15,
"points": [
{
"sequenceNumber": 0,
"latitude": 37.7749,
"longitude": -122.4194,
"pointType": "original",
"segmentIndex": 0,
"distanceFromPrevious": 0
},
{
"sequenceNumber": 1,
"latitude": 37.7765,
"longitude": -122.4177,
"pointType": "intermediate",
"segmentIndex": 0,
"distanceFromPrevious": 198.3
}
],
"regions": [
"region-id-1",
"region-id-2"
],
"tilesZipPath": "./ready/route_7fa85f64-5717-4562-b3fc-2c963f66afa6_tiles.zip",
"createdAt": "2024-11-20T10:30:00Z"
}
```
### Get Route
```http
GET /api/satellite/route/{id}
```
Retrieve route details including all interpolated points.
## Configuration
### Environment Variables
- `ASPNETCORE_ENVIRONMENT`: Development/Production
- `ASPNETCORE_URLS`: HTTP binding address (default: http://+:8080)
- `ConnectionStrings__DefaultConnection`: PostgreSQL connection string
- `MapConfig__ApiKey`: Google Maps API key
### appsettings.json
Configuration is managed through `appsettings.json`:
```json
{
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft.AspNetCore": "Warning"
}
},
"WriteTo": [
{ "Name": "Console" },
{
"Name": "File",
"Args": {
"path": "./logs/satellite-provider-.log",
"rollingInterval": "Day"
}
}
]
"ConnectionStrings": {
"DefaultConnection": "Host=localhost;Database=satelliteprovider;Username=postgres;Password=postgres"
},
"MapConfig": {
"Service": "GoogleMaps",
"ApiKey": "your-google-maps-api-key"
},
"StorageConfig": {
"TilesDirectory": "./tiles",
@@ -217,160 +284,187 @@ GET /api/satellite/region/{regionId}
},
"ProcessingConfig": {
"MaxConcurrentDownloads": 4,
"DefaultZoomLevel": 18,
"QueueCapacity": 100
"MaxConcurrentRegions": 20,
"DefaultZoomLevel": 20,
"QueueCapacity": 1000,
"DelayBetweenRequestsMs": 50,
"SessionTokenReuseCount": 100
}
}
```
## Running with Docker
### Environment Variables (Docker)
### Prerequisites
Override settings via environment variables in `docker-compose.yml`:
- Docker and Docker Compose
- Google Maps API key with Tile API enabled
### Setup
1. Create `.env` file in project root:
```bash
GOOGLE_MAPS_API_KEY=your_api_key_here
```yaml
environment:
- ConnectionStrings__DefaultConnection=Host=db;Database=satelliteprovider;Username=postgres;Password=postgres
- MapConfig__ApiKey=your-api-key-here
```
2. Start services:
```bash
docker-compose up -d
## File Storage
### Tiles
Downloaded tiles are stored in:
```
./tiles/{zoomLevel}/{x}/{y}.jpg
```
3. View logs:
```bash
docker-compose logs -f api
### Output Files
**CSV File** (`region_{id}_ready.csv`):
Comma-separated list of tiles covering the region, ordered top-left to bottom-right:
```csv
latitude,longitude,filepath
37.7750,-122.4195,18/158/90.jpg
37.7750,-122.4193,18/158/91.jpg
```
4. Stop services:
```bash
docker-compose down
**Summary File** (`region_{id}_summary.txt`):
Processing statistics:
```
Region: 3fa85f64-5717-4562-b3fc-2c963f66afa6
Size: 500m x 500m
Center: 37.7749, -122.4194
Zoom Level: 18
Tiles Downloaded: 12
Tiles Reused: 4
Total Tiles: 16
Processing Time: 3.5s
```
### Docker Compose Services
**Stitched Image** (`region_{id}_stitched.jpg`):
Single combined image when `stitchTiles: true`
- **postgres**: PostgreSQL database server
- **api**: Satellite Provider API service
**Route ZIP** (`route_{id}_tiles.zip`):
ZIP archive of all tiles for route when `createTilesZip: true` (max 50MB)
### Volume Mounts
## Database Schema
- `./tiles`: Downloaded tile images
- `./ready`: Processed region outputs (CSV, stitched images, summaries)
- `./logs`: Application logs
### Tables
**tiles**: Cached satellite tiles
- Unique per (latitude, longitude, tile_size_meters, zoom_level)
- Tracks maps version for cache invalidation
**regions**: Region requests and processing status
- Status tracking: pending → processing → completed/failed
- Links to output files when complete
**routes**: Route definitions
- Stores original configuration
- Links to calculated points and regions
**route_points**: All points (original + interpolated)
- Sequenced in route order
- Type: "original" or "intermediate"
- Includes distance calculations
**route_regions**: Junction table
- Links routes to regions for tile requests
- Tracks geofence status (inside/outside)
## Development
### Local Development Setup
### Building Locally
1. Install .NET 8.0 SDK
2. Install PostgreSQL 16
3. Set up database:
```bash
createdb satelliteprovider
```
4. Update connection string in `appsettings.Development.json`
5. Run migrations (automatic on startup)
6. Start API:
```bash
cd SatelliteProvider.Api
dotnet build
dotnet run
```
### Running Tests
**Unit Tests:**
```bash
cd SatelliteProvider.Tests
dotnet test
```
### Integration Tests
**Integration Tests:**
```bash
docker-compose -f docker-compose.tests.yml up --build
docker-compose -f docker-compose.yml -f docker-compose.tests.yml up --build --abort-on-container-exit
```
## Output Files
### Region Processing Outputs
For each processed region, the service generates:
1. **CSV File** (`region_{id}_ready.csv`):
- Tile coordinates and file paths
- Ordered by latitude (descending) and longitude (ascending)
2. **Stitched Image** (`region_{id}_stitched.jpg`):
- Combined satellite imagery for the entire region
- Red crosshair marking the requested center point
3. **Summary Report** (`region_{id}_summary.txt`):
- Region metadata and coordinates
- Processing statistics (tiles downloaded, tiles reused, total tiles)
- Processing time and timestamps
- File paths for all generated outputs
### Example Summary Report
### Database Migrations
Migrations run automatically on startup. SQL files are located in:
```
Region Processing Summary
========================
Region ID: 5d218e0d-92bc-483c-9e88-fd79200b84e6
Center: 47.461747, 37.647063
Size: 200 meters
Zoom Level: 18
Processing Statistics:
- Tiles Downloaded: 5
- Tiles Reused from Cache: 4
- Total Tiles: 9
- Processing Time: 0.50 seconds
- Started: 2025-10-29 10:32:56 UTC
- Completed: 2025-10-29 10:32:57 UTC
Files Created:
- CSV: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_ready.csv
- Stitched Image: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_stitched.jpg
- Summary: region_5d218e0d-92bc-483c-9e88-fd79200b84e6_summary.txt
SatelliteProvider.DataAccess/Migrations/
```
## Zoom Levels
To add a new migration:
1. Create `NNN_DescriptiveName.sql` (increment number)
2. Set Build Action to `EmbeddedResource`
3. Restart application
Supported zoom levels: **15, 16, 17, 18, 19**
### Logging
- **Zoom 15**: ~2,600m per tile (larger areas, less detail)
- **Zoom 16**: ~1,300m per tile
- **Zoom 17**: ~650m per tile
- **Zoom 18**: ~103m per tile (recommended for most uses)
- **Zoom 19**: ~52m per tile (maximum detail)
Logs are written to:
- Console (stdout)
- `./logs/satellite-provider-YYYYMMDD.log`
Log level can be adjusted in `appsettings.json` under `Serilog:MinimumLevel`.
## Performance Considerations
### Tile Caching
- Tiles are cached indefinitely (no expiration)
- Reused across multiple region/route requests
- Reduces redundant downloads significantly
- First request for a region downloads all required tiles
- Subsequent requests (same year) reuse cached tiles
- Typical 3x3 region (9 tiles) at zoom 18: ~300ms first request, ~50ms cached
### Concurrent Processing
- Configurable concurrent downloads (default: 4)
- Configurable concurrent regions (default: 20)
- Queue capacity: 1000 requests
### Region Processing
### Rate Limiting
- Delay between requests: 50ms default
- Session token reuse: 100 tiles per token
- Helps avoid Google Maps rate limits
- Processed asynchronously in background queue
- Status polling recommended at 1-second intervals
- Processing time depends on:
- Number of tiles required
- Cache hit rate
- Network latency to Google Maps API
- Image stitching complexity
### Geofencing
- Uses point-in-polygon algorithm
- Reduces unnecessary tile downloads
- Spatial index on polygon data
## Troubleshooting
### Service won't start
- Check Docker is running
- Verify ports 5100 and 5432 are available
- Check logs: `docker-compose logs api`
### Tiles not downloading
- Verify Google Maps API key is configured
- Check internet connectivity
- Review logs for HTTP errors
- Check rate limiting settings
### Region stuck in "processing"
- Check background service is running: `docker-compose logs api`
- Verify database connection
- Check queue capacity not exceeded
- Review logs for exceptions
### Tests failing
- Ensure all containers are healthy before tests run
- Check test container logs: `docker-compose logs tests`
- Verify database migrations completed
- Check network connectivity between containers
### Out of disk space
- Tiles accumulate in `./tiles/` directory
- Implement cleanup strategy for old tiles
- Monitor disk usage
## License
This project is proprietary software. All rights reserved.
This project is proprietary software.
## Support
For issues, questions, or feature requests, please contact the development team.
For issues and questions, please contact the development team.