satellite-provider/plan.md

Satellite Provider Service - Implementation Plan

Executive Summary

This document outlines the detailed implementation plan for the Satellite Provider Service, which downloads satellite map tiles from Google Maps, stores them with metadata in PostgreSQL, and provides an API for requesting and retrieving map regions.

1. Code Analysis

1.1 Legacy Code Comparison

Legacy Code (Azaion.Common/Services/SatelliteDownloader.cs):

• Downloads tiles at 256x256 pixels (Google Maps standard)
• Composes tiles into one large image
• Splits and resizes to 512x512 TIFF files
• Uses MediatR for status updates
• Stores tiles in memory using ConcurrentDictionary
• Session-based authentication with Google Maps API

Current Code (SatelliteProvider.Services/GoogleMapsDownloader.cs):

• Downloads tiles at 256x256 pixels
• Saves tiles directly to disk as JPG files
• Simpler implementation without composing/splitting
• Session-based authentication with Google Maps API
• Uses concurrent queue for downloading

Decision: Use current code as base. It's simpler and more aligned with requirements (store tiles "as is"). The legacy code's image composition and resizing is unnecessary overhead for our use case.

1.2 Current Solution Structure

SatelliteProvider/
├── SatelliteProvider.Api          # Web API (ASP.NET 8.0)
├── SatelliteProvider.Common       # DTOs, interfaces, utilities
├── SatelliteProvider.Services     # Business logic (GoogleMapsDownloader)
└── SatelliteProvider.Tests        # Unit tests (XUnit)

1.3 Gap Analysis

Missing Components:

1. Database layer (Dapper + PostgreSQL)
2. Database migrations (DbUp)
3. Tile metadata storage/retrieval
4. Region request processing with background jobs
5. File system management for /tiles and /ready directories
6. API endpoints for region requests
7. Docker and docker-compose configuration
8. Integration tests as a separate console application

2. Architecture Design

2.1 Solution Structure (Updated)

SatelliteProvider/
├── SatelliteProvider.Api                  # Web API
├── SatelliteProvider.Common               # Shared DTOs, interfaces, utilities
├── SatelliteProvider.Services             # Business logic services
├── SatelliteProvider.DataAccess           # Database access layer (NEW)
├── SatelliteProvider.Tests                # Unit tests
├── SatelliteProvider.IntegrationTests     # Integration tests console app (NEW)
├── docker-compose.svc.yml                 # Service + dependencies (NEW)
└── docker-compose.tests.yml               # Tests + service + dependencies (NEW)

2.2 Data Model

Tiles Table Schema:

CREATE TABLE tiles (
    id UUID PRIMARY KEY,
    zoom_level INT NOT NULL,
    latitude DOUBLE PRECISION NOT NULL,
    longitude DOUBLE PRECISION NOT NULL,
    tile_size_meters DOUBLE PRECISION NOT NULL,
    tile_size_pixels INT NOT NULL,
    image_type VARCHAR(10) NOT NULL,
    maps_version VARCHAR(50),
    file_path VARCHAR(500) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_tiles_composite ON tiles(latitude, longitude, tile_size_meters);
CREATE INDEX idx_tiles_zoom ON tiles(zoom_level);

Regions Table Schema:

CREATE TABLE regions (
    id UUID PRIMARY KEY,
    latitude DOUBLE PRECISION NOT NULL,
    longitude DOUBLE PRECISION NOT NULL,
    size_meters DOUBLE PRECISION NOT NULL,
    zoom_level INT NOT NULL,
    status VARCHAR(20) NOT NULL,
    csv_file_path VARCHAR(500),
    summary_file_path VARCHAR(500),
    tiles_downloaded INT DEFAULT 0,
    tiles_reused INT DEFAULT 0,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_regions_status ON regions(status);

2.3 Background Processing

In-Memory Queue Implementation:

• Use System.Threading.Channels for thread-safe, async-friendly queue
• Hosted service (IHostedService) to process queue items
• Status tracking in database (processing, completed, failed)
• Future migration path: Azure Storage Queue, AWS SQS, or RabbitMQ

2.4 Google Maps Tile Details

Tile Size:

• Google Maps returns tiles at 256x256 pixels (standard)
• We will store tiles as-is from Google Maps
• Store tile_size_pixels = 256 in database

Maps Version:

• Google Maps doesn't expose explicit version via API
• Solution: Use download timestamp as version identifier
• Store as maps_version = "downloaded_YYYY-MM-DD"
• Consider ETags if available in HTTP response headers

Image Format:

• Google Maps returns JPEG for satellite tiles
• Store as JPG with image_type = "jpg"

3. Implementation Plan

Phase 1: Database Layer (Priority: High)

Task 1.1: Create DataAccess Project

• Create new project SatelliteProvider.DataAccess
• Add Dapper NuGet package (version 2.1.35 to match existing dependencies)
• Add Npgsql NuGet package for PostgreSQL support
• Add DbUp NuGet package for migrations

Task 1.2: Implement Database Migrations

• Create Migrations folder in DataAccess project
• Script 001: Create tiles table
• Script 002: Create regions table
• Script 003: Create indexes
• Implement DbUp migration runner in Program.cs startup

Task 1.3: Implement Repository Pattern

• Create ITileRepository interface
• Create TileRepository class implementing Dapper queries
• Create IRegionRepository interface
• Create RegionRepository class implementing Dapper queries
• Add connection string management in appsettings.json

Task 1.4: Configuration

• Add PostgreSQL connection string to appsettings.json
• Add PostgreSQL connection string to appsettings.Development.json
• Create DatabaseConfig class in Common project

Estimated Time: 2-3 days

Phase 2: Enhanced Tile Downloading and Storage (Priority: High)

Task 2.1: Update GoogleMapsDownloader

• Modify to return list of downloaded tile metadata
• Add tile size calculation in meters (based on zoom level)
• Extract and store maps version (download timestamp)
• Improve error handling and retry logic
• Add progress tracking

Task 2.2: Implement Tile Service

• Create ITileService interface in Common
• Create TileService class in Services
• Method: DownloadAndStoreTiles(lat, lon, size, zoom)
• Check database for existing tiles before downloading
• Save tile metadata to database after download
• Return list of tile paths and metadata

Task 2.3: File System Management

• Configure /tiles directory path from appsettings.json
• Ensure directory creation on startup
• Implement tile naming convention: tile_{zoom}_{x}_{y}_{timestamp}.jpg
• Add file path storage in database

Estimated Time: 2-3 days

Phase 3: Background Processing System (Priority: High)

Task 3.1: Create Region Request Queue

• Create IRegionRequestQueue interface
• Implement using System.Threading.Channels
• Thread-safe enqueue/dequeue operations
• Capacity limits and overflow handling

Task 3.2: Create Background Worker

• Implement RegionProcessingService : IHostedService
• Dequeue region requests
• Process each region (download tiles, create CSV, create summary)
• Update region status in database
• Error handling and retry logic

Task 3.3: Implement Region Service

• Create IRegionService interface
• Create RegionService class
• Method: RequestRegion(id, lat, lon, size) - adds to queue and database
• Method: GetRegionStatus(id) - returns status and file paths
• Method: ProcessRegion(id) - called by background worker

Task 3.4: CSV and Summary File Generation

• Create /ready directory management
• Implement CSV writer with tile coordinates and file paths
• Generate region_{id}_ready.csv
• Generate region_{id}_summary.txt with statistics
• Format: tiles downloaded, tiles reused, processing time, region info

Estimated Time: 3-4 days

Phase 4: API Endpoints (Priority: High)

Task 4.1: Remove Old Endpoints

• Remove unused endpoints from Program.cs
• Clean up old DTOs and models

Task 4.2: Implement Region Request Endpoint

• POST /api/satellite/request
• Request body: { id: UUID, latitude: double, longitude: double, sizeMeters: double }
• Validation: size between 100-10000 meters
• Returns: { id: UUID, status: "queued" }
• Enqueue region processing

Task 4.3: Implement Region Status Endpoint

• GET /api/satellite/region/{id}
• Returns: { id: UUID, status: string, csvPath: string?, summaryPath: string?, tilesDownloaded: int, tilesReused: int }
• Status values: "queued", "processing", "completed", "failed"

Task 4.4: OpenAPI/Swagger Documentation

• Update Swagger configuration
• Add endpoint descriptions and examples
• Document request/response models

Estimated Time: 1-2 days

Phase 5: Docker Configuration (Priority: Medium)

Task 5.1: Create Dockerfiles

• Dockerfile for SatelliteProvider.Api
• Dockerfile for SatelliteProvider.IntegrationTests
• Multi-stage builds for optimization
• Base image: mcr.microsoft.com/dotnet/aspnet:8.0

Task 5.2: Create docker-compose.svc.yml

• Service: satellite-provider-api
• Service: postgres (official image)
• Volume: /tiles mounted to host
• Volume: /ready mounted to host
• Network configuration
• Environment variables for configuration

Task 5.3: Create docker-compose.tests.yml

• Extends docker-compose.svc.yml
• Service: satellite-provider-integration-tests
• Waits for API and database readiness
• Runs tests and exits

Task 5.4: Environment Configuration

• Development configuration
• Production configuration
• Environment-specific appsettings files
• Secrets management (API keys, connection strings)

Estimated Time: 2-3 days

Phase 6: Integration Tests (Priority: Medium)

Task 6.1: Create Integration Tests Project

• Create SatelliteProvider.IntegrationTests console app project
• Add necessary NuGet packages (XUnit, FluentAssertions, etc.)
• Configure to run as console application
• Setup TestContext for database and API access

Task 6.2: Implement Test Scenarios

• Test 1: Download tiles for a small region (100m)
• Test 2: Request region and verify CSV creation
• Test 3: Verify tile caching/reuse
• Test 4: Test concurrent region requests
• Test 5: Test region status endpoint
• Test 6: Test error handling and validation

Task 6.3: Test Data Management

• Database setup/teardown scripts
• Test data seeding
• Cleanup after tests
• Isolated test environments

Task 6.4: CI/CD Integration

• Configure test execution in CI/CD pipeline
• Test result reporting
• Code coverage analysis

Estimated Time: 3-4 days

Phase 7: Additional Features and Improvements (Priority: Low)

Task 7.1: Monitoring and Logging

• Structured logging with Serilog
• Application Insights or similar
• Performance metrics
• Error tracking

Task 7.2: Caching Strategy

• Implement tile expiration policy
• Cache invalidation logic
• Disk space management

Task 7.3: API Rate Limiting

• Implement rate limiting for API endpoints
• Google Maps API quota management
• Request throttling

Task 7.4: Documentation

• API documentation
• Deployment guide
• Configuration guide
• Architecture diagrams

Estimated Time: 2-3 days

4. Technical Specifications

4.1 Technology Stack

• Runtime: .NET 8.0
• Web Framework: ASP.NET Core Web API
• ORM: Dapper 2.1.35
• Database: PostgreSQL (latest stable)
• Migrations: DbUp
• Testing: XUnit, Moq, FluentAssertions
• Containerization: Docker, Docker Compose
• Image Processing: SixLabors.ImageSharp 3.1.11
• Logging: Microsoft.Extensions.Logging

4.2 NuGet Packages to Add

SatelliteProvider.DataAccess:

• Dapper (2.1.35)
• Npgsql (latest stable)
• DbUp (latest stable)
• Microsoft.Extensions.Configuration.Abstractions

SatelliteProvider.Api:

• Npgsql (for PostgreSQL connection)

SatelliteProvider.IntegrationTests:

• All existing test packages
• System.CommandLine (for console app)
• Testcontainers (optional, for Docker-based integration tests)

4.3 Configuration Structure

appsettings.json:

{
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Database=satelliteprovider;Username=postgres;Password=postgres"
  },
  "MapConfig": {
    "Service": "GoogleMaps",
    "ApiKey": "YOUR_API_KEY_HERE"
  },
  "StorageConfig": {
    "TilesDirectory": "/tiles",
    "ReadyDirectory": "/ready"
  },
  "ProcessingConfig": {
    "MaxConcurrentDownloads": 4,
    "DefaultZoomLevel": 20,
    "QueueCapacity": 100
  }
}

4.4 API Endpoints Summary

Method	Endpoint	Description
POST	/api/satellite/request	Request tiles for a region
GET	/api/satellite/region/{id}	Get region status and file paths

4.5 File Naming Conventions

Tiles:

/tiles/tile_{zoom}_{x}_{y}_{timestamp}.jpg
Example: /tiles/tile_20_12345_67890_20231015143022.jpg

CSV Files:

/ready/region_{id}_ready.csv
Example: /ready/region_550e8400-e29b-41d4-a716-446655440000_ready.csv

Summary Files:

/ready/region_{id}_summary.txt
Example: /ready/region_550e8400-e29b-41d4-a716-446655440000_summary.txt

4.6 CSV File Format

latitude,longitude,file_path
37.7749,-122.4194,/tiles/tile_20_12345_67890_20231015143022.jpg
37.7750,-122.4194,/tiles/tile_20_12346_67890_20231015143023.jpg

Rows ordered from top-left to bottom-right covering the requested region.

4.7 Summary File Format

Region Processing Summary
========================
Region ID: 550e8400-e29b-41d4-a716-446655440000
Center: 37.7749, -122.4194
Size: 500 meters
Zoom Level: 20

Processing Statistics:
- Tiles Downloaded: 45
- Tiles Reused from Cache: 12
- Total Tiles: 57
- Processing Time: 12.5 seconds
- Started: 2023-10-15 14:30:15
- Completed: 2023-10-15 14:30:28

Files Created:
- CSV: /ready/region_550e8400-e29b-41d4-a716-446655440000_ready.csv
- Summary: /ready/region_550e8400-e29b-41d4-a716-446655440000_summary.txt

5. Testing Strategy

5.1 Unit Tests

• Test GeoUtils calculations
• Test tile coordinate conversions
• Test repository methods (with mocked database)
• Test service business logic

5.2 Integration Tests

• End-to-end region request flow
• Database operations
• File system operations
• API endpoint testing
• Background processing

5.3 Test Coverage Goals

• Minimum 80% code coverage
• 100% coverage for critical paths (downloading, storage, region processing)

6. Deployment Strategy

6.1 Development Environment

docker-compose -f docker-compose.svc.yml up

6.2 Running Integration Tests

docker-compose -f docker-compose.tests.yml up --abort-on-container-exit

6.3 Production Deployment

• Use docker-compose.svc.yml as base
• Override with production-specific configuration
• Configure volumes for persistent storage
• Set up backup strategy for PostgreSQL
• Configure SSL/TLS for API
• Set up monitoring and alerting

7. Security Considerations

1. API Key Management: Store Google Maps API key in environment variables or secrets manager
2. Database Security: Use strong passwords, restrict network access
3. Input Validation: Validate all API inputs (coordinates, size limits)
4. File System Security: Restrict write access to /tiles and /ready directories
5. Rate Limiting: Implement to prevent abuse
6. HTTPS: Enforce HTTPS in production

8. Performance Considerations

1. Concurrent Downloads: Use 4 concurrent connections to Google Maps
2. Database Indexing: Composite index on (latitude, longitude, tile_size_meters)
3. Caching: Reuse existing tiles to minimize downloads
4. Async Operations: All I/O operations are async
5. Background Processing: Non-blocking region processing

9. Monitoring and Observability

1. Logging: Structured logging for all operations
2. Metrics: Track download counts, cache hit rates, processing times
3. Health Checks: API and database health endpoints
4. Error Tracking: Log errors with context for debugging

10. Future Enhancements

1. Cloud Queue Integration: Replace in-memory queue with Azure/AWS queue
2. CDN Integration: Serve tiles via CDN
3. Tile Expiration: Implement automatic tile refresh
4. Multiple Map Providers: Support for other satellite imagery providers
5. WebSocket Support: Real-time progress updates
6. Admin Dashboard: UI for monitoring and management
7. Tile Stitching: API to stitch tiles into single large image

11. Implementation Timeline

Total Estimated Time: 15-20 working days

Phase	Duration	Dependencies
Phase 1: Database Layer	2-3 days	None
Phase 2: Enhanced Tile Downloading	2-3 days	Phase 1
Phase 3: Background Processing	3-4 days	Phase 1, Phase 2
Phase 4: API Endpoints	1-2 days	Phase 3
Phase 5: Docker Configuration	2-3 days	Phase 4
Phase 6: Integration Tests	3-4 days	Phase 5
Phase 7: Additional Features	2-3 days	Phase 6

Critical Path: Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6

12. Risk Assessment

Risk	Impact	Probability	Mitigation
Google Maps API changes	High	Low	Monitor API changes, version locking
API quota exceeded	High	Medium	Implement caching, rate limiting
Disk space exhaustion	Medium	Medium	Implement cleanup policy, monitoring
Database performance	Medium	Low	Proper indexing, connection pooling
Concurrent processing issues	Medium	Low	Thorough testing, proper locking

13. Success Criteria

1. ✅ API successfully downloads and stores tiles
2. ✅ Metadata stored correctly in PostgreSQL
3. ✅ Region requests processed in background
4. ✅ CSV and summary files generated correctly
5. ✅ Tile caching works (reuses existing tiles)
6. ✅ Docker containers run successfully
7. ✅ Integration tests pass
8. ✅ API documentation complete
9. ✅ Performance meets requirements (< 30s for 1000m region)
10. ✅ Code coverage > 80%

14. Open Questions and Decisions Made

Questions from Analysis:

1. Legacy Code? ✅ Analyzed - Use current code as base
2. Background Processing? ✅ In-memory queue with Channels
3. Database Migrations? ✅ DbUp
4. Tile Size? ✅ 256x256 pixels (Google Maps standard)
5. Maps Version? ✅ Use download timestamp
6. Composite Key? ✅ Indexed composite key, UUID primary key
7. Integration Tests? ✅ Separate console app project

15. Next Steps

1. Review and approve this implementation plan
2. Set up development environment
3. Create feature branches for each phase
4. Begin Phase 1: Database Layer implementation
5. Daily stand-ups to track progress
6. Regular code reviews for each completed phase

---

Document Version: 1.0
Created: 2025-10-26
Last Updated: 2025-10-26
Author: AI Assistant
Approved By: [Pending]