azaion/satellite-provider
1
0
Fork 0
mirror of https://github.com/azaion/satellite-provider.git synced 2026-06-21 10:31:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-284] Autodev baseline + testability refactor

Browse Source 
Phase A baseline outputs from /autodev (Steps 1-5):
- Problem & solution docs (_docs/00_problem, _docs/01_solution)
- Codebase documentation (_docs/02_document) incl. architecture,
  module-layout, glossary, system-flows, baseline compliance scan
- Test specs (blackbox, performance, resilience, security, resource,
  traceability matrix)
- Test task decomposition (_docs/02_tasks/todo): AZ-285..AZ-290
- Testability refactor (_docs/04_refactoring/01-testability-refactoring):
  - TC-01 Move DownloadedTileInfoV2 + new ExistingTileInfo to Common.DTO
  - TC-02 Replace dead ISatelliteDownloader API with real signatures
  - TC-03 GoogleMapsDownloaderV2 implements ISatelliteDownloader
  - TC-04 TileService depends on ISatelliteDownloader (mockable)
  - TC-05 DI + endpoints use ISatelliteDownloader
- Test runner scripts (scripts/run-tests.sh, run-performance-tests.sh)
- Autodev state pointer (_docs/_autodev_state.md)

Prepares the codebase for AZ-285..AZ-290 unit/integration test work.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-10 04:44:08 +03:00
parent 25a644a9bf
commit b0fffa6d42
68 changed files with 4192 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_document/architecture.md
+182
View File
@@ -0,0 +1,182 @@
# Satellite Provider — Architecture
## Architecture Vision
Satellite Provider is a self-hosted .NET 8.0 backend service that pre-downloads, caches, and composites Google Maps satellite imagery for offline use. It runs as a single containerized monolith with PostgreSQL, processing requests asynchronously via in-process queues. The dominant pattern is a layered architecture (API → Services → DataAccess → PostgreSQL) with background hosted services for long-running work.
**Components & responsibilities**:
- **Common** — Shared contracts: DTOs, service interfaces, configuration models, geospatial math
- **DataAccess** — PostgreSQL persistence via Dapper + DbUp migrations
- **TileDownloader** — Provider-agnostic tile acquisition via `ISatelliteDownloader` interface (first implementation: Google Maps) with deduplication and concurrency control
- **RegionProcessing** — Batch tile downloads for geographic areas, stitching, output generation
- **RouteManagement** — Route interpolation, geofenced region generation, consolidated map output
**Major data flows**:
- *Tile acquisition*: HTTP request → cache check → Google Maps download → disk + DB persistence
- *Region processing*: Request queued → background worker calculates tile grid → downloads all tiles → produces CSV/summary/stitched image
- *Route expansion*: Waypoints → interpolated points every ~200m → geofence filtering → region requests per point → optional ZIP archive
**Architectural principles** (inferred):
- Single-instance deployment, no horizontal scaling requirements (`inferred-from: Channel-based queue, no distributed state`)
- Immutable tile storage with year-based versioning for cache invalidation (`inferred-from: version column + unique index`)
- Fire-and-forget async processing with status polling (`inferred-from: queue + background service + status endpoint`)
- No authentication layer — designed as an internal/trusted network service (`inferred-from: no auth middleware in Program.cs`)
**Planned features** (confirmed by user, currently stubs):
- MGRS endpoint — tile access via Military Grid Reference System coordinates
- Upload endpoint — UAV nadir camera tile ingestion (Layer 2: orthogonal tiles uploaded post-flight, stored alongside Google Maps Layer 1; most recent layer returned on access)
**Drift signals**:
- `geofence_polygons` mentioned in AGENTS.md as a routes table column but does not exist in schema or entity — documentation drift
## 1. System Context
**Problem being solved**: A GPS-denied UAV navigation service requires satellite imagery for positioning and route planning without GPS. This service pre-downloads Google Maps satellite tiles (Layer 1) for specified regions and routes, accepts UAV-captured nadir camera imagery uploaded post-flight (Layer 2), and serves the most recent tile layer on access. Tiles are stitched into composite images and packaged for offline use.
**System boundaries**: The Satellite Provider is a self-contained backend service. It receives HTTP requests (region/route definitions), downloads tiles from Google Maps, stores them on disk and in PostgreSQL, and produces output files (images, CSVs, ZIPs).
**External systems**:
| System | Integration Type | Direction | Purpose |
|--------|-----------------|-----------|---------|
| Satellite imagery provider (e.g., Google Maps) | HTTPS (tile download) | Outbound | Layer 1 satellite imagery source (provider-agnostic via `ISatelliteDownloader`) |
| GPS-Denied Service (UAV) | REST API | Inbound | Layer 2 nadir camera tile uploads post-flight |
| PostgreSQL | TCP (Npgsql) | Both | Tile metadata, region/route state |
| File System | Local disk | Both | Tile image storage, output artifacts |
| HTTP Clients | REST API | Inbound | Region/route requests, tile queries |
## 2. Technology Stack
| Layer | Technology | Version | Rationale |
|-------|-----------|---------|-----------|
| Language | C# | 12.0 | .NET ecosystem, strong typing |
| Framework | ASP.NET Core (Minimal API) | 8.0 | Lightweight HTTP hosting |
| Database | PostgreSQL | 15+ | Reliable RDBMS, spatial-friendly |
| ORM | Dapper | latest | Micro-ORM, raw SQL control |
| Migrations | DbUp | latest | Simple SQL-file-based schema migrations |
| Image Processing | SixLabors.ImageSharp | 3.1.11 | Cross-platform image manipulation |
| Logging | Serilog | 8.0.3 | Structured logging with file sinks |
| Hosting | Docker (docker-compose) | — | Containerized deployment |
| CI/CD | Woodpecker CI | — | Lightweight self-hosted CI |
## 3. Deployment Model
**Environments**: Development (docker-compose), Production (Docker)
**Infrastructure**:
- Docker-based containerized deployment
- PostgreSQL as a separate container
- Shared volumes for tile storage and output artifacts
- No cloud provider dependency (self-hosted capable)
**Environment-specific configuration**:
| Config | Development | Production |
|--------|-------------|------------|
| Database | localhost:5432 (Docker) | Container network `db:5432` |
| Secrets | appsettings.Development.json | Environment variables |
| Logging | Console + File | File (./logs/) |
| API URL | http://localhost:5100 | http://0.0.0.0:5100 |
## 4. Data Model Overview
**Core entities**:
| Entity | Description | Owned By Component |
|--------|-------------|--------------------|
| Tile | A single satellite image tile with coordinates and zoom | TileDownloader |
| Region | A square area request with processing status | RegionProcessing |
| Route | A named path with geofence polygons | RouteManagement |
| RoutePoint | An individual point (original or interpolated) on a route | RouteManagement |
**Key relationships**:
- Route → RoutePoint: one-to-many (a route has many sequential points)
- Route → Region: many-to-many via `route_regions` (each route point generates a region)
- Region → Tile: implicit (a processed region references tiles by coordinate/zoom)
**Data flow summary**:
- Client → API → Queue → BackgroundService → GoogleMaps → FileSystem + DB: tile acquisition pipeline
- Client → API → RouteService → PointInterpolation → RegionCreation → Queue: route-to-region expansion
## 5. Integration Points
### Internal Communication
| From | To | Protocol | Pattern | Notes |
|------|----|----------|---------|-------|
| WebApi | RegionProcessing | In-process queue (Channel) | Fire-and-forget | Request queued, status polled |
| WebApi | TileDownloader | Direct method call | Request-Response | Synchronous single-tile download |
| RegionProcessing | TileDownloader | Direct method call | Request-Response | Per-tile within region processing |
| RouteManagement | RegionProcessing | In-process queue (Channel) | Fire-and-forget | Route regions submitted to queue |
| All Services | DataAccess | Direct method call | Repository pattern | Dapper queries |
### External Integrations
| External System | Protocol | Auth | Rate Limits | Failure Mode |
|----------------|----------|------|-------------|--------------|
| Satellite imagery provider (abstracted via `ISatelliteDownloader`; first implementation: Google Maps) | HTTPS GET | Provider-specific (e.g., session token) | Configured concurrency (MaxConcurrentDownloads) | Retry with backoff, mark region failed |
## 6. Non-Functional Requirements
| Requirement | Target | Measurement | Priority |
|------------|--------|-------------|----------|
| Concurrent Downloads | 4 (configurable) | SemaphoreSlim limit | High |
| Concurrent Regions | 20 (configurable) | Processing config | Medium |
| Queue Capacity | 1000 requests | Channel bounded capacity | Medium |
| Tile Deduplication | 100% (no re-download) | DB lookup before fetch | High |
| Max Zip Size | 50 MB | Route zip output | Medium |
## 7. Security Architecture
**Authentication**: None (internal service, no auth layer)
**Authorization**: None (all endpoints are open)
**Data protection**:
- At rest: No encryption (tiles stored as plain JPEG files)
- In transit: HTTPS for Google Maps calls; API itself on HTTP
- Secrets management: Google Maps session token in appsettings / env vars
**Audit logging**: Serilog writes to file; logs exceptions and processing state transitions
## 8. Key Architectural Decisions
### ADR-001: Minimal API over Controller-based
**Context**: Project needed a lightweight HTTP layer for a small set of endpoints.
**Decision**: Use ASP.NET Core Minimal APIs (no controllers, no MVC).
**Consequences**: Less ceremony, all routing in `Program.cs`, but less structure for future growth.
### ADR-002: Dapper over Entity Framework
**Context**: Database access is straightforward CRUD with some spatial queries.
**Decision**: Use Dapper for raw SQL control and performance, paired with DbUp for schema migrations.
**Consequences**: Full SQL control, no ORM overhead; trade-off is manual mapping and no change tracking.
### ADR-003: In-Process Queue over External Message Broker
**Context**: Region/route processing needs to be asynchronous but the system is a single service.
**Decision**: Use `System.Threading.Channels` as an in-process bounded queue.
**Consequences**: Simple, no external dependencies; but limited to single-instance deployment — no horizontal scaling of workers.
### ADR-004: File-Based Tile Storage
**Context**: Tiles are immutable JPEG images that need fast random access.
**Decision**: Store tiles as files in a directory hierarchy (`./tiles/{zoom}/{x}/{y}.jpg`) with metadata in PostgreSQL.
**Consequences**: Fast reads, easy backup/migration, but requires shared filesystem for multi-instance (which is not currently needed).
### ADR-005: Background Hosted Services for Processing
**Context**: Region and route processing is long-running and should not block HTTP requests.
**Decision**: Use `IHostedService` implementations that consume from the in-process queue.
**Consequences**: Clean separation of request handling and processing; lifecycle managed by the host.