mirror of
https://github.com/azaion/satellite-provider.git
synced 2026-06-21 11:31:14 +00:00
[AZ-484] Cycle 1 Steps 12-16: docs, security, perf, deploy report
Captures the post-implementation autodev gates for AZ-484 multi-source tile storage: - Step 12 (Test-Spec Sync): added 7 AC rows (AZ-484 AC-1..AC-7) and a PT-07 NFR row to traceability-matrix.md; added PT-07 scenario to performance-tests.md. - Step 13 (Update Docs): refreshed data_model.md (tiles columns + indexes + selection rule + UPSERT contract + migrations 012/013), module-layout.md (Common/Enums section with L-001 guidance, DataAccess imports-from now lists 6 sites), 6 module / component docs to reflect the new repo signatures, source/captured_at fields, and Dapper enum bypass workaround. ripple_log_cycle1.md records zero out-of-scope ripple. - Step 14 (Security Audit): PASS_WITH_WARNINGS - 0 Critical, 0 High, 5 Medium, 5 Low. AZ-484 itself added zero new findings. Hardening items (Postgres default creds, .env in build context, GMaps key rotation, ASP.NET Core 8.0.21 -> 8.0.25, rate limiter) recorded for separate tickets. - Step 15 (Performance Test): all PT-01..PT-07 scenarios Unverified (non-blocking); PT-07 baseline-comparison harness deferred to a leftover for next cycle. - Step 16 (Deploy): cycle deploy report covering migration safety, rollback path, post-deploy verification, security caveats. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -23,7 +23,7 @@ Runs DbUp-based SQL migrations against PostgreSQL on application startup. Ensure
|
||||
## Consumers
|
||||
- `Program.cs` — instantiated directly (not via DI) and called during startup. If migration fails, the application throws and does not start.
|
||||
|
||||
## Migrations (11 scripts)
|
||||
## Migrations (13 scripts)
|
||||
1. `001_CreateTilesTable.sql`
|
||||
2. `002_CreateRegionsTable.sql`
|
||||
3. `003_CreateIndexes.sql`
|
||||
@@ -35,6 +35,8 @@ Runs DbUp-based SQL migrations against PostgreSQL on application startup. Ensure
|
||||
9. `009_AddGeofencePolygonIndex.sql`
|
||||
10. `010_AddTilesZipToRoutes.sql`
|
||||
11. `011_AddTileCoordinates.sql`
|
||||
12. `012_DropTileVersionConstraint.sql` — drops the legacy 5-col `(latitude, longitude, tile_zoom, tile_size_meters, version)` unique index, replaces with 4-col `idx_tiles_unique_location` (preparation for AZ-484).
|
||||
13. `013_AddTileSourceAndCapturedAt.sql` — AZ-484 multi-source tile storage. Transactional. Adds `source` (VARCHAR(32) NOT NULL DEFAULT 'google_maps') and `captured_at` (TIMESTAMP NOT NULL) columns; backfills existing rows with `source='google_maps'`, `captured_at=created_at`; drops `idx_tiles_unique_location` and creates 5-col `idx_tiles_unique_location_source` on `(latitude, longitude, tile_zoom, tile_size_meters, source)`. Idempotent against partial replays.
|
||||
|
||||
## Configuration
|
||||
Receives connection string directly as constructor parameter.
|
||||
|
||||
@@ -9,7 +9,9 @@ Database entity classes that map directly to PostgreSQL tables via Dapper. Prope
|
||||
Maps to `tiles` table.
|
||||
- `Id` (Guid), `TileZoom` (int), `TileX` (int), `TileY` (int)
|
||||
- `Latitude`, `Longitude` (double), `TileSizeMeters` (double), `TileSizePixels` (int)
|
||||
- `ImageType` (string), `MapsVersion` (string?), `Version` (int)
|
||||
- `ImageType` (string), `MapsVersion` (string?), `Version` (int) — `MapsVersion`/`Version` are vestigial post-AZ-484 (kept nullable for backward compatibility; no longer part of the unique key)
|
||||
- `Source` (string) — AZ-484 producer wire value, defaults to `TileSourceConverter.GoogleMapsWireValue` (`"google_maps"`). Stored as plain string (not the `TileSource` enum) due to Dapper issue #259 — see `_docs/LESSONS.md` L-001. Convert via `SatelliteProvider.Common.Enums.TileSourceConverter.{ToWireValue,FromWireValue}`.
|
||||
- `CapturedAt` (DateTime, UTC) — AZ-484 imagery acquisition timestamp; drives the most-recent-across-sources selection.
|
||||
- `FilePath` (string), `CreatedAt`, `UpdatedAt` (DateTime)
|
||||
|
||||
### RegionEntity
|
||||
|
||||
@@ -7,26 +7,31 @@ Dapper-based repository for the `tiles` table. Handles CRUD operations and spati
|
||||
|
||||
### ITileRepository (interface)
|
||||
- `GetByIdAsync(Guid id) → Task<TileEntity?>`
|
||||
- `GetByTileCoordinatesAsync(int tileZoom, int tileX, int tileY) → Task<TileEntity?>`: finds tile by slippy map coordinates, returns latest version
|
||||
- `FindExistingTileAsync(double lat, double lon, double tileSizeMeters, int zoomLevel, int version) → Task<TileEntity?>`: fuzzy coordinate match (tolerance: 0.0001° lat/lon, 1m tile size)
|
||||
- `GetTilesByRegionAsync(double lat, double lon, double sizeMeters, int zoomLevel) → Task<IEnumerable<TileEntity>>`: spatial bounding box query with expanded range to cover tile edges
|
||||
- `InsertAsync(TileEntity tile) → Task<Guid>`: upsert — `ON CONFLICT (latitude, longitude, tile_zoom, tile_size_meters, version) DO UPDATE` file_path, tile_x, tile_y, updated_at
|
||||
- `UpdateAsync(TileEntity tile) → Task<int>`
|
||||
- `GetByTileCoordinatesAsync(int tileZoom, int tileX, int tileY) → Task<TileEntity?>`: finds the most-recent tile across all sources for the given slippy coordinates. Selection rule: `ORDER BY captured_at DESC, updated_at DESC, id DESC LIMIT 1` (AZ-484 v1.0.0 contract).
|
||||
- `GetTilesByRegionAsync(double lat, double lon, double sizeMeters, int zoomLevel) → Task<IEnumerable<TileEntity>>`: spatial bounding box query (expanded by 2 × tile size to cover edges); applies `DISTINCT ON (latitude, longitude, tile_zoom, tile_size_meters)` per AZ-484 to return at most one row per cell — the most-recent across sources — preserving the historical caller-facing order `latitude DESC, longitude ASC`.
|
||||
- `InsertAsync(TileEntity tile) → Task<Guid>`: per-source UPSERT — `ON CONFLICT (latitude, longitude, tile_zoom, tile_size_meters, source) DO UPDATE file_path, tile_x, tile_y, captured_at, updated_at` (AZ-484 5-column unique key).
|
||||
- `UpdateAsync(TileEntity tile) → Task<int>`: full row update by `id` including `source` and `captured_at`.
|
||||
- `DeleteAsync(Guid id) → Task<int>`
|
||||
|
||||
### TileRepository (implementation)
|
||||
Constructs a new `NpgsqlConnection` per method call (no connection pooling at the repository level; Npgsql pools connections internally).
|
||||
Constructs a new `NpgsqlConnection` per method call (no connection pooling at the repository level; Npgsql pools connections internally). Logs a `Slow GetTilesByRegionAsync` warning when the region query exceeds 500 ms.
|
||||
|
||||
## Internal Logic
|
||||
- `GetTilesByRegionAsync` calculates a bounding box by expanding the requested region by 2 × tile size to ensure edge tiles are included. Uses meters-to-degrees approximation (111,000 m/degree latitude, adjusted for longitude).
|
||||
- `InsertAsync` uses an upsert pattern to handle duplicate tile downloads gracefully.
|
||||
- `GetByTileCoordinatesAsync` orders by `version DESC` and takes the latest.
|
||||
- `GetTilesByRegionAsync` calculates a bounding box by expanding the requested region by 2 × tile size to ensure edge tiles are included. Uses meters-to-degrees approximation via `GeoUtils` (post-AZ-377 — single source of truth for Earth constants).
|
||||
- `InsertAsync` uses a per-source UPSERT pattern keyed on the 5-column unique index `idx_tiles_unique_location_source` (created by migration 013). Two producers (e.g., `google_maps` + `uav`) coexist for the same cell; same-source re-insert refreshes `captured_at` and `updated_at`.
|
||||
- `GetByTileCoordinatesAsync` and `GetTilesByRegionAsync` apply the AZ-484 selection rule: most-recent across sources, deterministic tie-break on `(captured_at DESC, updated_at DESC, id DESC)`.
|
||||
- `TileEntity.Source` is a plain `string` storing the snake_case wire value (`'google_maps'` | `'uav'`); enum<->wire conversion happens via `SatelliteProvider.Common.Enums.TileSourceConverter`. This avoids Dapper issue #259 (TypeHandler<T> bypass for enum reads — see `_docs/LESSONS.md` L-001).
|
||||
- `FindExistingTileAsync` was removed by AZ-376 (see `_docs/04_refactoring/03-code-quality-refactoring/`).
|
||||
|
||||
## Dependencies
|
||||
- NuGet: `Dapper`, `Npgsql`
|
||||
- `SatelliteProvider.DataAccess.Models.TileEntity`
|
||||
- `SatelliteProvider.Common.GeoUtils` (Earth constants / meters-to-degrees)
|
||||
- `Microsoft.Extensions.Logging`
|
||||
|
||||
## Contract
|
||||
Implements the frozen v1.0.0 contract `_docs/02_document/contracts/data-access/tile-storage.md`. Schema invariants Inv-1..Inv-5 are enforced here (UPSERT key, selection rule, source value space).
|
||||
|
||||
## Consumers
|
||||
- `TileService` — all read/write operations
|
||||
- `Program.cs` (ServeTile, GetTileByLatLon handlers) — `GetByTileCoordinatesAsync`, `InsertAsync`
|
||||
|
||||
@@ -9,9 +9,9 @@ Orchestrates tile downloading and persistence. Bridges the downloader (Google Ma
|
||||
|
||||
### TileService (implements ITileService)
|
||||
- `DownloadAndStoreTilesAsync(double lat, double lon, double sizeMeters, int zoomLevel, CancellationToken) → Task<List<TileMetadata>>`:
|
||||
1. Queries existing tiles in the region from the repository (latest per `(latitude, longitude, zoom_level, tile_size_meters)` post-AZ-357)
|
||||
1. Queries existing tiles in the region from the repository — most-recent across sources per `(latitude, longitude, tile_zoom, tile_size_meters)` (AZ-484 selection rule applied by `TileRepository.GetTilesByRegionAsync` via `DISTINCT ON`)
|
||||
2. Calls `ISatelliteDownloader.GetTilesWithMetadataAsync` with existing tiles to skip
|
||||
3. Creates `TileEntity` for each newly downloaded tile and inserts via repository (upsert)
|
||||
3. Creates `TileEntity` for each newly downloaded tile and inserts via repository (per-source UPSERT keyed on `(latitude, longitude, tile_zoom, tile_size_meters, source)`)
|
||||
4. Returns combined list of existing + new tile metadata
|
||||
- `GetTileAsync(Guid id) → Task<TileMetadata?>`: single tile lookup
|
||||
- `GetTilesByRegionAsync(double lat, double lon, double sizeMeters, int zoomLevel) → Task<IEnumerable<TileMetadata>>`: query tiles in a region
|
||||
@@ -20,7 +20,8 @@ Orchestrates tile downloading and persistence. Bridges the downloader (Google Ma
|
||||
|
||||
## Internal Logic
|
||||
- New rows write `Version = null` and `MapsVersion = null` (post-AZ-357 / AZ-373); the `version` and `maps_version` columns are retained for backward compatibility with pre-existing rows
|
||||
- `MapToMetadata(TileEntity) → TileMetadata`: entity-to-DTO mapping (static helper); `MapsVersion` is no longer projected onto `TileMetadata` / `DownloadTileResponse`
|
||||
- AZ-484: `BuildTileEntity` stamps every newly downloaded row with `Source = TileSourceConverter.ToWireValue(TileSource.GoogleMaps)` (wire value `"google_maps"`) and `CapturedAt = DateTime.UtcNow`. The Google Maps download path is the only producer of `'google_maps'` rows; UAV ingestion (separate task) is the only producer of `'uav'` rows.
|
||||
- `MapToMetadata(TileEntity) → TileMetadata`: entity-to-DTO mapping (static helper); `MapsVersion` is no longer projected onto `TileMetadata` / `DownloadTileResponse`. `Source` and `CapturedAt` are not currently projected to the public DTO (no API contract change observable for AZ-484).
|
||||
- `TileSizePixels` sourced from `MapConfig.TileSizePixels` (default 256, post-AZ-371); image type fixed at `"jpg"`
|
||||
- `IMemoryCache` keyed by `(z, x, y)` with 1h absolute / 30min sliding expiration; populated on first hit and on downloader fallback
|
||||
|
||||
@@ -29,6 +30,7 @@ Orchestrates tile downloading and persistence. Bridges the downloader (Google Ma
|
||||
- `ITileRepository`
|
||||
- `IMemoryCache` (registered by `AddTileDownloader()`)
|
||||
- `SatelliteProvider.Common.DTO` — GeoPoint, TileMetadata, TileBytes
|
||||
- `SatelliteProvider.Common.Enums` — `TileSource`, `TileSourceConverter` (AZ-484)
|
||||
- `SatelliteProvider.DataAccess.Models` — TileEntity
|
||||
|
||||
## Consumers
|
||||
|
||||
@@ -11,6 +11,7 @@ Console application that runs end-to-end integration tests against a live API in
|
||||
- `BasicRouteTests` — route creation with intermediate points
|
||||
- `ComplexRouteTests` — routes with geofencing
|
||||
- `ExtendedRouteTests` — routes with `requestMaps: true` and tile ZIP creation
|
||||
- `MigrationTests` — direct PostgreSQL schema/index validation (no HTTP). AZ-484 cycle added: `NewUniqueConstraintIncludesSourceColumn_AZ484_AC1`, `BackfillUpdateAssignsGoogleMapsAndCapturedAt_AZ484_AC4`, `MultiSourceInsertCoexistsUnderNewIndex_AZ484_AC1`, `MostRecentAcrossSourcesSelection_AZ484_AC2`, `SameSourceUpsertReplacesPreviousRow_AZ484_AC3` (latter four use temp tables to keep production data untouched).
|
||||
|
||||
### Supporting Classes
|
||||
- `Models.cs` — HTTP response DTOs for deserialization
|
||||
|
||||
Reference in New Issue
Block a user