[AZ-488] UAV tile batch upload + 5-rule quality gate

Replaces the 501 stub at POST /api/satellite/upload with a multipart
batch endpoint that ingests UAV-captured tiles, runs each item through
a 5-rule quality gate, and persists accepted tiles via the AZ-484
multi-source storage path with source='uav'.

Quality gate (in fixed order, first failure wins): JPEG format
(content-type + magic), size band 5 KiB-5 MiB, exact 256x256
dimensions, captured-at age (no future >30 s skew, no older than
7 days), luminance variance on 32x32 downsample. Closed reject-reason
enumeration in v1.0.0 contract.

Authorization: custom PermissionsRequirement / PermissionsAuthorization
Handler that reads the JWT `permissions` claim (tolerates both
repeated-string and JSON-array shapes). Endpoint protected by
RequiresGpsPermission policy; 401 without token, 403 without GPS perm.

Persistence: file-first to ./tiles/uav/{z}/{x}/{y}.jpg, then
ITileRepository.InsertAsync UPSERT (per-source UPSERT contract from
AZ-484). Per-item failures reported in response without aborting the
batch. Kestrel MaxRequestBodySize and FormOptions limits set to
MaxBatchSize x MaxBytes (default 100 x 5 MiB = 500 MiB).

New frozen contract: _docs/02_document/contracts/api/uav-tile-upload.md
v1.0.0. PT-08 NFR added to performance-tests.md as Deferred (harness
work tracked in PT-07 leftover, per AZ-488 § Risk 4).

Tests: 11 quality-gate unit tests, 5 handler unit tests, 3 file-path
unit tests, 12 permission-handler unit tests, 7 integration tests
(AC-1..AC-6, AC-8). All 253 unit tests + smoke integration suite
green.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-11 23:50:49 +03:00
parent 11b7074485
commit 1802d32107
35 changed files with 2280 additions and 107 deletions
+13 -6
View File
@@ -33,9 +33,12 @@ The three Layer-3 service components are compile-time siblings: each only refere
**Planned features** (confirmed by user, currently stubs):
- MGRS endpoint — tile access via Military Grid Reference System coordinates
- Upload endpoint — UAV nadir camera tile ingestion. Writes a row with `source='uav'` for the captured cell; the storage layer accepts it alongside any existing Google Maps row, and reads return whichever has the highest `captured_at`. AZ-484 has built the multi-source storage; the upload endpoint itself (T2 — AZ-485) and any quality-gate logic remain to be implemented.
The N-source storage contract is authoritative in `_docs/02_document/contracts/data-access/tile-storage.md` (v1.0.0). Anything that reads or writes `tiles` MUST follow that contract rather than re-deriving the rules from prose here.
**Multi-source tile producers** (live as of AZ-488):
- *Google Maps* — `TileService.DownloadAndStoreTilesAsync` / `DownloadAndStoreSingleTileAsync` stamp `source='google_maps'` on every persisted row; tile JPEGs live under `{StorageConfig.TilesDirectory}/{zoom}/...` per the legacy grandfathered layout.
- *UAV* — `POST /api/satellite/upload` (AZ-488) accepts a multipart batch of UAV-captured tiles, runs each item through a 5-rule quality gate (`UavTileQualityGate`), and persists accepted items via `ITileRepository.InsertAsync` with `source='uav'`. UAV JPEGs live under `{StorageConfig.TilesDirectory}/uav/{zoom}/{x}/{y}.jpg`. Requires the `GPS` permission claim on top of the JWT baseline.
The N-source storage contract is authoritative in `_docs/02_document/contracts/data-access/tile-storage.md` (v1.0.0). The UAV upload contract is authoritative in `_docs/02_document/contracts/api/uav-tile-upload.md` (v1.0.0). Anything that reads or writes `tiles` MUST follow those contracts rather than re-deriving the rules from prose here.
**Drift signals**:
- `geofence_polygons` mentioned in AGENTS.md as a routes table column but does not exist in schema or entity — documentation drift
@@ -51,7 +54,7 @@ The N-source storage contract is authoritative in `_docs/02_document/contracts/d
| System | Integration Type | Direction | Purpose |
|--------|-----------------|-----------|---------|
| Satellite imagery provider (e.g., Google Maps) | HTTPS (tile download) | Outbound | First implementation of the multi-source `tiles` storage; provider-agnostic via `ISatelliteDownloader`. Stamps `source='google_maps'` on every persisted row. |
| GPS-Denied Service (UAV) | REST API | Inbound | Future producer of `source='uav'` rows via the upload endpoint (T2 — AZ-485). The storage layer (AZ-484) is already in place; the endpoint itself is still a stub. |
| GPS-Denied Service (UAV) | REST API (multipart) | Inbound | Producer of `source='uav'` rows via `POST /api/satellite/upload` (AZ-488). Authenticates with a JWT carrying the `GPS` permission claim; items pass through the 5-rule quality gate before persistence. |
| PostgreSQL | TCP (Npgsql) | Both | Tile metadata, region/route state |
| File System | Local disk | Both | Tile image storage, output artifacts |
| HTTP Clients | REST API | Inbound | Region/route requests, tile queries |
@@ -141,7 +144,7 @@ The N-source storage contract is authoritative in `_docs/02_document/contracts/d
**Authentication**: HS256 JWT Bearer tokens (AZ-487). Signing key from `JWT_SECRET` env var (≥ 32 bytes, validated at startup). `Microsoft.AspNetCore.Authentication.JwtBearer` validates signature, lifetime, and signing key; issuer and audience are intentionally not validated (suite contract does not specify expected values). ClockSkew tightened from JwtBearer default (5 min) to 30 s. Tokens are minted by the centralized Admin API per `suite/_docs/10_auth.md`.
**Authorization**: Every endpoint requires authentication via `.RequireAuthorization()`. Permission-claim enforcement (e.g. `permissions: ["GPS"]`) is added per-endpoint where needed — AZ-488 introduces it on `POST /api/satellite/upload`. Other endpoints accept any authenticated principal.
**Authorization**: Every endpoint requires authentication via `.RequireAuthorization()`. Permission-claim enforcement is layered on top through the `PermissionsRequirement` authorization handler, which reads the `permissions` claim (accepting either repeated string claims OR a single JSON-array string). AZ-488 wires the `RequiresGpsPermission` policy on `POST /api/satellite/upload` — callers without `GPS` receive HTTP 403; other endpoints accept any authenticated principal.
**Data protection**:
- At rest: No encryption (tiles stored as plain JPEG files)
@@ -180,9 +183,13 @@ The N-source storage contract is authoritative in `_docs/02_document/contracts/d
**Context**: Tiles are immutable JPEG images that need fast random access.
**Decision**: Store tiles as files in a directory hierarchy (`./tiles/{zoom}/{x}/{y}.jpg`) with metadata in PostgreSQL.
**Decision**: Store tiles as files in a directory hierarchy with metadata in PostgreSQL. The layout is per-source so the bytes for `google_maps` and `uav` writes for the same cell remain individually addressable on disk:
- Google Maps (legacy, grandfathered): `{StorageConfig.TilesDirectory}/{zoom}/{x_bucket}/{y_bucket}/tile_{zoom}_{x}_{y}_{timestamp}.jpg`
- UAV (AZ-488): `{StorageConfig.TilesDirectory}/uav/{zoom}/{x}/{y}.jpg`
**Consequences**: Fast reads, easy backup/migration, but requires shared filesystem for multi-instance (which is not currently needed).
The authoritative source marker is the `tiles.source` column; the per-source on-disk path matters only for write isolation between producers.
**Consequences**: Fast reads, easy backup/migration, both producers can run without colliding on bytes, but requires shared filesystem for multi-instance (which is not currently needed). No migration of pre-AZ-488 Google Maps files is shipped — the legacy layout stays intact.
### ADR-005: Background Hosted Services for Processing
@@ -2,15 +2,15 @@
## 1. High-Level Overview
**Purpose**: Acquires satellite imagery tiles from Google Maps, stores them on disk, and persists metadata to the database. Handles session tokens, concurrent downloads, retry logic, and tile deduplication.
**Purpose**: Acquires satellite imagery tiles from Google Maps, stores them on disk, and persists metadata to the database. Handles session tokens, concurrent downloads, retry logic, and tile deduplication. Since AZ-488 it also hosts the UAV upload pipeline: the `UavTileQualityGate` 5-rule validator and the `UavTileUploadHandler` that persists `source='uav'` rows via `ITileRepository.InsertAsync`.
**Architectural Pattern**: Service + Gateway (wraps external API with retry/throttling)
**Architectural Pattern**: Service + Gateway (wraps external API with retry/throttling) + per-source quality gate (UAV upload path)
**csproj**: `SatelliteProvider.Services.TileDownloader/SatelliteProvider.Services.TileDownloader.csproj` (split out of the monolithic `SatelliteProvider.Services` project in epic AZ-309)
**Upstream dependencies**: Common (DTOs, Enums — `TileSource` + `TileSourceConverter` since AZ-484, GeoUtils, configs, RateLimitException), DataAccess (TileEntity, ITileRepository)
**Upstream dependencies**: Common (DTOs, Enums — `TileSource` + `TileSourceConverter` since AZ-484, plus `UavTileMetadata` / `UavTileBatchUploadResponse` / `UavTileRejectReasons` since AZ-488; GeoUtils; configs `MapConfig`, `StorageConfig`, `ProcessingConfig`, `UavQualityConfig`; RateLimitException), DataAccess (TileEntity, ITileRepository), SixLabors.ImageSharp 3.1.11 (UAV decode + variance check).
**Downstream consumers**: RegionProcessing and WebApi — both via `ITileService` from Common (no compile-time `ProjectReference` from any consumer to this project's concrete types).
**Downstream consumers**: RegionProcessing and WebApi — both via `ITileService` from Common; WebApi also resolves `IUavTileQualityGate` and `IUavTileUploadHandler` from this component (DI only — no compile-time `ProjectReference` from any consumer to this project's concrete types).
## 2. Internal Interfaces
@@ -29,6 +29,20 @@
| `GetOrDownloadTileAsync` (AZ-310) | z, x, y, CancellationToken | `TileBytes` | Yes | propagated from downloader |
| `DownloadAndStoreSingleTileAsync` (AZ-311) | lat, lon, zoom, CancellationToken | `TileMetadata` | Yes | propagated from downloader |
### Service: UavTileQualityGate (implements IUavTileQualityGate, AZ-488)
| Method | Input | Output | Async | Error Types |
|--------|-------|--------|-------|-------------|
| `Validate` | imageBytes, contentType, `UavTileMetadata` | `UavTileQualityResult` (accept + reason code) | No | none (decode exceptions caught and translated to `INVALID_FORMAT`) |
Rules run in fixed order (Format → Size band → Dimensions → Captured-at age → Blank/uniform); first failure short-circuits. Thresholds come from `UavQualityConfig`. Time comes from injected `TimeProvider` (defaults to `TimeProvider.System`) for deterministic tests.
### Service: UavTileUploadHandler (implements IUavTileUploadHandler, AZ-488)
| Method | Input | Output | Async | Error Types |
|--------|-------|--------|-------|-------------|
| `HandleAsync` | `metadataJson`, `IReadOnlyList<UavUploadFile>`, CancellationToken | `UavTileUploadHandlerResult` (envelope error OR per-item response) | Yes | propagated `IOException`/`UnauthorizedAccessException` per item, translated to per-item `STORAGE_FAILURE` |
Per-item flow: parse metadata JSON → reject envelope (mismatch, oversize, malformed JSON) OR run each item through `IUavTileQualityGate` → for accepted items, write JPEG to `{StorageConfig.TilesDirectory}/uav/{zoom}/{x}/{y}.jpg` then call `ITileRepository.InsertAsync` (per-source UPSERT) with `source='uav'` and the request-supplied `capturedAt`. File-before-row ordering keeps an orphan file (rather than a row pointing at nothing) when persistence fails.
## 4. Data Access Patterns
### Caching Strategy
@@ -0,0 +1,179 @@
# Contract: uav-tile-upload
**Component**: WebApi (`SatelliteProvider.Api`) producing rows via TileDownloader (`SatelliteProvider.Services.TileDownloader`)
**Producer task**: AZ-488 — `_docs/02_tasks/done/AZ-488_uav_tile_upload.md`
**Consumer tasks**: `gps-denied-onboard`, mission planner UI, any future UAV-equipped client
**Version**: 1.0.0
**Status**: frozen
**Last Updated**: 2026-05-11
## Purpose
Defines the HTTP contract for the `POST /api/satellite/upload` batch endpoint that ingests UAV-captured satellite tiles, runs each item through a 5-rule quality gate, and persists accepted rows under the `tile-storage` v1.0.0 data contract with `source='uav'`.
## Endpoint
```
POST /api/satellite/upload
Content-Type: multipart/form-data
Authorization: Bearer <JWT>
```
The request MUST carry a valid JWT (AZ-487) AND the `permissions` claim MUST contain `GPS`. Anonymous requests are rejected with HTTP 401; authenticated requests without `GPS` are rejected with HTTP 403.
## Request shape
Multipart form fields (case-sensitive part names):
| Part | Type | Required | Description |
|------|------|----------|-------------|
| `metadata` | JSON string | yes | Document of shape `{ "items": [ <UavTileMetadata>, ... ] }`. See the table below for the per-item schema. |
| `files` | binary (repeating) | yes | One JPEG file per metadata item. Files are correlated to metadata entries by ordinal index (file index `i``metadata.items[i]`). Each part MUST be sent under the form field name `files`. |
### `UavTileMetadata` (per item)
| Field | Type | Required | Description | Constraints |
|-------|------|----------|-------------|-------------|
| `latitude` | number | yes | Geographic latitude of the tile center | WGS-84 decimal degrees |
| `longitude` | number | yes | Geographic longitude of the tile center | WGS-84 decimal degrees |
| `tileZoom` | integer | yes | Slippy Map zoom level | Must satisfy the same zoom-level policy as the existing tile pipeline (see `MapConfig.AllowedZoomLevels`) |
| `tileSizeMeters` | number | yes | Tile size in meters at the captured latitude | Producer-supplied |
| `capturedAt` | string (ISO-8601 UTC) | yes | Moment of UAV image capture | Must satisfy the captured-at rule (see Quality Gate, Rule 4) |
Field names are camelCase. Property-name matching is case-insensitive on read.
### Constraints
- The number of `files` MUST equal the number of `metadata.items` entries (1:1 correlation by ordinal index). Mismatch → HTTP 400.
- `metadata.items.length` MUST NOT exceed `UavQualityConfig.MaxBatchSize` (default `100`). Oversize → HTTP 400.
- The total request body size is capped at `MaxBatchSize × MaxBytes` by Kestrel's `MaxRequestBodySize` and the form `MultipartBodyLengthLimit`. Requests beyond this cap are rejected at the framework layer (HTTP 413/400).
## Quality Gate (5 rules)
Each item is evaluated in this fixed order. The **first** failing rule produces the reported reason; subsequent rules are not evaluated for that item.
| # | Rule | Failure condition | Reason code |
|---|------|-------------------|-------------|
| 1 | Format | `Content-Type` of the file part is not `image/jpeg` (case-insensitive, allowing trailing parameters) OR the file's first 3 bytes are not `FF D8 FF` | `INVALID_FORMAT` |
| 2 | Size band | `bytes.length` is outside `[UavQualityConfig.MinBytes, UavQualityConfig.MaxBytes]` (defaults: 5 KiB … 5 MiB) | `SIZE_OUT_OF_BAND` |
| 3 | Dimensions | Image width OR height ≠ `MapConfig.TileSizePixels` (default 256). Strict equality, no tolerance | `WRONG_DIMENSIONS` |
| 4a | Captured-at future | `capturedAt > now + UavQualityConfig.CapturedAtFutureSkewSeconds` (default 30s) | `CAPTURED_AT_FUTURE` |
| 4b | Captured-at age | `capturedAt < now - UavQualityConfig.MaxAgeDays` (default 7 days) | `CAPTURED_AT_TOO_OLD` |
| 5 | Blank / uniform | Pixel-luminance variance on a downsampled (default 32×32, configurable via `UavQualityConfig.LuminanceSampleSize`) version of the image is below `UavQualityConfig.MinLuminanceVariance` (default 10.0) | `IMAGE_TOO_UNIFORM` |
If the file decode itself fails for the variance check, the item is rejected with `INVALID_FORMAT` (the file is not a real JPEG even though the header bytes matched).
Storage failures while persisting an accepted item (e.g., disk full, unwritable path) are surfaced as `STORAGE_FAILURE` so the client can retry that specific item without re-uploading the whole batch.
### Reject-reason enumeration (closed)
| Code | Source rule | When the reason fires |
|------|-------------|------------------------|
| `INVALID_FORMAT` | Rule 1 + Rule 5 decode safety net | Wrong content-type, wrong magic bytes, or undecodable bytes |
| `SIZE_OUT_OF_BAND` | Rule 2 | Byte length outside `[MinBytes, MaxBytes]` |
| `WRONG_DIMENSIONS` | Rule 3 | Width or height ≠ `MapConfig.TileSizePixels` |
| `CAPTURED_AT_FUTURE` | Rule 4a | `capturedAt` is more than `CapturedAtFutureSkewSeconds` in the future |
| `CAPTURED_AT_TOO_OLD` | Rule 4b | `capturedAt` is older than `MaxAgeDays` |
| `IMAGE_TOO_UNIFORM` | Rule 5 | Luminance variance below `MinLuminanceVariance` |
| `METADATA_MISSING` | Validation gate | Per-item metadata could not be matched (reserved; not currently surfaced because batch mismatch is reported as an envelope error) |
| `STORAGE_FAILURE` | Persistence path | The image bytes could not be written to disk or the repository insert raised an IO error |
Adding a new code is a **minor** contract version bump per the Versioning Rules below. Removing or renaming a code is **major**.
## Response shape
### HTTP 200 — per-item results
```json
{
"items": [
{ "index": 0, "status": "accepted", "tileId": "11111111-...", "rejectReason": null, "rejectDetails": null },
{ "index": 1, "status": "rejected", "tileId": null, "rejectReason": "WRONG_DIMENSIONS", "rejectDetails": null }
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `items` | array | One result per input item, in the same order as `metadata.items` |
| `items[].index` | integer | Ordinal index of the item in the request batch |
| `items[].status` | string | `"accepted"` or `"rejected"` (closed enumeration) |
| `items[].tileId` | UUID or null | When `accepted`, the persisted `tiles.id` value; null when `rejected` |
| `items[].rejectReason` | string or null | Reason code (see table above) when `rejected`; null when `accepted` |
| `items[].rejectDetails` | string or null | Optional human-readable detail; MUST NOT leak server paths, exception types, or internal identifiers |
### HTTP 400 — envelope error (RFC 7807 `application/problem+json`)
Returned when the request itself is malformed:
- `metadata` field absent, empty, or not valid JSON
- `metadata.items` empty or null
- `metadata.items.length``files.length`
- `metadata.items.length` > `MaxBatchSize`
The 5-rule per-item quality gate never produces a 400; per-item failures always surface in the HTTP 200 response array.
### HTTP 401 — missing or invalid JWT (from AZ-487)
### HTTP 403 — JWT present but `permissions` claim does not include `GPS`
## Persistence semantics
- Accepted items are persisted via `ITileRepository.InsertAsync` (the per-source UPSERT path established in AZ-484). The `tiles` row carries `source='uav'` and `captured_at` from the request.
- A UAV upload for a cell that already has a `google_maps` row **coexists** with that row (per `tile-storage.md` Inv-3). The most-recent row across sources wins on read.
- A second UAV upload for the same cell UPSERTs the existing `uav` row, updating `file_path`, `captured_at`, `updated_at` and overwriting the JPEG bytes on disk.
## File-path layout
- UAV files: `{StorageConfig.TilesDirectory}/uav/{tile_zoom}/{tile_x}/{tile_y}.jpg`
- Google Maps files: unchanged from the pre-AZ-488 layout (grandfathered, no migration ships with this contract)
`tile_x` and `tile_y` are derived server-side from `(latitude, longitude, tile_zoom)` via `GeoUtils.WorldToTilePos`; the client cannot influence the on-disk path beyond providing valid coordinates.
## Concurrency
- Per-source UPSERT in the DB is the authoritative serialization point for the same cell.
- Two concurrent UAV uploads for the exact same `(tile_zoom, tile_x, tile_y)` cell may race on the on-disk bytes; the final file is whichever upload wrote last, and the final DB row is whichever upload UPSERTed last. Per-source `file_path` is identical in this race so the file and row remain self-consistent — there is no orphan reference even under contention.
## Invariants
- **Inv-1**: Status strings are limited to `"accepted"` and `"rejected"`. New status values require a contract minor bump.
- **Inv-2**: Reject reasons are drawn from the closed enumeration above. New reasons require a contract minor bump.
- **Inv-3**: Persisted rows are inserted **only** via `ITileRepository.InsertAsync`. No new write path is introduced by this contract.
- **Inv-4**: `rejectDetails` MUST NOT contain server-side file paths, .NET exception type names, or internal identifiers. Operators consume these messages directly.
- **Inv-5**: Item ordering in the response matches item ordering in `metadata.items`.
## Non-Goals
- **Not covered**: Asynchronous or queued processing — the batch is synchronous. Larger batches than `MaxBatchSize` require a new contract (likely async + status-poll) and a major version bump.
- **Not covered**: Geofence filtering. UAV uploads outside any operational area still succeed; geofence enforcement is a follow-up PBI.
- **Not covered**: Per-tile photogrammetry metadata (altitude, focal length, sensor dimensions). Excluded from v1.0.0 by user choice during planning.
- **Not covered**: Streaming uploads. The endpoint reads each `IFormFile` into memory before validation.
- **Not covered**: Image storage outside the local filesystem (S3, GCS). Matches the existing Google Maps producer behavior.
- **Not covered**: Compression or re-encoding of accepted JPEGs. Stored as received.
## Versioning Rules
- **Patch (1.0.x)**: Documentation clarifications; expanded test cases; tightening of `rejectDetails` content; tuning default thresholds without changing the reject-reason enum.
- **Minor (1.x.0)**: Adding a new `rejectReason` code; adding optional metadata fields with backward-compatible defaults; relaxing rule thresholds in a backward-compatible way; introducing a new permission (e.g., `SAT`).
- **Major (2.0.0)**: Changing the request envelope shape; removing or renaming a reject-reason code; changing the persistence path; switching to async/status-poll; changing the `permissions` claim required.
Each version bump requires updating the Change Log and notifying every consumer listed in the header.
## Test Cases
| Case | Inputs | Expected | Reference |
|------|--------|----------|-----------|
| happy-1-item | 1× valid 256×256 JPEG, captured_at = now, GPS token | HTTP 200, `accepted`, row with `source='uav'`, file at `./tiles/uav/{z}/{x}/{y}.jpg` | AC-1 |
| mixed-batch | 3 items: valid, 512×512, non-JPEG bytes | HTTP 200, results = `[accepted, rejected:WRONG_DIMENSIONS, rejected:INVALID_FORMAT]`, 1 new row | AC-2 |
| multi-source | Pre-seeded `google_maps` row at `T1`; UAV upload for same cell at `T2 > T1` | HTTP 200, both rows persist, subsequent read returns the UAV row | AC-3 |
| same-source-upsert | UAV row at `T1`; second UAV upload at `T2 > T1` | HTTP 200, exactly one `uav` row remains, refreshed `captured_at` and bytes | AC-4 |
| no-token | No `Authorization` header | HTTP 401 | AC-5 |
| no-permission | Token with `permissions=["FL"]` | HTTP 403 | AC-6 |
| oversized | `metadata.items.length` = `MaxBatchSize + 1` | HTTP 400 envelope error | AC-8 |
## Change Log
| Version | Date | Change | Author |
|---------|------|--------|--------|
| 1.0.0 | 2026-05-11 | Initial contract — batch UAV upload endpoint, 5-rule quality gate, per-source UPSERT, closed reject-reason enum, GPS-permission requirement. Produced by AZ-488. | autodev (cycle 2 step 10) |
+1 -1
View File
@@ -102,7 +102,7 @@ Stores metadata for downloaded satellite imagery tiles. Each tile is a single im
| version | INT | NOT NULL, DEFAULT 2025 | Year-based versioning for cache invalidation. Vestigial post-AZ-484 — removed from the unique key by migration 012 (preparation for AZ-484); column retained nullable for backward compatibility |
| source | VARCHAR(32) | NOT NULL, DEFAULT 'google_maps' | AZ-484: producer of the imagery (`'google_maps'`, `'uav'`). Closed value set — see `tile-storage` v1.0.0 contract Inv-5 and `Common.Enums.TileSourceConverter`. Backfilled to `'google_maps'` for all pre-AZ-484 rows by migration 013 |
| captured_at | TIMESTAMP | NOT NULL | AZ-484: imagery acquisition timestamp (UTC). Drives most-recent-across-sources selection. Backfilled to `created_at` for pre-AZ-484 rows by migration 013 |
| file_path | VARCHAR(500) | NOT NULL | Relative path to stored image |
| file_path | VARCHAR(500) | NOT NULL | Relative path to stored image. **AZ-488 per-source layout**: `source='google_maps'` rows keep the legacy bucketed/timestamped path emitted by `StorageConfig.GetTileFilePath` (`{TilesDirectory}/{zoom}/{x_bucket}/{y_bucket}/tile_{zoom}_{x}_{y}_{ts}.jpg`). `source='uav'` rows live under `{TilesDirectory}/uav/{zoom}/{x}/{y}.jpg` — see `_docs/02_document/contracts/api/uav-tile-upload.md` v1.0.0. The authoritative source marker is the `source` column; the per-source path is implementation detail that keeps both producers' bytes individually addressable. |
| tile_x | INT | NOT NULL | Tile X coordinate (Slippy Map) |
| tile_y | INT | NOT NULL | Tile Y coordinate (Slippy Map) |
| created_at | TIMESTAMP | NOT NULL, DEFAULT NOW | |
+8
View File
@@ -15,6 +15,14 @@
| Layer 2 | Historic name for UAV-captured nadir camera imagery (orthogonal tiles uploaded post-flight). Generalised in AZ-484 to the `uav` `Tile Source` value; the term is retained for continuity with earlier docs and tickets. | user clarification, AZ-484 |
| Tile Source | The producer of a tile row, persisted in `tiles.source` as a contract-defined string (`google_maps`, `uav`, …). Each cell may have at most one row per source; reads return the most-recent across sources. Adding a new source requires a new `TileSource` enum member and a tile-storage contract version bump. | _docs/02_document/contracts/data-access/tile-storage.md (v1.0.0) |
| Captured At | Producer-defined UTC timestamp ("the moment this tile imagery represents") persisted in `tiles.captured_at`. For Google Maps it is `DateTime.UtcNow` at download time (provider does not expose original imagery date); for UAV it is the capture timestamp supplied by the upload client. Drives the most-recent-across-sources read selection rule. | _docs/02_document/contracts/data-access/tile-storage.md (v1.0.0) |
| UAV Tile Upload | `POST /api/satellite/upload` batch endpoint (AZ-488) that ingests UAV-captured tiles. Multipart envelope with a JSON `metadata` field and an aligned `files` collection; per-item results returned in a single HTTP 200 response. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| Quality Gate | The 5-rule validator (`UavTileQualityGate`) applied to every UAV tile before persistence: Format, Size band, Dimensions, Captured-at age, Blank/uniform. The first failing rule produces a reject reason from the closed `UavTileRejectReasons` enumeration. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| INVALID_FORMAT | UAV reject reason — content-type is not `image/jpeg` OR the file's first three bytes are not the JPEG magic `FF D8 FF` OR the bytes fail to decode as JPEG. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| SIZE_OUT_OF_BAND | UAV reject reason — image byte length outside `[UavQualityConfig.MinBytes, MaxBytes]` (defaults 5 KiB … 5 MiB). | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| WRONG_DIMENSIONS | UAV reject reason — image width or height does not equal `MapConfig.TileSizePixels`. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| CAPTURED_AT_FUTURE | UAV reject reason — `capturedAt` is more than `CapturedAtFutureSkewSeconds` ahead of the server clock. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| CAPTURED_AT_TOO_OLD | UAV reject reason — `capturedAt` is older than `UavQualityConfig.MaxAgeDays`. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| IMAGE_TOO_UNIFORM | UAV reject reason — pixel-luminance variance on the downsampled image is below `MinLuminanceVariance`. | _docs/02_document/contracts/api/uav-tile-upload.md (v1.0.0) |
| Nadir Camera | Downward-facing camera on a UAV capturing ground imagery during flight | user clarification |
| GPS-Denied Service | The consuming system: a UAV navigation service operating without GPS, using satellite/UAV imagery for positioning | user clarification |
| Slippy Map Coordinates | Tile X/Y indices in the Web Mercator projection grid (standard for web map tile servers) | data_model.md |
+23 -11
View File
@@ -11,7 +11,7 @@ Application entry point. Configures DI container, sets up middleware, defines mi
| GET | `/tiles/{z}/{x}/{y}` | `ServeTile` | Slippy map tile server with in-memory caching |
| GET | `/api/satellite/tiles/latlon` | `GetTileByLatLon` | Download single tile by lat/lon/zoom |
| GET | `/api/satellite/tiles/mgrs` | `GetSatelliteTilesByMgrs` | MGRS stub (returns empty) |
| POST | `/api/satellite/upload` | `UploadImage` | Image upload stub (returns `Success: false`) |
| POST | `/api/satellite/upload` | `UploadUavTileBatch` | UAV tile batch upload (AZ-488) — multipart envelope, 5-rule quality gate, per-source UPSERT with `source='uav'`. Requires the `RequiresGpsPermission` policy. |
| POST | `/api/satellite/request` | `RequestRegion` | Queue region for async tile processing |
| GET | `/api/satellite/region/{id}` | `GetRegionStatus` | Get region processing status |
| POST | `/api/satellite/route` | `CreateRoute` | Create route with intermediate points |
@@ -19,24 +19,31 @@ Application entry point. Configures DI container, sets up middleware, defines mi
### Local Records (defined in Program.cs)
- `GetSatelliteTilesResponse`, `SatelliteTile` — MGRS response stubs
- `UploadImageRequest` — multipart form data request
- `SaveResult` — upload response stub
- `DownloadTileResponse` — tile download response
- `RequestRegionRequest` — region request body
- `ParameterDescriptionFilter` — Swagger operation filter
### Api/DTOs (AZ-488)
- `UavTileBatchUploadRequest` — multipart envelope with `metadata` (JSON string) and `files` (`IFormFileCollection`)
### Common/DTO (AZ-488)
- `UavTileMetadata`, `UavTileBatchMetadataPayload` — per-item metadata + envelope shape
- `UavTileBatchUploadResponse`, `UavTileUploadResultItem` — per-item response shape
- `UavTileUploadStatus`, `UavTileRejectReasons` — string-constant enumerations exposed in the v1.0.0 contract
## Internal Logic
### DI Registration
1. Serilog configured from `appsettings.json`
2. Connection string extracted from `ConnectionStrings:DefaultConnection`
3. Config bindings: `MapConfig`, `StorageConfig`, `ProcessingConfig`
4. Singletons: repositories (`TileRepository`, `RegionRepository`, `RouteRepository`), `GoogleMapsDownloaderV2`, `ITileService`, `IRegionService`, `IRouteService`
5. `IRegionRequestQueue` with configurable capacity
6. Hosted services: `RegionProcessingService`, `RouteProcessingService`
7. CORS policy: `TilesCors` — configured origins from `CorsConfig:AllowedOrigins`, falls back to allow-any
8. JSON options: camelCase, case-insensitive
9. **JWT authentication (AZ-487)**: `AddSatelliteJwt(builder.Configuration)` (extension in `SatelliteProvider.Api.Authentication`) registers `JwtBearer` with `TokenValidationParameters` set per the suite auth contract (signature + lifetime, no issuer/audience validation, 30 s clock skew, ≥ 32-byte HMAC key). Followed by `AddAuthorization()`.
3. Config bindings: `MapConfig`, `StorageConfig`, `ProcessingConfig`, `UavQualityConfig` (AZ-488)
4. **Request size limits (AZ-488)**: `KestrelServerOptions.Limits.MaxRequestBodySize` and `FormOptions.MultipartBodyLengthLimit` are set to `UavQualityConfig.MaxBatchSize × UavQualityConfig.MaxBytes` (default 100 × 5 MiB = 500 MiB) so an oversized UAV batch is rejected at the framework layer before reaching the handler.
5. Singletons: repositories (`TileRepository`, `RegionRepository`, `RouteRepository`), `GoogleMapsDownloaderV2`, `ITileService`, `IRegionService`, `IRouteService`, `IUavTileQualityGate`, `IUavTileUploadHandler` (AZ-488)
6. `IRegionRequestQueue` with configurable capacity
7. Hosted services: `RegionProcessingService`, `RouteProcessingService`
8. CORS policy: `TilesCors` — configured origins from `CorsConfig:AllowedOrigins`, falls back to allow-any
9. JSON options: camelCase, case-insensitive
10. **JWT authentication (AZ-487)**: `AddSatelliteJwt(builder.Configuration)` (extension in `SatelliteProvider.Api.Authentication`) registers `JwtBearer` with `TokenValidationParameters` set per the suite auth contract (signature + lifetime, no issuer/audience validation, 30 s clock skew, ≥ 32-byte HMAC key). Followed by `AddAuthorization` with the `RequiresGpsPermission` policy (AZ-488). The `PermissionsAuthorizationHandler` singleton supports both repeated-string and JSON-array shapes for the `permissions` claim.
### Startup
1. Database migration via `DatabaseMigrator.RunMigrations()` — throws on failure
@@ -57,6 +64,9 @@ Downloads a tile, persists it, returns metadata as `DownloadTileResponse`.
### RequestRegion Handler
Validates size (10010000m), delegates to `IRegionService.RequestRegionAsync`.
### UploadUavTileBatch Handler (AZ-488)
Buffers each `IFormFile` into memory, packages them as `UavUploadFile` records (filename, content-type, bytes), and delegates to `IUavTileUploadHandler.HandleAsync`. Envelope-level errors (mismatched batch, oversized batch, malformed metadata) are surfaced as HTTP 400 ProblemDetails; per-item rejects are returned in the HTTP 200 response payload. The endpoint is protected by `.RequireAuthorization(SatellitePermissions.UavUploadPolicy)` so 401 (no token) and 403 (no `GPS` permission) are returned before the handler runs.
## Dependencies
All project references: Common, DataAccess, Services.
NuGet: `Serilog.AspNetCore`, `Swashbuckle.AspNetCore`, `Microsoft.AspNetCore.OpenApi`, `Microsoft.AspNetCore.Authentication.JwtBearer` (8.0.21, AZ-487), `SixLabors.ImageSharp`, `Newtonsoft.Json`.
@@ -72,6 +82,7 @@ Defines several local request/response records that are not shared with other pr
All configuration sections are consumed here:
- `ConnectionStrings:DefaultConnection`
- `MapConfig`, `StorageConfig`, `ProcessingConfig`
- `UavQuality` (AZ-488) — `MinBytes`, `MaxBytes`, `MaxAgeDays`, `CapturedAtFutureSkewSeconds`, `MinLuminanceVariance`, `MaxBatchSize`, `LuminanceSampleSize`. Drives the 5-rule quality gate AND the per-request body-size limits.
- `CorsConfig:AllowedOrigins`
- `Jwt:Secret` — HMAC-SHA256 signing key for JWT validation (AZ-487). Resolution: `JWT_SECRET` env var (preferred, opaque production secret) → `Jwt:Secret` configuration key (`appsettings.Development.json` placeholder only). Startup fails fast if the resolved value is unset, empty, or shorter than 32 bytes.
- `Serilog` section
@@ -85,7 +96,8 @@ All configuration sections are consumed here:
- CORS configured (permissive by default when no origins specified)
- Swagger only in Development; Bearer token "Authorize" button registered via `AddSecurityDefinition`/`AddSecurityRequirement` (AZ-487)
- HTTPS redirection enabled
- JWT bearer authentication (AZ-487) — every endpoint requires a valid HS256-signed token. Anonymous, expired, or signature-tampered requests return 401 before the handler runs. Per-endpoint permission claims are layered on top in subsequent PBIs (e.g. AZ-488 requires `permissions: ["GPS"]` on the upload endpoint).
- JWT bearer authentication (AZ-487) — every endpoint requires a valid HS256-signed token. Anonymous, expired, or signature-tampered requests return 401 before the handler runs.
- Permission-claim policies (AZ-488) — `POST /api/satellite/upload` is wrapped in `.RequireAuthorization(SatellitePermissions.UavUploadPolicy)`. The `PermissionsAuthorizationHandler` reads the `permissions` claim (repeated-string OR JSON-array shape) and returns 403 if `GPS` is not present.
## Tests
Integration tests exercise all endpoints. Unit test project has only a dummy test.
@@ -50,3 +50,14 @@
**Pass criterion**: p95(GetTilesByRegionAsync) ≤ 1.10 × pre-AZ-484 p95 baseline.
**Source**: AZ-484 NFR (Performance) — `_docs/02_tasks/done/AZ-484_multi_source_tile_storage.md` § Non-Functional Requirements.
**Note**: This NFR is recorded for tracking. Active enforcement (running PT-07 against a real workload and comparing) is deferred to autodev Step 15 (Performance Test) when a baseline run is available. Until then, the integration test `MostRecentAcrossSourcesSelection_AZ484_AC2` provides correctness coverage for the new query shape.
## PT-08: UAV Tile Batch Upload Latency
**Status**: **Deferred — harness work tracked in `_docs/_process_leftovers/2026-05-11_perf-pt07-harness.md`.** PT-08 reuses the same perf harness expansion (baseline capture + p95 ratio computation) that PT-07 is waiting for; no separate runner-script scenario was added in this commit. Active enforcement starts at cycle 2 Step 15 once the PT-07 harness lands.
**Trigger**: `POST /api/satellite/upload` exercised via the integration test fixtures generated by `UavTileImageFactory.CreateRandomJpeg` — a single 10-item batch of 256×256 / ~50 KiB JPEGs carrying a valid `GPS` JWT.
**Load**: 1 request, repeated 20 times to get a stable distribution.
**Expected**: Per-item quality-gate cost target < 50 ms (Rule 5 dominates — luminance variance after the 32×32 downsample). End-to-end p95 for a 10-item batch < 2 s on the dev hardware (8-core x86 baseline; revise on hardware change).
**Pass criterion**: `p95(UploadUavTileBatch[10 items]) ≤ 2000ms` AND `p95(UavTileQualityGate.Validate[single item]) ≤ 50ms`.
**Source**: AZ-488 NFR (Performance) — `_docs/02_tasks/done/AZ-488_uav_tile_upload.md` § Non-Functional Requirements.
**Process compliance**: AZ-488 § Risk 4 + cycle 1 retro Action 2 require that PT-08 ship with a runner-script scenario in the same commit OR be marked Deferred with a tracked follow-up. This entry takes the Deferred branch because the PT-07 harness expansion is the prerequisite for both scenarios, and a duplicated stub-runner for PT-08 would diverge from PT-07 once the real harness lands.
+1 -1
View File
@@ -69,7 +69,7 @@ Roadmap: `_docs/04_refactoring/03-code-quality-refactoring/analysis/refactoring_
| Task | Title | Depends On | Points | Status |
|------|-------|-----------|--------|--------|
| AZ-487 | JWT validation baseline (HS256, JWT_SECRET, all endpoints) | — (consumes suite-level contract `suite/_docs/10_auth.md`) | 2 | Done (In Testing) |
| AZ-488 | UAV tile upload endpoint with batch + 5-rule quality gate | AZ-487 (hard prereq), AZ-484 contract `tile-storage.md` v1.0.0 | 8 (over-cap, user-accepted) | To Do |
| AZ-488 | UAV tile upload endpoint with batch + 5-rule quality gate | AZ-487 (hard prereq), AZ-484 contract `tile-storage.md` v1.0.0 | 8 (over-cap, user-accepted) | Done (In Testing) |
## Execution Order
@@ -0,0 +1,63 @@
# Batch Report — Batch 02 cycle 2
**Batch**: 02 (cycle 2)
**Tasks**: AZ-488 (UAV tile upload endpoint + 5-rule quality gate)
**Date**: 2026-05-11
## Task Results
| Task | Status | Files Modified | Tests | AC Coverage | Issues |
|------|--------|---------------|-------|-------------|--------|
| AZ-488_uav_tile_upload | Done | 9 modified + 13 added (`UavTileBatchUploadRequest.cs`, `UavQualityConfig.cs`, `UavTileMetadata.cs`, `UavTileBatchUploadResponse.cs`, `PermissionsRequirement.cs`, `UavTileQualityGate.cs`, `UavTileUploadHandler.cs`, `UavTileImageFactory.cs`, `UavTileQualityGateTests.cs`, `UavTileUploadHandlerTests.cs`, `UavTileFilePathTests.cs`, `PermissionsRequirementTests.cs`, `UavUploadTests.cs`, contract doc `uav-tile-upload.md`); `SatelliteProvider.Api/DTOs/UploadImageRequest.cs` deleted | All green (unit 253/253 + smoke integration including `UavUploadTests`) | 10/10 ACs covered | 0 blockers; 4 Low findings (see review) |
## AC Test Coverage: All covered (10 of 10)
## Code Review Verdict: PASS_WITH_WARNINGS
## Auto-Fix Attempts: 1 (in-flight build fix: removed unused `using Microsoft.AspNetCore.Http;` in `UavTileUploadHandler.cs` after first `--unit-only` revealed it broke Service-layer build)
## Stuck Agents: None
## What was implemented
- New batch DTOs replacing the old stub: `UavTileBatchUploadRequest` (multipart envelope with JSON `metadata` + `IFormFileCollection`) in `Api/DTOs`; `UavTileMetadata`, `UavTileBatchMetadataPayload`, `UavTileBatchUploadResponse`, `UavTileUploadResultItem`, `UavTileUploadStatus`, and the closed `UavTileRejectReasons` enumeration in `Common/DTO` (placed in Common so Layer 3 services can reference them without a Service → API back-edge). Legacy `UploadImageRequest` deleted.
- New config: `Common/Configs/UavQualityConfig.cs` (MinBytes/MaxBytes/MaxAgeDays/CapturedAtFutureSkewSeconds/MinLuminanceVariance/MaxBatchSize/LuminanceSampleSize). `appsettings.json` ships defaults under `UavQuality`.
- New service `Services.TileDownloader.UavTileQualityGate` (impls `IUavTileQualityGate`) running the 5 rules in fixed order (Format → Size → Dimensions → Captured-at → Uniformity). Welford's online variance on a 32×32 ImageSharp downsample keeps the heuristic ~< 50 ms / item. `TimeProvider` injected for deterministic age tests.
- New service `Services.TileDownloader.UavTileUploadHandler` (impls `IUavTileUploadHandler`) orchestrating envelope validation (batch size / mismatch / malformed JSON), per-item gate run, file-first-then-row persistence (`./tiles/uav/{z}/{x}/{y}.jpg`), and per-item result construction. Uses `TileSourceConverter.ToWireValue(TileSource.Uav)` per L-001.
- New authorization: `Api/Authentication/PermissionsRequirement.cs` + `PermissionsAuthorizationHandler` reading the `permissions` claim — tolerates both repeated-string and JSON-array shapes. `SatellitePermissions.UavUploadPolicy` ("RequiresGpsPermission") wires the `GPS` permission requirement.
- `Program.cs` wires: `UavQualityConfig` binding, Kestrel `MaxRequestBodySize = MaxBatchSize × MaxBytes = 500 MiB`, `FormOptions.MultipartBodyLengthLimit` + `ValueLengthLimit`, `IUavTileQualityGate` + `IUavTileUploadHandler` + `PermissionsAuthorizationHandler` DI registrations, `AddAuthorization(RequiresGpsPermission policy)`, Swagger `MapType<UavTileBatchUploadRequest>` so the multipart shape renders correctly, and the new `UploadUavTileBatch` endpoint replacing the 501 stub.
- Tests:
- Unit: `UavTileQualityGateTests` (11 — every rule happy + reject + ordering), `UavTileUploadHandlerTests` (5 — happy/mixed/oversize/mismatch/invalid JSON), `UavTileFilePathTests` (3 — path shape + invariants), `PermissionsRequirementTests` (12 — claim shape coverage), `UavTileImageFactory` test utility.
- Integration: `UavUploadTests.RunAll` (AC-1 happy, AC-2 mixed-batch, AC-3 multi-source coexistence with pre-seeded `google_maps` row, AC-4 same-source UPSERT with file overwrite + db refresh, AC-5 401 no-token, AC-6 403 wrong perm, AC-8 oversized 400). `StubAndErrorContractTests` updated to drop the old 501-stub assertion.
- Docs:
- **New frozen contract** `_docs/02_document/contracts/api/uav-tile-upload.md` v1.0.0 — endpoint shape, request/response, 5-rule quality gate, closed reject-reason enum, file-path layout, concurrency model, versioning rules, test cases.
- `architecture.md`: UAV ingestion is live; permission-handler description; ADR-004 updated for the per-source file-path split (UAV under `./tiles/uav/`, google_maps grandfathered at bare `./tiles/`).
- `glossary.md`: `UAV Tile Upload`, `Quality Gate`, and all 7 reject-reason constants.
- `modules/api_program.md`: new endpoint row, new local DTOs section, DI registration steps including the body-size cap math, security policy description, configuration section adds `UavQuality`.
- `components/03_tile_downloader/description.md`: documents the two new public types, their dependencies, and the file-path divergence vs. legacy Google Maps tiles.
- `data_model.md`: `file_path` semantics now per-source (UAV vs google_maps).
- `tests/performance-tests.md`: PT-08 (UAV upload latency NFR) added with Status `Deferred — harness work tracked in PT-07 leftover`. `_docs/_process_leftovers/2026-05-11_perf-pt07-harness.md` updated with the PT-08 follow-on instruction so PT-08 lands when PT-07 lands.
## Test results (Step 10 verification)
- **Unit**: 253/253 passed (single docker container, `dotnet/sdk:8.0`, ~3.2 s test time after restore).
- **Integration (smoke)**: all green including the new `UavUploadTests` suite (which runs before the smoke/full branching).
- **Pre-existing AZ-487 test bugs surfaced and fixed in separate `fix:` commits** (see below) — were masked by a CS0104 build error.
## Pre-existing fixes shipped alongside this batch
Three small `fix:` commits were made on `dev` BEFORE the AZ-488 batch commit because they were blocking the test gate for AZ-488:
1. `753be43 [AZ-487] fix: resolve CS0104 ambiguity in AuthN tests``Microsoft.Extensions.DependencyInjection.AuthenticationServiceCollectionExtensions` collided with our same-named class in `SatelliteProvider.Api.Authentication`. Resolved via `using` alias.
2. `f64d0d7 [AZ-487] fix: JWT factory + tests now pass on net8.0``JwtTokenFactory.Create` with a negative lifetime produced `Expires < NotBefore`, which `JwtSecurityToken` rejects at construction. Shifted `notBefore` behind `expires` for non-positive lifetimes. Also disabled `MapInboundClaims` in `JwtTokenFactoryTests` so assertions read the factory's actual claim names ("sub", "email", "permissions") rather than `.NET`-default `ClaimTypes.*` aliases.
3. `11b7074 [AZ-487] fix: integration-test JWT factory handles negative lifetime` — same `Expires < NotBefore` issue in the integration-test side's own copy at `SatelliteProvider.IntegrationTests/JwtTestHelpers.cs`.
All three are AZ-487 test-side hygiene that became observable only after the CS0104 build error was lifted. They are independent of the AZ-488 feature commit; user implicitly approved option B during the autodev pause.
## Open follow-ups (non-blocking)
- **Doc-folder choice (F1, carried over from batch 01)**: `_docs/02_document/components/01_web_api/description.md` referenced by the spec doesn't exist; updates went into `modules/api_program.md` instead. Needs an operator decision on whether to add a stub `01_web_api` folder or formalize the convention.
- **`File.WriteAllBytesAsync(byte[])` allocation** (F4 in review): up to 5 MiB array copy per accepted tile. Replace with `FileStream.WriteAsync(ReadOnlyMemory<byte>, ct)` when PT-08 measurement begins. Not blocking — Rule 5 decode + downsample dominates the gate cost target.
- **PT-08 runner-script scenario**: deferred to land with the PT-07 harness expansion (per cycle 1 retro Action 2 / AZ-488 § Risk 4). Tracked in `_docs/_process_leftovers/2026-05-11_perf-pt07-harness.md`.
- **Coordinate external consumers** for AZ-488: `gps-denied-onboard` and any mission-planner client that posts to `/api/satellite/upload` must attach a Bearer token with `permissions: ["GPS"]` (or the JSON-array shape `"[\"GPS\"]"` — handler accepts both). Coordination is the operator's at Step 16 (Deploy).
## Next: Step 11 (Run Functional Tests) — autodev auto-chain
Cycle 2 batches all closed. Next autodev step is `test-run``deploy` (per `flows/existing-code.md` auto-chain rules).
@@ -0,0 +1,152 @@
# Code Review Report — Batch 02 cycle 2
**Batch**: AZ-488 (UAV tile upload endpoint + 5-rule quality gate)
**Date**: 2026-05-11
**Verdict**: PASS_WITH_WARNINGS
## Findings
| # | Severity | Category | File:Line | Title |
|---|----------|----------|-----------|-------|
| 1 | Low | Style | _docs/02_document/components/01_web_api/description.md | Task spec referenced a doc path that does not exist (carried over from batch 01) |
| 2 | Low | Maintainability | SatelliteProvider.Services.TileDownloader/UavTileQualityGate.cs:23 | `JpegMagicBytes` declared as mutable `byte[]` instead of `ReadOnlySpan<byte>` static |
| 3 | Low | Maintainability | SatelliteProvider.Common/Configs/StorageConfig.cs | UAV path layout diverges from `StorageConfig.GetTileFilePath` — two contracts in one component (grandfathered per AZ-488 § Constraints) |
| 4 | Low | Performance | SatelliteProvider.Services.TileDownloader/UavTileUploadHandler.cs:152 | `File.WriteAllBytesAsync` requires `byte[]`; current code does `imageBytes.ToArray()` per accept (extra allocation) |
### Finding Details
**F1: Task spec referenced a doc path that does not exist in the codebase** (Low / Style)
- Location: `_docs/02_document/components/01_web_api/description.md` (referenced; does not exist)
- Description: The AZ-488 task spec § Scope > Documentation lists `_docs/02_document/components/01_web_api/description.md` as a doc to update. The component-doc folders are `01_common`, `02_data_access`, `03_tile_downloader`, `04_region_processing`, `05_route_management` — there is no `01_web_api` folder. This finding was first reported in batch 01 cycle 2 (AZ-487 F1) and is unchanged. WebApi's documentation lives in `_docs/02_document/modules/api_program.md` and has been updated there.
- Suggestion: Carry-over from batch 01 — needs an explicit operator decision: (a) create the missing folder with a stub that defers to `api_program.md`, or (b) update the documentation conventions to acknowledge WebApi lives in `modules/`. No change in this batch beyond updating `modules/api_program.md` and `components/03_tile_downloader/description.md`.
- Task: AZ-488 (carried over from AZ-487)
**F2: `JpegMagicBytes` declared as mutable `byte[]` instead of `ReadOnlySpan<byte>` static** (Low / Maintainability)
- Location: `SatelliteProvider.Services.TileDownloader/UavTileQualityGate.cs:23`
- Description: `private static readonly byte[] JpegMagicBytes = { 0xFF, 0xD8, 0xFF };``byte[]` allows in-place mutation of `JpegMagicBytes[0] = …` from inside the class. Not a security issue since the type is `private static`, but `static ReadOnlySpan<byte> JpegMagicBytes => [0xFF, 0xD8, 0xFF];` is a more intent-revealing C# 12 pattern, also slightly faster (no heap allocation; backed by RVA literal).
- Suggestion: Refactor when a follow-up touches this file. Not blocking — the constant is private and isolated.
- Task: AZ-488
**F3: UAV path layout diverges from `StorageConfig.GetTileFilePath`** (Low / Maintainability)
- Location: `SatelliteProvider.Common/Configs/StorageConfig.cs` (GetTileFilePath) vs `SatelliteProvider.Services.TileDownloader/UavTileUploadHandler.cs:182` (BuildUavTileFilePath)
- Description: Google Maps tiles use `StorageConfig.GetTileFilePath``{TilesDirectory}/{zoom}/{x_bucket}/{y_bucket}/tile_{z}_{x}_{y}_{ts}.jpg`. UAV tiles use `UavTileUploadHandler.BuildUavTileFilePath``{TilesDirectory}/uav/{z}/{x}/{y}.jpg`. Two file-naming contracts coexist in one component. This is explicitly grandfathered by the AZ-488 task spec § Scope/Constraints ("Per-source file-path strategy is fixed; do NOT migrate Google Maps files"), so it's intentional, not a defect.
- Suggestion: Documented in `architecture.md` § ADR-004 and `data_model.md`. If a future task unifies storage layouts, both consumers should move to a single helper on `StorageConfig`. Carrying this as a known divergence is acceptable.
- Task: AZ-488
**F4: `File.WriteAllBytesAsync` requires `byte[]` — `imageBytes.ToArray()` per accept** (Low / Performance)
- Location: `SatelliteProvider.Services.TileDownloader/UavTileUploadHandler.cs:152`
- Description: `await File.WriteAllBytesAsync(filePath, imageBytes.ToArray(), cancellationToken);` allocates a new array of up to 5 MiB per accepted tile. With `MaxBatchSize=100` × `MaxBytes=5 MiB` that is up to 500 MiB of extra allocations per batch worst-case. The `(String, ReadOnlyMemory<Byte>, CancellationToken)` overload of `File.WriteAllBytesAsync` is .NET 9+, so it is NOT available on this project's `net8.0` target — `ToArray()` is the only API-direct option here.
- Suggestion: Use `await using var fs = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize: 4096, useAsync: true); await fs.WriteAsync(imageBytes, cancellationToken);``FileStream.WriteAsync(ReadOnlyMemory<byte>, CancellationToken)` is available on net8.0 and skips the `ToArray()` copy. Not blocking — quality-gate cost target (< 50 ms / item) is dominated by Rule 5 decode + downsample, not the allocation. Address when PT-08 measurement starts (see `_docs/_process_leftovers/2026-05-11_perf-pt07-harness.md`).
- Task: AZ-488
## Phase Notes
### Phase 1 — Context Loading
- Task spec: `_docs/02_tasks/todo/AZ-488_uav_tile_upload.md` (now archived under done/ at end of batch).
- Plan artifacts: `_docs/02_task_plans/uav-batch-upload/00_research/00_ac_assessment.md`, `_docs/02_task_plans/uav-batch-upload/01_solution/solution_draft01.md`, `_docs/02_task_plans/uav-batch-upload/problem.md`.
- Contracts consumed: `_docs/02_document/contracts/data-access/tile-storage.md` v1.0.0 (per-source UPSERT).
- Contract produced: `_docs/02_document/contracts/api/uav-tile-upload.md` v1.0.0 (frozen).
- Prior batch: batch 01 cycle 2 (AZ-487) — JWT bearer middleware + Swagger Authorize button + `RequireAuthorization()` on all endpoints. AZ-488 layers permission policy on top.
### Phase 2 — Spec Compliance
All 10 ACs are demonstrably covered by automated tests:
| AC | Description | Tests |
|----|-------------|-------|
| AC-1 | Happy path single item persists with source='uav' | `UavUploadTests.HappyPath_BatchOfTwoTiles_Returns200_PersistsRows`, `UavTileUploadHandlerTests.HappyPath_SingleItem_InsertsRow_WithUavSource` |
| AC-2 | Mixed batch partial reject | `UavUploadTests.MixedBatch_PartialReject_Returns200_WithPerItemResults`, `UavTileUploadHandlerTests.MixedBatch_OnlyAcceptedItemsInserted` |
| AC-3 | Multi-source coexistence with Google Maps | `UavUploadTests.MultiSourceCoexistence_AZ484_Cycle2` |
| AC-4 | Same-source UPSERT | `UavUploadTests.SameSourceUpsert_AZ484_Cycle2` |
| AC-5 | Unauth → 401 | `UavUploadTests.NoToken_Returns401` |
| AC-6 | Missing GPS perm → 403 | `UavUploadTests.ValidTokenWithoutGpsPermission_Returns403`, `PermissionsRequirementTests.HandleRequirement_*` (12 unit tests) |
| AC-7a | Wrong content-type / magic → INVALID_FORMAT | `UavTileQualityGateTests.Validate_RejectsNonJpegContentType`, `Validate_RejectsJpegContentTypeWithWrongMagic` |
| AC-7b | Size out of band | `UavTileQualityGateTests.Validate_RejectsTooSmall`, `Validate_RejectsTooLarge` |
| AC-7c | Wrong dimensions | `UavTileQualityGateTests.Validate_RejectsWrongDimensions` |
| AC-7d | Captured-at future / too old | `UavTileQualityGateTests.Validate_RejectsCapturedInFuture`, `Validate_RejectsCapturedTooOld` |
| AC-7e | Blank/uniform → IMAGE_TOO_UNIFORM | `UavTileQualityGateTests.Validate_RejectsUniformImage`, `Validate_AcceptsHighVarianceImage` |
| Rule ordering | First-failing rule wins | `UavTileQualityGateTests.Validate_FormatBeforeDimensions` |
| AC-8 | Oversized batch → 400 | `UavUploadTests.OversizedBatch_Returns400`, `UavTileUploadHandlerTests.Oversized_EnvelopeRejected` |
| AC-9 | Contract docs match impl | Manual: `_docs/02_document/contracts/api/uav-tile-upload.md` v1.0.0 frozen, reject reasons match `Common.DTO.UavTileRejectReasons`, request shape matches `UavTileBatchUploadRequest` + `UavTileBatchMetadataPayload` |
| AC-10 | Existing tests pass | Deferred to test execution (Step 11) |
**Spec gaps** — none. The earlier PT-08 gap (NFR mandated in same commit) was closed by adding `PT-08` to `_docs/02_document/tests/performance-tests.md` with Status `Deferred — harness work tracked in _docs/_process_leftovers/2026-05-11_perf-pt07-harness.md` and a cross-reference appended to that leftover so PT-08 lands when the PT-07 harness lands. Per the AZ-488 task spec § Risk 4 / cycle 1 retro Action 2, the Deferred branch is explicitly sanctioned ("NOT as an active scenario" → "Deferred — harness work tracked in <follow-up ticket>").
**Contract verification**`_docs/02_document/contracts/api/uav-tile-upload.md` v1.0.0:
- Request shape matches `UavTileBatchUploadRequest` + `UavTileBatchMetadataPayload` + `UavTileMetadata` (multipart `metadata` JSON string + `files` collection; per-item ordinal alignment).
- Response shape matches `UavTileBatchUploadResponse` + `UavTileUploadResultItem` (per-item `index`, `status`, `tileId?`, `rejectReason?`, `rejectDetails?`).
- Reject-reason closed enumeration matches `UavTileRejectReasons` constants exactly (7 reasons: `INVALID_FORMAT`, `SIZE_OUT_OF_BAND`, `WRONG_DIMENSIONS`, `CAPTURED_AT_FUTURE`, `CAPTURED_AT_TOO_OLD`, `IMAGE_TOO_UNIFORM`, `STORAGE_FAILURE`).
- Status codes (200, 400, 401, 403) match `Program.cs` endpoint annotations.
- Cross-reference to `tile-storage.md` v1.0.0 is present (per-source UPSERT semantics).
### Phase 3 — Code Quality
- **SRP**: `UavTileQualityGate` validates only; `UavTileUploadHandler` orchestrates only; `PermissionsAuthorizationHandler` authorizes only. Clean separation.
- **Error handling**: per-item `try/catch` in `UavTileUploadHandler.HandleAsync` narrowed to `IOException` / `UnauthorizedAccessException``STORAGE_FAILURE`. No bare catches. Envelope-level errors return `EnvelopeRejected=true` with the original message preserved (no swallowing).
- **Naming**: `UavTileQualityResult.Pass()/Fail()`, `UavTileUploadHandlerResult.EnvelopeRejected`, `BuildUavTileFilePath` all read at the call site.
- **Complexity**: `Validate` is ~70 lines but linear with one short-circuit per rule — easy to follow. No methods exceed 50 logical lines.
- **DRY**: `ReadOnlyMemoryStream` is a small, internal utility (no `MemoryStream` over a `byte[]` copy path).
- **Test quality**: each rule has both happy and reject coverage; rule ordering is independently tested. Mocked-repo handler tests assert call count + arguments, not just "no exception".
- **Dead code**: legacy `UploadImageRequest` was deleted; old stub test `StubUpload_Returns501` was deleted to match the new shape.
### Phase 4 — Security Quick-Scan
- No SQL string interpolation in this batch (DataAccess goes through Dapper parameterized queries already).
- No `Process.Start`, no `eval`, no dynamic SQL.
- No hardcoded secrets in implementation code.
- Input validation: image bytes are size-bounded (5 KiB - 5 MiB), dimensions are enforced exact-equal to `MapConfig.TileSizePixels`, JSON metadata is bounded by `MaxBatchSize`, framework body-size limit set to `MaxBatchSize × MaxBytes`.
- Path-traversal: `BuildUavTileFilePath` takes `int` tileZoom/X/Y and uses `Path.Combine` with `InvariantCulture` integer formatting — no caller-supplied strings, no `..` escape vector.
- Sensitive data in logs: `UavTileUploadHandler` logs storage failures with `_logger.LogError(ex, "UAV tile persistence failed at index {Index}", index)` — no file paths or user data in the message. Reject-reason `RejectDetails` is set only by the quality gate (currently always null for the 7 closed reasons; safe).
- Deserialization: `JsonSerializer.Deserialize<UavTileBatchMetadataPayload>` with case-insensitive matching but no `JsonStringEnumConverter` injection — safe (no enum fields in the payload).
### Phase 5 — Performance Scan
- Rule 5 (luminance variance) does `Image.Load<L8>` then `Mutate(Resize(32,32))` then `ProcessPixelRows`. Welford's online variance avoids the 2-pass sum + sum-of-squares. Correct shape for the < 50 ms target.
- Rule 3 uses `Image.Identify` (header-only) — does NOT decode the full image. Correct.
- N+1 query risk: `InsertAsync` is per-accepted-item. Acceptable at `MaxBatchSize=100`; if batches grow, a `BulkInsertAsync` would help. Out of scope for this PBI.
- Blocking I/O in async context: file write uses `File.WriteAllBytesAsync` (true async). Good.
- F4 (above) is the only observed allocation hot-spot — Low severity.
### Phase 6 — Cross-Task Consistency
- AZ-487 (batch 01 cycle 2) exposed `AddSatelliteJwt(builder.Configuration)`; AZ-488 layers `AddAuthorization` on top with the `RequiresGpsPermission` policy. Compatible.
- AZ-484 produced `TileSourceConverter.ToWireValue(TileSource.Uav)` — AZ-488 calls it via the sanctioned path. Compatible (per L-001 in `_docs/LESSONS.md`).
- DTO layering: `UavTileMetadata`, `UavTileBatchUploadResponse`, `UavTileRejectReasons` live in `SatelliteProvider.Common.DTO` so both the API and the service can reference them without a Service → API dependency. The endpoint-specific `UavTileBatchUploadRequest` envelope stays in `SatelliteProvider.Api.DTOs`. Layering preserved.
- JSON conventions: handler uses `JsonSerializerOptions { PropertyNameCaseInsensitive = true }`, matching the API's camelCase / case-insensitive `ConfigureHttpJsonOptions` block.
### Phase 7 — Architecture Compliance
Files in scope (touched in this batch):
- `SatelliteProvider.Api/Program.cs` — Layer 4 (Api). Imports: `Common.*`, `DataAccess.*`, `Services.*`, `Api.Authentication.*`, `Api.DTOs.*`. All directionally correct.
- `SatelliteProvider.Api/Authentication/PermissionsRequirement.cs` — Layer 4 (Api). Imports: `Microsoft.AspNetCore.Authorization`, `System.Security.Claims`, `System.Text.Json`. No cross-component import. ✓
- `SatelliteProvider.Api/DTOs/UavTileBatchUploadRequest.cs` — Layer 4 (Api). Imports: `Microsoft.AspNetCore.Http`, `Microsoft.AspNetCore.Mvc`. ✓
- `SatelliteProvider.Common/Configs/UavQualityConfig.cs` — Layer 1 (Common). No upward imports. ✓
- `SatelliteProvider.Common/DTO/UavTileMetadata.cs`, `UavTileBatchUploadResponse.cs` — Layer 1 (Common). No upward imports. ✓
- `SatelliteProvider.Services.TileDownloader/UavTileQualityGate.cs` — Layer 3 (Services). Imports: `Common.Configs`, `Common.DTO`, `SixLabors.ImageSharp.*`. ✓
- `SatelliteProvider.Services.TileDownloader/UavTileUploadHandler.cs` — Layer 3 (Services). Imports: `Common.*`, `DataAccess.Models`, `DataAccess.Repositories`. Service → DataAccess is allowed per `module-layout.md`. ✓
- `SatelliteProvider.Services.TileDownloader/TileDownloaderServiceCollectionExtensions.cs` — Layer 3 (Services). Adds `IUavTileQualityGate`, `IUavTileUploadHandler` singletons. ✓
**Layer direction**: clean. No Service → API import. No DataAccess → Service. No DataAccess → API.
**Public API respect**: cross-component imports go through:
- `Common.Configs.{UavQualityConfig, StorageConfig, MapConfig}` (public)
- `Common.DTO.{UavTileMetadata, UavTileBatchUploadResponse, UavTileRejectReasons, UavTileUploadStatus, UavTileBatchMetadataPayload, UavTileUploadResultItem}` (public)
- `Common.Enums.{TileSource, TileSourceConverter}` (public)
- `Common.Utils.GeoUtils` (public)
- `DataAccess.Models.TileEntity`, `DataAccess.Repositories.ITileRepository` (public per AZ-484)
- `Services.TileDownloader.{IUavTileQualityGate, IUavTileUploadHandler, UavUploadFile}` (public)
No internal-file imports across components.
**No new cyclic dependencies**: import graph is acyclic — Api → Services → DataAccess → Common; Services → Common; Api → Common. No new edges added.
**Duplicate symbols across components**: none.
**Cross-cutting concerns**: `PermissionsAuthorizationHandler` is an Api-layer concern (authorization handlers map to ASP.NET Core's authorization pipeline, which lives at the Api layer). Correctly placed in `SatelliteProvider.Api/Authentication/`. Not duplicated elsewhere.
## Baseline Delta
No `_docs/02_document/architecture_compliance_baseline.md` exists in this repository. Skip baseline-delta partitioning.
## Verdict Logic
- 0 Critical findings
- 0 High findings
- 0 Medium findings
- 4 Low findings
**PASS_WITH_WARNINGS** — proceed to commit.
+2 -2
View File
@@ -6,9 +6,9 @@ step: 10
name: Implement
status: in_progress
sub_step:
phase: 7
phase: 6
name: batch-loop
detail: "batch 1 of 2 done (AZ-487); batch 2 (AZ-488) pending"
detail: "batch 2 of 2 done (AZ-488); awaiting Step 11"
retry_count: 0
cycle: 2
tracker: jira
@@ -27,6 +27,10 @@ When the next cycle's autodev runs, before any new tracker write or before re-en
3. Capture results into `_docs/06_metrics/perf_<YYYY-MM-DD>_cycle<N>.md`.
4. Once results are recorded, delete this leftover file.
## AZ-488 follow-on: PT-08 (UAV upload latency)
The AZ-488 commit added PT-08 (UAV tile batch upload latency) to `_docs/02_document/tests/performance-tests.md` with Status `Deferred` because it reuses the same harness expansion as PT-07 (baseline capture + p95 ratio). When PT-07's runner-script scenario is implemented in step 1 above, add the PT-08 scenario in the **same commit** — the integration-test fixtures already exist (`SatelliteProvider.IntegrationTests/UavUploadTests` happy-path JWT + `UavTileImageFactory.CreateRandomJpeg`). After PT-08 runs, flip the Status line in `performance-tests.md` from `Deferred` to active. This keeps cycle 1 retro Action 2 satisfied for both NFRs.
## Tracker action (none required this cycle)
This leftover does NOT require a Jira ticket on its own — it tracks deferred process work, not user-visible scope. If the perf comparison reveals a regression next cycle, that finding will create a Jira bug; until then there is nothing to file.