mirror of
https://github.com/azaion/satellite-provider.git
synced 2026-06-21 23:11:14 +00:00
490902c80a
Adds the per-endpoint child of AZ-795 ("Strict Input Validation Epic")
for the UAV upload multipart endpoint. Three new validators land under
SatelliteProvider.Api/Validators/:
- UavTileBatchMetadataPayloadValidator: items NotNull + NotEmpty +
count <= MaxBatchSize + RuleForEach dispatching to the per-item
validator.
- UavTileMetadataValidator: lat / lon / tileZoom range, tileSizeMeters
> 0, capturedAt within [now - MaxAgeDays, now + future-skew]; uses an
injectable TimeProvider so unit tests can drive a fixed clock.
- UavUploadValidationFilter: IEndpointFilter that reads the multipart
`metadata` form field, deserializes it with the strict global
JsonSerializerOptions (so UnmappedMemberHandling.Disallow +
[JsonRequired] from AZ-795 are honored), runs the FluentValidation
chain, and enforces the cross-field `items.Count == files.Count`
envelope rule. FluentValidation errors are prefixed with `metadata.`
so wire keys look like `errors["metadata.items[0].latitude"]`.
[JsonRequired] is added to every non-optional axis on
UavTileMetadata and UavTileBatchMetadataPayload; FlightId stays
nullable per AZ-503 anonymous-flight semantics.
Coverage: 13 unit tests + 16 integration tests + 1 curl probe script
exercise the happy path and every failure mode. All 9 ACs covered;
no regression in AZ-488 UavUploadTests payloads (traced against the
new rules).
Documentation: uav-tile-upload.md bumped v1.1.0 -> v1.2.0 with the
new validation rules section + 400-shape examples + changelog entry.
api_program.md updated to describe the three new validators + filter
+ the AddTransient<UavUploadValidationFilter>() DI registration.
Reports: batch_04_cycle8_report.md + reviews/batch_04_cycle8_review.md
record the PASS_WITH_WARNINGS verdict (2 Low DRY-in-tests findings:
FixedTimeProvider duplication crossed the cycle-2 "promote to shared"
threshold; PostBatch helper duplicated between two integration
suites). Both deferred to follow-up PBIs.
Task spec archived: _docs/02_tasks/todo/AZ-810... -> done/.
Jira: AZ-810 transitioned In Progress -> In Testing.
Co-authored-by: Cursor <cursoragent@cursor.com>
241 lines
19 KiB
Markdown
241 lines
19 KiB
Markdown
# Contract: uav-tile-upload
|
||
|
||
**Component**: WebApi (`SatelliteProvider.Api`) producing rows via TileDownloader (`SatelliteProvider.Services.TileDownloader`)
|
||
**Producer task**: AZ-488 — `_docs/02_tasks/done/AZ-488_uav_tile_upload.md`
|
||
**Extended by**: AZ-503 — `_docs/02_tasks/done/AZ-503_tile_identity_uuidv5_bulk_list.md` (added optional `flightId` per-item field; per-flight on-disk path; deterministic `tileId`)
|
||
**Consumer tasks**: `gps-denied-onboard`, mission planner UI, any future UAV-equipped client
|
||
**Version**: 1.2.0
|
||
**Status**: frozen
|
||
**Last Updated**: 2026-05-23
|
||
|
||
## Purpose
|
||
|
||
Defines the HTTP contract for the `POST /api/satellite/upload` batch endpoint that ingests UAV-captured satellite tiles, runs each item through a 5-rule quality gate, and persists accepted rows under the `tile-storage` v1.0.0 data contract with `source='uav'`.
|
||
|
||
## Endpoint
|
||
|
||
```
|
||
POST /api/satellite/upload
|
||
Content-Type: multipart/form-data
|
||
Authorization: Bearer <JWT>
|
||
```
|
||
|
||
The request MUST carry a valid JWT (AZ-487) AND the `permissions` claim MUST contain `GPS`. Anonymous requests are rejected with HTTP 401; authenticated requests without `GPS` are rejected with HTTP 403.
|
||
|
||
## Request shape
|
||
|
||
Multipart form fields (case-sensitive part names):
|
||
|
||
| Part | Type | Required | Description |
|
||
|------|------|----------|-------------|
|
||
| `metadata` | JSON string | yes | Document of shape `{ "items": [ <UavTileMetadata>, ... ] }`. See the table below for the per-item schema. |
|
||
| `files` | binary (repeating) | yes | One JPEG file per metadata item. Files are correlated to metadata entries by ordinal index (file index `i` ↔ `metadata.items[i]`). Each part MUST be sent under the form field name `files`. |
|
||
|
||
### `UavTileMetadata` (per item)
|
||
|
||
| Field | Type | Required | Description | Constraints |
|
||
|-------|------|----------|-------------|-------------|
|
||
| `latitude` | number | yes | Geographic latitude of the tile center | WGS-84 decimal degrees |
|
||
| `longitude` | number | yes | Geographic longitude of the tile center | WGS-84 decimal degrees |
|
||
| `tileZoom` | integer | yes | Slippy Map zoom level | Must satisfy the same zoom-level policy as the existing tile pipeline (see `MapConfig.AllowedZoomLevels`) |
|
||
| `tileSizeMeters` | number | yes | Tile size in meters at the captured latitude | Producer-supplied |
|
||
| `capturedAt` | string (ISO-8601 UTC) | yes | Moment of UAV image capture | Must satisfy the captured-at rule (see Quality Gate, Rule 4) |
|
||
| `flightId` | string (UUID) | no | AZ-503: optional flight identifier. When present, two flights uploading the same cell coexist as separate rows; absent uploads share a single anonymous row per cell. Omitting the field is fully backward-compatible with v1.0.0 clients. | RFC 4122 UUID. Backward-compatible default: `null` |
|
||
|
||
Field names are camelCase. Property-name matching is case-insensitive on read.
|
||
|
||
### Constraints
|
||
|
||
- The number of `files` MUST equal the number of `metadata.items` entries (1:1 correlation by ordinal index). Mismatch → HTTP 400.
|
||
- `metadata.items.length` MUST NOT exceed `UavQualityConfig.MaxBatchSize` (default `100`). Oversize → HTTP 400.
|
||
- The total request body size is capped at `MaxBatchSize × MaxBytes` by Kestrel's `MaxRequestBodySize` and the form `MultipartBodyLengthLimit`. Requests beyond this cap are rejected at the framework layer (HTTP 413/400).
|
||
|
||
## Metadata validation (14 rules, v1.2.0)
|
||
|
||
Before any file bytes are inspected by the Quality Gate below, the `metadata` envelope is run through a strict validator chain. This is the **metadata layer**; the **file layer** (see Quality Gate) is unchanged.
|
||
|
||
The validator is split into three composing layers and runs inside a custom multipart endpoint filter (`UavUploadValidationFilter`):
|
||
|
||
1. **Deserializer layer** — `JsonSerializerOptions.UnmappedMemberHandling.Disallow` + `[JsonRequired]` on every non-optional axis of `UavTileBatchMetadataPayload` / `UavTileMetadata`. Catches missing-required, unknown fields (root and nested), JSON type mismatches, and malformed UUIDs. Errors surface under `errors["metadata"]`.
|
||
2. **FluentValidation layer** — `UavTileBatchMetadataPayloadValidator` (envelope rules) and `UavTileMetadataValidator` (per-item rules). Errors surface under `errors["metadata.items"]` / `errors["metadata.items[i].<field>"]`.
|
||
3. **Cross-field envelope rule** — `items.Count == files.Count`, evaluated in the filter after the FluentValidation result is clean. Errors surface under **both** `errors["metadata.items"]` AND `errors["files"]`.
|
||
|
||
Any failing rule short-circuits with HTTP 400 + RFC 7807 `ValidationProblemDetails` per `error-shape.md` v1.0.0. The body never reaches the Quality Gate or the persistence path on a metadata validation failure.
|
||
|
||
| # | Rule | Failure condition | Error path | Layer |
|
||
|---|------|-------------------|------------|-------|
|
||
| 1 | Multipart envelope present | Request `Content-Type` is not `multipart/form-data` | `errors["metadata"]` | filter |
|
||
| 2 | `metadata` form field present | Multipart form has no part named `metadata` | `errors["metadata"]` | filter |
|
||
| 3 | `metadata` parses as JSON | Malformed JSON body | `errors["metadata"]` | deserializer |
|
||
| 4 | `items` required + non-empty | `items` missing OR `items: []` | `errors["metadata.items"]` | FluentValidation |
|
||
| 5 | `items.Count` ≤ `UavQualityConfig.MaxBatchSize` | `items.Count > MaxBatchSize` (default 100) | `errors["metadata.items"]` | FluentValidation |
|
||
| 6 | `items.Count == files.Count` | Per-item file count differs from metadata count | `errors["metadata.items"]` + `errors["files"]` | filter |
|
||
| 7 | `latitude` ∈ [-90, +90] | Out of range | `errors["metadata.items[i].latitude"]` | FluentValidation |
|
||
| 8 | `longitude` ∈ [-180, +180] | Out of range | `errors["metadata.items[i].longitude"]` | FluentValidation |
|
||
| 9 | `tileZoom` ∈ [0, 22] | Out of range | `errors["metadata.items[i].tileZoom"]` | FluentValidation |
|
||
| 10 | `tileSizeMeters` > 0 | Zero or negative | `errors["metadata.items[i].tileSizeMeters"]` | FluentValidation |
|
||
| 11 | `capturedAt` within freshness window | `capturedAt > now + CapturedAtFutureSkewSeconds` OR `capturedAt < now - MaxAgeDays` | `errors["metadata.items[i].capturedAt"]` | FluentValidation |
|
||
| 12 | `flightId` parses as UUID | Non-UUID string (`null`/missing is valid per AZ-503) | `errors["metadata"]` | deserializer |
|
||
| 13 | Unknown fields rejected (root + nested) | Any field not declared on the DTO | `errors["metadata"]` | deserializer |
|
||
| 14 | Type mismatch | e.g. `"latitude": "fifty"`, `"tileZoom": 18.5` | `errors["metadata"]` | deserializer |
|
||
|
||
### Relationship to the Quality Gate
|
||
|
||
The Quality Gate's Rule 4 (captured-at freshness) is preserved exactly as documented below. It runs **after** the metadata validator and provides defence-in-depth against handler callers that bypass the filter (unit tests of `IUavTileUploadHandler`, future internal call paths). Operators consuming the public API will see the metadata validator's verdict first.
|
||
|
||
The Quality Gate's Rules 1, 2, 3, 5 (file-level: format, size, dimensions, luminance) are unchanged and still produce per-item rejections via the existing HTTP 200 + `rejectReason` envelope — they have no metadata-validator equivalent.
|
||
|
||
## Quality Gate (5 rules)
|
||
|
||
Each item is evaluated in this fixed order. The **first** failing rule produces the reported reason; subsequent rules are not evaluated for that item.
|
||
|
||
| # | Rule | Failure condition | Reason code |
|
||
|---|------|-------------------|-------------|
|
||
| 1 | Format | `Content-Type` of the file part is not `image/jpeg` (case-insensitive, allowing trailing parameters) OR the file's first 3 bytes are not `FF D8 FF` | `INVALID_FORMAT` |
|
||
| 2 | Size band | `bytes.length` is outside `[UavQualityConfig.MinBytes, UavQualityConfig.MaxBytes]` (defaults: 5 KiB … 5 MiB) | `SIZE_OUT_OF_BAND` |
|
||
| 3 | Dimensions | Image width OR height ≠ `MapConfig.TileSizePixels` (default 256). Strict equality, no tolerance | `WRONG_DIMENSIONS` |
|
||
| 4a | Captured-at future | `capturedAt > now + UavQualityConfig.CapturedAtFutureSkewSeconds` (default 30s) | `CAPTURED_AT_FUTURE` |
|
||
| 4b | Captured-at age | `capturedAt < now - UavQualityConfig.MaxAgeDays` (default 7 days) | `CAPTURED_AT_TOO_OLD` |
|
||
| 5 | Blank / uniform | Pixel-luminance variance on a downsampled (default 32×32, configurable via `UavQualityConfig.LuminanceSampleSize`) version of the image is below `UavQualityConfig.MinLuminanceVariance` (default 10.0) | `IMAGE_TOO_UNIFORM` |
|
||
|
||
If the file decode itself fails for the variance check, the item is rejected with `INVALID_FORMAT` (the file is not a real JPEG even though the header bytes matched).
|
||
|
||
Storage failures while persisting an accepted item (e.g., disk full, unwritable path) are surfaced as `STORAGE_FAILURE` so the client can retry that specific item without re-uploading the whole batch.
|
||
|
||
### Reject-reason enumeration (closed)
|
||
|
||
| Code | Source rule | When the reason fires |
|
||
|------|-------------|------------------------|
|
||
| `INVALID_FORMAT` | Rule 1 + Rule 5 decode safety net | Wrong content-type, wrong magic bytes, or undecodable bytes |
|
||
| `SIZE_OUT_OF_BAND` | Rule 2 | Byte length outside `[MinBytes, MaxBytes]` |
|
||
| `WRONG_DIMENSIONS` | Rule 3 | Width or height ≠ `MapConfig.TileSizePixels` |
|
||
| `CAPTURED_AT_FUTURE` | Rule 4a | `capturedAt` is more than `CapturedAtFutureSkewSeconds` in the future |
|
||
| `CAPTURED_AT_TOO_OLD` | Rule 4b | `capturedAt` is older than `MaxAgeDays` |
|
||
| `IMAGE_TOO_UNIFORM` | Rule 5 | Luminance variance below `MinLuminanceVariance` |
|
||
| `METADATA_MISSING` | Validation gate | Per-item metadata could not be matched (reserved; not currently surfaced because batch mismatch is reported as an envelope error) |
|
||
| `STORAGE_FAILURE` | Persistence path | The image bytes could not be written to disk or the repository insert raised an IO error |
|
||
|
||
Adding a new code is a **minor** contract version bump per the Versioning Rules below. Removing or renaming a code is **major**.
|
||
|
||
## Response shape
|
||
|
||
### HTTP 200 — per-item results
|
||
|
||
```json
|
||
{
|
||
"items": [
|
||
{ "index": 0, "status": "accepted", "tileId": "11111111-...", "rejectReason": null, "rejectDetails": null },
|
||
{ "index": 1, "status": "rejected", "tileId": null, "rejectReason": "WRONG_DIMENSIONS", "rejectDetails": null }
|
||
]
|
||
}
|
||
```
|
||
|
||
| Field | Type | Description |
|
||
|-------|------|-------------|
|
||
| `items` | array | One result per input item, in the same order as `metadata.items` |
|
||
| `items[].index` | integer | Ordinal index of the item in the request batch |
|
||
| `items[].status` | string | `"accepted"` or `"rejected"` (closed enumeration) |
|
||
| `items[].tileId` | UUID or null | When `accepted`, the persisted `tiles.id` value; null when `rejected` |
|
||
| `items[].rejectReason` | string or null | Reason code (see table above) when `rejected`; null when `accepted` |
|
||
| `items[].rejectDetails` | string or null | Optional human-readable detail; MUST NOT leak server paths, exception types, or internal identifiers |
|
||
|
||
### HTTP 400 — envelope error (RFC 7807 `application/problem+json`)
|
||
|
||
Returned when the request itself is malformed. As of v1.2.0 every 400 body conforms to the shared `ValidationProblemDetails` shape in `error-shape.md` v1.0.0, with the `errors` map keys listed in the "Metadata validation" rule table above. Triggers include:
|
||
|
||
- `metadata` field absent, empty, or not valid JSON
|
||
- `metadata.items` empty or null
|
||
- `metadata.items.length` ≠ `files.length`
|
||
- `metadata.items.length` > `MaxBatchSize`
|
||
- Per-item `latitude`/`longitude`/`tileZoom`/`tileSizeMeters` out of declared range
|
||
- Per-item `capturedAt` outside the freshness window
|
||
- Unknown root or nested fields
|
||
- Type mismatches and malformed UUIDs
|
||
|
||
Sample body:
|
||
|
||
```json
|
||
{
|
||
"type": "https://tools.ietf.org/html/rfc9110#section-15.5.1",
|
||
"title": "One or more validation errors occurred.",
|
||
"status": 400,
|
||
"errors": {
|
||
"metadata.items[0].latitude": ["`latitude` must be between -90 and 90."]
|
||
}
|
||
}
|
||
```
|
||
|
||
The 5-rule per-item quality gate never produces a 400; per-item file rejections always surface in the HTTP 200 response array.
|
||
|
||
### HTTP 401 — missing or invalid JWT (from AZ-487)
|
||
|
||
### HTTP 403 — JWT present but `permissions` claim does not include `GPS`
|
||
|
||
## Persistence semantics
|
||
|
||
- Accepted items are persisted via `ITileRepository.InsertAsync` (the integer-only flight-aware UPSERT path established in AZ-503; supersedes the AZ-484 5-column float-based UPSERT). The `tiles` row carries `source='uav'`, `captured_at` from the request, `flight_id` from the optional `metadata.flightId`, and a deterministic `id = Uuidv5(TileNamespace, "{z}/{x}/{y}/uav/{flight_id or zero-uuid}")`.
|
||
- A UAV upload for a cell that already has a `google_maps` row **coexists** with that row (per `tile-storage.md` Inv-3). The most-recent row across sources wins on read.
|
||
- Two UAV uploads with **different** `flightId` for the same cell coexist as separate rows (one row per flight, sharing `location_hash`). Two UAV uploads with the **same** `flightId` for the same cell UPSERT the existing row, updating `file_path, captured_at, location_hash, content_sha256, updated_at` and overwriting the JPEG bytes on disk. `id` is intentionally NOT regenerated on conflict — re-uploading identical bytes returns the same `tileId` (AZ-503 AC-2).
|
||
- `content_sha256` is computed from the JPEG body on every persisted row.
|
||
|
||
## File-path layout
|
||
|
||
- UAV files (AZ-503): `{StorageConfig.TilesDirectory}/uav/{flightId or 'none'}/{tile_zoom}/{tile_x}/{tile_y}.jpg`
|
||
- Anonymous uploads (`flightId` absent or null) use the literal `none` segment.
|
||
- Per-flight uploads use the full `flightId` UUID as a directory name, so `rm -rf ./tiles/uav/{flightId}/` cleanly removes one flight's evidence without touching other flights or sources.
|
||
- Google Maps files: unchanged from the pre-AZ-488 layout (grandfathered, no migration ships with this contract).
|
||
|
||
`tile_x` and `tile_y` are derived server-side from `(latitude, longitude, tile_zoom)` via `GeoUtils.WorldToTilePos`; the client cannot influence the on-disk path beyond providing valid coordinates and an optional `flightId`.
|
||
|
||
Pre-AZ-503 UAV files written at `./tiles/uav/{z}/{x}/{y}.jpg` (no flight segment) are not relocated. Post-AZ-503 anonymous uploads write to `./tiles/uav/none/{z}/{x}/{y}.jpg`. This split is acceptable because AZ-488 only shipped in cycle 2 and the pre-AZ-503 UAV file count is small.
|
||
|
||
## Concurrency
|
||
|
||
- Per-source UPSERT in the DB is the authoritative serialization point for the same cell.
|
||
- Two concurrent UAV uploads for the exact same `(tile_zoom, tile_x, tile_y)` cell may race on the on-disk bytes; the final file is whichever upload wrote last, and the final DB row is whichever upload UPSERTed last. Per-source `file_path` is identical in this race so the file and row remain self-consistent — there is no orphan reference even under contention.
|
||
|
||
## Invariants
|
||
|
||
- **Inv-1**: Status strings are limited to `"accepted"` and `"rejected"`. New status values require a contract minor bump.
|
||
- **Inv-2**: Reject reasons are drawn from the closed enumeration above. New reasons require a contract minor bump.
|
||
- **Inv-3**: Persisted rows are inserted **only** via `ITileRepository.InsertAsync`. No new write path is introduced by this contract.
|
||
- **Inv-4**: `rejectDetails` MUST NOT contain server-side file paths, .NET exception type names, or internal identifiers. Operators consume these messages directly.
|
||
- **Inv-5**: Item ordering in the response matches item ordering in `metadata.items`.
|
||
|
||
## Non-Goals
|
||
|
||
- **Not covered**: Asynchronous or queued processing — the batch is synchronous. Larger batches than `MaxBatchSize` require a new contract (likely async + status-poll) and a major version bump.
|
||
- **Not covered**: Geofence filtering. UAV uploads outside any operational area still succeed; geofence enforcement is a follow-up PBI.
|
||
- **Not covered**: Per-tile photogrammetry metadata (altitude, focal length, sensor dimensions). Excluded from v1.0.0 by user choice during planning.
|
||
- **Not covered**: Streaming uploads. The endpoint reads each `IFormFile` into memory before validation.
|
||
- **Not covered**: Image storage outside the local filesystem (S3, GCS). Matches the existing Google Maps producer behavior.
|
||
- **Not covered**: Compression or re-encoding of accepted JPEGs. Stored as received.
|
||
|
||
## Versioning Rules
|
||
|
||
- **Patch (1.0.x)**: Documentation clarifications; expanded test cases; tightening of `rejectDetails` content; tuning default thresholds without changing the reject-reason enum.
|
||
- **Minor (1.x.0)**: Adding a new `rejectReason` code; adding optional metadata fields with backward-compatible defaults; relaxing rule thresholds in a backward-compatible way; introducing a new permission (e.g., `SAT`).
|
||
- **Major (2.0.0)**: Changing the request envelope shape; removing or renaming a reject-reason code; changing the persistence path; switching to async/status-poll; changing the `permissions` claim required.
|
||
|
||
Each version bump requires updating the Change Log and notifying every consumer listed in the header.
|
||
|
||
## Test Cases
|
||
|
||
| Case | Inputs | Expected | Reference |
|
||
|------|--------|----------|-----------|
|
||
| happy-1-item | 1× valid 256×256 JPEG, captured_at = now, GPS token | HTTP 200, `accepted`, row with `source='uav'`, file at `./tiles/uav/{z}/{x}/{y}.jpg` | AC-1 |
|
||
| mixed-batch | 3 items: valid, 512×512, non-JPEG bytes | HTTP 200, results = `[accepted, rejected:WRONG_DIMENSIONS, rejected:INVALID_FORMAT]`, 1 new row | AC-2 |
|
||
| multi-source | Pre-seeded `google_maps` row at `T1`; UAV upload for same cell at `T2 > T1` | HTTP 200, both rows persist, subsequent read returns the UAV row | AC-3 |
|
||
| same-source-upsert | UAV row at `T1`; second UAV upload at `T2 > T1` | HTTP 200, exactly one `uav` row remains, refreshed `captured_at` and bytes | AC-4 |
|
||
| no-token | No `Authorization` header | HTTP 401 | AC-5 |
|
||
| no-permission | Token with `permissions=["FL"]` | HTTP 403 | AC-6 |
|
||
| oversized | `metadata.items.length` = `MaxBatchSize + 1` | HTTP 400 envelope error | AC-8 |
|
||
|
||
## Change Log
|
||
|
||
| Version | Date | Change | Author |
|
||
|---------|------|--------|--------|
|
||
| 1.0.0 | 2026-05-11 | Initial contract — batch UAV upload endpoint, 5-rule quality gate, per-source UPSERT, closed reject-reason enum, GPS-permission requirement. Produced by AZ-488. | autodev (cycle 2 step 10) |
|
||
| 1.1.0 | 2026-05-12 | Minor bump for AZ-503: added optional `flightId` per-item metadata field (backward-compatible default `null`); `tileId` in the response is now a deterministic UUIDv5 derived from `(z, x, y, source, flightId)` instead of a random Guid; on-disk path adds a `{flightId or 'none'}` segment for per-flight evidence isolation. No reject-reason changes, no envelope changes, no permission changes. v1.0.0 clients omitting `flightId` keep working unchanged. | autodev (cycle 5 step 13) |
|
||
| 1.2.0 | 2026-05-23 | Minor bump for AZ-810: added the **metadata validation** layer (14 rules) wired through a new `UavUploadValidationFilter` that runs BEFORE the per-item Quality Gate. Marks every non-optional metadata axis with `[JsonRequired]`; uses `UnmappedMemberHandling.Disallow` so unknown root/nested fields are rejected. HTTP 400 envelope-error body now matches the shared `ValidationProblemDetails` shape per `error-shape.md` v1.0.0. **Behavior change**: callers previously sending malformed metadata that silently coerced (e.g. `latitude: 0` for a missing field) now receive HTTP 400 instead of HTTP 200 + per-item rejection. No wire-format renames, no reject-reason changes, no permission changes. The Quality Gate (5 rules) is unchanged and continues as defence-in-depth. | autodev (cycle 8 batch 4) |
|