Files
satellite-provider/_docs/02_document/contracts/api/uav-tile-upload.md
T
Oleksandr Bezdieniezhnykh 490902c80a [AZ-810] Strict validation for POST /api/satellite/upload metadata
Adds the per-endpoint child of AZ-795 ("Strict Input Validation Epic")
for the UAV upload multipart endpoint. Three new validators land under
SatelliteProvider.Api/Validators/:

- UavTileBatchMetadataPayloadValidator: items NotNull + NotEmpty +
  count <= MaxBatchSize + RuleForEach dispatching to the per-item
  validator.
- UavTileMetadataValidator: lat / lon / tileZoom range, tileSizeMeters
  > 0, capturedAt within [now - MaxAgeDays, now + future-skew]; uses an
  injectable TimeProvider so unit tests can drive a fixed clock.
- UavUploadValidationFilter: IEndpointFilter that reads the multipart
  `metadata` form field, deserializes it with the strict global
  JsonSerializerOptions (so UnmappedMemberHandling.Disallow +
  [JsonRequired] from AZ-795 are honored), runs the FluentValidation
  chain, and enforces the cross-field `items.Count == files.Count`
  envelope rule. FluentValidation errors are prefixed with `metadata.`
  so wire keys look like `errors["metadata.items[0].latitude"]`.

[JsonRequired] is added to every non-optional axis on
UavTileMetadata and UavTileBatchMetadataPayload; FlightId stays
nullable per AZ-503 anonymous-flight semantics.

Coverage: 13 unit tests + 16 integration tests + 1 curl probe script
exercise the happy path and every failure mode. All 9 ACs covered;
no regression in AZ-488 UavUploadTests payloads (traced against the
new rules).

Documentation: uav-tile-upload.md bumped v1.1.0 -> v1.2.0 with the
new validation rules section + 400-shape examples + changelog entry.
api_program.md updated to describe the three new validators + filter
+ the AddTransient<UavUploadValidationFilter>() DI registration.

Reports: batch_04_cycle8_report.md + reviews/batch_04_cycle8_review.md
record the PASS_WITH_WARNINGS verdict (2 Low DRY-in-tests findings:
FixedTimeProvider duplication crossed the cycle-2 "promote to shared"
threshold; PostBatch helper duplicated between two integration
suites). Both deferred to follow-up PBIs.

Task spec archived: _docs/02_tasks/todo/AZ-810... -> done/.
Jira: AZ-810 transitioned In Progress -> In Testing.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-23 13:32:19 +03:00

241 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Contract: uav-tile-upload
**Component**: WebApi (`SatelliteProvider.Api`) producing rows via TileDownloader (`SatelliteProvider.Services.TileDownloader`)
**Producer task**: AZ-488 — `_docs/02_tasks/done/AZ-488_uav_tile_upload.md`
**Extended by**: AZ-503 — `_docs/02_tasks/done/AZ-503_tile_identity_uuidv5_bulk_list.md` (added optional `flightId` per-item field; per-flight on-disk path; deterministic `tileId`)
**Consumer tasks**: `gps-denied-onboard`, mission planner UI, any future UAV-equipped client
**Version**: 1.2.0
**Status**: frozen
**Last Updated**: 2026-05-23
## Purpose
Defines the HTTP contract for the `POST /api/satellite/upload` batch endpoint that ingests UAV-captured satellite tiles, runs each item through a 5-rule quality gate, and persists accepted rows under the `tile-storage` v1.0.0 data contract with `source='uav'`.
## Endpoint
```
POST /api/satellite/upload
Content-Type: multipart/form-data
Authorization: Bearer <JWT>
```
The request MUST carry a valid JWT (AZ-487) AND the `permissions` claim MUST contain `GPS`. Anonymous requests are rejected with HTTP 401; authenticated requests without `GPS` are rejected with HTTP 403.
## Request shape
Multipart form fields (case-sensitive part names):
| Part | Type | Required | Description |
|------|------|----------|-------------|
| `metadata` | JSON string | yes | Document of shape `{ "items": [ <UavTileMetadata>, ... ] }`. See the table below for the per-item schema. |
| `files` | binary (repeating) | yes | One JPEG file per metadata item. Files are correlated to metadata entries by ordinal index (file index `i``metadata.items[i]`). Each part MUST be sent under the form field name `files`. |
### `UavTileMetadata` (per item)
| Field | Type | Required | Description | Constraints |
|-------|------|----------|-------------|-------------|
| `latitude` | number | yes | Geographic latitude of the tile center | WGS-84 decimal degrees |
| `longitude` | number | yes | Geographic longitude of the tile center | WGS-84 decimal degrees |
| `tileZoom` | integer | yes | Slippy Map zoom level | Must satisfy the same zoom-level policy as the existing tile pipeline (see `MapConfig.AllowedZoomLevels`) |
| `tileSizeMeters` | number | yes | Tile size in meters at the captured latitude | Producer-supplied |
| `capturedAt` | string (ISO-8601 UTC) | yes | Moment of UAV image capture | Must satisfy the captured-at rule (see Quality Gate, Rule 4) |
| `flightId` | string (UUID) | no | AZ-503: optional flight identifier. When present, two flights uploading the same cell coexist as separate rows; absent uploads share a single anonymous row per cell. Omitting the field is fully backward-compatible with v1.0.0 clients. | RFC 4122 UUID. Backward-compatible default: `null` |
Field names are camelCase. Property-name matching is case-insensitive on read.
### Constraints
- The number of `files` MUST equal the number of `metadata.items` entries (1:1 correlation by ordinal index). Mismatch → HTTP 400.
- `metadata.items.length` MUST NOT exceed `UavQualityConfig.MaxBatchSize` (default `100`). Oversize → HTTP 400.
- The total request body size is capped at `MaxBatchSize × MaxBytes` by Kestrel's `MaxRequestBodySize` and the form `MultipartBodyLengthLimit`. Requests beyond this cap are rejected at the framework layer (HTTP 413/400).
## Metadata validation (14 rules, v1.2.0)
Before any file bytes are inspected by the Quality Gate below, the `metadata` envelope is run through a strict validator chain. This is the **metadata layer**; the **file layer** (see Quality Gate) is unchanged.
The validator is split into three composing layers and runs inside a custom multipart endpoint filter (`UavUploadValidationFilter`):
1. **Deserializer layer**`JsonSerializerOptions.UnmappedMemberHandling.Disallow` + `[JsonRequired]` on every non-optional axis of `UavTileBatchMetadataPayload` / `UavTileMetadata`. Catches missing-required, unknown fields (root and nested), JSON type mismatches, and malformed UUIDs. Errors surface under `errors["metadata"]`.
2. **FluentValidation layer**`UavTileBatchMetadataPayloadValidator` (envelope rules) and `UavTileMetadataValidator` (per-item rules). Errors surface under `errors["metadata.items"]` / `errors["metadata.items[i].<field>"]`.
3. **Cross-field envelope rule**`items.Count == files.Count`, evaluated in the filter after the FluentValidation result is clean. Errors surface under **both** `errors["metadata.items"]` AND `errors["files"]`.
Any failing rule short-circuits with HTTP 400 + RFC 7807 `ValidationProblemDetails` per `error-shape.md` v1.0.0. The body never reaches the Quality Gate or the persistence path on a metadata validation failure.
| # | Rule | Failure condition | Error path | Layer |
|---|------|-------------------|------------|-------|
| 1 | Multipart envelope present | Request `Content-Type` is not `multipart/form-data` | `errors["metadata"]` | filter |
| 2 | `metadata` form field present | Multipart form has no part named `metadata` | `errors["metadata"]` | filter |
| 3 | `metadata` parses as JSON | Malformed JSON body | `errors["metadata"]` | deserializer |
| 4 | `items` required + non-empty | `items` missing OR `items: []` | `errors["metadata.items"]` | FluentValidation |
| 5 | `items.Count``UavQualityConfig.MaxBatchSize` | `items.Count > MaxBatchSize` (default 100) | `errors["metadata.items"]` | FluentValidation |
| 6 | `items.Count == files.Count` | Per-item file count differs from metadata count | `errors["metadata.items"]` + `errors["files"]` | filter |
| 7 | `latitude` ∈ [-90, +90] | Out of range | `errors["metadata.items[i].latitude"]` | FluentValidation |
| 8 | `longitude` ∈ [-180, +180] | Out of range | `errors["metadata.items[i].longitude"]` | FluentValidation |
| 9 | `tileZoom` ∈ [0, 22] | Out of range | `errors["metadata.items[i].tileZoom"]` | FluentValidation |
| 10 | `tileSizeMeters` > 0 | Zero or negative | `errors["metadata.items[i].tileSizeMeters"]` | FluentValidation |
| 11 | `capturedAt` within freshness window | `capturedAt > now + CapturedAtFutureSkewSeconds` OR `capturedAt < now - MaxAgeDays` | `errors["metadata.items[i].capturedAt"]` | FluentValidation |
| 12 | `flightId` parses as UUID | Non-UUID string (`null`/missing is valid per AZ-503) | `errors["metadata"]` | deserializer |
| 13 | Unknown fields rejected (root + nested) | Any field not declared on the DTO | `errors["metadata"]` | deserializer |
| 14 | Type mismatch | e.g. `"latitude": "fifty"`, `"tileZoom": 18.5` | `errors["metadata"]` | deserializer |
### Relationship to the Quality Gate
The Quality Gate's Rule 4 (captured-at freshness) is preserved exactly as documented below. It runs **after** the metadata validator and provides defence-in-depth against handler callers that bypass the filter (unit tests of `IUavTileUploadHandler`, future internal call paths). Operators consuming the public API will see the metadata validator's verdict first.
The Quality Gate's Rules 1, 2, 3, 5 (file-level: format, size, dimensions, luminance) are unchanged and still produce per-item rejections via the existing HTTP 200 + `rejectReason` envelope — they have no metadata-validator equivalent.
## Quality Gate (5 rules)
Each item is evaluated in this fixed order. The **first** failing rule produces the reported reason; subsequent rules are not evaluated for that item.
| # | Rule | Failure condition | Reason code |
|---|------|-------------------|-------------|
| 1 | Format | `Content-Type` of the file part is not `image/jpeg` (case-insensitive, allowing trailing parameters) OR the file's first 3 bytes are not `FF D8 FF` | `INVALID_FORMAT` |
| 2 | Size band | `bytes.length` is outside `[UavQualityConfig.MinBytes, UavQualityConfig.MaxBytes]` (defaults: 5 KiB … 5 MiB) | `SIZE_OUT_OF_BAND` |
| 3 | Dimensions | Image width OR height ≠ `MapConfig.TileSizePixels` (default 256). Strict equality, no tolerance | `WRONG_DIMENSIONS` |
| 4a | Captured-at future | `capturedAt > now + UavQualityConfig.CapturedAtFutureSkewSeconds` (default 30s) | `CAPTURED_AT_FUTURE` |
| 4b | Captured-at age | `capturedAt < now - UavQualityConfig.MaxAgeDays` (default 7 days) | `CAPTURED_AT_TOO_OLD` |
| 5 | Blank / uniform | Pixel-luminance variance on a downsampled (default 32×32, configurable via `UavQualityConfig.LuminanceSampleSize`) version of the image is below `UavQualityConfig.MinLuminanceVariance` (default 10.0) | `IMAGE_TOO_UNIFORM` |
If the file decode itself fails for the variance check, the item is rejected with `INVALID_FORMAT` (the file is not a real JPEG even though the header bytes matched).
Storage failures while persisting an accepted item (e.g., disk full, unwritable path) are surfaced as `STORAGE_FAILURE` so the client can retry that specific item without re-uploading the whole batch.
### Reject-reason enumeration (closed)
| Code | Source rule | When the reason fires |
|------|-------------|------------------------|
| `INVALID_FORMAT` | Rule 1 + Rule 5 decode safety net | Wrong content-type, wrong magic bytes, or undecodable bytes |
| `SIZE_OUT_OF_BAND` | Rule 2 | Byte length outside `[MinBytes, MaxBytes]` |
| `WRONG_DIMENSIONS` | Rule 3 | Width or height ≠ `MapConfig.TileSizePixels` |
| `CAPTURED_AT_FUTURE` | Rule 4a | `capturedAt` is more than `CapturedAtFutureSkewSeconds` in the future |
| `CAPTURED_AT_TOO_OLD` | Rule 4b | `capturedAt` is older than `MaxAgeDays` |
| `IMAGE_TOO_UNIFORM` | Rule 5 | Luminance variance below `MinLuminanceVariance` |
| `METADATA_MISSING` | Validation gate | Per-item metadata could not be matched (reserved; not currently surfaced because batch mismatch is reported as an envelope error) |
| `STORAGE_FAILURE` | Persistence path | The image bytes could not be written to disk or the repository insert raised an IO error |
Adding a new code is a **minor** contract version bump per the Versioning Rules below. Removing or renaming a code is **major**.
## Response shape
### HTTP 200 — per-item results
```json
{
"items": [
{ "index": 0, "status": "accepted", "tileId": "11111111-...", "rejectReason": null, "rejectDetails": null },
{ "index": 1, "status": "rejected", "tileId": null, "rejectReason": "WRONG_DIMENSIONS", "rejectDetails": null }
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `items` | array | One result per input item, in the same order as `metadata.items` |
| `items[].index` | integer | Ordinal index of the item in the request batch |
| `items[].status` | string | `"accepted"` or `"rejected"` (closed enumeration) |
| `items[].tileId` | UUID or null | When `accepted`, the persisted `tiles.id` value; null when `rejected` |
| `items[].rejectReason` | string or null | Reason code (see table above) when `rejected`; null when `accepted` |
| `items[].rejectDetails` | string or null | Optional human-readable detail; MUST NOT leak server paths, exception types, or internal identifiers |
### HTTP 400 — envelope error (RFC 7807 `application/problem+json`)
Returned when the request itself is malformed. As of v1.2.0 every 400 body conforms to the shared `ValidationProblemDetails` shape in `error-shape.md` v1.0.0, with the `errors` map keys listed in the "Metadata validation" rule table above. Triggers include:
- `metadata` field absent, empty, or not valid JSON
- `metadata.items` empty or null
- `metadata.items.length``files.length`
- `metadata.items.length` > `MaxBatchSize`
- Per-item `latitude`/`longitude`/`tileZoom`/`tileSizeMeters` out of declared range
- Per-item `capturedAt` outside the freshness window
- Unknown root or nested fields
- Type mismatches and malformed UUIDs
Sample body:
```json
{
"type": "https://tools.ietf.org/html/rfc9110#section-15.5.1",
"title": "One or more validation errors occurred.",
"status": 400,
"errors": {
"metadata.items[0].latitude": ["`latitude` must be between -90 and 90."]
}
}
```
The 5-rule per-item quality gate never produces a 400; per-item file rejections always surface in the HTTP 200 response array.
### HTTP 401 — missing or invalid JWT (from AZ-487)
### HTTP 403 — JWT present but `permissions` claim does not include `GPS`
## Persistence semantics
- Accepted items are persisted via `ITileRepository.InsertAsync` (the integer-only flight-aware UPSERT path established in AZ-503; supersedes the AZ-484 5-column float-based UPSERT). The `tiles` row carries `source='uav'`, `captured_at` from the request, `flight_id` from the optional `metadata.flightId`, and a deterministic `id = Uuidv5(TileNamespace, "{z}/{x}/{y}/uav/{flight_id or zero-uuid}")`.
- A UAV upload for a cell that already has a `google_maps` row **coexists** with that row (per `tile-storage.md` Inv-3). The most-recent row across sources wins on read.
- Two UAV uploads with **different** `flightId` for the same cell coexist as separate rows (one row per flight, sharing `location_hash`). Two UAV uploads with the **same** `flightId` for the same cell UPSERT the existing row, updating `file_path, captured_at, location_hash, content_sha256, updated_at` and overwriting the JPEG bytes on disk. `id` is intentionally NOT regenerated on conflict — re-uploading identical bytes returns the same `tileId` (AZ-503 AC-2).
- `content_sha256` is computed from the JPEG body on every persisted row.
## File-path layout
- UAV files (AZ-503): `{StorageConfig.TilesDirectory}/uav/{flightId or 'none'}/{tile_zoom}/{tile_x}/{tile_y}.jpg`
- Anonymous uploads (`flightId` absent or null) use the literal `none` segment.
- Per-flight uploads use the full `flightId` UUID as a directory name, so `rm -rf ./tiles/uav/{flightId}/` cleanly removes one flight's evidence without touching other flights or sources.
- Google Maps files: unchanged from the pre-AZ-488 layout (grandfathered, no migration ships with this contract).
`tile_x` and `tile_y` are derived server-side from `(latitude, longitude, tile_zoom)` via `GeoUtils.WorldToTilePos`; the client cannot influence the on-disk path beyond providing valid coordinates and an optional `flightId`.
Pre-AZ-503 UAV files written at `./tiles/uav/{z}/{x}/{y}.jpg` (no flight segment) are not relocated. Post-AZ-503 anonymous uploads write to `./tiles/uav/none/{z}/{x}/{y}.jpg`. This split is acceptable because AZ-488 only shipped in cycle 2 and the pre-AZ-503 UAV file count is small.
## Concurrency
- Per-source UPSERT in the DB is the authoritative serialization point for the same cell.
- Two concurrent UAV uploads for the exact same `(tile_zoom, tile_x, tile_y)` cell may race on the on-disk bytes; the final file is whichever upload wrote last, and the final DB row is whichever upload UPSERTed last. Per-source `file_path` is identical in this race so the file and row remain self-consistent — there is no orphan reference even under contention.
## Invariants
- **Inv-1**: Status strings are limited to `"accepted"` and `"rejected"`. New status values require a contract minor bump.
- **Inv-2**: Reject reasons are drawn from the closed enumeration above. New reasons require a contract minor bump.
- **Inv-3**: Persisted rows are inserted **only** via `ITileRepository.InsertAsync`. No new write path is introduced by this contract.
- **Inv-4**: `rejectDetails` MUST NOT contain server-side file paths, .NET exception type names, or internal identifiers. Operators consume these messages directly.
- **Inv-5**: Item ordering in the response matches item ordering in `metadata.items`.
## Non-Goals
- **Not covered**: Asynchronous or queued processing — the batch is synchronous. Larger batches than `MaxBatchSize` require a new contract (likely async + status-poll) and a major version bump.
- **Not covered**: Geofence filtering. UAV uploads outside any operational area still succeed; geofence enforcement is a follow-up PBI.
- **Not covered**: Per-tile photogrammetry metadata (altitude, focal length, sensor dimensions). Excluded from v1.0.0 by user choice during planning.
- **Not covered**: Streaming uploads. The endpoint reads each `IFormFile` into memory before validation.
- **Not covered**: Image storage outside the local filesystem (S3, GCS). Matches the existing Google Maps producer behavior.
- **Not covered**: Compression or re-encoding of accepted JPEGs. Stored as received.
## Versioning Rules
- **Patch (1.0.x)**: Documentation clarifications; expanded test cases; tightening of `rejectDetails` content; tuning default thresholds without changing the reject-reason enum.
- **Minor (1.x.0)**: Adding a new `rejectReason` code; adding optional metadata fields with backward-compatible defaults; relaxing rule thresholds in a backward-compatible way; introducing a new permission (e.g., `SAT`).
- **Major (2.0.0)**: Changing the request envelope shape; removing or renaming a reject-reason code; changing the persistence path; switching to async/status-poll; changing the `permissions` claim required.
Each version bump requires updating the Change Log and notifying every consumer listed in the header.
## Test Cases
| Case | Inputs | Expected | Reference |
|------|--------|----------|-----------|
| happy-1-item | 1× valid 256×256 JPEG, captured_at = now, GPS token | HTTP 200, `accepted`, row with `source='uav'`, file at `./tiles/uav/{z}/{x}/{y}.jpg` | AC-1 |
| mixed-batch | 3 items: valid, 512×512, non-JPEG bytes | HTTP 200, results = `[accepted, rejected:WRONG_DIMENSIONS, rejected:INVALID_FORMAT]`, 1 new row | AC-2 |
| multi-source | Pre-seeded `google_maps` row at `T1`; UAV upload for same cell at `T2 > T1` | HTTP 200, both rows persist, subsequent read returns the UAV row | AC-3 |
| same-source-upsert | UAV row at `T1`; second UAV upload at `T2 > T1` | HTTP 200, exactly one `uav` row remains, refreshed `captured_at` and bytes | AC-4 |
| no-token | No `Authorization` header | HTTP 401 | AC-5 |
| no-permission | Token with `permissions=["FL"]` | HTTP 403 | AC-6 |
| oversized | `metadata.items.length` = `MaxBatchSize + 1` | HTTP 400 envelope error | AC-8 |
## Change Log
| Version | Date | Change | Author |
|---------|------|--------|--------|
| 1.0.0 | 2026-05-11 | Initial contract — batch UAV upload endpoint, 5-rule quality gate, per-source UPSERT, closed reject-reason enum, GPS-permission requirement. Produced by AZ-488. | autodev (cycle 2 step 10) |
| 1.1.0 | 2026-05-12 | Minor bump for AZ-503: added optional `flightId` per-item metadata field (backward-compatible default `null`); `tileId` in the response is now a deterministic UUIDv5 derived from `(z, x, y, source, flightId)` instead of a random Guid; on-disk path adds a `{flightId or 'none'}` segment for per-flight evidence isolation. No reject-reason changes, no envelope changes, no permission changes. v1.0.0 clients omitting `flightId` keep working unchanged. | autodev (cycle 5 step 13) |
| 1.2.0 | 2026-05-23 | Minor bump for AZ-810: added the **metadata validation** layer (14 rules) wired through a new `UavUploadValidationFilter` that runs BEFORE the per-item Quality Gate. Marks every non-optional metadata axis with `[JsonRequired]`; uses `UnmappedMemberHandling.Disallow` so unknown root/nested fields are rejected. HTTP 400 envelope-error body now matches the shared `ValidationProblemDetails` shape per `error-shape.md` v1.0.0. **Behavior change**: callers previously sending malformed metadata that silently coerced (e.g. `latitude: 0` for a missing field) now receive HTTP 400 instead of HTTP 200 + per-item rejection. No wire-format renames, no reject-reason changes, no permission changes. The Quality Gate (5 rules) is unchanged and continues as defence-in-depth. | autodev (cycle 8 batch 4) |