[AZ-1074] [AZ-1075] Cycle 9 closeout: security, tests, metrics

Resolve F-AZ1074-1/2 (collection caps, generic gRPC internal errors).
Standalone integration compose stack, docs, security audit, perf and retro.

Co-authored-by: Cursor <cursoragent@cursor.com>

_docs/02_document/modules/api_program.md

@@ -18,6 +18,11 @@ Application entry point. Configures DI container, sets up middleware, defines mi
 | POST | `/api/satellite/route` | `CreateRoute` | Create route with intermediate points. AZ-809 (cycle 8) added strict pre-handler validation via `WithValidation<CreateRouteRequest>()`: non-zero `id`, name length ∈ \[1, 200\], description length ≤ 1000, `regionSizeMeters` ∈ \[100, 10000\], `zoomLevel` ∈ \[0, 22\], `points` count ∈ \[2, 500\] with per-point lat/lon range checks, per-polygon NW-of-SE invariants, and the `createTilesZip ⇒ requestMaps` cross-field rule. Deserializer-layer failures (missing `[JsonRequired]` axes, unknown fields, type mismatches) are caught by `GlobalExceptionHandler` and produce the same RFC 7807 envelope. Contract: `_docs/02_document/contracts/api/route-creation.md` v1.0.0 + `_docs/02_document/contracts/api/error-shape.md` v1.0.0. |
 | GET | `/api/satellite/route/{id}` | `GetRoute` | Get route with all points |
 
+### gRPC Services (AZ-1074, cycle 9)
+| Service | RPC | Handler | Description |
+|---------|-----|---------|-------------|
+| `RouteTileDelivery` | `DeliverRouteTiles(DeliverRouteTilesRequest) → stream RouteTileEvent` | `RouteTileDeliveryGrpcService` | Server-streaming route tile provisioning for gps-denied-onboard pre-flight cache fill. Reuses `RouteTileDeliveryOrchestrator` + tile download pipeline. Auth: gRPC metadata `authorization: Bearer <JWT>` (same HS256 contract as REST). Proto: `SatelliteProvider.GrpcContracts/tile_provision.proto` (`package satellite.v1`). Contract doc: `_docs/02_document/contracts/c11_tilemanager/tile_provision_grpc.md`. Invalid requests (single waypoint, out-of-range lat/lon/zoom) map to gRPC `InvalidArgument`. |
+
 ### Local Records (defined in Program.cs)
 - `GetSatelliteTilesResponse`, `SatelliteTile` — MGRS response stubs
 - `DownloadTileResponse` — tile download response
@@ -74,11 +79,12 @@ Application entry point. Configures DI container, sets up middleware, defines mi
 8. CORS policy: `TilesCors` — configured origins from `CorsConfig:AllowedOrigins`, falls back to allow-any
 9. JSON options: camelCase, case-insensitive
 10. **JWT authentication (AZ-487 + AZ-494)**: `AddSatelliteJwt(builder.Configuration)` (extension in `SatelliteProvider.Api.Authentication`) registers `JwtBearer` with `TokenValidationParameters` set per the suite auth contract: signature + lifetime + issuer + audience validation, 30 s clock skew, ≥ 32-byte HMAC key. The `iss` value comes from `JWT_ISSUER` env (fallback `Jwt:Issuer` config); the `aud` value comes from `JWT_AUDIENCE` env (fallback `Jwt:Audience` config). All three values (secret, iss, aud) are fail-fast — the API throws `InvalidOperationException` at startup if any is unset or whitespace-only. Production deploys MUST set the env vars with admin-team-confirmed values; `appsettings.json` ships empty so the fail-fast triggers. `appsettings.Development.json` ships clearly-tagged DEV-ONLY values (`DEV-ONLY-iss-admin-azaion-local` / `DEV-ONLY-aud-satellite-provider`) so local dev works out-of-the-box. Followed by `AddAuthorization` with the `RequiresGpsPermission` policy (AZ-488).
-11. **Kestrel HTTP/2 (AZ-505)**: `builder.WebHost.ConfigureKestrel(opts => opts.ConfigureEndpointDefaults(lo => lo.Protocols = HttpProtocols.Http1AndHttp2))`. The dev listener is now `https://+:8080` with a self-signed cert (`./certs/api.pfx`, generated idempotently by `scripts/run-tests.sh` and bound via `ASPNETCORE_Kestrel__Certificates__Default__Path` / `__Password` in `docker-compose.yml`). Kestrel needs TLS for HTTP/2 protocol negotiation; ALPN advertises both `h2` and `http/1.1` so HTTP/2-capable clients (browser Leaflet, `HttpClient` with `Version20` + `RequestVersionExact`, httpx `http2=True`) multiplex tile reads on a single TLS connection, and legacy clients fall back to HTTP/1.1. The integration-test container trusts the dev cert via `/usr/local/share/ca-certificates/` + `update-ca-certificates`. AZ-505 AC-5 verifies the multiplex semantics here; production termination is expected at the ingress (Envoy / nginx / ALB) — Kestrel can then drop to HTTP/2 cleartext behind it without changing this code.
-12. **ProblemDetails + global exception handler (AZ-795, cycle 7)**: `AddProblemDetails()` + `AddExceptionHandler<GlobalExceptionHandler>()` register the uniform RFC 7807 error pipeline. `app.UseExceptionHandler()` (in the middleware chain) routes unhandled exceptions through `GlobalExceptionHandler`, which converts `BadHttpRequestException(JsonException)` (unknown-member rejection, missing-required-field, JSON type mismatch) into `ValidationProblemDetails` with the same `errors[]` map shape that FluentValidation produces. This is the deserializer-layer half of the strict-validation contract — `error-shape.md` v1.0.0 §"Two collaborating pieces of shared infrastructure".
-13. **Strict JSON parsing (AZ-795, cycle 7)**: `ConfigureHttpJsonOptions` sets `PropertyNamingPolicy = CamelCase`, `PropertyNameCaseInsensitive = true`, `UnmappedMemberHandling = Disallow`, and adds `JsonStringEnumConverter` with camelCase naming. `UnmappedMemberHandling.Disallow` is the key strict-parsing knob: any unknown root or nested field is rejected at the deserializer rather than silently dropped. Catches typos (`{"Z":12}` uppercase, `{"tileZoom":...}` post-rename) that no FluentValidation rule can see after deserialization.
-14. **FluentValidation registration (AZ-795 + AZ-796, cycle 7)**: `AddValidatorsFromAssemblyContaining<Program>()` auto-registers every `IValidator<T>` in the API assembly (currently `InventoryRequestValidator` + `TileCoordValidator`, AZ-808 `RegionRequestValidator`, AZ-809 `CreateRouteRequestValidator` + `RoutePointValidator` + `GeofencePolygonValidator`, AZ-810 `UavTileBatchMetadataPayloadValidator` + `UavTileMetadataValidator`, AZ-811 `GetTileByLatLonQueryValidator`). `GlobalValidatorConfig.ApplyOnce()` runs the idempotent process-wide config — sets `ValidatorOptions.Global.PropertyNameResolver` so `errors` map keys are camelCase (matches the request body's casing per `error-shape.md` Inv-4). Per-endpoint opt-in via `.WithValidation<T>()` on the JSON-body endpoints — the generic `ValidationEndpointFilter<T>` resolves the validator from DI at request time and returns `Results.ValidationProblem` on failure.
-15. **AZ-810 multipart validation filter (cycle 8)**: `AddTransient<UavUploadValidationFilter>()` registers the bespoke filter used by `POST /api/satellite/upload`. The endpoint is `multipart/form-data` so the generic `.WithValidation<T>()` JSON-body filter cannot bind; this filter reads the `metadata` form field, deserializes it via the strict global `JsonSerializerOptions`, runs the FluentValidation chain, and enforces the cross-field `items.Count == files.Count` envelope rule. Wired on the endpoint with `.AddEndpointFilter<UavUploadValidationFilter>()` between `.RequireAuthorization(SatellitePermissions.UavUploadPolicy)` and the metadata accept/produces annotations.
+11. **Kestrel HTTP/2 (AZ-505)**: `builder.WebHost.ConfigureKestrel(opts => opts.ConfigureEndpointDefaults(lo => lo.Protocols = HttpProtocols.Http1AndHttp2))`. The dev listener is now `https://+:8080` with a self-signed cert (`./certs/api.pfx`, generated idempotently by `scripts/run-tests.sh` and bound via `ASPNETCORE_Kestrel__Certificates__Default__Path` / `__Password` in `docker-compose.yml`). Kestrel needs TLS for HTTP/2 protocol negotiation; ALPN advertises both `h2` and `http/1.1` so HTTP/2-capable clients (browser Leaflet, `HttpClient` with `Version20` + `RequestVersionExact`, httpx `http2=True`, gRPC over HTTP/2) multiplex tile reads on a single TLS connection, and legacy clients fall back to HTTP/1.1. The integration-test container trusts the dev cert via `/usr/local/share/ca-certificates/` + `update-ca-certificates`. AZ-505 AC-5 verifies the multiplex semantics here; production termination is expected at the ingress (Envoy / nginx / ALB) — Kestrel can then drop to HTTP/2 cleartext behind it without changing this code.
+12. **gRPC (AZ-1074, cycle 9)**: `AddGrpc()` + `MapGrpcService<RouteTileDeliveryGrpcService>()`. Shares JWT auth middleware with REST — callers pass `authorization: Bearer <token>` in gRPC metadata. Server-streaming RPC delegates to `IRouteTileDeliveryOrchestrator.DeliverAsync`.
+13. **ProblemDetails + global exception handler (AZ-795, cycle 7)**: `AddProblemDetails()` + `AddExceptionHandler<GlobalExceptionHandler>()` register the uniform RFC 7807 error pipeline. `app.UseExceptionHandler()` (in the middleware chain) routes unhandled exceptions through `GlobalExceptionHandler`, which converts `BadHttpRequestException(JsonException)` (unknown-member rejection, missing-required-field, JSON type mismatch) into `ValidationProblemDetails` with the same `errors[]` map shape that FluentValidation produces. This is the deserializer-layer half of the strict-validation contract — `error-shape.md` v1.0.0 §"Two collaborating pieces of shared infrastructure".
+14. **Strict JSON parsing (AZ-795, cycle 7)**: `ConfigureHttpJsonOptions` sets `PropertyNamingPolicy = CamelCase`, `PropertyNameCaseInsensitive = true`, `UnmappedMemberHandling = Disallow`, and adds `JsonStringEnumConverter` with camelCase naming. `UnmappedMemberHandling.Disallow` is the key strict-parsing knob: any unknown root or nested field is rejected at the deserializer rather than silently dropped. Catches typos (`{"Z":12}` uppercase, `{"tileZoom":...}` post-rename) that no FluentValidation rule can see after deserialization.
+15. **FluentValidation registration (AZ-795 + AZ-796, cycle 7)**: `AddValidatorsFromAssemblyContaining<Program>()` auto-registers every `IValidator<T>` in the API assembly (currently `InventoryRequestValidator` + `TileCoordValidator`, AZ-808 `RegionRequestValidator`, AZ-809 `CreateRouteRequestValidator` + `RoutePointValidator` + `GeofencePolygonValidator`, AZ-810 `UavTileBatchMetadataPayloadValidator` + `UavTileMetadataValidator`, AZ-811 `GetTileByLatLonQueryValidator`). `GlobalValidatorConfig.ApplyOnce()` runs the idempotent process-wide config — sets `ValidatorOptions.Global.PropertyNameResolver` so `errors` map keys are camelCase (matches the request body's casing per `error-shape.md` Inv-4). Per-endpoint opt-in via `.WithValidation<T>()` on the JSON-body endpoints — the generic `ValidationEndpointFilter<T>` resolves the validator from DI at request time and returns `Results.ValidationProblem` on failure.
+16. **AZ-810 multipart validation filter (cycle 8)**: `AddTransient<UavUploadValidationFilter>()` registers the bespoke filter used by `POST /api/satellite/upload`. The endpoint is `multipart/form-data` so the generic `.WithValidation<T>()` JSON-body filter cannot bind; this filter reads the `metadata` form field, deserializes it via the strict global `JsonSerializerOptions`, runs the FluentValidation chain, and enforces the cross-field `items.Count == files.Count` envelope rule. Wired on the endpoint with `.AddEndpointFilter<UavUploadValidationFilter>()` between `.RequireAuthorization(SatellitePermissions.UavUploadPolicy)` and the metadata accept/produces annotations.
 
 ### Startup
 1. Database migration via `DatabaseMigrator.RunMigrations()` — throws on failure
@@ -128,7 +134,8 @@ NuGet: `Serilog.AspNetCore` (8.0.3 — fallback retained on .NET 10 per AZ-500 R
 
 ## Consumers
 - HTTP clients (external)
-- Integration tests (via HTTP)
+- gRPC clients (gps-denied-onboard C11 — AZ-1074)
+- Integration tests (via HTTP + gRPC)
 
 ## Data Models
 Defines several local request/response records that are not shared with other projects.

_docs/02_document/modules/tests_integration.md

@@ -18,6 +18,7 @@ Console application that runs end-to-end integration tests against a live API in
 - `TileInventoryTests` (added cycle 6 — AZ-505) — `OrderingAndPresentAbsentShaping_AC1`, `LeafletReadReturnsMostRecentViaLocationHash_AC2`, `ValidationRejectsBothPopulated_AC6`, `ValidationRejectsNeitherPopulated_AC6`, `ValidationRejectsOversizedBatch_AC6`, `UnauthenticatedRequestReturns401_AC6`, `PerformanceBudget_AC4` (full-suite only). Tests are cycle-7-stable — they use the post-AZ-794 `{z, x, y}` wire shape and a minor x/y reduction was applied in cycle 7 to keep the synthetic coords within the z=18 slippy bounds enforced by `TileCoordValidator`.
 - `TileInventoryValidationTests` (added cycle 7 — AZ-796) — 16 tests: `HappyPath_Returns200`, `EmptyBody_Returns400`, `NeitherPopulated_Returns400`, `BothPopulated_Returns400`, `EmptyTilesArray_Returns400`, `TilesOverCap_Returns400`, `MissingZ_Returns400WithFieldPath`, `MissingXAndY_Returns400`, `ZoomOutOfRange_Returns400WithFieldPath`, `XBeyondZoomBounds_Returns400`, `YBeyondZoomBounds_Returns400`, `NegativeAxis_Returns400`, `UnknownRootField_Returns400`, `UnknownNestedField_Returns400`, `OldV1FieldName_Returns400` (AZ-794 + AZ-796 intersection — exact AZ-777 Phase 1 reproducer body, asserts legacy `tileZoom/tileX/tileY` now yields 400), `TypeMismatch_Returns400`. Each test exercises one of the 9 validation rules end-to-end through `ValidationEndpointFilter<TileInventoryRequest>` + `GlobalExceptionHandler`, asserts HTTP 400 + RFC 7807 `ValidationProblemDetails` shape via the shared `ProblemDetailsAssertions` helper.
 - `IdempotentPostTests` — pre-existing; cycle 7 adjusted the route-point payload from PascalCase (`Latitude`/`Longitude`) to camelCase (`lat`/`lon`) because the post-AZ-795 `UnmappedMemberHandling.Disallow` would otherwise reject the previously-silently-ignored fields. The `RoutePoint` DTO has carried `JsonPropertyName("lat"/"lon")` since AZ-309; cycle 7's strict JSON parsing exposed the test was sending the wrong shape and getting away with it via the pre-cycle-7 permissive deserializer.
+- `RouteTileDeliveryGrpcTests` (added cycle 9 — AZ-1074/AZ-1075) — `RunHappyPath`, `RunInvalidRequests` (single waypoint / lat out of range / zoom out of range → `InvalidArgument`), `RunBackpressureSafe` (slow consumer preserves JPEG + SHA256), `RunRestConsistency` (REST route CSV tile keys overlap gRPC stream keys). Wired into both smoke and full suites via `Program.cs`.
 
 ### Supporting Classes
 - `Models.cs` — HTTP response DTOs for deserialization
@@ -34,9 +35,11 @@ Console application that runs end-to-end integration tests against a live API in
 - `IntegrationTestDatabaseReset.cs` (AZ-493) — instance class with a single `EnsureCleanStateAsync()` method that truncates the integration-test target tables in FK-safe order. Guarded via `SatelliteProvider.TestSupport.IntegrationTestResetGuard` (env + Host allowlist) so it cannot run against a non-test database.
 - `PerfBootstrap.cs` (AZ-492) — static helpers for the perf harness bootstrap subcommands. `MintToken()` mints a 4-hour HS256 token with subject `perf-tests` and a `permissions: GPS` claim via the canonical `SatelliteProvider.TestSupport.JwtTokenFactory.Create`; `GenerateUavFixture(args)` writes a 256×256 random-noise JPEG via `SixLabors.ImageSharp` to the path passed on the CLI. Invoked from `scripts/run-performance-tests.sh` via `dotnet <SatelliteProvider.IntegrationTests.dll> --mint-only` and `--gen-uav-fixture <path>`.
 - `ProblemDetailsAssertions.cs` (added cycle 7 — AZ-795) — shared static helpers for asserting RFC 7807 ProblemDetails bodies on integration-test responses. `ReadProblemDetailsAsync(HttpResponseMessage, label)` deserialises the response body into a `JsonElement` with helpful failure messages when the content-type / shape doesn't match. `AssertProblemDetails(problem, expectedStatus, label)` asserts the base ProblemDetails shape (`type`, `title`, `status`). `AssertValidationProblem(problem, expectedStatus, label, expectedErrorPath?, expectedErrorContains?)` extends the base assertion to require the `errors` map per `error-shape.md` Inv-2 and optionally checks a specific field path / message substring. Consumed by `TileInventoryValidationTests`; designed to be reused by every future per-endpoint child task under AZ-795.
+- `GrpcTestHelpers.cs` (added cycle 9 — AZ-1075) — gRPC client factory (`CreateClient` over TLS with dev cert trust), request builders, stream collector, and `ExpectInvalidArgumentAsync` assertion helper. Consumed exclusively by `RouteTileDeliveryGrpcTests`.
 
 ## Internal Logic
-- Makes HTTP calls to the API at `API_URL` environment variable (default: `http://api:8080`)
+- Makes HTTP calls to the API at `API_URL` environment variable (default: `https://api:8080` post-AZ-505 TLS)
+- Makes gRPC calls to the same host via `Grpc.Net.Client` + generated stubs from `SatelliteProvider.GrpcContracts`
 - Tests are methods called sequentially from `Program.cs` (not xUnit — plain console app)
 - Poll-based waiting for async operations (region/route completion)
 - Validates response structure, status transitions, file creation
@@ -45,13 +48,13 @@ Console application that runs end-to-end integration tests against a live API in
 - `ProjectReference` to `SatelliteProvider.TestSupport` (added by AZ-491; provides `JwtTokenFactory`. Added by AZ-493; provides `IntegrationTestResetGuard`).
 - Communicates with the API exclusively via HTTP for end-to-end tests; communicates with PostgreSQL directly only via the dedicated DB-reset hook + the existing `MigrationTests` schema assertions.
 - NuGet: `Npgsql` 9.0.2 (Postgres client for DB-reset + MigrationTests), `SixLabors.ImageSharp` 3.1.11 (UAV fixture image generation).
-- ProjectReferences: `SatelliteProvider.Api` (running service for the integration runner), `SatelliteProvider.TestSupport` (canonical `JwtTokenFactory` + `IntegrationTestResetGuard`), `SatelliteProvider.Common` (added by AZ-503 so the `MultiSourceCoexistence_AZ484_Cycle2` seeder can compute `location_hash` via `Uuidv5.Create` instead of duplicating the UUIDv5 algorithm in T-SQL fixtures).
+- ProjectReferences: `SatelliteProvider.Api` (running service for the integration runner), `SatelliteProvider.TestSupport` (canonical `JwtTokenFactory` + `IntegrationTestResetGuard`), `SatelliteProvider.Common` (added by AZ-503 so the `MultiSourceCoexistence_AZ484_Cycle2` seeder can compute `location_hash` via `Uuidv5.Create` instead of duplicating the UUIDv5 algorithm in T-SQL fixtures), `SatelliteProvider.GrpcContracts` (added cycle 9 — generated gRPC client stubs for `RouteTileDeliveryGrpcTests`).
 
 ## Consumers
 - `docker-compose.tests.yml` — runs as a container that depends on the API service
 
 ## Configuration
-- `API_URL` environment variable (set in docker-compose.tests.yml to `http://api:8080`)
+- `API_URL` environment variable (set in docker-compose.tests.yml to `https://api:8080`)
 - `INTEGRATION_TESTS_MODE` — `smoke` or `full` (default `full`). Drives `TestRunMode.Smoke`.
 - `INTEGRATION_KEEP_STATE` — set to `1` or `true` (or pass `--keep-state` to `Program.cs` / `scripts/run-tests.sh`) to skip the AZ-493 DB-reset hook. Useful for debugging a failed run.
 - `ASPNETCORE_ENVIRONMENT=Testing` — guard for the DB-reset hook. The reset refuses to run unless this is set (see Reliability § Test isolation below).