diff --git a/_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md b/_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md index 6c6a38e..ea4dbf5 100644 --- a/_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md +++ b/_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md @@ -7,9 +7,9 @@ - AZ-TBD-c6-freshness-gate (insert hook + sector classification reader) - AZ-TBD-c6-cache-budget-eviction (LRU candidate enumeration + delete coordination) - TBD at decompose time: E-C10 (AZ-252 — manifest + provisioning), E-C11 (AZ-251 — both `TileDownloader` insert and `TileUploader` reader queries), E-C12 (AZ-253 — operator pre-flight tooling) -**Version**: 1.0.0 +**Version**: 1.1.0 **Status**: draft -**Last Updated**: 2026-05-10 +**Last Updated**: 2026-05-12 ## Purpose @@ -81,7 +81,7 @@ class SectorBoundary: ## Invariants -- **I-1 (composite key uniqueness):** `(zoom_level, lat, lon, source)` is the unique key in the `tiles` table. Re-inserting the same key with different content_sha256 raises `TileMetadataError` — no silent overwrite. +- **I-1 (natural-key uniqueness, per-flight separated):** the storage's unique key is `(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, '00000000-0000-0000-0000-000000000000'::uuid))`. The integer slippy-tile coordinates `(tile_x, tile_y)` are derived from the DTO's WGS84 `(lat, lon)` and `zoom_level` via the project's shared Web-Mercator helper at insert time; `lat` / `lon` are persisted advisory-only and are NOT part of the uniqueness predicate. The `flight_id` coalesce term means two `ONBOARD_INGEST` rows for the same cell from different flights coexist (required by the future D-PROJ-2 voting layer), while two `GOOGLEMAPS` rows for the same cell (both `flight_id` = NULL → both coalesce to the zero UUID) cannot. Re-inserting an identical natural key with different `content_sha256_hex` raises `TileMetadataError` — no silent overwrite. - **I-2 (freshness gate at insert):** `insert_metadata` rejects (raises `FreshnessRejectionError`) iff the tile's `(lat, lon)` falls inside an `ACTIVE_CONFLICT` sector AND `capture_timestamp < now() - active_conflict_max_age`. The freshness rules table is configured per-flight (default 6 months for active_conflict; 12 months for stable_rear which downgrades rather than rejects). - **I-3 (downgrade marking):** when a tile in a `STABLE_REAR` sector is older than `stable_rear_max_age`, the row is inserted with `freshness_label=DOWNGRADED` (NOT rejected). `query_by_bbox` returns the downgrade flag intact so consumers (C2 / C3 spoof-rejection) can act on it. - **I-4 (LRU clock):** `record_lru_access` updates `accessed_at = max(current accessed_at, supplied timestamp)`; clock skew never sets `accessed_at` backward. `lru_candidates` returns oldest-first. @@ -111,7 +111,8 @@ Same rules as `tile_store.md` § Versioning Rules. | protocol-conformance-full | A class implementing all 9 methods | `isinstance(impl, TileMetadataStore) == True` | Producer AC-1 | | query-by-bbox-basic | bbox covering 100 inserted tiles at zoom=18 | Returns exactly the 100 tiles; `voting_filter=None` returns all statuses | Smoke | | query-by-bbox-voting-filter | Same with `voting_filter=TRUSTED` | Returns only TRUSTED tiles in bbox | Used by C10 manifest builder | -| insert-duplicate-key | Insert (z=18, lat, lon, src=GOOGLEMAPS) twice with different content_sha256 | First succeeds; second raises `TileMetadataError` | I-1 | +| insert-duplicate-key | Insert (z=18, tile_x, tile_y, tile_size_meters, src=GOOGLEMAPS, flight_id=NULL) twice with different content_sha256 | First succeeds; second raises `TileMetadataError` | I-1 | +| insert-per-flight-coexists | Insert (z=18, tile_x, tile_y, tile_size_meters, src=ONBOARD_INGEST) twice with different `flight_id` values | Both succeed; rows share the same derived `location_hash` cell-bag identifier | I-1 / D-PROJ-2 | | insert-active-conflict-stale | Insert into ACTIVE_CONFLICT sector, capture_timestamp = now - 7 months | `FreshnessRejectionError`; row not committed | I-2 / C6-IT-02 | | insert-stable-rear-stale | Insert into STABLE_REAR sector, capture_timestamp = now - 13 months | Row inserted with `freshness_label=DOWNGRADED` | I-3 | | update-voting-status-forward | PENDING → TRUSTED | Succeeds | I-8 | @@ -130,3 +131,4 @@ Same rules as `tile_store.md` § Versioning Rules. | Version | Date | Change | Author | |---------|------|--------|--------| | 1.0.0 | 2026-05-10 | Initial contract — 9-method Protocol + LRU/disk-budget extensions + freshness gate semantics + composite-key uniqueness invariant. | autodev (decompose Step 2 of AZ-250 / E-C6) | +| 1.1.0 | 2026-05-12 | Non-breaking refinement of Invariant I-1: natural key switched from `(zoom_level, lat, lon, source)` (float-based) to `(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, zero_uuid))` (integer + per-flight separated). Protocol surface unchanged; consumers gain the ability to observe multiple ONBOARD_INGEST rows for the same cell from different flights (required by D-PROJ-2 voting). Driven by `_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md` and the cross-workspace satellite-provider task `AZ-TBD_tile_identity_uuidv5_bulk_list`. | autodev (AZ-304 batch 27 of cycle 1) | diff --git a/_docs/02_tasks/todo/AZ-304_c6_postgres_schema.md b/_docs/02_tasks/todo/AZ-304_c6_postgres_schema.md index d5acf48..7d5dbad 100644 --- a/_docs/02_tasks/todo/AZ-304_c6_postgres_schema.md +++ b/_docs/02_tasks/todo/AZ-304_c6_postgres_schema.md @@ -1,139 +1,160 @@ -# C6 Postgres Schema — Tiles Table + Sector Boundaries + Migration Script +# C6 Postgres Schema — Additive 0002 Migration (Identity + LRU + Freshness Rules) **Task**: AZ-304_c6_postgres_schema -**Name**: C6 Postgres Schema -**Description**: Author the canonical Postgres schema for `c6_tile_cache`: `tiles` (composite key + spatial btree + LRU + voting state + onboard-ingest provenance + per-row JPEG disk size + content-hash chain), `sector_boundaries` (operator-set classification rectangles), `tile_freshness_rules` (per-flight thresholds the freshness gate reads). Ship the initial Alembic migration `_alembic/0001_initial.sql` (forward + reversible down), the schema dataclass mappings used by `PostgresFilesystemStore`, and the per-flight bootstrap migration runner that the composition root invokes at startup. -**Complexity**: 2 points -**Dependencies**: AZ-303_c6_storage_interfaces, AZ-263_initial_structure, AZ-269_config_loader, AZ-266_log_module +**Name**: C6 Postgres Schema — Identity, LRU, and Freshness Rules (additive 0002) +**Description**: Author the additive Alembic migration `0002_c6_tile_identity_and_lru.py` on top of the AZ-263 bootstrap baseline (`db/migrations/versions/0001_initial.py`). This migration is strictly additive per `data_model.md` § 6.1 / § 6.3: every existing AZ-263 column, table, index, and CHECK is preserved unchanged. The migration adds the deterministic-identity columns (`tile_uuid`, `location_hash`), the content-hash chain (`content_sha256`), the LRU + disk-budget bookkeeping columns (`disk_bytes`, `accessed_at`, `uploaded_at`), the per-flight natural-key UNIQUE constraint with COALESCE-zero-uuid semantics, the sector-boundary geometry columns (NULLable additions on the existing `sector_classifications` table), the new `tile_freshness_rules` table seeded with the default thresholds, and the additive widening of the `tiles.freshness_status` CHECK to accept the AZ-303 DTO `FreshnessLabel` vocabulary alongside the existing AZ-263 values. Ships the pinned UUIDv5 namespace module used by both onboard inserts and cross-workspace `satellite-provider` integrations, the migration runner the composition root invokes at startup, the schema-shape diff test, and the AZ-303 contract bump from v1.1.0 → v1.2.0 (adds `location_hash` field to `TileMetadata`). +**Complexity**: 3 points +**Dependencies**: AZ-303_c6_storage_interfaces (DTO contract — bumped here), AZ-263_initial_structure (baseline 0001_initial migration this builds on), AZ-269_config_loader, AZ-266_log_module **Component**: c6_tile_cache (epic AZ-250 / E-C6) **Tracker**: AZ-304 **Epic**: AZ-250 (E-C6) ### Document Dependencies -- `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` — defines the `TileMetadata` / `Bbox` / `SectorBoundary` shapes the schema must persist; defines the LRU + disk-budget contract. -- `_docs/02_document/contracts/c6_tile_cache/tile_store.md` — defines the `content_sha256_hex` invariant the `tiles.content_sha256` column carries. +- `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` — bumped here v1.1.0 → v1.2.0 (adds `location_hash` field). Invariant I-1 already expresses the integer-slippy-tile + COALESCE-zero-uuid natural key this task wires into the DB. +- `_docs/02_document/contracts/c6_tile_cache/tile_store.md` — defines `content_sha256_hex` (str) the `tiles.content_sha256` column carries; unchanged this cycle. - `_docs/02_document/contracts/shared_config/composition_root_protocol.md` — `config.tile_cache.postgres_dsn` field. - `_docs/02_document/contracts/shared_logging/log_record_schema.md` — INFO log shape on migration apply / no-op. -- `_docs/02_document/data_model.md` — system-wide data model the schema must align with (`tiles`, `flight_id` provenance, `quality_metadata` JSONB shape). +- `_docs/02_document/data_model.md` — system-wide data model; § 4.4 (migration baseline ordering), § 6.1 (additive-only by default), § 6.3 (`tiles` canonical columns frozen) are the governing rules this task obeys. +- `db/migrations/versions/0001_initial.py` — AZ-263 baseline this migration extends additively. +- `_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md` — cross-workspace scenario analysis that established the UUIDv5 + `location_hash` design; the cross-workspace counterpart `AZ-TBD_tile_identity_uuidv5_bulk_list` lives in `satellite-provider/_docs/02_tasks/todo/`. ## Problem -Without a frozen Postgres schema: +The AZ-263 bootstrap baseline shipped the `tiles`, `flights`, `sector_classifications`, `manifests`, and `engine_cache_entries` tables, but the canonical AZ-303 `TileMetadataStore` contract (v1.1.0) requires the schema to also carry: -- `PostgresFilesystemStore` has nothing to insert against — `insert_metadata` cannot land any row. -- `query_by_bbox` has no btree to index against — even a 1k-row corpus will table-scan, blowing the C6-PT-01 latency budget. -- The composite-key uniqueness invariant from `tile_metadata_store.md` § I-1 is unenforced — duplicate-key inserts would silently corrupt the cache. -- `lru_candidates` cannot order by `accessed_at` without a column; `total_disk_bytes` cannot SUM without a `disk_bytes` column. -- The freshness gate (separate task) cannot read sector boundaries without a `sector_boundaries` table. -- The C11 `TileUploader` cannot drive its loop off `pending_uploads()` without an `uploaded_at` column. -- Re-running the companion against a stale DB has no migration runner — the operator would have to manually rebuild. +- a deterministic UUIDv5 identifier byte-equal to `satellite-provider`'s tile id, so cross-workspace correlation (D-PROJ-2 ingest, voting, post-landing upload) does not need a separate join table; +- a `location_hash` cell-bag identifier for `WHERE location_hash = ?` cell-bag queries (Scenario 1 UI lookup + Scenario 6 voting query in the 2026-05-12 leftover); +- the per-flight natural-key UNIQUE constraint (`(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, zero_uuid))`) — without it, two `onboard_ingest` rows for the same cell from different flights collapse, destroying the per-flight evidence the future D-PROJ-2 voting layer needs; +- the `content_sha256` content-hash column the AZ-280 atomic-write/sidecar pattern chains into; +- the LRU + disk-budget bookkeeping columns (`disk_bytes`, `accessed_at`, `uploaded_at`) the `TileMetadataStore.record_lru_access` / `total_disk_bytes` / `pending_uploads` methods read and write; +- a freshness-rules table the freshness gate (separate task) reads at insert time; +- geometry on `sector_boundaries` records so the freshness gate can perform tile-in-sector containment checks against `SectorBoundary` DTO instances — AZ-263 shipped `sector_classifications` with a flat `sector_id` text column and no bbox geometry. -This task delivers the on-disk shape that every other C6 task and every consumer depends on. It writes no Python logic beyond the Alembic env + the schema-validation helper — concrete `PostgresFilesystemStore` is a separate task. +Additionally, the AZ-303 DTO vocabulary for `FreshnessLabel` (`fresh`, `stale_active_conflict`, `stale_rear`, `downgraded`) differs from the AZ-263 baseline `freshness_status` CHECK (`fresh`, `stale_warn`, `stale_reject`). Because `data_model.md` § 6.1 forbids removing CHECK enum values without an ADR, this task widens the CHECK to the UNION of both vocabularies (a CHECK loosening, which IS additive); a future cycle may deprecate the legacy values via ADR-gated cleanup. + +This task delivers the strictly-additive `0002` migration that closes those gaps without renaming, retyping, or dropping any AZ-263 column, table, index, or CHECK. ## Outcome -- A migration script at `src/gps_denied_onboard/components/c6_tile_cache/_alembic/versions/0001_initial.py` (Alembic Python migration; the project's existing Alembic env is bootstrap-task-owned per AZ-263). Forward migration `upgrade()` creates three tables and four indexes; reverse `downgrade()` drops them in reverse order. The migration is idempotent against a clean DB and is rejected (Alembic's standard behaviour) if applied to a DB at a later revision. -- A migration runner `apply_migrations(config) -> MigrationResult` at `src/gps_denied_onboard/components/c6_tile_cache/migrations.py` invoked by the composition root at startup AFTER config load and BEFORE `PostgresFilesystemStore` construction. Returns `MigrationResult(applied: list[str], current_revision: str, no_op: bool)`. Logs INFO on every applied revision; logs INFO with `no_op=True` when the DB is already at head. -- Three tables exist after `upgrade()`: - 1. `tiles` — see Schema below. - 2. `sector_boundaries` — see Schema below. - 3. `tile_freshness_rules` — see Schema below. -- Four indexes exist after `upgrade()`: - - `tiles_pkey` — `PRIMARY KEY (zoom_level, lat, lon, source)` (composite, enforces I-1 from the metadata-store contract). - - `idx_tiles_spatial` — btree over `(zoom_level, lat, lon)` for `query_by_bbox`. - - `idx_tiles_pending_upload` — partial btree over `(uploaded_at) WHERE source = 'onboard_ingest' AND uploaded_at IS NULL` for `pending_uploads`. - - `idx_tiles_lru` — btree over `accessed_at` for `lru_candidates`. -- `quality_metadata` is JSONB (NOT a separate table) — matches description.md § 2 and `data_model.md`. The JSONB shape is validated at the application layer (the `TileQualityMetadata` dataclass). -- A schema fixture `tests/fixtures/c6_postgres_schema_v1.sql` is the human-readable expected DDL used by the schema-shape test (AC-3). +- An Alembic migration script at `db/migrations/versions/0002_c6_tile_identity_and_lru.py` (forward `upgrade()` is purely additive; reverse `downgrade()` drops the additions and restores the original AZ-263 `freshness_status` CHECK). The migration is idempotent against a DB at AZ-263 head; Alembic rejects double-application via the standard `alembic_version` row. +- The migration runner `apply_migrations(config) -> MigrationResult` at `src/gps_denied_onboard/components/c6_tile_cache/migrations.py`, invoked by the composition root at startup AFTER config load and BEFORE `PostgresFilesystemStore` construction. Returns `MigrationResult(applied: list[str], current_revision: str, no_op: bool)`. Logs INFO on every applied revision; logs INFO with `no_op=True` when the DB is already at head. +- The pinned UUIDv5 namespace module at `src/gps_denied_onboard/components/c6_tile_cache/_uuid_namespace.py` exporting `TILE_NAMESPACE_UUID = UUID("5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0")`, `derive_tile_id(zoom_level, tile_x, tile_y, source, flight_id) -> UUID`, and `derive_location_hash(zoom_level, tile_x, tile_y) -> UUID`. The namespace value is cross-repo coordinated with `satellite-provider/SatelliteProvider.Common/Utils/Uuidv5.cs` per `AZ-TBD_tile_identity_uuidv5_bulk_list`; the same `uuidv5(NAMESPACE, name)` MUST produce byte-identical output on both sides. +- The psycopg_pool connection helper at `src/gps_denied_onboard/components/c6_tile_cache/connection.py` (`psycopg_pool(config) -> psycopg_pool.ConnectionPool`), used by both this task's runner and the future `PostgresFilesystemStore`. +- After `apply_migrations(config)` on an AZ-263-baselined DB: + - The `tiles` table has six additional columns (`tile_uuid`, `location_hash`, `content_sha256`, `disk_bytes`, `accessed_at`, `uploaded_at`); the AZ-263 columns are unchanged. + - The `tiles` table has a new UNIQUE index `idx_tiles_natural_key` over the COALESCE-zero-uuid natural key; the AZ-263 indices are unchanged. + - The `tiles` table has four new indices (`idx_tiles_location_hash`, `idx_tiles_tile_uuid` from the UNIQUE, `idx_tiles_accessed_at`, `idx_tiles_pending_upload` partial, `idx_tiles_flight_captured` partial — see Schema Additions). + - The `tiles.freshness_status` CHECK is widened to the UNION vocabulary; AZ-263 rows (none exist on greenfield apply) would continue to validate. + - The `sector_classifications` table has four additional NULLable bbox columns; the AZ-263 columns are unchanged. + - A new `tile_freshness_rules` table exists, seeded with the two default rows. +- The AZ-303 contract `tile_metadata_store.md` is bumped v1.1.0 → v1.2.0 with the `location_hash: UUID | None` field added to the documented `TileMetadata` shape (non-breaking minor — Optional default `None`, populated by `PostgresFilesystemStore.insert_metadata` from `_uuid_namespace.derive_location_hash` when not supplied). +- The DTO `TileMetadata` in `src/gps_denied_onboard/components/c6_tile_cache/_types.py` gains the same `location_hash: UUID | None = None` field (positional last, default value preserves existing constructor call sites). +- A schema fixture `tests/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test. ## Scope ### Included -- The Alembic migration `0001_initial.py` covering three tables + four indexes. -- A `MigrationResult` dataclass `@dataclass(frozen=True)`. -- The `apply_migrations(config)` runner using the project-pinned Alembic version (already in the bootstrap dependency set per AZ-263). -- The schema-shape test (`tests/unit/c6_tile_cache/test_postgres_schema.py`) that introspects a freshly-migrated test DB and asserts the documented column types, nullable flags, default values, primary keys, and indexes (Postgres `information_schema` queries; no FAISS / no Python logic). -- The `_alembic/env.py` bootstrap (registers the migration directory with the existing project Alembic env; no NEW alembic config). -- The schema fixture `tests/fixtures/c6_postgres_schema_v1.sql` — copy-pastable DDL the test diffs against. -- Postgres connection helper `c6_tile_cache.connection.psycopg_pool(config) -> psycopg_pool.ConnectionPool` (used by both this task's runner and the future `PostgresFilesystemStore`); the helper is a thin wrapper over `psycopg_pool.ConnectionPool` that takes the DSN from config. +- The Alembic migration `0002_c6_tile_identity_and_lru.py` (additive only; the migration body uses `op.add_column`, `op.create_unique_constraint` / `op.create_index` for the UNIQUE expression index, `op.create_index` for new btree indices, `op.create_table` for `tile_freshness_rules`, `op.bulk_insert` for the seed rows, and a single `op.drop_constraint` + `op.create_check_constraint` pair to widen `ck_tiles_freshness_status`). +- A `MigrationResult` dataclass `@dataclass(frozen=True)` at `c6_tile_cache.migrations`. +- The `apply_migrations(config) -> MigrationResult` runner using the existing project-pinned Alembic env at `db/migrations/` (no new alembic.ini, no new env.py — AZ-263 bootstrap owns those; this task only wires `target_metadata` into `db/migrations/env.py` so future autogenerate diffs work). +- The pinned UUIDv5 namespace module `_uuid_namespace.py` with `TILE_NAMESPACE_UUID`, `derive_tile_id`, `derive_location_hash`. No Postgres dependency; pure stdlib `uuid.uuid5`. +- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/fixtures/c6_postgres_schema_v2.sql`. Test uses `testcontainers`-managed Postgres 16; no Python logic beyond `information_schema` queries. +- The UUIDv5 determinism test `tests/unit/c6_tile_cache/test_uuid_namespace.py` that locks `TILE_NAMESPACE_UUID` and verifies ≥5 fixed `(z, x, y, source, flight_id)` input vectors produce the documented UUIDv5 outputs. These vectors are the cross-repo coordination evidence — the corresponding `satellite-provider` test MUST produce byte-identical UUIDs. +- The connection helper `connection.psycopg_pool(config) -> psycopg_pool.ConnectionPool`. +- Wiring of `db/migrations/env.py` `target_metadata` to a `c6_tile_cache.metadata` SQLAlchemy `MetaData` object that reflects both AZ-263 and AZ-304 schema (so future autogenerate diffs are mechanically comparable). +- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against. +- DTO extension in `_types.py`: `TileMetadata.location_hash: UUID | None = None` (positional last, default `None`). +- Contract bump `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` v1.1.0 → v1.2.0 with a Change Log entry. ### Excluded -- Concrete `PostgresFilesystemStore` (insert / query / mark methods) — separate task (`c6_postgres_filesystem_store`). -- The freshness gate logic that reads `sector_boundaries` / `tile_freshness_rules` — separate task (`c6_freshness_gate`). -- The LRU eviction policy that reads `accessed_at` — separate task (`c6_cache_budget_eviction`). +- Concrete `PostgresFilesystemStore` (insert / query / mark methods) — separate task (`c6_postgres_filesystem_store`). That task is responsible for: (a) computing `tile_x` / `tile_y` from the `TileId` WGS84 coordinates via the project's shared Web-Mercator helper, (b) deriving `tile_uuid` / `location_hash` via `_uuid_namespace.derive_*`, (c) mapping DTO field names to AZ-263 column names (`quality_metadata` → `tile_quality_metadata`, `freshness_label` → `freshness_status`, `(lat, lon)` → `(latitude, longitude)`), and (d) validating cross-side UUIDv5 parity with `satellite-provider`. +- The freshness gate logic reading `sector_classifications` + `tile_freshness_rules` — separate task (`c6_freshness_gate`). +- The LRU eviction policy reading `accessed_at` — separate task (`c6_cache_budget_eviction`). - FAISS index file format — separate task (`c6_faiss_descriptor_index`). -- Sector-boundary CRUD (operator-side INSERT/UPDATE) — owned by C12. +- Sector-boundary CRUD (operator-side INSERT/UPDATE of geometry columns) — owned by C12; this task only adds the columns as NULLable. - Per-flight DB lifecycle (drop-and-rebuild between flights, freshness-rules reload) — owned by the composition root's startup orchestration; this task only applies migrations idempotently. -- A second migration revision — every future schema change is a NEW migration file; this task only ships `0001_initial.py`. -- Postgres tuning (work_mem, shared_buffers) — handled by the deployment / Dockerfile (E-DEPLOY); the schema is portable across reasonable Postgres 16 configurations. +- A third migration revision — every future schema change is a NEW migration file; this task only ships `0002`. +- Postgres tuning (work_mem, shared_buffers) — handled by deployment / Dockerfile (E-DEPLOY); the schema is portable across reasonable Postgres 16 configurations. - Postgres-version migration (16 → 17) — out of scope this cycle; the schema MUST work on 16.x. +- Backfill of the legacy `freshness_status` enum values (`stale_warn`, `stale_reject`) to the new vocabulary — deferred until an ADR explicitly decides the deprecation path; this task widens the CHECK without committing to a value-mapping rewrite. +- Renaming any AZ-263 column or table — forbidden per `coderule.mdc` and `data_model.md` § 6.2; the DTO-to-column mapping lives in `PostgresFilesystemStore`. -## Schema +## Schema Additions (0002 on top of AZ-263 0001_initial) -### Table: `tiles` +All additions below are strictly additive. The migration MUST NOT drop, rename, or retype any AZ-263 column, table, index, or CHECK; the only constraint change is a CHECK widening (loosening) for `tiles.freshness_status`, which is additive per `data_model.md` § 6.1. + +### `tiles` — additive columns | Column | Type | Nullable | Default | Notes | |--------|------|----------|---------|-------| -| `zoom_level` | `INTEGER` | NO | — | composite PK | -| `lat` | `DOUBLE PRECISION` | NO | — | composite PK; centre latitude | -| `lon` | `DOUBLE PRECISION` | NO | — | composite PK; centre longitude | -| `source` | `TEXT` | NO | — | composite PK; CHECK `source IN ('googlemaps', 'onboard_ingest')` | -| `tile_size_meters` | `DOUBLE PRECISION` | NO | — | | -| `tile_size_pixels` | `INTEGER` | NO | — | | -| `capture_timestamp` | `TIMESTAMPTZ` | NO | — | UTC | -| `content_sha256` | `TEXT` | NO | — | 64 hex chars; matches the JPEG body hash from AZ-280's atomic-write/sidecar pattern | -| `freshness_label` | `TEXT` | NO | `'fresh'` | CHECK `freshness_label IN ('fresh', 'stale_active_conflict', 'stale_rear', 'downgraded')` | -| `flight_id` | `UUID` | YES | NULL | non-NULL when `source = 'onboard_ingest'` (CHECK enforces) | -| `companion_id` | `TEXT` | YES | NULL | non-NULL when `source = 'onboard_ingest'` (CHECK enforces) | -| `quality_metadata` | `JSONB` | YES | NULL | non-NULL when `source = 'onboard_ingest'` (CHECK enforces); shape validated app-side | -| `voting_status` | `TEXT` | NO | `'trusted'` for googlemaps; `'pending'` for onboard_ingest | CHECK `voting_status IN ('pending', 'trusted', 'rejected')`; default per-source via trigger | -| `disk_bytes` | `BIGINT` | NO | — | byte size of the on-disk JPEG; populated by `write_tile` | -| `accessed_at` | `TIMESTAMPTZ` | NO | `now()` | LRU clock — updated by `record_lru_access` | -| `uploaded_at` | `TIMESTAMPTZ` | YES | NULL | set by `mark_uploaded`; remains NULL until C11 `TileUploader` confirms post-flight upload | -| `created_at` | `TIMESTAMPTZ` | NO | `now()` | row-create timestamp; immutable | +| `tile_uuid` | `UUID` | NO | — | Deterministic `uuidv5(TILE_NAMESPACE_UUID, "{zoom_level}/{tile_x}/{tile_y}/{source}/{flight_id_or_zero_uuid}")`. Cross-repo correlation key with `satellite-provider`. Populated at INSERT by `PostgresFilesystemStore`. UNIQUE. | +| `location_hash` | `UUID` | NO | — | `uuidv5(TILE_NAMESPACE_UUID, "{zoom_level}/{tile_x}/{tile_y}")`. Identifies a Web-Mercator cell across sources / flights. Populated at INSERT. | +| `content_sha256` | `TEXT` | NO | — | 64 hex chars; matches `tile_store.md` `content_sha256_hex` invariant. CHECK `length(content_sha256) = 64`. | +| `disk_bytes` | `BIGINT` | NO | — | On-disk JPEG body size; populated by `write_tile`. CHECK `disk_bytes >= 0`. | +| `accessed_at` | `TIMESTAMPTZ` | NO | `now()` | LRU clock; updated by `record_lru_access`. | +| `uploaded_at` | `TIMESTAMPTZ` | YES | NULL | Set by `mark_uploaded`; NULL until C11 `TileUploader` confirms post-flight upload. | -Constraints: +**Empty-DB assumption**: at 0002 apply time the `tiles` table is empty (greenfield); the four NOT-NULL columns without server defaults are added with `nullable=False` directly. On a hypothetical non-empty DB the migration would need a 2-phase split (add nullable → backfill → alter NOT NULL); this is documented but explicitly out of scope for this cycle. -- `PRIMARY KEY (zoom_level, lat, lon, source)` -- `CHECK (zoom_level BETWEEN 0 AND 21)` -- `CHECK (source IN ('googlemaps', 'onboard_ingest'))` -- `CHECK (freshness_label IN ('fresh', 'stale_active_conflict', 'stale_rear', 'downgraded'))` -- `CHECK (voting_status IN ('pending', 'trusted', 'rejected'))` -- `CHECK (disk_bytes >= 0)` -- `CHECK (length(content_sha256) = 64)` -- `CHECK ((source = 'onboard_ingest' AND flight_id IS NOT NULL AND companion_id IS NOT NULL AND quality_metadata IS NOT NULL) OR (source = 'googlemaps'))` +### `tiles` — natural-key UNIQUE (additive index) -### Table: `sector_boundaries` +A UNIQUE expression index `idx_tiles_natural_key` over + +``` +(zoom_level, tile_x, tile_y, tile_size_meters, source, + COALESCE(flight_id, '00000000-0000-0000-0000-000000000000'::uuid)) +``` + +Two `onboard_ingest` rows for the same `(z, tile_x, tile_y, tile_size_meters)` from different `flight_id` values coexist (required by the future D-PROJ-2 voting layer). Two `googlemaps` rows for the same cell (both `flight_id = NULL` → both coalesce to the zero UUID) cannot coexist. The index is implemented via `op.create_index(..., unique=True, postgresql_using='btree')` with the expression list literal. + +### `tiles` — CHECK widening (additive loosen) + +The existing `ck_tiles_freshness_status` CHECK is dropped and recreated with the UNION vocabulary: + +```sql +freshness_status IN ( + 'fresh', 'stale_warn', 'stale_reject', -- AZ-263 vocabulary (legacy) + 'stale_active_conflict', 'stale_rear', 'downgraded' -- AZ-303 FreshnessLabel vocabulary +) +``` + +A CHECK widening (more values accepted) is an additive loosening; any row that satisfied the prior CHECK still satisfies the new one. The legacy values are retained until an ADR-gated future cycle deprecates them. The constraint name stays `ck_tiles_freshness_status` to keep the AZ-263 fixture diffable. + +### `tiles` — new indices (additive) + +| Index | Definition | Purpose | +|-------|-----------|---------| +| `idx_tiles_natural_key` | UNIQUE btree over the COALESCE natural-key expression (above) | I-1 of `tile_metadata_store.md` | +| `idx_tiles_location_hash` | btree over `(location_hash)` | Scenario 1 UI cell-bag lookup, Scenario 6 voting query | +| `idx_tiles_accessed_at` | btree over `(accessed_at)` | `lru_candidates` ordering | +| `idx_tiles_pending_upload` | partial btree over `(uploaded_at)` WHERE `source = 'onboard_ingest' AND uploaded_at IS NULL` | `pending_uploads` | +| `idx_tiles_flight_captured` | partial btree over `(flight_id, capture_timestamp)` WHERE `flight_id IS NOT NULL` | Scenario 8 per-flight inspection | + +The AZ-263 indices (`ix_tiles_zxy`, `ix_tiles_lat_lon`, `ix_tiles_voting_status_onboard`, `ix_tiles_flight_id`, `ix_tiles_created_at`) are unchanged. + +### `sector_classifications` — additive columns (geometry, NULLable) | Column | Type | Nullable | Default | Notes | |--------|------|----------|---------|-------| -| `boundary_id` | `UUID` | NO | `gen_random_uuid()` | PK | -| `min_lat` | `DOUBLE PRECISION` | NO | — | | -| `min_lon` | `DOUBLE PRECISION` | NO | — | | -| `max_lat` | `DOUBLE PRECISION` | NO | — | | -| `max_lon` | `DOUBLE PRECISION` | NO | — | | -| `classification` | `TEXT` | NO | — | CHECK `classification IN ('active_conflict', 'stable_rear')` | -| `set_by_operator` | `TEXT` | NO | — | operator handle for audit | -| `set_at` | `TIMESTAMPTZ` | NO | `now()` | | +| `min_lat` | `DOUBLE PRECISION` | YES | NULL | bbox south edge — populated pre-flight by C12 | +| `min_lon` | `DOUBLE PRECISION` | YES | NULL | bbox west edge | +| `max_lat` | `DOUBLE PRECISION` | YES | NULL | bbox north edge | +| `max_lon` | `DOUBLE PRECISION` | YES | NULL | bbox east edge | -Constraints: +All NULLable because existing AZ-263 rows (if any) lack geometry; C12 populates pre-flight before the freshness gate reads. The AZ-263 columns (`sector_id`, `classification`, `freshness_threshold_days`) are unchanged. No CHECK on min ≤ max yet — that arrives when C12 enforces population (would be a tightening, ADR-gated). -- `PRIMARY KEY (boundary_id)` -- `CHECK (min_lat <= max_lat AND min_lon <= max_lon)` -- `CHECK (classification IN ('active_conflict', 'stable_rear'))` +The `SectorBoundary` DTO from `_types.py` maps `bbox.min_lat → min_lat`, etc.; `classification` maps directly. The DTO field `sector_id` is not present in the DTO today because the in-memory `SectorBoundary` does not need it; the impl task resolves DB row ↔ DTO via the `sector_id` text PK component. -NO spatial index this cycle — the row count is small (≤ a few hundred per flight), and the freshness gate reads them all into memory at flight start. - -### Table: `tile_freshness_rules` +### New table: `tile_freshness_rules` | Column | Type | Nullable | Default | Notes | |--------|------|----------|---------|-------| -| `classification` | `TEXT` | NO | — | PK; matches `sector_boundaries.classification` | -| `max_age_seconds` | `BIGINT` | NO | — | seconds; per `STABLE_REAR` is the downgrade threshold; per `ACTIVE_CONFLICT` is the rejection threshold | -| `action` | `TEXT` | NO | — | CHECK `action IN ('reject', 'downgrade')` | +| `classification` | `TEXT` | NO | — | PK; matches `sector_classifications.classification` | +| `max_age_seconds` | `BIGINT` | NO | — | Per-classification threshold | +| `action` | `TEXT` | NO | — | CHECK `action IN ('reject','downgrade')` | | `set_at` | `TIMESTAMPTZ` | NO | `now()` | | Constraints: @@ -142,130 +163,182 @@ Constraints: - `CHECK (action IN ('reject', 'downgrade'))` - `CHECK (max_age_seconds > 0)` -Default rows seeded by the migration: -- `('active_conflict', 6 * 30 * 86400, 'reject')` — 6 months, AC-8.2. -- `('stable_rear', 12 * 30 * 86400, 'downgrade')` — 12 months, AC-8.2. +Default rows seeded by the migration (AC-7): + +- `('active_conflict', 15552000, 'reject')` — 6 × 30 × 86400 seconds = 6 months +- `('stable_rear', 31104000, 'downgrade')` — 12 × 30 × 86400 seconds = 12 months + +### AZ-303 contract bump v1.1.0 → v1.2.0: `TileMetadata.location_hash` + +- DTO change in `_types.py`: add `location_hash: UUID | None = None` to `TileMetadata` (positional last, default `None`). `PostgresFilesystemStore.insert_metadata` computes the value via `_uuid_namespace.derive_location_hash(...)` when `None`; uses the supplied value when present. +- Contract change in `tile_metadata_store.md`: bump version to v1.2.0; add `location_hash: UUID | None` row to the DTO field table; add a "v1.2.0" Change Log row referencing this task and the leftover. +- Non-breaking: existing constructors and `insinstance(impl, TileMetadataStore)` checks continue to work. ## Acceptance Criteria -**AC-1: Migration is idempotent against a clean DB** -Given a fresh Postgres 16 database with no `alembic_version` row +**AC-1: Migration is idempotent against an AZ-263-baselined DB** +Given a Postgres 16 database at AZ-263 head (`alembic_version = '0001_initial'`) When `apply_migrations(config)` runs -Then all three tables and all four indexes exist; the `alembic_version` row carries `0001_initial`; `MigrationResult.applied == ['0001_initial']`; `MigrationResult.no_op == False` +Then the additive columns / indices / table / CHECK widening exist; the `alembic_version` row carries `0002_c6_tile_identity_and_lru`; `MigrationResult.applied == ['0002_c6_tile_identity_and_lru']`; `MigrationResult.no_op == False`; AZ-263 columns / indices / CHECKs are byte-identical to their pre-migration state. **AC-2: Migration is no-op when at head** -Given a Postgres DB already at `0001_initial` +Given a Postgres DB already at `0002_c6_tile_identity_and_lru` When `apply_migrations(config)` runs again -Then `MigrationResult.applied == []`; `MigrationResult.no_op == True`; no DDL is emitted (verifiable via `pg_stat_user_tables` row counts unchanged) +Then `MigrationResult.applied == []`; `MigrationResult.no_op == True`; no DDL is emitted (verifiable via `pg_stat_user_tables` row counts unchanged). **AC-3: Schema shape matches the documented DDL** -Given a freshly-migrated DB -When the schema-shape test introspects `information_schema.columns` and `pg_indexes` -Then every column matches the `Schema` section above (name, data type, nullability, default expression); every index matches (name, columns, partial-index predicate where applicable); every CHECK constraint exists with the documented expression +Given a DB upgraded through 0001 + 0002 +When the schema-shape diff test introspects `information_schema.columns` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` row contents +Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/fixtures/c6_postgres_schema_v2.sql` is empty. -**AC-4: Composite primary key enforces uniqueness** -Given an empty `tiles` table -When two INSERTs with the same `(zoom_level, lat, lon, source)` are attempted with different `content_sha256` values -Then the second INSERT raises a Postgres unique-constraint violation; the first row is unaffected; the application layer translates this to `TileMetadataError` (in the `PostgresFilesystemStore` task — this task surfaces only the raw Postgres error) +**AC-4: Natural-key uniqueness enforces per-flight separation** +Given a `tiles` table after 0002 +When two INSERTs with the same `(zoom_level, tile_x, tile_y, tile_size_meters, source='onboard_ingest')` and **different** `flight_id` values are attempted +Then both INSERTs succeed; the resulting rows have different `tile_uuid` values (different UUIDv5 inputs) and the same `location_hash` (UUIDv5 inputs differ only in `flight_id`/`source`). -**AC-5: CHECK constraint enforces source-aware mandatory fields** -Given an `onboard_ingest` row with `flight_id = NULL` -When the INSERT is attempted -Then the row is rejected by the CHECK constraint at the DB layer +**AC-4b: Natural-key uniqueness rejects duplicate flight inserts** +Given a `tiles` table after 0002 +When two INSERTs with the same `(zoom_level, tile_x, tile_y, tile_size_meters, source, flight_id)` (or both `flight_id = NULL` for `googlemaps`) are attempted with different `content_sha256` values +Then the second INSERT raises `psycopg.errors.UniqueViolation` (`idx_tiles_natural_key`); the first row is unaffected. + +**AC-5: AZ-263 CHECK constraints survive widening** +Given the post-0002 `tiles` table +When `INSERT INTO tiles (...freshness_status...) VALUES ('fresh')` runs (and separately for `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`) +Then all six values are accepted; an `INSERT ... VALUES ('bogus')` is rejected by the widened `ck_tiles_freshness_status` CHECK. **AC-6: Down migration reverses cleanly** -Given a DB at `0001_initial` +Given a DB at `0002_c6_tile_identity_and_lru` When `alembic downgrade -1` runs (operator-only command; not exercised by the runtime) -Then all three tables and all four indexes are dropped; the DB returns to the empty pre-migration state; subsequent `upgrade` re-applies cleanly +Then the additive columns / indices / `tile_freshness_rules` table are dropped; the `ck_tiles_freshness_status` CHECK is restored to the AZ-263 vocabulary; the DB returns to byte-identical AZ-263 state; subsequent `upgrade` re-applies cleanly. **AC-7: Default freshness rules are seeded** -Given a freshly-migrated DB +Given a DB upgraded through 0002 When the schema-shape test queries `tile_freshness_rules` -Then exactly two rows exist: `('active_conflict', 15552000, 'reject')` and `('stable_rear', 31104000, 'downgrade')` +Then exactly two rows exist: `('active_conflict', 15552000, 'reject')` and `('stable_rear', 31104000, 'downgrade')`. **AC-8: Migration runner logs INFO on apply and no-op** -Given a clean DB +Given a DB at AZ-263 head When `apply_migrations` runs and then runs again -Then the first call emits an INFO log with `kind="c6.migration.applied"` carrying `revisions=['0001_initial']`; the second call emits an INFO log with `kind="c6.migration.no_op"` +Then the first call emits an INFO log with `kind="c6.migration.applied"` carrying `revisions=['0002_c6_tile_identity_and_lru']`; the second call emits an INFO log with `kind="c6.migration.no_op"`. Both calls also log the resolved `TILE_NAMESPACE_UUID` value once for post-mortem drift detection. -**AC-9: Quality metadata JSONB is validated app-side, NOT DB-side** -Given an `onboard_ingest` row with `quality_metadata = '{}'::jsonb` (empty JSONB but non-NULL) -When the INSERT runs at the DB layer -Then the INSERT succeeds (DB CHECK does not validate the JSONB shape); the application-layer validation (in `PostgresFilesystemStore`'s `insert_metadata`) is what would reject it. This task documents the boundary: the schema enforces presence/non-NULL only; shape is the impl task's responsibility. +**AC-9: AZ-263 columns are byte-identical after upgrade** +Given a DB at AZ-263 head + a recorded snapshot of `information_schema.columns` for `tiles`, `flights`, `sector_classifications`, `manifests`, `engine_cache_entries` +When `apply_migrations(config)` runs +Then for every AZ-263 column the snapshot and the post-0002 row are byte-identical (column name, data type, nullability, default expression). New columns appear only as additions. + +**AC-10: UUIDv5 derivation is deterministic and cross-repo coordinated** +Given the pinned `TILE_NAMESPACE_UUID = 5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` +When `derive_tile_id` and `derive_location_hash` are called for the documented fixed test vectors (≥5 `(zoom_level, tile_x, tile_y, source, flight_id)` tuples in `tests/unit/c6_tile_cache/test_uuid_namespace.py`) +Then each call returns the documented UUID byte-for-byte; the same Python expression run twice produces identical output; the locked test vectors are the cross-repo coordination evidence used to verify `satellite-provider`'s C# implementation produces the same UUIDs. + +**AC-11: `location_hash` is invariant across `source` and `flight_id`** +Given three rows for the same `(zoom_level, tile_x, tile_y)` — one `source='googlemaps'` and two `source='onboard_ingest'` from different `flight_id` values +When the test queries `SELECT DISTINCT location_hash FROM tiles WHERE zoom_level=? AND tile_x=? AND tile_y=?` +Then exactly ONE `location_hash` value is returned (the three rows share the same cell-bag identifier). + +**AC-12: `TileMetadata.location_hash` default is `None` (non-breaking DTO bump)** +Given the v1.2.0 `TileMetadata` DTO +When existing AZ-303-style positional constructors run (without supplying `location_hash`) +Then construction succeeds with `location_hash = None`; when constructors supply an explicit `UUID`, the value is preserved on the frozen instance. ## Non-Functional Requirements **Performance** -- Migration apply ≤ 5 s on an empty Postgres 16 database. Schema is small (3 tables, 4 indexes) and the runner uses a single connection. +- Migration apply ≤ 5 s on an AZ-263-baselined Postgres 16 database with empty `tiles`. The migration body is bounded by index creation on an empty table, four `ALTER TABLE … ADD COLUMN`s, one `op.create_table`, two `op.bulk_insert` rows, and one CHECK widening — all O(1) in row count. - `apply_migrations` no-op call (DB at head) ≤ 100 ms. -- Idempotency: re-running `apply_migrations` is bound only by the head-detection query (single SELECT against `alembic_version`). +- The new UNIQUE expression index `idx_tiles_natural_key` does not regress AZ-263 query plans on the existing `ix_tiles_zxy` btree; both coexist and the planner selects based on predicate shape. **Compatibility** -- Postgres 16.x (matches `satellite-provider`'s pin per description.md § 5). -- `psycopg_pool` 3.x — already pinned by AZ-263 bootstrap. -- Alembic 1.13+ — already pinned by AZ-263 bootstrap. +- Postgres 16.x. +- `psycopg_pool` 3.x — pinned by AZ-263; this task adds no new third-party dependencies. +- Alembic 1.13+ — pinned by AZ-263. +- Cross-workspace UUIDv5 namespace: `TILE_NAMESPACE_UUID` MUST be byte-identical to the satellite-provider C# constant; any change requires a coordinated cross-repo release. **Reliability** -- The migration is wrapped in a single transaction (Alembic's default for non-DDL-batched migrations on Postgres). A crash mid-migration leaves the DB at the prior revision. -- The runner catches `psycopg.errors.SerializationFailure` and retries once with exponential backoff; after the second failure, raises a `MigrationError` (NEW error type defined here, NOT in `TileCacheError` — migrations are bootstrap-time, not runtime). +- The migration is wrapped in a single Alembic transaction (Postgres default). A crash mid-migration leaves the DB at the prior revision (`0001_initial`). +- The runner catches `psycopg.errors.SerializationFailure` and retries once with exponential backoff; after the second failure, raises `MigrationError` (defined in `c6_tile_cache.migrations`, NOT in `TileCacheError` — migrations run before any runtime error consumer is constructed). ## Unit Tests | AC Ref | What to Test | Required Outcome | |--------|-------------|-----------------| -| AC-1 | `apply_migrations` against fresh testcontainer DB | Three tables + four indexes exist; alembic_version='0001_initial'; result.applied=['0001_initial'] | -| AC-2 | `apply_migrations` against already-migrated DB | result.applied=[]; result.no_op=True; no DDL emitted | -| AC-3 | Introspect information_schema after migration; diff against `tests/fixtures/c6_postgres_schema_v1.sql` | Zero diff; every column / index / CHECK matches | -| AC-4 | Two INSERTs with same `(zoom, lat, lon, source)` | Second INSERT raises `psycopg.errors.UniqueViolation` | -| AC-5 | INSERT `onboard_ingest` row with `flight_id=NULL` | Raises `psycopg.errors.CheckViolation` | -| AC-6 | `alembic downgrade -1` then `upgrade` | DB returns to empty state then re-applies cleanly | -| AC-7 | SELECT `tile_freshness_rules` after migration | Exactly 2 rows with documented values | -| AC-8 | Capture log records during migration apply + no-op | Two INFO records with `kind="c6.migration.applied"` and `kind="c6.migration.no_op"` | -| AC-9 | INSERT row with `quality_metadata='{}'::jsonb` | DB-layer accepts; documented as app-side responsibility | -| NFR-perf-apply | Migration apply on empty 16.x | Wall ≤ 5 s | +| AC-1 | `apply_migrations` against fresh `testcontainer` DB previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. | +| AC-2 | `apply_migrations` against DB already at 0002 head | `result.applied=[]`; `result.no_op=True`; no DDL emitted. | +| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/fixtures/c6_postgres_schema_v2.sql` | Zero diff. | +| AC-4 | Two `onboard_ingest` INSERTs with same `(z, tile_x, tile_y, tile_size_meters)` and different `flight_id` | Both succeed; same `location_hash`; different `tile_uuid`. | +| AC-4b | Two INSERTs with identical natural-key tuple (same `flight_id` or both NULL for `googlemaps`) | Second INSERT raises `psycopg.errors.UniqueViolation`. | +| AC-5 | INSERT one row each for `'fresh'`, `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`; INSERT one row with `'bogus'` | First six succeed; last raises `psycopg.errors.CheckViolation`. | +| AC-6 | `alembic downgrade -1` then `upgrade head` | DB returns to AZ-263-byte-identical state; subsequent upgrade re-applies cleanly. | +| AC-7 | SELECT `tile_freshness_rules` after migration | Exactly 2 rows with documented values. | +| AC-8 | Capture log records during migration apply + no-op | INFO records with `kind="c6.migration.applied"` and `kind="c6.migration.no_op"`; namespace UUID emitted on apply. | +| AC-9 | Snapshot AZ-263 `information_schema.columns` before 0002; compare after | Snapshot rows byte-identical post-migration; new column rows are additions only. | +| AC-10 | `derive_tile_id` / `derive_location_hash` for 5 fixed input vectors | Outputs match the documented UUIDs byte-for-byte; idempotent on second call. | +| AC-11 | INSERT three rows sharing `(z, tile_x, tile_y)` from different `(source, flight_id)` | Exactly one distinct `location_hash` value. | +| AC-12 | Construct `TileMetadata` without `location_hash` and with explicit `location_hash=uuid4()` | First yields `location_hash=None`; second preserves the supplied value; both instances are frozen. | +| NFR-perf-apply | Migration apply on AZ-263-baselined empty `tiles` | Wall ≤ 5 s | | NFR-perf-noop | `apply_migrations` no-op timing | Wall ≤ 100 ms | -| NFR-reliability-retry | Inject `SerializationFailure` once, then succeed | Migration succeeds on retry; on second failure raises `MigrationError` | +| NFR-reliability-retry | Inject `SerializationFailure` once, then succeed | Migration succeeds on retry; on second failure raises `MigrationError`. | ## Constraints - Postgres 16.x ONLY this cycle; no SQLite / no MySQL fallback. -- Alembic + `psycopg_pool` are already pinned by AZ-263; this task does NOT introduce new third-party dependencies. -- The migration MUST be reversible (`downgrade` drops cleanly) — operator post-flight tooling depends on it for "drop-and-rebuild" flows. -- The schema MUST mirror `data_model.md` exactly (especially the `quality_metadata` JSONB shape and the `voting_status` enum). Any deviation requires a `data_model.md` update first; this task does NOT silently extend the data model. -- The `quality_metadata` JSONB shape is NOT validated at the DB layer (no domain types, no CHECK on JSON structure). That validation is `PostgresFilesystemStore.insert_metadata` (separate task) — documented in AC-9. -- `gen_random_uuid()` requires the `pgcrypto` extension; the migration's `upgrade()` runs `CREATE EXTENSION IF NOT EXISTS pgcrypto` as its first statement. +- Alembic + `psycopg_pool` are pinned by AZ-263; this task does NOT introduce new third-party dependencies. +- The migration MUST be reversible (`downgrade` drops the additions cleanly and restores the AZ-263 CHECK) — operator post-flight tooling depends on it for "drop-and-rebuild" flows. +- The migration MUST be strictly additive on every AZ-263 column, table, index, and CHECK per `data_model.md` § 6.1 / § 6.3 and `coderule.mdc`. The single allowed constraint mutation is the `ck_tiles_freshness_status` CHECK widening, which is additive in semantic effect. +- `pgcrypto` extension is NOT required by 0002 (no `gen_random_uuid()` use); `tile_freshness_rules.classification` is a TEXT PK and the seeded rows are static. - `MigrationError` is NOT a member of the `TileCacheError` family — migrations run before any `c6_tile_cache.errors` consumer is constructed. -- The schema-fixture file `tests/fixtures/c6_postgres_schema_v1.sql` is the diff target; updating it without a migration revision is a Spec-Gap finding (High) at code-review time. +- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time. +- The pinned `TILE_NAMESPACE_UUID` MUST NOT be regenerated by this task. The value `5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` is locked here; subsequent edits require a coordinated cross-workspace release. +- The `latitude` / `longitude` columns (AZ-263 names) remain advisory; `lat` / `lon` are NOT introduced. The DTO `TileId(zoom_level, lat, lon)` maps via `PostgresFilesystemStore` serialisation; the schema is NOT changed to match the DTO field name. +- The `freshness_status` column name (AZ-263) is NOT renamed to `freshness_label`. DTO field `freshness_label: FreshnessLabel` maps via `PostgresFilesystemStore` serialisation. +- The `tile_quality_metadata` column name (AZ-263) is NOT renamed to `quality_metadata`. DTO field `quality_metadata: TileQualityMetadata | None` maps via `PostgresFilesystemStore` serialisation. +- The `tiles.id BIGSERIAL` PK is NOT replaced. The deterministic UUIDv5 lives in the additive `tile_uuid` column; cross-workspace correlation uses `tile_uuid` (not `id`). Satellite-provider may use UUIDv5 as its own PK; the wire-format correlation key is the UUIDv5 value, not the column name. +- The migration assumes empty `tiles` at apply time (greenfield). NOT-NULL adds on existing rows would require a 2-phase migration (add nullable + backfill + alter NOT NULL); explicitly out of scope. +- AZ-263 legacy `freshness_status` values (`stale_warn`, `stale_reject`) are NOT deprecated by this task. A future ADR-gated migration may tighten the CHECK once any rows carrying those values are backfilled. ## Risks & Mitigation -**Risk 1: `quality_metadata` JSONB silently malformed** -- *Risk*: An impl task writes a `quality_metadata` JSONB that doesn't match `TileQualityMetadata` shape; the DB accepts it; downstream consumers crash on read. -- *Mitigation*: AC-9 documents the boundary — DB only enforces presence; shape is `insert_metadata`'s job. The future `c6_postgres_filesystem_store` task's tests cover round-trip of every documented shape. +**Risk 1: AZ-263 baseline drift between dev and CI breaks the additive assumption** +- *Risk*: A developer manually edits `0001_initial.py` after AZ-263 merged; 0002 fails to apply because expected base state diverges. +- *Mitigation*: AC-9 snapshots AZ-263 column metadata before 0002 and asserts byte-identity post-migration. AC-3 diffs full schema against `c6_postgres_schema_v2.sql`. CI catches drift before merge. -**Risk 2: Alembic version drift between dev and CI** -- *Risk*: Developer pins different Alembic minor and migrations apply differently in CI. -- *Mitigation*: AZ-263 bootstrap pins Alembic to a single minor; this task adds no version constraints of its own. +**Risk 2: Empty-DB assumption violated in dev fixtures** +- *Risk*: A dev or test fixture inserts `tiles` rows between `0001_initial` and `0002_c6_tile_identity_and_lru`; the NOT-NULL additive columns fail. +- *Mitigation*: Document the empty-DB assumption in Constraints and in the migration's docstring. The composition root applies migrations BEFORE any seed step. Test fixtures use `apply_migrations` first, then INSERT. -**Risk 3: Down-migration data loss is irreversible** -- *Risk*: Operator runs `alembic downgrade -1` on a DB with live data; tiles are lost. -- *Mitigation*: Down-migration is documented as operator-only and destructive; the runner does NOT auto-downgrade. The composition root's startup runner only ever calls `upgrade head`. +**Risk 3: Down-migration data loss is irreversible if rows exist** +- *Risk*: Operator runs `alembic downgrade -1` on a DB with live tiles; the `tile_uuid` / `location_hash` / `content_sha256` columns are dropped, destroying cross-workspace correlation evidence. +- *Mitigation*: Downgrade is documented operator-only and destructive. The composition root only calls `upgrade head` at runtime. Document in the migration docstring. -**Risk 4: Spatial-index strategy is wrong for high-zoom queries** -- *Risk*: `(zoom_level, lat, lon)` btree may not be optimal for a tight bbox at zoom 21. -- *Mitigation*: AC-3 fixes the index shape; if `query_by_bbox` benchmarks fail at takeoff load, a follow-up migration adds a GIST index. Not blocking this cycle (description.md notes the row count is bounded; btree is sufficient). +**Risk 4: UNIQUE expression index on COALESCE is slow or unsupported** +- *Risk*: The COALESCE-zero-uuid expression index is unusual; Postgres 16 supports it but plan-tooling may not recognise it for query optimisation. +- *Mitigation*: AC-4 / AC-4b exercise the index directly. The expression is deterministic and Postgres documents COALESCE-expression indices as supported since v8.0; no version-specific risk on 16.x. -**Risk 5: `pgcrypto` extension not available on a deployment** -- *Risk*: A Tier-1 Postgres deployment ships without `pgcrypto`; `gen_random_uuid()` fails. -- *Mitigation*: The migration's first statement is `CREATE EXTENSION IF NOT EXISTS pgcrypto`; if the deployment lacks the extension package, `apply_migrations` raises `MigrationError` early — surfaced to the operator at composition. +**Risk 5: CHECK widening creates ambiguity in downstream consumers** +- *Risk*: Code reading `freshness_status` may not handle the new 4-value subset and crash on legacy `stale_warn`/`stale_reject` rows (or vice-versa). +- *Mitigation*: No legacy rows exist on greenfield. The DTO `FreshnessLabel` enum is the canonical onboard vocabulary; `PostgresFilesystemStore` is responsible for the value-mapping policy. Document the legacy values as deprecated-pending-ADR in the contract bump. + +**Risk 6: UUIDv5 namespace divergence between onboard Python and satellite-provider C#** +- *Risk*: A subtle bug in the cross-repo C# UUIDv5 implementation produces UUIDs that differ from Python's `uuid.uuid5`. Cross-repo lookups fail silently. +- *Mitigation*: AC-10 locks 5+ fixed vectors with documented expected output. The corresponding `satellite-provider` test (per `AZ-TBD_tile_identity_uuidv5_bulk_list` AC-1) asserts the same vectors produce byte-identical output. + +**Risk 7: AZ-303 DTO `location_hash` default `None` shadowing impl bugs** +- *Risk*: `PostgresFilesystemStore.insert_metadata` forgets to derive `location_hash` when the DTO field is `None`; INSERT fails the NOT-NULL DB constraint at runtime. +- *Mitigation*: The impl task's AC includes a "derive-on-None" unit test. The DB-side NOT-NULL is the safety net — fail-fast, not silent. Documented in the contract bump. + +**Risk 8: Future cross-workspace renegotiation of the namespace UUID** +- *Risk*: A future cycle wants to rotate `TILE_NAMESPACE_UUID`; doing so without coordination invalidates every existing `tile_uuid` and `location_hash`. +- *Mitigation*: Constraint forbids regeneration in this task; future change requires an ADR + coordinated cross-workspace release. The migration runner logs the namespace value used at apply time. ## Runtime Completeness -- **Named capability**: Postgres 16 spatial metadata index + per-flight schema bootstrap + LRU/upload bookkeeping columns + sector-boundary classification table + per-classification freshness rules table (description.md / data_model.md / AC-NEW-3 / AC-NEW-6 / RESTRICT-SAT-2). -- **Production code that must exist**: real Alembic migration `0001_initial.py`, real `apply_migrations` runner, real schema-fixture diff test, real `psycopg_pool` connection helper. +- **Named capability**: Postgres 16 deterministic-identity columns + per-flight natural-key UNIQUE + LRU/upload bookkeeping + sector-boundary geometry + per-classification freshness rules — all additive on the AZ-263 baseline (description.md / data_model.md / AC-NEW-3 / AC-NEW-6 / RESTRICT-SAT-2 / 2026-05-12 leftover § 4–5). +- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner, real schema-fixture diff test, real `psycopg_pool` connection helper, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`. - **Allowed external stubs**: tests use `testcontainers`-managed Postgres 16 instances (already in the project's test infra per AZ-263); production wiring uses the operator's deployed Postgres. -- **Unacceptable substitutes**: SQLite "for testing only" — `production` and `test` environments MUST both be Postgres 16 (test environment as close to production as possible per coderule.mdc); raw SQL DDL applied without Alembic (would defeat the version-tracking the runner depends on); a `quality_metadata` validation at the DB layer (would lock the schema to the JSONB shape — the application-side validation is the single source of truth). +- **Unacceptable substitutes**: SQLite "for testing only" — production and test environments MUST both be Postgres 16; raw SQL DDL applied without Alembic (would defeat the version-tracking the runner depends on); a `tile_quality_metadata` validation at the DB layer (would lock the schema to the JSONB shape — the application-side validation is the single source of truth); a non-deterministic `tile_uuid` strategy (would defeat the cross-workspace coordination the namespace pin establishes); any operation that renames, retypes, or drops an AZ-263 column / table / index / CHECK (forbidden per `coderule.mdc` and `data_model.md` § 6.2 / § 6.3); a parallel Alembic env at `src/.../c6_tile_cache/_alembic/` (forbidden — the project uses one alembic env at `db/migrations/` per AZ-263 + `alembic.ini`). ## Contract -This task does NOT produce a new contract file — it implements the `tile_metadata_store.md` contract's persistence surface. The schema-fixture file `tests/fixtures/c6_postgres_schema_v1.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract. +This task does NOT produce a new contract file — it implements the `tile_metadata_store.md` contract's persistence surface and bumps its version v1.1.0 → v1.2.0 with one non-breaking minor addition (`TileMetadata.location_hash: UUID | None = None`). + +The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract. diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 42feace..1ec74d7 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -6,9 +6,9 @@ step: 7 name: Implement status: in_progress sub_step: - phase: 7 - name: archive-and-loop - detail: "batch 26/cycle1 complete and pushed to origin/dev (commit 49a67f7): AZ-302 (ThermalStatePublisher, 3pt) implemented and tests green (1140 passed, 11 Tier-2/CUDA skipped). Added ThermalStatePublisher + ThermalSource Protocol + ThermalReading DTO + _JtopSource/_PynvmlSource backends; registered c7.thermal_transition in FDR schema; new [telemetry] optional dep group in pyproject.toml. Spec moved to done/ with as-built notes; cycle report at _docs/03_implementation/batch_26_cycle1_report.md. AZ-302 transitioned to In Testing in Jira. Next: compute batch 27 — lead candidate AZ-304 (C6 Postgres schema, 2pt); 15 other ready tasks queued (AZ-306, AZ-317, AZ-318, AZ-323, AZ-324, AZ-326, AZ-327, AZ-333, AZ-334, AZ-358, AZ-389, AZ-399, AZ-400, AZ-406, AZ-408)." + phase: 3 + name: compute-batch + detail: "batch 27/cycle1: AZ-304 spec rewritten per Option A (strict-additive 0002 on AZ-263)" retry_count: 0 cycle: 1 tracker: jira diff --git a/_docs/_process_leftovers/2026-05-09_satellite-provider-design-tasks.md b/_docs/_process_leftovers/2026-05-09_satellite-provider-design-tasks.md deleted file mode 100644 index 523caf9..0000000 --- a/_docs/_process_leftovers/2026-05-09_satellite-provider-design-tasks.md +++ /dev/null @@ -1,103 +0,0 @@ -# Parent-suite design tasks for `satellite-provider` - -**Date created**: 2026-05-09 (Plan Phase 2a.0 outcome — `gps-denied-onboard` workspace) -**Workspace this leftover lives in**: `gps-denied-onboard` -**Workspace work needs to happen in**: `/Users/obezdienie001/dev/azaion/suite/satellite-provider/` -**Type**: cross-workspace dependency surfaced from this Plan cycle, NOT a tracker write blocker - ---- - -## Why this is a leftover - -During Plan Phase 2a.0 (Glossary + Architecture Vision) for `gps-denied-onboard`, two assumptions in `_docs/01_solution/solution.md` were validated against the actual `satellite-provider` codebase and found broken: - -1. **AC-8.4 — mid-flight tile upload to the Service**: `solution.md` and `acceptance_criteria.md` both assume the onboard system uploads orthorectified mid-flight tiles to `satellite-provider` after landing. **`satellite-provider` has no inbound ingest endpoint.** It is read-only from the onboard side (downloads tiles from Google Maps + serves them). - -2. **AC-NEW-7 — multi-flight ingest-side voting / trust layer**: `solution.md` assumes the Service operates "a multi-flight ingest-side voting layer that gates onboard-tile promotion to 'trusted basemap' until multiple independent flights agree on geo-alignment". **No such layer exists in `satellite-provider`.** - -Both gaps are parent-suite design / build tasks. They are tracked in this onboard workspace as **D-PROJ-2** and surfaced to the parent suite via this leftover file. - -`gps-denied-onboard` will proceed in this Plan cycle treating both as planned external capabilities; the architecture document references them as such. - ---- - -## Why these are NOT replayed automatically - -Per `.cursor/rules/tracker.mdc` § Leftovers Mechanism, this leftover does NOT block onboard progress and does NOT auto-replay because: - -- Replay requires writes against a different workspace's `_docs/` (and tracker entries against `satellite-provider`'s tracker scope). -- The next `/autodev` invocation in the **`satellite-provider`** workspace should pick this up at its own Bootstrap step. Cross-workspace leftover replay is intentionally human-gated. - -If you (the human) explicitly want me to write these design tasks into `satellite-provider/_docs/` from this conversation, say so — I have the user's permission from the 2026-05-09 turn ("If it doesn't provide sufficient information, then analyze the repository, think about the best solution to tile selection process, and document it there"). I held back to respect the workspace boundary discipline this autodev session was operating under. - ---- - -## Design task #1 — Inbound tile ingest endpoint - -**Trigger**: AC-8.4 (mid-flight tile generation, post-landing upload) per `gps-denied-onboard/_docs/00_problem/acceptance_criteria.md`. - -**Contract sketch (from the onboard side)**: - -``` -POST /api/satellite/tiles/ingest -Content-Type: multipart/form-data - -Fields per tile (one or more per request, batched): - - tile_blob: JPEG body, byte-identical to satellite-provider's existing tile format - - zoomLevel: int — same semantics as satellite-provider's existing tiles table - - latitude: double — center latitude (composite key element) - - longitude: double — center longitude - - tile_size_meters: double - - tile_size_pixels: int - - capture_timestamp: ISO 8601 — when the onboard companion generated the tile - - flight_id: UUID — which flight this tile came from - - companion_id: string — which deployed unit produced it - - quality_metadata: JSON blob (per AC-8.4 quality metadata for the Service's voting pipeline): - - estimator_label: "satellite_anchored" | "visual_propagated" | "dead_reckoned" - - covariance_2x2: [[σ_xx, σ_xy], [σ_yx, σ_yy]] — horizontal sub-matrix at tile-emit time - - last_anchor_age_ms: int — AC-1.3 binning input - - mre_px: double — reprojection error at the contributing match - - imu_bias_norm: double — VIO health proxy - - signature: optional — onboard companion's per-flight key signature over the payload (for source authentication; Plan Phase 2a.0 carryforward) - -Response: 202 Accepted with batch UUID + per-tile ingest status (queued / rejected / duplicate / superseded). -``` - -**On-disk persistence**: tiles stored in the same `./tiles/{zoomLevel}/{x}/{y}.jpg` layout as existing Google-Maps-sourced tiles. Service's existing `tiles` table extended with: `flight_id`, `companion_id`, `capture_timestamp`, `source` (`googlemaps | onboard_ingest`), `quality_metadata` (jsonb), `voting_status` (`pending | trusted | rejected`). - -**Design questions for `satellite-provider`'s Plan phase**: -- How to authenticate the onboard companion (mTLS? per-flight ephemeral keys? signed payload?). Companion is a remote untrusted endpoint by threat model. -- How to rate-limit ingest (a compromised companion could DOS the basemap). -- How to expose an admin/operator UI to inspect ingested-but-not-yet-trusted tiles. - ---- - -## Design task #2 — Multi-flight trust / voting layer - -**Trigger**: AC-NEW-7 (cache-poisoning safety budget; cross-flight error compounding) per `gps-denied-onboard/_docs/00_problem/acceptance_criteria.md`. - -**Goal (from the onboard side)**: when `satellite-provider` serves tiles to a future flight's pre-flight cache build, tiles ingested from prior flights must NOT be served as "trusted basemap" until multiple independent flights agree on geo-alignment for the same area. - -**Algorithmic intent (not prescriptive — Service team owns the design)**: -- Tiles enter with `voting_status = pending`. -- A tile is promoted to `voting_status = trusted` when ≥N independent companions (different `companion_id`) have ingested geometrically-consistent tiles covering the same lat/lon/zoom cell, weighted by the quality metadata above. -- The pre-flight cache builder (operator-side tool) consumes only `trusted` tiles by default; can be overridden to accept `pending` tiles for stale-area refresh, with explicit operator confirmation. -- Stale tiles (per AC-8.2 freshness) are demoted on age regardless of trust status. - -**Design questions for `satellite-provider`'s Plan phase**: -- N (votes-required threshold) — driven by AC-NEW-7's safety budget back-solved against measured per-flight pose error CDF. -- How to detect adversarial agreement (multiple compromised companions colluding) — out-of-band integrity checks against Google Maps ground truth? -- What "geometric consistency" means quantitatively (pixel-level RANSAC on overlapping tiles? GTSAM factor-graph over multi-flight poses?). -- What happens when `trusted` tiles disagree with newly ingested `pending` tiles in active-conflict sectors (legitimate scene change vs. cache poisoning). - ---- - -## Hand-off - -Next time `/autodev` runs in the **`satellite-provider`** workspace: - -1. Bootstrap should detect this leftover via cross-workspace search (`/Users/obezdienie001/dev/azaion/suite/gps-denied-onboard/_docs/_process_leftovers/`) — NOTE: cross-workspace leftover detection is not yet implemented in autodev; human operator must surface this manually for now. -2. The Plan skill should add Design Task #1 + Design Task #2 to the satellite-provider Plan cycle as new components / endpoints. -3. After both are implemented, this leftover can be deleted from `gps-denied-onboard`. - -Until then, `gps-denied-onboard` Plan / Decompose / Implement phases will proceed with the architecture vision treating both capabilities as **planned external dependencies** (not yet available, but contract is sketched above). diff --git a/_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md b/_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md new file mode 100644 index 0000000..92bb542 --- /dev/null +++ b/_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md @@ -0,0 +1,272 @@ +# Tile schema — usage-scenario analysis & cross-workspace design decision + +**Date**: 2026-05-12 +**Trigger**: `AZ-304` (C6 Postgres schema) found contradictory inputs (AZ-263 bootstrap migration vs. AZ-303 contract vs. AZ-304 spec). Mid-Implement step the user requested a research pass over all tile-utilisation scenarios to verify whether the proposed schema is sufficient before committing. +**Type**: design / research artefact. Replays at next `/autodev` invocation as input to AZ-304 implementation. Spawns ONE cross-workspace task in `satellite-provider`. +**Related leftover**: `2026-05-09_satellite-provider-design-tasks.md` (D-PROJ-2 inbound ingest + voting layer — STILL OPEN). This document supersedes the "tile identity" portion of that leftover and re-uses the same multi-flight trust framing. + +--- + +## 0. TL;DR + +The proposed two-hash schema (`id = uuidv5(z, x, y, source, flight_id)` + `location_hash = uuidv5(z, x, y)` + integer-only UPSERT key + `content_sha256`) is **sufficient and strictly better than current** for all eight scenarios analysed. Two changes are required outside the schema itself: + +1. **`satellite-provider`** — replace `Guid.NewGuid()` with UUIDv5, replace the float-based UPSERT conflict key with the integer slippy-tile key extended by `flight_id`, add `content_sha256`. Tracked as a new todo task in that workspace (see § 5 hand-off). +2. **`satellite-provider`** — add the `enumerate_remote_coverage` GET surface that the onboard `TileDownloader` (AZ-316) already references (`GET /api/satellite/tiles?bbox=...&zoom=...&list-only=true`). This endpoint does NOT exist in `SatelliteProvider.Api/Program.cs` today. Folded into the same satellite-provider todo. + +No change needed to AZ-304's already-drafted schema for the onboard side — the new columns are additive. + +--- + +## 1. Scenario inventory & access patterns + +Researched in parallel across `gps-denied-onboard`, `ui/`, `ui/mission-planner/`, and `satellite-provider/`. + +| # | Scenario | Driver | Access pattern | Where it lives | +|---|----------|--------|----------------|----------------| +| 1 | UI tile display (Leaflet) | Operator pans the map | `GET /tiles/{z}/{x}/{y}` — one tile per request, no filters | `ui/src/features/flights/{FlightMap,MiniMap}.tsx`, `mission-planner/src/flightPlanning/MapView.tsx`. Backed by `satellite-provider/.../Program.cs` MapGet `/tiles/{z:int}/{x:int}/{y:int}` → `TileService.GetOrDownloadTileAsync` → `TileRepository.GetByTileCoordinatesAsync` (AZ-484 "most-recent across sources, deterministic tie-break") | +| 2 | Pre-flight cache provisioning (operator) | Operator submits flight plan JSON (geofences + waypoints), C11 downloads required tiles, C10 builds engines/manifest | bbox + zoom_levels → enumerate (z, x, y) → bulk fetch from satellite-provider, write to local C6 | UI side: `mission-planner/.../flightPlan.tsx` exports `{ geofences.polygons[].{northWest, southEast}, action_points[] }` JSON. Onboard: `C11 TileDownloader.enumerate_remote_coverage` + `download_tiles_for_area` (AZ-316); `C10 CacheProvisioner.build_cache_artifacts` (AZ-325). Persistence: AZ-303 + AZ-304/305 (this repo). | +| 3 | Post-flight UAV tile upload | UAV captured mid-flight tiles after landing | Onboard: enumerate pending tiles by `(source='onboard_ingest', voting_status='pending')` → POST multipart → satellite-provider stores with `source='uav'` | Onboard: `C13 FDR tile_snapshot_sink.py` (capture) + `C11 TileUploader` (AZ-319 task spec). Satellite-provider: `/api/satellite/upload` → `UavTileUploadHandler` → `TileRepository.InsertAsync` (currently UPSERT on `(latitude, longitude, tile_zoom, tile_size_meters, source)`). | +| 4 | Onboard nav-time tile fetch | C2 SuperGlue / C2.5 rerank needs a local tile under the UAV's current pose estimate | `(zoom, lat, lon)` neighborhood lookup OR pre-computed `(zoom, tile_x, tile_y)` — local-only, no network | C6 in this repo. Hot path. | +| 5 | Freshness gate / eviction | AZ-307 freshness rules + AZ-308 budget enforcement | Range scans on `captured_at`, `sector_class`, `last_accessed_at`, `byte_count` | C6 in this repo. | +| 6 | Multi-flight trust promotion (future) | D-PROJ-2 voting layer in satellite-provider | "All `source='uav'` tiles for this `(z, x, y)` grouped by flight_id, run voting" | Satellite-provider future work — currently NO implementation. See `2026-05-09_satellite-provider-design-tasks.md` Design Task #2. | +| 7 | Bulk replay / re-grade | Operator re-runs quality grading or manifest rebuild | Stream by `id` or `captured_at` | Either repo. | +| 8 | Per-flight tile inspection UI | Operator wants to see what flight X uploaded | `WHERE flight_id = ? ORDER BY captured_at` | Satellite-provider read path; future UI page. | + +--- + +## 2. Schema sufficiency per scenario + +Proposed schema columns: + +``` +id uuid PRIMARY KEY = uuidv5(NAMESPACE, "${z}/${x}/${y}/${source}/${flight_id or 'none'}") +location_hash uuid NOT NULL = uuidv5(NAMESPACE, "${z}/${x}/${y}") +zoom_level smallint NOT NULL +tile_x integer NOT NULL +tile_y integer NOT NULL +latitude double precision NOT NULL -- center, derived, advisory +longitude double precision NOT NULL -- center, derived, advisory +tile_size_meters double precision NOT NULL +source text NOT NULL CHECK (source IN ('google_maps', 'uav', 'onboard_ingest')) +flight_id uuid NULL +content_sha256 bytea NOT NULL -- 32 bytes; tile JPEG digest +captured_at timestamptz NOT NULL +... +UNIQUE (zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, '00000000-0000-0000-0000-000000000000'::uuid)) +INDEX btree (location_hash) -- scenario 1, 6 +INDEX btree (zoom_level, tile_x, tile_y) -- scenario 2, 4 +INDEX btree (flight_id, captured_at) -- scenario 8 +INDEX btree (sector_class, captured_at) -- scenario 5 +``` + +### Scenario 1 — UI tile display ✅ +- Current: `WHERE tile_zoom=? AND tile_x=? AND tile_y=? ORDER BY captured_at DESC, ... LIMIT 1` — 3-column compound predicate. +- Proposed: `WHERE location_hash=? ORDER BY captured_at DESC LIMIT 1` — single equality probe on a uuid column (hash-index-friendly). +- Modest performance win; major correctness win: the cell-bag "give me everything for this (z, x, y)" becomes natural for future season-toggle UI. + +### Scenario 2 — Pre-flight provisioning ✅ +- Mission-planner submits `{ northWest: {lat, lon}, southEast: {lat, lon} }` rectangles. Onboard converts to slippy-tile ranges via `helpers/wgs_converter.py.latlon_to_tile_xy` (same formula as C# `GeoUtils.WorldToTilePos`, verified deterministic in the prior turn). +- The enumerated `(z, x, y)` list goes to satellite-provider's bulk-fetch endpoint, which today **does not exist**. The closest is `GET /api/satellite/tiles/latlon` (single tile by lat/lon) and there's a private `GetTilesByRegionAsync` (uses double-comparison `latitude BETWEEN ... AND longitude BETWEEN ...` — also imprecise on floats and not exposed over HTTP). +- The schema supports this fine via `(zoom_level, tile_x, tile_y)` btree. The missing piece is the HTTP surface — that's part of the new satellite-provider task. + +### Scenario 3 — UAV tile upload ⚠️ requires UPSERT-key change +- **Current bug**: UPSERT conflict key is `(latitude, longitude, tile_zoom, tile_size_meters, source)`. The first two are `double precision`. Two POSTs of the "same" tile from independently computed `tile_center_latitude` values can hash-equal (per Postgres semantics) only when bit-identical — which is fragile. AZ-484 partially papered over this with `DISTINCT ON` + ORDER BY tie-break on read, but the duplicate row still exists. +- **Bigger issue**: even if floats were stable, the UPSERT collapses multi-flight `uav` rows for the same cell into ONE row, with `DO UPDATE` overwriting `file_path` + `tile_x` + `tile_y` + `captured_at`. **This destroys per-flight evidence that D-PROJ-2's voting layer (Design Task #2 from the prior leftover) needs.** Flight A's tile is lost the moment Flight B uploads the same cell. +- **Fix** (part of new satellite-provider task): switch UPSERT to integer-only key `(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, '00000000-0000-0000-0000-000000000000'::uuid))`. Now Flight A and Flight B each get their own row; voting can see both. + +### Scenario 4 — Onboard nav-time fetch ✅ +- C6-local: `(zoom_level, tile_x, tile_y)` btree gives O(log N) point lookup. +- A small neighborhood query (e.g. fetch 3×3 tiles around current UAV pose) is a range scan on the same index. **Schema sufficient.** + +### Scenario 5 — Freshness gate & eviction ✅ +- Btree on `(sector_class, captured_at)` covers AZ-307's "active_conflict ≥30d → reject / DOWNGRADE" rule. +- Btree on `(last_accessed_at)` covers AZ-308's LRU eviction. +- Schema sufficient (these indices already in `0001_initial.py`). + +### Scenario 6 — Multi-flight trust promotion ✅ +- Voting query: `SELECT flight_id, file_path, quality_metadata FROM tiles WHERE location_hash=? AND source='uav'`. +- Single hash-index hit on `location_hash` returns the candidate set; aggregation in app code. **Schema sufficient.** Note: the trust layer is owned by satellite-provider (Design Task #2 — still open). + +### Scenario 7 — Bulk replay ✅ — standard. +### Scenario 8 — Per-flight inspection ✅ — `(flight_id, captured_at)` btree. + +--- + +## 3. Gaps surfaced by the analysis (NOT schema gaps — surrounding code/API gaps) + +| # | Gap | Owner | Effort | +|---|-----|-------|--------| +| G1 | satellite-provider uses `Guid.NewGuid()` for tile id (non-deterministic). | satellite-provider | small (replace with UUIDv5 generator) | +| G2 | satellite-provider UPSERT key uses doubles + omits `flight_id`. | satellite-provider | small (rewrite SQL + migration) | +| G3 | satellite-provider lacks `content_sha256` for content-addressable dedup. | satellite-provider | small (column + compute on insert) | +| G4 | satellite-provider lacks `GET /api/satellite/tiles?bbox=...&zoom=...&list-only=true` — referenced by onboard `TileDownloader` (AZ-316) but not implemented. | satellite-provider | medium (new MapGet + service method + tile-coord enumeration shared with onboard) | +| G5 | satellite-provider lacks the inbound ingest + voting endpoints from D-PROJ-2 leftover. | satellite-provider | large — already filed in `2026-05-09_satellite-provider-design-tasks.md` | +| G6 | Cross-workspace WGS↔tile coordinate math is duplicated (`helpers/wgs_converter.py` + `GeoUtils.cs`) with identical formulas. | suite | low priority — both sides match within 1 ULP. Document equivalence; do not refactor. | + +G1–G4 form ONE coherent satellite-provider task (the schema/UPSERT change + the bulk-list endpoint). +G5 stays as its own task per the prior leftover. + +--- + +## 4. Onboard impact + +For this repo (`gps-denied-onboard`), the conclusion is: + +- **AZ-304 still goes forward with the proposed schema** — it's strictly compatible with all scenarios above. +- **AZ-304 should add `location_hash` and `content_sha256` columns** to the `tiles` table — additive migration on top of `0001_initial.py`. This was already in the proposed Option C from the prior turn; this analysis confirms it. +- **No `flight_id` column changes** required onboard — already present. +- **AZ-303 contract update**: the `TileMetadataStore` Protocol's DTO must learn `location_hash: uuid.UUID` and `content_sha256: bytes` fields. Marked as a non-breaking minor bump (per the contract's versioning rules). +- **AZ-316 TileDownloader** (not yet implemented) will continue to call `enumerate_remote_coverage` against satellite-provider's planned `GET /api/satellite/tiles?bbox=...&list-only=true`. Until that endpoint exists onboard the downloader must use per-tile GETs via `/tiles/{z}/{x}/{y}` — slower but functional. Add this caveat to AZ-316. + +--- + +## 5. Hand-off — what happens next + +### Inside `gps-denied-onboard` (this repo) +- This document remains in `_docs/_process_leftovers/` until AZ-304 implementation incorporates it. Next `/autodev` invocation should: + 1. Read this file. + 2. Confirm the proposed schema (Option C from the prior turn + `location_hash`/`content_sha256`) with the user. + 3. Execute AZ-304 as an **additive** `0002` migration on top of `0001_initial.py`. +- The autodev state file is updated to reflect parking AZ-304 implementation pending user confirmation of the design. + +### Inside `satellite-provider` (cross-workspace) +A new todo task spec is written at: + +`/Users/obezdienie001/dev/azaion/suite/satellite-provider/_docs/02_tasks/todo/AZ-TBD_tile_identity_uuidv5_bulk_list.md` + +Next `/autodev` in that workspace must: +1. Claim a real `AZ-NNN` Jira ID for it. +2. Pull G1–G4 from § 3 above into a single PBI (estimated 3pt — moderate scope, low risk, additive migration). +3. Coordinate the migration via a feature flag: dual-write both `Guid.NewGuid()` and `uuidv5(...)` for a deprecation window, OR a one-time backfill that recomputes `id` and `location_hash` for all existing rows. + +--- + +## 6. Retrieval efficiency & on-disk layout (added in the same session) + +### 6.1 Why "multiple versions per cell" stays cheap + +The realistic row count per `location_hash` is bounded: **1 `google_maps` + ~1–5 `uav` rows from prior flights = 2–6 rows total**. That bound is what keeps "find best" cheap; a materialised pointer table is NOT needed at the bounded scale. + +**Q1 — Leaflet `/tiles/{z}/{x}/{y}` hot path** — required covering index: + +```sql +CREATE INDEX tiles_leaflet_path +ON tiles (location_hash, captured_at DESC, updated_at DESC, id DESC) +INCLUDE (file_path, content_type, etag, voting_status); +``` + +Query: + +```sql +SELECT file_path, content_type, etag FROM tiles +WHERE location_hash = $1 + AND voting_status IN ('trusted', NULL) +ORDER BY captured_at DESC, updated_at DESC, id DESC +LIMIT 1; +``` + +Layers above the DB do most of the work: `MemoryCache` keyed `tile_{z}_{x}_{y}` (L1, µs) → OS page cache (L2, sub-ms) → DB only on cold path. Index-only scan at N=6 ≈ 6 µs. + +**Q2 — Pre-flight provisioning** — replace the proposed `GET /api/satellite/tiles?bbox=...&list-only=true` with a POST inventory endpoint. The onboard side already has `helpers/wgs_converter.py.latlon_to_tile_xy` (deterministic, identical to C# `GeoUtils.WorldToTilePos`), so it can enumerate `(z, x, y)` locally and ask the server only "what do you have for THIS list?": + +``` +POST /api/satellite/tiles/inventory +Body: { "tiles": [{"z":18,"x":12345,"y":23456}, ...] } +Response: per-tile { id, location_hash, captured_at, resolution_m_per_px, estimated_bytes, source } | null +``` + +Server query (one round-trip, indexed via `location_hash` btree): + +```sql +SELECT DISTINCT ON (location_hash) + location_hash, id, captured_at, resolution_m_per_px, estimated_bytes, source +FROM tiles +WHERE location_hash = ANY($1::uuid[]) + AND voting_status IN ('trusted', NULL) +ORDER BY location_hash, captured_at DESC, updated_at DESC, id DESC; +``` + +At 2500 cells × N=6 = 15k rows scanned, `DISTINCT ON` collapses to 2500. Postgres handles this ~100–200 ms cold, ~30 ms warm — well inside the 500 ms AC-9 budget. + +**Defer the materialised `tile_current` pointer table** until production profiling demands it. Pre-optimisation is rejected. + +### 6.2 On-disk layout — source-segregated, flight-keyed + +Current state (already partially correct): +- `./tiles/{z}/{x}/{y}/...jpg` — Google Maps +- `./tiles/uav/{z}/{x}/{y}.jpg` — UAV (per `uav-tile-upload.md` v1.0.0) + +Target layout: + +``` +./tiles/ + google_maps/{z}/{x}/{y}.jpg # 1 file per cell, no flight_id + uav/{flight_id}/{z}/{x}/{y}.jpg # 1 file per flight per cell + onboard_ingest/{flight_id}/{z}/{x}/{y}.jpg # D-PROJ-2 future +``` + +Properties: +- The DB `file_path` column is authoritative; FS layout is informational/predictable but never parsed by code. +- Natural sharding on `x` keeps leaf directories bounded. +- `flight_id` in the path lets ops nuke "everything from flight X" with `rm -r ./tiles/uav/{flight_id}/` — the same primitive D-PROJ-2 voting needs for adversarial rollback. +- **No content-addressable / blob storage.** At our scale CAS adds complexity without measurable benefit. `content_sha256` gives dedup *detection* without forcing dedup *storage*. +- Page cache + `MemoryCache` carry the hot path; SSDs make adjacent-file locality irrelevant. + +### 6.3 Bulk retrieval — HTTP/2 multiplexing, NOT application-level batching + +The conventional "bulk fetch" instinct (one POST returns N tiles in a multipart body) is rejected. The right "bulk" mechanism is **HTTP/2 multiplexing** for both scenarios. + +#### Leaflet path (`/tiles/{z}/{x}/{y}`) + +Custom-TileLayer batching is rejected. Every per-tile property we want to keep — ETag-based 304s, browser cache, CDN/nginx edge cache, independent retry, display-as-arrives latency, conventional `image/jpeg` content-type — is lost the moment you bundle. HTTP/2 multiplexing gives the same wire-level efficiency (one TCP connection, header compression, response interleaving) while preserving every property above. This is what every production tile server (Mapbox, OSM, ArcGIS) does. + +**Action**: enable HTTP/2 (and HTTP/3 when feasible) at the nginx + Kestrel boundary in `satellite-provider`. Zero application code changes. + +Current state: neither side has HTTP/2 enabled today. ASP.NET Kestrel defaults to HTTP/1.1; flip `EndpointDefaults.Protocols = HttpProtocols.Http1AndHttp2` (or `Http1AndHttp2AndHttp3` over TLS). One-line change at the host config layer. + +#### UAV provisioning path (no Leaflet constraint) + +Multipart / tar / zip bundle responses are also rejected. Their downsides: + +- Server holds N file handles open and streams serially within one response → less parallelism than HTTP/2 multistream. +- Resume granularity collapses to "the whole bundle". +- Cache key becomes the bundle, not the tile. +- Client needs a parser. + +Per-tile GET over HTTP/2 wins on every axis, and the journal-based resume already designed in AZ-316 keeps its semantics intact. + +PMTiles is excellent for STATIC pre-built tile sets (Cloudflare/Protomaps); our DB is dynamic (UAV uploads invalidate any pre-built archive). Defer PMTiles until profiling demands it. + +The real efficiency win on this path is the **inventory pre-filter step** (already designed in § 6.1). Refined `TileDownloader` (AZ-316) flow: + +1. C11 computes the `(z, x, y)` list locally (deterministic slippy math — verified in the prior turn). +2. `POST /api/satellite/tiles/inventory` with the coord list → metadata-only response (`captured_at`, `estimated_bytes`, `resolution_m_per_px`). +3. Run AZ-307 freshness gate + RESTRICT-SAT-4 resolution gate + journal-skip against metadata → produce a "keep list". +4. Issue per-tile `GET /tiles/{z}/{x}/{y}` for the keep list over an HTTP/2 multiplexed connection: `httpx.Client(http2=True, limits=httpx.Limits(max_connections=10, max_keepalive_connections=10))`. +5. Append journal per successful write; per-tile retry on transient failure; resume on partial-success. + +In active-conflict zones where many tiles are stale or sub-spec, the inventory step lets C11 skip downloading bytes for 30–50% of the candidate set. That dwarfs any wire-level bulk-format gain. + +### 6.4 Existing test that needs updating in the new satellite-provider task + +`SatelliteProvider.Tests/UavTileFilePathTests.cs:23` — current expectation: + +``` +"UAV file paths follow `./tiles/uav/{zoom}/{x}/{y}.jpg` per `uav-tile-upload.md` v1.0.0" +``` + +becomes: + +``` +"UAV file paths follow `./tiles/uav/{flight_id}/{zoom}/{x}/{y}.jpg`" +``` + +The migration moves existing files into per-flight directories during the same backfill that adds `location_hash`/`content_sha256`. Mechanical operation; DB `file_path` column rewritten in the same transaction. + +--- + +## 7. Why this is parked as a leftover (not blocking) + +- The user explicitly asked for the analysis before deciding the AZ-304 reconciliation option. Decision is now informed, but the user did not yet pick A/B/C/D. The session is being closed for token reasons. +- Next `/autodev` will re-enter at the same decision point with this document and the prior turn's recommendation as input.