mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 17:21:13 +00:00
[AZ-304] C6 Postgres schema: additive 0002 migration + UUIDv5
Strictly additive Alembic migration on the AZ-263 baseline (data_model
.md § 6.1 / § 6.3): six new tiles columns (tile_uuid UNIQUE,
location_hash, content_sha256, disk_bytes, accessed_at, uploaded_at),
four new btree indices, one UNIQUE expression index over the
COALESCE-zero-uuid natural key, CHECK widening of
ck_tiles_freshness_status to the AZ-263 + AZ-303 vocabulary UNION,
four NULLable bbox columns on sector_classifications, and a new
tile_freshness_rules table seeded with the two default thresholds.
Pinned UUIDv5 namespace (TILE_NAMESPACE_UUID =
5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0) + derive_tile_id /
derive_location_hash helpers cross-coordinated with
satellite-provider. Migration runner apply_migrations(config) drives
Alembic command.upgrade("head") against the AZ-263 env with one
retry on PG SQLSTATE 40001 and structured INFO logs on apply / no-op.
Contract bump tile_metadata_store.md v1.1.0 -> v1.2.0 adds
TileMetadata.location_hash: UUID | None = None (non-breaking).
module-layout.md updated so c6_tile_cache explicitly Owns
db/migrations/**.
Tier-1 tests: UUIDv5 determinism + locked vectors + DSN resolution +
retry mocked DBAPIError -> 1180 passed, 32 skipped. Tier-2 docker
schema tests gated by @pytest.mark.docker run against the existing
docker-compose.test.yml db service.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -1,344 +0,0 @@
|
||||
# C6 Postgres Schema — Additive 0002 Migration (Identity + LRU + Freshness Rules)
|
||||
|
||||
**Task**: AZ-304_c6_postgres_schema
|
||||
**Name**: C6 Postgres Schema — Identity, LRU, and Freshness Rules (additive 0002)
|
||||
**Description**: Author the additive Alembic migration `0002_c6_tile_identity_and_lru.py` on top of the AZ-263 bootstrap baseline (`db/migrations/versions/0001_initial.py`). This migration is strictly additive per `data_model.md` § 6.1 / § 6.3: every existing AZ-263 column, table, index, and CHECK is preserved unchanged. The migration adds the deterministic-identity columns (`tile_uuid`, `location_hash`), the content-hash chain (`content_sha256`), the LRU + disk-budget bookkeeping columns (`disk_bytes`, `accessed_at`, `uploaded_at`), the per-flight natural-key UNIQUE constraint with COALESCE-zero-uuid semantics, the sector-boundary geometry columns (NULLable additions on the existing `sector_classifications` table), the new `tile_freshness_rules` table seeded with the default thresholds, and the additive widening of the `tiles.freshness_status` CHECK to accept the AZ-303 DTO `FreshnessLabel` vocabulary alongside the existing AZ-263 values. Ships the pinned UUIDv5 namespace module used by both onboard inserts and cross-workspace `satellite-provider` integrations, the migration runner the composition root invokes at startup, the schema-shape diff test, and the AZ-303 contract bump from v1.1.0 → v1.2.0 (adds `location_hash` field to `TileMetadata`).
|
||||
**Complexity**: 3 points
|
||||
**Dependencies**: AZ-303_c6_storage_interfaces (DTO contract — bumped here), AZ-263_initial_structure (baseline 0001_initial migration this builds on), AZ-269_config_loader, AZ-266_log_module
|
||||
**Component**: c6_tile_cache (epic AZ-250 / E-C6)
|
||||
**Tracker**: AZ-304
|
||||
**Epic**: AZ-250 (E-C6)
|
||||
|
||||
### Document Dependencies
|
||||
|
||||
- `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` — bumped here v1.1.0 → v1.2.0 (adds `location_hash` field). Invariant I-1 already expresses the integer-slippy-tile + COALESCE-zero-uuid natural key this task wires into the DB.
|
||||
- `_docs/02_document/contracts/c6_tile_cache/tile_store.md` — defines `content_sha256_hex` (str) the `tiles.content_sha256` column carries; unchanged this cycle.
|
||||
- `_docs/02_document/contracts/shared_config/composition_root_protocol.md` — `config.tile_cache.postgres_dsn` field.
|
||||
- `_docs/02_document/contracts/shared_logging/log_record_schema.md` — INFO log shape on migration apply / no-op.
|
||||
- `_docs/02_document/data_model.md` — system-wide data model; § 4.4 (migration baseline ordering), § 6.1 (additive-only by default), § 6.3 (`tiles` canonical columns frozen) are the governing rules this task obeys.
|
||||
- `db/migrations/versions/0001_initial.py` — AZ-263 baseline this migration extends additively.
|
||||
- `_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md` — cross-workspace scenario analysis that established the UUIDv5 + `location_hash` design; the cross-workspace counterpart `AZ-TBD_tile_identity_uuidv5_bulk_list` lives in `satellite-provider/_docs/02_tasks/todo/`.
|
||||
|
||||
## Problem
|
||||
|
||||
The AZ-263 bootstrap baseline shipped the `tiles`, `flights`, `sector_classifications`, `manifests`, and `engine_cache_entries` tables, but the canonical AZ-303 `TileMetadataStore` contract (v1.1.0) requires the schema to also carry:
|
||||
|
||||
- a deterministic UUIDv5 identifier byte-equal to `satellite-provider`'s tile id, so cross-workspace correlation (D-PROJ-2 ingest, voting, post-landing upload) does not need a separate join table;
|
||||
- a `location_hash` cell-bag identifier for `WHERE location_hash = ?` cell-bag queries (Scenario 1 UI lookup + Scenario 6 voting query in the 2026-05-12 leftover);
|
||||
- the per-flight natural-key UNIQUE constraint (`(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, zero_uuid))`) — without it, two `onboard_ingest` rows for the same cell from different flights collapse, destroying the per-flight evidence the future D-PROJ-2 voting layer needs;
|
||||
- the `content_sha256` content-hash column the AZ-280 atomic-write/sidecar pattern chains into;
|
||||
- the LRU + disk-budget bookkeeping columns (`disk_bytes`, `accessed_at`, `uploaded_at`) the `TileMetadataStore.record_lru_access` / `total_disk_bytes` / `pending_uploads` methods read and write;
|
||||
- a freshness-rules table the freshness gate (separate task) reads at insert time;
|
||||
- geometry on `sector_boundaries` records so the freshness gate can perform tile-in-sector containment checks against `SectorBoundary` DTO instances — AZ-263 shipped `sector_classifications` with a flat `sector_id` text column and no bbox geometry.
|
||||
|
||||
Additionally, the AZ-303 DTO vocabulary for `FreshnessLabel` (`fresh`, `stale_active_conflict`, `stale_rear`, `downgraded`) differs from the AZ-263 baseline `freshness_status` CHECK (`fresh`, `stale_warn`, `stale_reject`). Because `data_model.md` § 6.1 forbids removing CHECK enum values without an ADR, this task widens the CHECK to the UNION of both vocabularies (a CHECK loosening, which IS additive); a future cycle may deprecate the legacy values via ADR-gated cleanup.
|
||||
|
||||
This task delivers the strictly-additive `0002` migration that closes those gaps without renaming, retyping, or dropping any AZ-263 column, table, index, or CHECK.
|
||||
|
||||
## Outcome
|
||||
|
||||
- An Alembic migration script at `db/migrations/versions/0002_c6_tile_identity_and_lru.py` (forward `upgrade()` is purely additive; reverse `downgrade()` drops the additions and restores the original AZ-263 `freshness_status` CHECK). The migration is idempotent against a DB at AZ-263 head; Alembic rejects double-application via the standard `alembic_version` row.
|
||||
- The migration runner `apply_migrations(config) -> MigrationResult` at `src/gps_denied_onboard/components/c6_tile_cache/migrations.py`, invoked by the composition root at startup AFTER config load and BEFORE `PostgresFilesystemStore` construction. Returns `MigrationResult(applied: list[str], current_revision: str, no_op: bool)`. Logs INFO on every applied revision; logs INFO with `no_op=True` when the DB is already at head.
|
||||
- The pinned UUIDv5 namespace module at `src/gps_denied_onboard/components/c6_tile_cache/_uuid_namespace.py` exporting `TILE_NAMESPACE_UUID = UUID("5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0")`, `derive_tile_id(zoom_level, tile_x, tile_y, source, flight_id) -> UUID`, and `derive_location_hash(zoom_level, tile_x, tile_y) -> UUID`. The namespace value is cross-repo coordinated with `satellite-provider/SatelliteProvider.Common/Utils/Uuidv5.cs` per `AZ-TBD_tile_identity_uuidv5_bulk_list`; the same `uuidv5(NAMESPACE, name)` MUST produce byte-identical output on both sides.
|
||||
- The psycopg_pool connection helper at `src/gps_denied_onboard/components/c6_tile_cache/connection.py` (`psycopg_pool(config) -> psycopg_pool.ConnectionPool`), used by both this task's runner and the future `PostgresFilesystemStore`.
|
||||
- After `apply_migrations(config)` on an AZ-263-baselined DB:
|
||||
- The `tiles` table has six additional columns (`tile_uuid`, `location_hash`, `content_sha256`, `disk_bytes`, `accessed_at`, `uploaded_at`); the AZ-263 columns are unchanged.
|
||||
- The `tiles` table has a new UNIQUE index `idx_tiles_natural_key` over the COALESCE-zero-uuid natural key; the AZ-263 indices are unchanged.
|
||||
- The `tiles` table has four new indices (`idx_tiles_location_hash`, `idx_tiles_tile_uuid` from the UNIQUE, `idx_tiles_accessed_at`, `idx_tiles_pending_upload` partial, `idx_tiles_flight_captured` partial — see Schema Additions).
|
||||
- The `tiles.freshness_status` CHECK is widened to the UNION vocabulary; AZ-263 rows (none exist on greenfield apply) would continue to validate.
|
||||
- The `sector_classifications` table has four additional NULLable bbox columns; the AZ-263 columns are unchanged.
|
||||
- A new `tile_freshness_rules` table exists, seeded with the two default rows.
|
||||
- The AZ-303 contract `tile_metadata_store.md` is bumped v1.1.0 → v1.2.0 with the `location_hash: UUID | None` field added to the documented `TileMetadata` shape (non-breaking minor — Optional default `None`, populated by `PostgresFilesystemStore.insert_metadata` from `_uuid_namespace.derive_location_hash` when not supplied).
|
||||
- The DTO `TileMetadata` in `src/gps_denied_onboard/components/c6_tile_cache/_types.py` gains the same `location_hash: UUID | None = None` field (positional last, default value preserves existing constructor call sites).
|
||||
- A schema fixture `tests/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test.
|
||||
|
||||
## Scope
|
||||
|
||||
### Included
|
||||
|
||||
- The Alembic migration `0002_c6_tile_identity_and_lru.py` (additive only; the migration body uses `op.add_column`, `op.create_unique_constraint` / `op.create_index` for the UNIQUE expression index, `op.create_index` for new btree indices, `op.create_table` for `tile_freshness_rules`, `op.bulk_insert` for the seed rows, and a single `op.drop_constraint` + `op.create_check_constraint` pair to widen `ck_tiles_freshness_status`).
|
||||
- A `MigrationResult` dataclass `@dataclass(frozen=True)` at `c6_tile_cache.migrations`.
|
||||
- The `apply_migrations(config) -> MigrationResult` runner using the existing project-pinned Alembic env at `db/migrations/` (no new alembic.ini, no new env.py — AZ-263 bootstrap owns those; this task only wires `target_metadata` into `db/migrations/env.py` so future autogenerate diffs work).
|
||||
- The pinned UUIDv5 namespace module `_uuid_namespace.py` with `TILE_NAMESPACE_UUID`, `derive_tile_id`, `derive_location_hash`. No Postgres dependency; pure stdlib `uuid.uuid5`.
|
||||
- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/fixtures/c6_postgres_schema_v2.sql`. Test uses `testcontainers`-managed Postgres 16; no Python logic beyond `information_schema` queries.
|
||||
- The UUIDv5 determinism test `tests/unit/c6_tile_cache/test_uuid_namespace.py` that locks `TILE_NAMESPACE_UUID` and verifies ≥5 fixed `(z, x, y, source, flight_id)` input vectors produce the documented UUIDv5 outputs. These vectors are the cross-repo coordination evidence — the corresponding `satellite-provider` test MUST produce byte-identical UUIDs.
|
||||
- The connection helper `connection.psycopg_pool(config) -> psycopg_pool.ConnectionPool`.
|
||||
- Wiring of `db/migrations/env.py` `target_metadata` to a `c6_tile_cache.metadata` SQLAlchemy `MetaData` object that reflects both AZ-263 and AZ-304 schema (so future autogenerate diffs are mechanically comparable).
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against.
|
||||
- DTO extension in `_types.py`: `TileMetadata.location_hash: UUID | None = None` (positional last, default `None`).
|
||||
- Contract bump `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` v1.1.0 → v1.2.0 with a Change Log entry.
|
||||
|
||||
### Excluded
|
||||
|
||||
- Concrete `PostgresFilesystemStore` (insert / query / mark methods) — separate task (`c6_postgres_filesystem_store`). That task is responsible for: (a) computing `tile_x` / `tile_y` from the `TileId` WGS84 coordinates via the project's shared Web-Mercator helper, (b) deriving `tile_uuid` / `location_hash` via `_uuid_namespace.derive_*`, (c) mapping DTO field names to AZ-263 column names (`quality_metadata` → `tile_quality_metadata`, `freshness_label` → `freshness_status`, `(lat, lon)` → `(latitude, longitude)`), and (d) validating cross-side UUIDv5 parity with `satellite-provider`.
|
||||
- The freshness gate logic reading `sector_classifications` + `tile_freshness_rules` — separate task (`c6_freshness_gate`).
|
||||
- The LRU eviction policy reading `accessed_at` — separate task (`c6_cache_budget_eviction`).
|
||||
- FAISS index file format — separate task (`c6_faiss_descriptor_index`).
|
||||
- Sector-boundary CRUD (operator-side INSERT/UPDATE of geometry columns) — owned by C12; this task only adds the columns as NULLable.
|
||||
- Per-flight DB lifecycle (drop-and-rebuild between flights, freshness-rules reload) — owned by the composition root's startup orchestration; this task only applies migrations idempotently.
|
||||
- A third migration revision — every future schema change is a NEW migration file; this task only ships `0002`.
|
||||
- Postgres tuning (work_mem, shared_buffers) — handled by deployment / Dockerfile (E-DEPLOY); the schema is portable across reasonable Postgres 16 configurations.
|
||||
- Postgres-version migration (16 → 17) — out of scope this cycle; the schema MUST work on 16.x.
|
||||
- Backfill of the legacy `freshness_status` enum values (`stale_warn`, `stale_reject`) to the new vocabulary — deferred until an ADR explicitly decides the deprecation path; this task widens the CHECK without committing to a value-mapping rewrite.
|
||||
- Renaming any AZ-263 column or table — forbidden per `coderule.mdc` and `data_model.md` § 6.2; the DTO-to-column mapping lives in `PostgresFilesystemStore`.
|
||||
|
||||
## Schema Additions (0002 on top of AZ-263 0001_initial)
|
||||
|
||||
All additions below are strictly additive. The migration MUST NOT drop, rename, or retype any AZ-263 column, table, index, or CHECK; the only constraint change is a CHECK widening (loosening) for `tiles.freshness_status`, which is additive per `data_model.md` § 6.1.
|
||||
|
||||
### `tiles` — additive columns
|
||||
|
||||
| Column | Type | Nullable | Default | Notes |
|
||||
|--------|------|----------|---------|-------|
|
||||
| `tile_uuid` | `UUID` | NO | — | Deterministic `uuidv5(TILE_NAMESPACE_UUID, "{zoom_level}/{tile_x}/{tile_y}/{source}/{flight_id_or_zero_uuid}")`. Cross-repo correlation key with `satellite-provider`. Populated at INSERT by `PostgresFilesystemStore`. UNIQUE. |
|
||||
| `location_hash` | `UUID` | NO | — | `uuidv5(TILE_NAMESPACE_UUID, "{zoom_level}/{tile_x}/{tile_y}")`. Identifies a Web-Mercator cell across sources / flights. Populated at INSERT. |
|
||||
| `content_sha256` | `TEXT` | NO | — | 64 hex chars; matches `tile_store.md` `content_sha256_hex` invariant. CHECK `length(content_sha256) = 64`. |
|
||||
| `disk_bytes` | `BIGINT` | NO | — | On-disk JPEG body size; populated by `write_tile`. CHECK `disk_bytes >= 0`. |
|
||||
| `accessed_at` | `TIMESTAMPTZ` | NO | `now()` | LRU clock; updated by `record_lru_access`. |
|
||||
| `uploaded_at` | `TIMESTAMPTZ` | YES | NULL | Set by `mark_uploaded`; NULL until C11 `TileUploader` confirms post-flight upload. |
|
||||
|
||||
**Empty-DB assumption**: at 0002 apply time the `tiles` table is empty (greenfield); the four NOT-NULL columns without server defaults are added with `nullable=False` directly. On a hypothetical non-empty DB the migration would need a 2-phase split (add nullable → backfill → alter NOT NULL); this is documented but explicitly out of scope for this cycle.
|
||||
|
||||
### `tiles` — natural-key UNIQUE (additive index)
|
||||
|
||||
A UNIQUE expression index `idx_tiles_natural_key` over
|
||||
|
||||
```
|
||||
(zoom_level, tile_x, tile_y, tile_size_meters, source,
|
||||
COALESCE(flight_id, '00000000-0000-0000-0000-000000000000'::uuid))
|
||||
```
|
||||
|
||||
Two `onboard_ingest` rows for the same `(z, tile_x, tile_y, tile_size_meters)` from different `flight_id` values coexist (required by the future D-PROJ-2 voting layer). Two `googlemaps` rows for the same cell (both `flight_id = NULL` → both coalesce to the zero UUID) cannot coexist. The index is implemented via `op.create_index(..., unique=True, postgresql_using='btree')` with the expression list literal.
|
||||
|
||||
### `tiles` — CHECK widening (additive loosen)
|
||||
|
||||
The existing `ck_tiles_freshness_status` CHECK is dropped and recreated with the UNION vocabulary:
|
||||
|
||||
```sql
|
||||
freshness_status IN (
|
||||
'fresh', 'stale_warn', 'stale_reject', -- AZ-263 vocabulary (legacy)
|
||||
'stale_active_conflict', 'stale_rear', 'downgraded' -- AZ-303 FreshnessLabel vocabulary
|
||||
)
|
||||
```
|
||||
|
||||
A CHECK widening (more values accepted) is an additive loosening; any row that satisfied the prior CHECK still satisfies the new one. The legacy values are retained until an ADR-gated future cycle deprecates them. The constraint name stays `ck_tiles_freshness_status` to keep the AZ-263 fixture diffable.
|
||||
|
||||
### `tiles` — new indices (additive)
|
||||
|
||||
| Index | Definition | Purpose |
|
||||
|-------|-----------|---------|
|
||||
| `idx_tiles_natural_key` | UNIQUE btree over the COALESCE natural-key expression (above) | I-1 of `tile_metadata_store.md` |
|
||||
| `idx_tiles_location_hash` | btree over `(location_hash)` | Scenario 1 UI cell-bag lookup, Scenario 6 voting query |
|
||||
| `idx_tiles_accessed_at` | btree over `(accessed_at)` | `lru_candidates` ordering |
|
||||
| `idx_tiles_pending_upload` | partial btree over `(uploaded_at)` WHERE `source = 'onboard_ingest' AND uploaded_at IS NULL` | `pending_uploads` |
|
||||
| `idx_tiles_flight_captured` | partial btree over `(flight_id, capture_timestamp)` WHERE `flight_id IS NOT NULL` | Scenario 8 per-flight inspection |
|
||||
|
||||
The AZ-263 indices (`ix_tiles_zxy`, `ix_tiles_lat_lon`, `ix_tiles_voting_status_onboard`, `ix_tiles_flight_id`, `ix_tiles_created_at`) are unchanged.
|
||||
|
||||
### `sector_classifications` — additive columns (geometry, NULLable)
|
||||
|
||||
| Column | Type | Nullable | Default | Notes |
|
||||
|--------|------|----------|---------|-------|
|
||||
| `min_lat` | `DOUBLE PRECISION` | YES | NULL | bbox south edge — populated pre-flight by C12 |
|
||||
| `min_lon` | `DOUBLE PRECISION` | YES | NULL | bbox west edge |
|
||||
| `max_lat` | `DOUBLE PRECISION` | YES | NULL | bbox north edge |
|
||||
| `max_lon` | `DOUBLE PRECISION` | YES | NULL | bbox east edge |
|
||||
|
||||
All NULLable because existing AZ-263 rows (if any) lack geometry; C12 populates pre-flight before the freshness gate reads. The AZ-263 columns (`sector_id`, `classification`, `freshness_threshold_days`) are unchanged. No CHECK on min ≤ max yet — that arrives when C12 enforces population (would be a tightening, ADR-gated).
|
||||
|
||||
The `SectorBoundary` DTO from `_types.py` maps `bbox.min_lat → min_lat`, etc.; `classification` maps directly. The DTO field `sector_id` is not present in the DTO today because the in-memory `SectorBoundary` does not need it; the impl task resolves DB row ↔ DTO via the `sector_id` text PK component.
|
||||
|
||||
### New table: `tile_freshness_rules`
|
||||
|
||||
| Column | Type | Nullable | Default | Notes |
|
||||
|--------|------|----------|---------|-------|
|
||||
| `classification` | `TEXT` | NO | — | PK; matches `sector_classifications.classification` |
|
||||
| `max_age_seconds` | `BIGINT` | NO | — | Per-classification threshold |
|
||||
| `action` | `TEXT` | NO | — | CHECK `action IN ('reject','downgrade')` |
|
||||
| `set_at` | `TIMESTAMPTZ` | NO | `now()` | |
|
||||
|
||||
Constraints:
|
||||
|
||||
- `PRIMARY KEY (classification)`
|
||||
- `CHECK (action IN ('reject', 'downgrade'))`
|
||||
- `CHECK (max_age_seconds > 0)`
|
||||
|
||||
Default rows seeded by the migration (AC-7):
|
||||
|
||||
- `('active_conflict', 15552000, 'reject')` — 6 × 30 × 86400 seconds = 6 months
|
||||
- `('stable_rear', 31104000, 'downgrade')` — 12 × 30 × 86400 seconds = 12 months
|
||||
|
||||
### AZ-303 contract bump v1.1.0 → v1.2.0: `TileMetadata.location_hash`
|
||||
|
||||
- DTO change in `_types.py`: add `location_hash: UUID | None = None` to `TileMetadata` (positional last, default `None`). `PostgresFilesystemStore.insert_metadata` computes the value via `_uuid_namespace.derive_location_hash(...)` when `None`; uses the supplied value when present.
|
||||
- Contract change in `tile_metadata_store.md`: bump version to v1.2.0; add `location_hash: UUID | None` row to the DTO field table; add a "v1.2.0" Change Log row referencing this task and the leftover.
|
||||
- Non-breaking: existing constructors and `insinstance(impl, TileMetadataStore)` checks continue to work.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: Migration is idempotent against an AZ-263-baselined DB**
|
||||
Given a Postgres 16 database at AZ-263 head (`alembic_version = '0001_initial'`)
|
||||
When `apply_migrations(config)` runs
|
||||
Then the additive columns / indices / table / CHECK widening exist; the `alembic_version` row carries `0002_c6_tile_identity_and_lru`; `MigrationResult.applied == ['0002_c6_tile_identity_and_lru']`; `MigrationResult.no_op == False`; AZ-263 columns / indices / CHECKs are byte-identical to their pre-migration state.
|
||||
|
||||
**AC-2: Migration is no-op when at head**
|
||||
Given a Postgres DB already at `0002_c6_tile_identity_and_lru`
|
||||
When `apply_migrations(config)` runs again
|
||||
Then `MigrationResult.applied == []`; `MigrationResult.no_op == True`; no DDL is emitted (verifiable via `pg_stat_user_tables` row counts unchanged).
|
||||
|
||||
**AC-3: Schema shape matches the documented DDL**
|
||||
Given a DB upgraded through 0001 + 0002
|
||||
When the schema-shape diff test introspects `information_schema.columns` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` row contents
|
||||
Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/fixtures/c6_postgres_schema_v2.sql` is empty.
|
||||
|
||||
**AC-4: Natural-key uniqueness enforces per-flight separation**
|
||||
Given a `tiles` table after 0002
|
||||
When two INSERTs with the same `(zoom_level, tile_x, tile_y, tile_size_meters, source='onboard_ingest')` and **different** `flight_id` values are attempted
|
||||
Then both INSERTs succeed; the resulting rows have different `tile_uuid` values (different UUIDv5 inputs) and the same `location_hash` (UUIDv5 inputs differ only in `flight_id`/`source`).
|
||||
|
||||
**AC-4b: Natural-key uniqueness rejects duplicate flight inserts**
|
||||
Given a `tiles` table after 0002
|
||||
When two INSERTs with the same `(zoom_level, tile_x, tile_y, tile_size_meters, source, flight_id)` (or both `flight_id = NULL` for `googlemaps`) are attempted with different `content_sha256` values
|
||||
Then the second INSERT raises `psycopg.errors.UniqueViolation` (`idx_tiles_natural_key`); the first row is unaffected.
|
||||
|
||||
**AC-5: AZ-263 CHECK constraints survive widening**
|
||||
Given the post-0002 `tiles` table
|
||||
When `INSERT INTO tiles (...freshness_status...) VALUES ('fresh')` runs (and separately for `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`)
|
||||
Then all six values are accepted; an `INSERT ... VALUES ('bogus')` is rejected by the widened `ck_tiles_freshness_status` CHECK.
|
||||
|
||||
**AC-6: Down migration reverses cleanly**
|
||||
Given a DB at `0002_c6_tile_identity_and_lru`
|
||||
When `alembic downgrade -1` runs (operator-only command; not exercised by the runtime)
|
||||
Then the additive columns / indices / `tile_freshness_rules` table are dropped; the `ck_tiles_freshness_status` CHECK is restored to the AZ-263 vocabulary; the DB returns to byte-identical AZ-263 state; subsequent `upgrade` re-applies cleanly.
|
||||
|
||||
**AC-7: Default freshness rules are seeded**
|
||||
Given a DB upgraded through 0002
|
||||
When the schema-shape test queries `tile_freshness_rules`
|
||||
Then exactly two rows exist: `('active_conflict', 15552000, 'reject')` and `('stable_rear', 31104000, 'downgrade')`.
|
||||
|
||||
**AC-8: Migration runner logs INFO on apply and no-op**
|
||||
Given a DB at AZ-263 head
|
||||
When `apply_migrations` runs and then runs again
|
||||
Then the first call emits an INFO log with `kind="c6.migration.applied"` carrying `revisions=['0002_c6_tile_identity_and_lru']`; the second call emits an INFO log with `kind="c6.migration.no_op"`. Both calls also log the resolved `TILE_NAMESPACE_UUID` value once for post-mortem drift detection.
|
||||
|
||||
**AC-9: AZ-263 columns are byte-identical after upgrade**
|
||||
Given a DB at AZ-263 head + a recorded snapshot of `information_schema.columns` for `tiles`, `flights`, `sector_classifications`, `manifests`, `engine_cache_entries`
|
||||
When `apply_migrations(config)` runs
|
||||
Then for every AZ-263 column the snapshot and the post-0002 row are byte-identical (column name, data type, nullability, default expression). New columns appear only as additions.
|
||||
|
||||
**AC-10: UUIDv5 derivation is deterministic and cross-repo coordinated**
|
||||
Given the pinned `TILE_NAMESPACE_UUID = 5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0`
|
||||
When `derive_tile_id` and `derive_location_hash` are called for the documented fixed test vectors (≥5 `(zoom_level, tile_x, tile_y, source, flight_id)` tuples in `tests/unit/c6_tile_cache/test_uuid_namespace.py`)
|
||||
Then each call returns the documented UUID byte-for-byte; the same Python expression run twice produces identical output; the locked test vectors are the cross-repo coordination evidence used to verify `satellite-provider`'s C# implementation produces the same UUIDs.
|
||||
|
||||
**AC-11: `location_hash` is invariant across `source` and `flight_id`**
|
||||
Given three rows for the same `(zoom_level, tile_x, tile_y)` — one `source='googlemaps'` and two `source='onboard_ingest'` from different `flight_id` values
|
||||
When the test queries `SELECT DISTINCT location_hash FROM tiles WHERE zoom_level=? AND tile_x=? AND tile_y=?`
|
||||
Then exactly ONE `location_hash` value is returned (the three rows share the same cell-bag identifier).
|
||||
|
||||
**AC-12: `TileMetadata.location_hash` default is `None` (non-breaking DTO bump)**
|
||||
Given the v1.2.0 `TileMetadata` DTO
|
||||
When existing AZ-303-style positional constructors run (without supplying `location_hash`)
|
||||
Then construction succeeds with `location_hash = None`; when constructors supply an explicit `UUID`, the value is preserved on the frozen instance.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
**Performance**
|
||||
- Migration apply ≤ 5 s on an AZ-263-baselined Postgres 16 database with empty `tiles`. The migration body is bounded by index creation on an empty table, four `ALTER TABLE … ADD COLUMN`s, one `op.create_table`, two `op.bulk_insert` rows, and one CHECK widening — all O(1) in row count.
|
||||
- `apply_migrations` no-op call (DB at head) ≤ 100 ms.
|
||||
- The new UNIQUE expression index `idx_tiles_natural_key` does not regress AZ-263 query plans on the existing `ix_tiles_zxy` btree; both coexist and the planner selects based on predicate shape.
|
||||
|
||||
**Compatibility**
|
||||
- Postgres 16.x.
|
||||
- `psycopg_pool` 3.x — pinned by AZ-263; this task adds no new third-party dependencies.
|
||||
- Alembic 1.13+ — pinned by AZ-263.
|
||||
- Cross-workspace UUIDv5 namespace: `TILE_NAMESPACE_UUID` MUST be byte-identical to the satellite-provider C# constant; any change requires a coordinated cross-repo release.
|
||||
|
||||
**Reliability**
|
||||
- The migration is wrapped in a single Alembic transaction (Postgres default). A crash mid-migration leaves the DB at the prior revision (`0001_initial`).
|
||||
- The runner catches `psycopg.errors.SerializationFailure` and retries once with exponential backoff; after the second failure, raises `MigrationError` (defined in `c6_tile_cache.migrations`, NOT in `TileCacheError` — migrations run before any runtime error consumer is constructed).
|
||||
|
||||
## Unit Tests
|
||||
|
||||
| AC Ref | What to Test | Required Outcome |
|
||||
|--------|-------------|-----------------|
|
||||
| AC-1 | `apply_migrations` against fresh `testcontainer` DB previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. |
|
||||
| AC-2 | `apply_migrations` against DB already at 0002 head | `result.applied=[]`; `result.no_op=True`; no DDL emitted. |
|
||||
| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/fixtures/c6_postgres_schema_v2.sql` | Zero diff. |
|
||||
| AC-4 | Two `onboard_ingest` INSERTs with same `(z, tile_x, tile_y, tile_size_meters)` and different `flight_id` | Both succeed; same `location_hash`; different `tile_uuid`. |
|
||||
| AC-4b | Two INSERTs with identical natural-key tuple (same `flight_id` or both NULL for `googlemaps`) | Second INSERT raises `psycopg.errors.UniqueViolation`. |
|
||||
| AC-5 | INSERT one row each for `'fresh'`, `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`; INSERT one row with `'bogus'` | First six succeed; last raises `psycopg.errors.CheckViolation`. |
|
||||
| AC-6 | `alembic downgrade -1` then `upgrade head` | DB returns to AZ-263-byte-identical state; subsequent upgrade re-applies cleanly. |
|
||||
| AC-7 | SELECT `tile_freshness_rules` after migration | Exactly 2 rows with documented values. |
|
||||
| AC-8 | Capture log records during migration apply + no-op | INFO records with `kind="c6.migration.applied"` and `kind="c6.migration.no_op"`; namespace UUID emitted on apply. |
|
||||
| AC-9 | Snapshot AZ-263 `information_schema.columns` before 0002; compare after | Snapshot rows byte-identical post-migration; new column rows are additions only. |
|
||||
| AC-10 | `derive_tile_id` / `derive_location_hash` for 5 fixed input vectors | Outputs match the documented UUIDs byte-for-byte; idempotent on second call. |
|
||||
| AC-11 | INSERT three rows sharing `(z, tile_x, tile_y)` from different `(source, flight_id)` | Exactly one distinct `location_hash` value. |
|
||||
| AC-12 | Construct `TileMetadata` without `location_hash` and with explicit `location_hash=uuid4()` | First yields `location_hash=None`; second preserves the supplied value; both instances are frozen. |
|
||||
| NFR-perf-apply | Migration apply on AZ-263-baselined empty `tiles` | Wall ≤ 5 s |
|
||||
| NFR-perf-noop | `apply_migrations` no-op timing | Wall ≤ 100 ms |
|
||||
| NFR-reliability-retry | Inject `SerializationFailure` once, then succeed | Migration succeeds on retry; on second failure raises `MigrationError`. |
|
||||
|
||||
## Constraints
|
||||
|
||||
- Postgres 16.x ONLY this cycle; no SQLite / no MySQL fallback.
|
||||
- Alembic + `psycopg_pool` are pinned by AZ-263; this task does NOT introduce new third-party dependencies.
|
||||
- The migration MUST be reversible (`downgrade` drops the additions cleanly and restores the AZ-263 CHECK) — operator post-flight tooling depends on it for "drop-and-rebuild" flows.
|
||||
- The migration MUST be strictly additive on every AZ-263 column, table, index, and CHECK per `data_model.md` § 6.1 / § 6.3 and `coderule.mdc`. The single allowed constraint mutation is the `ck_tiles_freshness_status` CHECK widening, which is additive in semantic effect.
|
||||
- `pgcrypto` extension is NOT required by 0002 (no `gen_random_uuid()` use); `tile_freshness_rules.classification` is a TEXT PK and the seeded rows are static.
|
||||
- `MigrationError` is NOT a member of the `TileCacheError` family — migrations run before any `c6_tile_cache.errors` consumer is constructed.
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time.
|
||||
- The pinned `TILE_NAMESPACE_UUID` MUST NOT be regenerated by this task. The value `5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` is locked here; subsequent edits require a coordinated cross-workspace release.
|
||||
- The `latitude` / `longitude` columns (AZ-263 names) remain advisory; `lat` / `lon` are NOT introduced. The DTO `TileId(zoom_level, lat, lon)` maps via `PostgresFilesystemStore` serialisation; the schema is NOT changed to match the DTO field name.
|
||||
- The `freshness_status` column name (AZ-263) is NOT renamed to `freshness_label`. DTO field `freshness_label: FreshnessLabel` maps via `PostgresFilesystemStore` serialisation.
|
||||
- The `tile_quality_metadata` column name (AZ-263) is NOT renamed to `quality_metadata`. DTO field `quality_metadata: TileQualityMetadata | None` maps via `PostgresFilesystemStore` serialisation.
|
||||
- The `tiles.id BIGSERIAL` PK is NOT replaced. The deterministic UUIDv5 lives in the additive `tile_uuid` column; cross-workspace correlation uses `tile_uuid` (not `id`). Satellite-provider may use UUIDv5 as its own PK; the wire-format correlation key is the UUIDv5 value, not the column name.
|
||||
- The migration assumes empty `tiles` at apply time (greenfield). NOT-NULL adds on existing rows would require a 2-phase migration (add nullable + backfill + alter NOT NULL); explicitly out of scope.
|
||||
- AZ-263 legacy `freshness_status` values (`stale_warn`, `stale_reject`) are NOT deprecated by this task. A future ADR-gated migration may tighten the CHECK once any rows carrying those values are backfilled.
|
||||
|
||||
## Risks & Mitigation
|
||||
|
||||
**Risk 1: AZ-263 baseline drift between dev and CI breaks the additive assumption**
|
||||
- *Risk*: A developer manually edits `0001_initial.py` after AZ-263 merged; 0002 fails to apply because expected base state diverges.
|
||||
- *Mitigation*: AC-9 snapshots AZ-263 column metadata before 0002 and asserts byte-identity post-migration. AC-3 diffs full schema against `c6_postgres_schema_v2.sql`. CI catches drift before merge.
|
||||
|
||||
**Risk 2: Empty-DB assumption violated in dev fixtures**
|
||||
- *Risk*: A dev or test fixture inserts `tiles` rows between `0001_initial` and `0002_c6_tile_identity_and_lru`; the NOT-NULL additive columns fail.
|
||||
- *Mitigation*: Document the empty-DB assumption in Constraints and in the migration's docstring. The composition root applies migrations BEFORE any seed step. Test fixtures use `apply_migrations` first, then INSERT.
|
||||
|
||||
**Risk 3: Down-migration data loss is irreversible if rows exist**
|
||||
- *Risk*: Operator runs `alembic downgrade -1` on a DB with live tiles; the `tile_uuid` / `location_hash` / `content_sha256` columns are dropped, destroying cross-workspace correlation evidence.
|
||||
- *Mitigation*: Downgrade is documented operator-only and destructive. The composition root only calls `upgrade head` at runtime. Document in the migration docstring.
|
||||
|
||||
**Risk 4: UNIQUE expression index on COALESCE is slow or unsupported**
|
||||
- *Risk*: The COALESCE-zero-uuid expression index is unusual; Postgres 16 supports it but plan-tooling may not recognise it for query optimisation.
|
||||
- *Mitigation*: AC-4 / AC-4b exercise the index directly. The expression is deterministic and Postgres documents COALESCE-expression indices as supported since v8.0; no version-specific risk on 16.x.
|
||||
|
||||
**Risk 5: CHECK widening creates ambiguity in downstream consumers**
|
||||
- *Risk*: Code reading `freshness_status` may not handle the new 4-value subset and crash on legacy `stale_warn`/`stale_reject` rows (or vice-versa).
|
||||
- *Mitigation*: No legacy rows exist on greenfield. The DTO `FreshnessLabel` enum is the canonical onboard vocabulary; `PostgresFilesystemStore` is responsible for the value-mapping policy. Document the legacy values as deprecated-pending-ADR in the contract bump.
|
||||
|
||||
**Risk 6: UUIDv5 namespace divergence between onboard Python and satellite-provider C#**
|
||||
- *Risk*: A subtle bug in the cross-repo C# UUIDv5 implementation produces UUIDs that differ from Python's `uuid.uuid5`. Cross-repo lookups fail silently.
|
||||
- *Mitigation*: AC-10 locks 5+ fixed vectors with documented expected output. The corresponding `satellite-provider` test (per `AZ-TBD_tile_identity_uuidv5_bulk_list` AC-1) asserts the same vectors produce byte-identical output.
|
||||
|
||||
**Risk 7: AZ-303 DTO `location_hash` default `None` shadowing impl bugs**
|
||||
- *Risk*: `PostgresFilesystemStore.insert_metadata` forgets to derive `location_hash` when the DTO field is `None`; INSERT fails the NOT-NULL DB constraint at runtime.
|
||||
- *Mitigation*: The impl task's AC includes a "derive-on-None" unit test. The DB-side NOT-NULL is the safety net — fail-fast, not silent. Documented in the contract bump.
|
||||
|
||||
**Risk 8: Future cross-workspace renegotiation of the namespace UUID**
|
||||
- *Risk*: A future cycle wants to rotate `TILE_NAMESPACE_UUID`; doing so without coordination invalidates every existing `tile_uuid` and `location_hash`.
|
||||
- *Mitigation*: Constraint forbids regeneration in this task; future change requires an ADR + coordinated cross-workspace release. The migration runner logs the namespace value used at apply time.
|
||||
|
||||
## Runtime Completeness
|
||||
|
||||
- **Named capability**: Postgres 16 deterministic-identity columns + per-flight natural-key UNIQUE + LRU/upload bookkeeping + sector-boundary geometry + per-classification freshness rules — all additive on the AZ-263 baseline (description.md / data_model.md / AC-NEW-3 / AC-NEW-6 / RESTRICT-SAT-2 / 2026-05-12 leftover § 4–5).
|
||||
- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner, real schema-fixture diff test, real `psycopg_pool` connection helper, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`.
|
||||
- **Allowed external stubs**: tests use `testcontainers`-managed Postgres 16 instances (already in the project's test infra per AZ-263); production wiring uses the operator's deployed Postgres.
|
||||
- **Unacceptable substitutes**: SQLite "for testing only" — production and test environments MUST both be Postgres 16; raw SQL DDL applied without Alembic (would defeat the version-tracking the runner depends on); a `tile_quality_metadata` validation at the DB layer (would lock the schema to the JSONB shape — the application-side validation is the single source of truth); a non-deterministic `tile_uuid` strategy (would defeat the cross-workspace coordination the namespace pin establishes); any operation that renames, retypes, or drops an AZ-263 column / table / index / CHECK (forbidden per `coderule.mdc` and `data_model.md` § 6.2 / § 6.3); a parallel Alembic env at `src/.../c6_tile_cache/_alembic/` (forbidden — the project uses one alembic env at `db/migrations/` per AZ-263 + `alembic.ini`).
|
||||
|
||||
## Contract
|
||||
|
||||
This task does NOT produce a new contract file — it implements the `tile_metadata_store.md` contract's persistence surface and bumps its version v1.1.0 → v1.2.0 with one non-breaking minor addition (`TileMetadata.location_hash: UUID | None = None`).
|
||||
|
||||
The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract.
|
||||
Reference in New Issue
Block a user