mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 11:31:13 +00:00
[AZ-304] C6 Postgres schema: additive 0002 migration + UUIDv5
Strictly additive Alembic migration on the AZ-263 baseline (data_model
.md § 6.1 / § 6.3): six new tiles columns (tile_uuid UNIQUE,
location_hash, content_sha256, disk_bytes, accessed_at, uploaded_at),
four new btree indices, one UNIQUE expression index over the
COALESCE-zero-uuid natural key, CHECK widening of
ck_tiles_freshness_status to the AZ-263 + AZ-303 vocabulary UNION,
four NULLable bbox columns on sector_classifications, and a new
tile_freshness_rules table seeded with the two default thresholds.
Pinned UUIDv5 namespace (TILE_NAMESPACE_UUID =
5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0) + derive_tile_id /
derive_location_hash helpers cross-coordinated with
satellite-provider. Migration runner apply_migrations(config) drives
Alembic command.upgrade("head") against the AZ-263 env with one
retry on PG SQLSTATE 40001 and structured INFO logs on apply / no-op.
Contract bump tile_metadata_store.md v1.1.0 -> v1.2.0 adds
TileMetadata.location_hash: UUID | None = None (non-breaking).
module-layout.md updated so c6_tile_cache explicitly Owns
db/migrations/**.
Tier-1 tests: UUIDv5 determinism + locked vectors + DSN resolution +
retry mocked DBAPIError -> 1180 passed, 32 skipped. Tier-2 docker
schema tests gated by @pytest.mark.docker run against the existing
docker-compose.test.yml db service.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -7,7 +7,7 @@
|
||||
- AZ-TBD-c6-freshness-gate (insert hook + sector classification reader)
|
||||
- AZ-TBD-c6-cache-budget-eviction (LRU candidate enumeration + delete coordination)
|
||||
- TBD at decompose time: E-C10 (AZ-252 — manifest + provisioning), E-C11 (AZ-251 — both `TileDownloader` insert and `TileUploader` reader queries), E-C12 (AZ-253 — operator pre-flight tooling)
|
||||
**Version**: 1.1.0
|
||||
**Version**: 1.2.0
|
||||
**Status**: draft
|
||||
**Last Updated**: 2026-05-12
|
||||
|
||||
@@ -63,6 +63,17 @@ class TileMetadataPersistent:
|
||||
|
||||
The Protocol returns `TileMetadata` from queries. `TileMetadataPersistent` is the in-process view of LRU and disk-budget state, accessible only via `lru_candidates` / `record_lru_access` / `total_disk_bytes`.
|
||||
|
||||
#### TileMetadata.location_hash (v1.2.0)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class TileMetadata:
|
||||
# ...existing AZ-303 v1.1.0 fields unchanged...
|
||||
location_hash: UUID | None = None # uuidv5(TILE_NAMESPACE_UUID, "{zoom}/{tile_x}/{tile_y}")
|
||||
```
|
||||
|
||||
`location_hash` is a deterministic per-cell-bag identifier (UUIDv5, namespace-pinned in `c6_tile_cache._uuid_namespace.TILE_NAMESPACE_UUID`) shared by every row at the same `(zoom_level, tile_x, tile_y)` regardless of source or flight (Scenario 1 UI lookup, Scenario 6 voting query of the 2026-05-12 tile-schema scenario analysis). Defaults to `None` so AZ-303-era constructors continue to work; `PostgresFilesystemStore.insert_metadata` derives the value via `derive_location_hash(zoom_level, tile_x, tile_y)` when `None`, and the DB-side NOT-NULL constraint is the safety net. Cross-repo coordinated with `satellite-provider` per `AZ-TBD_tile_identity_uuidv5_bulk_list`.
|
||||
|
||||
### Sector classification (read-only input to the freshness gate)
|
||||
|
||||
```python
|
||||
@@ -77,7 +88,7 @@ class SectorBoundary:
|
||||
classification: SectorClassification
|
||||
```
|
||||
|
||||
`SectorClassification` is set pre-flight by the operator via C12; the metadata store reads `SectorBoundary` rows from a sibling table (`sector_boundaries`) at insert-time to decide which freshness rule to apply. The Protocol does NOT expose insert-side methods for `SectorBoundary` rows — that surface lives in C12.
|
||||
`SectorClassification` is set pre-flight by the operator via C12; the metadata store reads `SectorBoundary` rows from the sibling table `sector_classifications` (per the AZ-263 baseline schema; AZ-304 adds the NULLable `min_lat` / `min_lon` / `max_lat` / `max_lon` bbox columns operators populate) at insert-time to decide which freshness rule to apply. The Protocol does NOT expose insert-side methods for `SectorBoundary` rows — that surface lives in C12.
|
||||
|
||||
## Invariants
|
||||
|
||||
@@ -98,7 +109,7 @@ class SectorBoundary:
|
||||
- **Not covered: sector boundary insert / update.** Owned by C12 operator-tooling against a sibling table; this Protocol is read-only on `SectorBoundary` and does NOT expose CRUD.
|
||||
- **Not covered: cross-flight aggregation / voting threshold computation.** That's `satellite-provider`'s D-PROJ-2 trust layer (parent suite); C6 just stamps the per-row `voting_status`.
|
||||
- **Not covered: full-text search / arbitrary-WHERE queries.** Only the methods above; ad-hoc queries go through DBA tooling, not this Protocol.
|
||||
- **Not covered: schema migrations.** Migration scripts live in `c6_tile_cache/_alembic/`; the Protocol is shape-only.
|
||||
- **Not covered: schema migrations.** Migration scripts live in `db/migrations/versions/` (project-level Alembic env owned by c6_tile_cache per `module-layout.md`; `0001_initial.py` shipped by AZ-263, `0002_c6_tile_identity_and_lru.py` by AZ-304); the Protocol is shape-only.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
@@ -132,3 +143,4 @@ Same rules as `tile_store.md` § Versioning Rules.
|
||||
|---------|------|--------|--------|
|
||||
| 1.0.0 | 2026-05-10 | Initial contract — 9-method Protocol + LRU/disk-budget extensions + freshness gate semantics + composite-key uniqueness invariant. | autodev (decompose Step 2 of AZ-250 / E-C6) |
|
||||
| 1.1.0 | 2026-05-12 | Non-breaking refinement of Invariant I-1: natural key switched from `(zoom_level, lat, lon, source)` (float-based) to `(zoom_level, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, zero_uuid))` (integer + per-flight separated). Protocol surface unchanged; consumers gain the ability to observe multiple ONBOARD_INGEST rows for the same cell from different flights (required by D-PROJ-2 voting). Driven by `_docs/_process_leftovers/2026-05-12_tile-schema-scenario-analysis.md` and the cross-workspace satellite-provider task `AZ-TBD_tile_identity_uuidv5_bulk_list`. | autodev (AZ-304 batch 27 of cycle 1) |
|
||||
| 1.2.0 | 2026-05-12 | Non-breaking addition of `TileMetadata.location_hash: UUID \| None = None` (cross-source/cross-flight cell-bag identifier; UUIDv5 over `(zoom, tile_x, tile_y)`). Corrected stale references: sector table name (`sector_boundaries` → `sector_classifications`) and Alembic env path (`c6_tile_cache/_alembic/` → `db/migrations/versions/`). Protocol surface unchanged; existing constructors continue to work because the field defaults to `None`. Shipped by AZ-304 alongside the additive `0002_c6_tile_identity_and_lru` migration. | autodev (AZ-304 batch 27 of cycle 1) |
|
||||
|
||||
@@ -143,8 +143,10 @@ Bootstrap reference: `_docs/02_tasks/todo/AZ-263_initial_structure.md`. Architec
|
||||
- **Internal**:
|
||||
- `postgres_filesystem_store.py` (Postgres mirror + filesystem mmap + FAISS HNSW; production-default)
|
||||
- `_native/` (`cpp/faiss_index/` wrapper)
|
||||
- `_alembic/` (migration scripts; `0001_initial.sql` shipped in bootstrap)
|
||||
- **Owns**: `src/gps_denied_onboard/components/c6_tile_cache/**`, `cpp/faiss_index/**`, `tests/unit/c6_tile_cache/**`
|
||||
- `migrations.py` (`apply_migrations(config) -> MigrationResult` runner invoked by the composition root at startup; AZ-304 + later)
|
||||
- `_uuid_namespace.py` (pinned `TILE_NAMESPACE_UUID` + `derive_tile_id` / `derive_location_hash` helpers; cross-repo coordinated with `satellite-provider`; AZ-304)
|
||||
- `connection.py` (`psycopg_pool` ConnectionPool helper; AZ-304)
|
||||
- **Owns**: `src/gps_denied_onboard/components/c6_tile_cache/**`, `cpp/faiss_index/**`, `tests/unit/c6_tile_cache/**`, `db/migrations/**` (project-level Alembic env owned by c6 — `alembic.ini` at repo root points here; `0001_initial.py` shipped by AZ-263 bootstrap, `0002_c6_tile_identity_and_lru.py` and forward owned by AZ-304+ migrations)
|
||||
- **Imports from**: `_types`, `helpers.sha256_sidecar`, `helpers.wgs_converter`, `config`, `logging`, `fdr_client`
|
||||
- **Consumed by**: `c2_vpr`, `c2_5_rerank`, `c3_matcher`, `c10_provisioning`, `c11_tile_manager`, `runtime_root`
|
||||
|
||||
|
||||
+14
-16
@@ -40,7 +40,7 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- An Alembic migration script at `db/migrations/versions/0002_c6_tile_identity_and_lru.py` (forward `upgrade()` is purely additive; reverse `downgrade()` drops the additions and restores the original AZ-263 `freshness_status` CHECK). The migration is idempotent against a DB at AZ-263 head; Alembic rejects double-application via the standard `alembic_version` row.
|
||||
- The migration runner `apply_migrations(config) -> MigrationResult` at `src/gps_denied_onboard/components/c6_tile_cache/migrations.py`, invoked by the composition root at startup AFTER config load and BEFORE `PostgresFilesystemStore` construction. Returns `MigrationResult(applied: list[str], current_revision: str, no_op: bool)`. Logs INFO on every applied revision; logs INFO with `no_op=True` when the DB is already at head.
|
||||
- The pinned UUIDv5 namespace module at `src/gps_denied_onboard/components/c6_tile_cache/_uuid_namespace.py` exporting `TILE_NAMESPACE_UUID = UUID("5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0")`, `derive_tile_id(zoom_level, tile_x, tile_y, source, flight_id) -> UUID`, and `derive_location_hash(zoom_level, tile_x, tile_y) -> UUID`. The namespace value is cross-repo coordinated with `satellite-provider/SatelliteProvider.Common/Utils/Uuidv5.cs` per `AZ-TBD_tile_identity_uuidv5_bulk_list`; the same `uuidv5(NAMESPACE, name)` MUST produce byte-identical output on both sides.
|
||||
- The psycopg_pool connection helper at `src/gps_denied_onboard/components/c6_tile_cache/connection.py` (`psycopg_pool(config) -> psycopg_pool.ConnectionPool`), used by both this task's runner and the future `PostgresFilesystemStore`.
|
||||
- **No `psycopg_pool` helper ships in this task.** The migration runner relies on Alembic's existing SQLAlchemy engine (`engine_from_config(..., poolclass=pool.NullPool)`) already wired in `db/migrations/env.py` by AZ-263. `psycopg_pool` is NOT pinned by AZ-263 (only `psycopg[binary]>=3.1` is), so a runtime connection-pool helper (`PostgresFilesystemStore` use-case) introduces a new dependency and is the responsibility of AZ-305 (`c6_postgres_filesystem_store`).
|
||||
- After `apply_migrations(config)` on an AZ-263-baselined DB:
|
||||
- The `tiles` table has six additional columns (`tile_uuid`, `location_hash`, `content_sha256`, `disk_bytes`, `accessed_at`, `uploaded_at`); the AZ-263 columns are unchanged.
|
||||
- The `tiles` table has a new UNIQUE index `idx_tiles_natural_key` over the COALESCE-zero-uuid natural key; the AZ-263 indices are unchanged.
|
||||
@@ -50,7 +50,7 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- A new `tile_freshness_rules` table exists, seeded with the two default rows.
|
||||
- The AZ-303 contract `tile_metadata_store.md` is bumped v1.1.0 → v1.2.0 with the `location_hash: UUID | None` field added to the documented `TileMetadata` shape (non-breaking minor — Optional default `None`, populated by `PostgresFilesystemStore.insert_metadata` from `_uuid_namespace.derive_location_hash` when not supplied).
|
||||
- The DTO `TileMetadata` in `src/gps_denied_onboard/components/c6_tile_cache/_types.py` gains the same `location_hash: UUID | None = None` field (positional last, default value preserves existing constructor call sites).
|
||||
- A schema fixture `tests/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test.
|
||||
- A schema fixture `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test (kept inside c6_tile_cache's owned test directory).
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -60,11 +60,10 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- A `MigrationResult` dataclass `@dataclass(frozen=True)` at `c6_tile_cache.migrations`.
|
||||
- The `apply_migrations(config) -> MigrationResult` runner using the existing project-pinned Alembic env at `db/migrations/` (no new alembic.ini, no new env.py — AZ-263 bootstrap owns those; this task only wires `target_metadata` into `db/migrations/env.py` so future autogenerate diffs work).
|
||||
- The pinned UUIDv5 namespace module `_uuid_namespace.py` with `TILE_NAMESPACE_UUID`, `derive_tile_id`, `derive_location_hash`. No Postgres dependency; pure stdlib `uuid.uuid5`.
|
||||
- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/fixtures/c6_postgres_schema_v2.sql`. Test uses `testcontainers`-managed Postgres 16; no Python logic beyond `information_schema` queries.
|
||||
- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql`. Test is `@pytest.mark.docker` (auto-skipped on Tier-1 per `tests/conftest.py`); it consumes the `db` service from `docker-compose.test.yml` via the `DB_URL` env var. No Python logic beyond `information_schema` queries.
|
||||
- The UUIDv5 determinism test `tests/unit/c6_tile_cache/test_uuid_namespace.py` that locks `TILE_NAMESPACE_UUID` and verifies ≥5 fixed `(z, x, y, source, flight_id)` input vectors produce the documented UUIDv5 outputs. These vectors are the cross-repo coordination evidence — the corresponding `satellite-provider` test MUST produce byte-identical UUIDs.
|
||||
- The connection helper `connection.psycopg_pool(config) -> psycopg_pool.ConnectionPool`.
|
||||
- Wiring of `db/migrations/env.py` `target_metadata` to a `c6_tile_cache.metadata` SQLAlchemy `MetaData` object that reflects both AZ-263 and AZ-304 schema (so future autogenerate diffs are mechanically comparable).
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against.
|
||||
- The `db/migrations/env.py` minimum-touch policy: AZ-263 already wires `engine_from_config` + DB_URL fallback for online mode; this task does NOT add `target_metadata` (we use Alembic `op.*` directly, never `autogenerate`). A `target_metadata` wiring lands when the first task adds SQLAlchemy ORM models (none in this task or AZ-305 as currently scoped).
|
||||
- The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against (lives inside c6_tile_cache's owned test glob per `module-layout.md`).
|
||||
- DTO extension in `_types.py`: `TileMetadata.location_hash: UUID | None = None` (positional last, default `None`).
|
||||
- Contract bump `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` v1.1.0 → v1.2.0 with a Change Log entry.
|
||||
|
||||
@@ -189,7 +188,7 @@ Then `MigrationResult.applied == []`; `MigrationResult.no_op == True`; no DDL is
|
||||
**AC-3: Schema shape matches the documented DDL**
|
||||
Given a DB upgraded through 0001 + 0002
|
||||
When the schema-shape diff test introspects `information_schema.columns` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` row contents
|
||||
Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/fixtures/c6_postgres_schema_v2.sql` is empty.
|
||||
Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is empty.
|
||||
|
||||
**AC-4: Natural-key uniqueness enforces per-flight separation**
|
||||
Given a `tiles` table after 0002
|
||||
@@ -250,8 +249,7 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
|
||||
**Compatibility**
|
||||
- Postgres 16.x.
|
||||
- `psycopg_pool` 3.x — pinned by AZ-263; this task adds no new third-party dependencies.
|
||||
- Alembic 1.13+ — pinned by AZ-263.
|
||||
- `psycopg[binary]>=3.1` and `alembic>=1.13` — both pinned by AZ-263. This task adds **no** new third-party dependencies. `psycopg_pool` is explicitly deferred to AZ-305 since AZ-263 did not pin it.
|
||||
- Cross-workspace UUIDv5 namespace: `TILE_NAMESPACE_UUID` MUST be byte-identical to the satellite-provider C# constant; any change requires a coordinated cross-repo release.
|
||||
|
||||
**Reliability**
|
||||
@@ -262,9 +260,9 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
|
||||
| AC Ref | What to Test | Required Outcome |
|
||||
|--------|-------------|-----------------|
|
||||
| AC-1 | `apply_migrations` against fresh `testcontainer` DB previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. |
|
||||
| AC-1 | `apply_migrations` against the `db` Postgres service (docker-compose.test.yml) previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. |
|
||||
| AC-2 | `apply_migrations` against DB already at 0002 head | `result.applied=[]`; `result.no_op=True`; no DDL emitted. |
|
||||
| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/fixtures/c6_postgres_schema_v2.sql` | Zero diff. |
|
||||
| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` | Zero diff. |
|
||||
| AC-4 | Two `onboard_ingest` INSERTs with same `(z, tile_x, tile_y, tile_size_meters)` and different `flight_id` | Both succeed; same `location_hash`; different `tile_uuid`. |
|
||||
| AC-4b | Two INSERTs with identical natural-key tuple (same `flight_id` or both NULL for `googlemaps`) | Second INSERT raises `psycopg.errors.UniqueViolation`. |
|
||||
| AC-5 | INSERT one row each for `'fresh'`, `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`; INSERT one row with `'bogus'` | First six succeed; last raises `psycopg.errors.CheckViolation`. |
|
||||
@@ -282,12 +280,12 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
## Constraints
|
||||
|
||||
- Postgres 16.x ONLY this cycle; no SQLite / no MySQL fallback.
|
||||
- Alembic + `psycopg_pool` are pinned by AZ-263; this task does NOT introduce new third-party dependencies.
|
||||
- Alembic + `psycopg[binary]` are pinned by AZ-263; this task does NOT introduce new third-party dependencies. `psycopg_pool` is NOT introduced here (deferred to AZ-305).
|
||||
- The migration MUST be reversible (`downgrade` drops the additions cleanly and restores the AZ-263 CHECK) — operator post-flight tooling depends on it for "drop-and-rebuild" flows.
|
||||
- The migration MUST be strictly additive on every AZ-263 column, table, index, and CHECK per `data_model.md` § 6.1 / § 6.3 and `coderule.mdc`. The single allowed constraint mutation is the `ck_tiles_freshness_status` CHECK widening, which is additive in semantic effect.
|
||||
- `pgcrypto` extension is NOT required by 0002 (no `gen_random_uuid()` use); `tile_freshness_rules.classification` is a TEXT PK and the seeded rows are static.
|
||||
- `MigrationError` is NOT a member of the `TileCacheError` family — migrations run before any `c6_tile_cache.errors` consumer is constructed.
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time.
|
||||
- The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time.
|
||||
- The pinned `TILE_NAMESPACE_UUID` MUST NOT be regenerated by this task. The value `5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` is locked here; subsequent edits require a coordinated cross-workspace release.
|
||||
- The `latitude` / `longitude` columns (AZ-263 names) remain advisory; `lat` / `lon` are NOT introduced. The DTO `TileId(zoom_level, lat, lon)` maps via `PostgresFilesystemStore` serialisation; the schema is NOT changed to match the DTO field name.
|
||||
- The `freshness_status` column name (AZ-263) is NOT renamed to `freshness_label`. DTO field `freshness_label: FreshnessLabel` maps via `PostgresFilesystemStore` serialisation.
|
||||
@@ -333,12 +331,12 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
## Runtime Completeness
|
||||
|
||||
- **Named capability**: Postgres 16 deterministic-identity columns + per-flight natural-key UNIQUE + LRU/upload bookkeeping + sector-boundary geometry + per-classification freshness rules — all additive on the AZ-263 baseline (description.md / data_model.md / AC-NEW-3 / AC-NEW-6 / RESTRICT-SAT-2 / 2026-05-12 leftover § 4–5).
|
||||
- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner, real schema-fixture diff test, real `psycopg_pool` connection helper, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`.
|
||||
- **Allowed external stubs**: tests use `testcontainers`-managed Postgres 16 instances (already in the project's test infra per AZ-263); production wiring uses the operator's deployed Postgres.
|
||||
- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner driving Alembic's `command.upgrade(cfg, "head")` against the AZ-263 env, real schema-fixture diff test, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`. **NOT included** in this task: a `psycopg_pool` connection helper (deferred to AZ-305).
|
||||
- **Allowed external stubs**: tests requiring a real Postgres are marked `@pytest.mark.docker` and run against the `db` service from `docker-compose.test.yml` (per the existing AZ-263 test-infra convention — see `tests/conftest.py` auto-skip logic). The schema-shape test reads `DB_URL` from the env at run time; on Tier-1 (`GPS_DENIED_TIER != 2`) the test auto-skips. Production wiring uses the operator's deployed Postgres.
|
||||
- **Unacceptable substitutes**: SQLite "for testing only" — production and test environments MUST both be Postgres 16; raw SQL DDL applied without Alembic (would defeat the version-tracking the runner depends on); a `tile_quality_metadata` validation at the DB layer (would lock the schema to the JSONB shape — the application-side validation is the single source of truth); a non-deterministic `tile_uuid` strategy (would defeat the cross-workspace coordination the namespace pin establishes); any operation that renames, retypes, or drops an AZ-263 column / table / index / CHECK (forbidden per `coderule.mdc` and `data_model.md` § 6.2 / § 6.3); a parallel Alembic env at `src/.../c6_tile_cache/_alembic/` (forbidden — the project uses one alembic env at `db/migrations/` per AZ-263 + `alembic.ini`).
|
||||
|
||||
## Contract
|
||||
|
||||
This task does NOT produce a new contract file — it implements the `tile_metadata_store.md` contract's persistence surface and bumps its version v1.1.0 → v1.2.0 with one non-breaking minor addition (`TileMetadata.location_hash: UUID | None = None`).
|
||||
|
||||
The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract.
|
||||
The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract.
|
||||
@@ -0,0 +1,236 @@
|
||||
# Batch 27 / Cycle 1 — Implementation Report
|
||||
|
||||
**Date**: 2026-05-12
|
||||
**Tasks**: AZ-304 (C6 Postgres schema — additive 0002 migration: identity, LRU, freshness rules)
|
||||
**Story points landed**: 3
|
||||
**Status**: complete (AZ-304 → In Testing)
|
||||
|
||||
## Scope summary
|
||||
|
||||
Single-task batch — final queued 1-pointer of cycle 1. AZ-304 ships the
|
||||
strictly-additive `0002_c6_tile_identity_and_lru.py` migration on top of
|
||||
the AZ-263 baseline, the pinned UUIDv5 namespace + derivation helpers
|
||||
cross-coordinated with `satellite-provider`, the migration runner the
|
||||
composition root invokes at startup, and bumps the AZ-303
|
||||
`TileMetadataStore` contract v1.1.0 → v1.2.0 by adding the
|
||||
`TileMetadata.location_hash: UUID | None` field.
|
||||
|
||||
Before implementation, the spec was rewritten under user-approved
|
||||
Option A to drop the renames / retypes / drops the original draft
|
||||
called for. The as-built migration only adds columns, indices, the
|
||||
`tile_freshness_rules` table, and widens (loosens) the
|
||||
`ck_tiles_freshness_status` CHECK to the UNION of AZ-263 and AZ-303
|
||||
vocabularies. No AZ-263 column, table, index, or CHECK is renamed,
|
||||
retyped, or dropped (data_model.md § 6.1 / § 6.3).
|
||||
|
||||
## Files added / modified
|
||||
|
||||
### New (production)
|
||||
|
||||
- `src/gps_denied_onboard/components/c6_tile_cache/_uuid_namespace.py` —
|
||||
pinned `TILE_NAMESPACE_UUID = 5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0`,
|
||||
`derive_tile_id(zoom, x, y, source, flight_id)` (UUIDv5 over
|
||||
`"{z}/{x}/{y}/{source}/{flight_id_or_zero_uuid}"`),
|
||||
`derive_location_hash(z, x, y)`. Cross-workspace coordination point
|
||||
with `satellite-provider/SatelliteProvider.Common/Utils/Uuidv5.cs`.
|
||||
- `src/gps_denied_onboard/components/c6_tile_cache/migrations.py` —
|
||||
`apply_migrations(config) -> MigrationResult` runner, frozen
|
||||
`MigrationResult` dataclass, `MigrationError`. Drives Alembic
|
||||
`command.upgrade(cfg, "head")` against the AZ-263 env at
|
||||
`db/migrations/`; one retry on PG SQLSTATE `40001`
|
||||
(serialization failure); structured INFO log on apply / no-op with
|
||||
the resolved namespace UUID emitted for post-mortem drift detection.
|
||||
- `db/migrations/versions/0002_c6_tile_identity_and_lru.py` — additive
|
||||
Alembic migration: six new `tiles` columns (`tile_uuid` UNIQUE,
|
||||
`location_hash`, `content_sha256` w/ length-64 CHECK, `disk_bytes`
|
||||
w/ nonneg CHECK, `accessed_at`, `uploaded_at`); four new btree
|
||||
indices on `tiles`; one UNIQUE expression index
|
||||
`idx_tiles_natural_key` over the COALESCE-zero-uuid natural key;
|
||||
CHECK-widening of `ck_tiles_freshness_status` to the AZ-263 +
|
||||
AZ-303 UNION vocabulary; four NULLable bbox columns on
|
||||
`sector_classifications`; new `tile_freshness_rules` table seeded
|
||||
with the two default thresholds (active_conflict / reject @ 6m,
|
||||
stable_rear / downgrade @ 12m). Reverse `downgrade()` drops every
|
||||
addition and restores the AZ-263 freshness CHECK to its original
|
||||
vocabulary.
|
||||
|
||||
### Modified (production)
|
||||
|
||||
- `src/gps_denied_onboard/components/c6_tile_cache/_types.py` —
|
||||
`TileMetadata.location_hash: UUID | None = None` (positional last,
|
||||
default `None` so AZ-303 v1.1.0 constructors keep working; the
|
||||
impl-task's `PostgresFilesystemStore.insert_metadata` derives the
|
||||
value when `None` and the DB NOT-NULL constraint is the safety
|
||||
net).
|
||||
- `db/migrations/env.py` — docstring tightened to document the
|
||||
`target_metadata` deferral (we use Alembic `op.*` directly; ORM
|
||||
models land in a later cycle).
|
||||
|
||||
### New (tests)
|
||||
|
||||
- `tests/unit/c6_tile_cache/test_uuid_namespace.py` — 8 deterministic
|
||||
Tier-1 tests covering AC-10 (5 locked tile-id vectors, 2 locked
|
||||
location-hash vectors, idempotent calls, `TileSource` enum /
|
||||
`UUID`-typed / `None`-flight inputs, malformed-flight rejection)
|
||||
and AC-11 (location-hash invariance across source/flight, distinct
|
||||
cells distinct hashes).
|
||||
- `tests/unit/c6_tile_cache/test_postgres_schema.py` — `@pytest.mark.
|
||||
docker` integration tests covering AC-1 through AC-9 + NFR-perf-
|
||||
apply / NFR-perf-noop + `MigrationResult` frozen smoke. Reads
|
||||
`DB_URL` env var; consumes the `db` service from `docker-compose.
|
||||
test.yml`. Auto-skipped on Tier-1 by `tests/conftest.py`.
|
||||
- `tests/unit/c6_tile_cache/test_migrations_runner.py` — 5 Tier-1
|
||||
tests covering DSN resolution (config-block, env fallback, missing
|
||||
raises `MigrationError`) and the SQLSTATE-40001 retry / terminal
|
||||
paths (mocked DBAPIError to keep them deterministic and DB-less).
|
||||
- `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` —
|
||||
copy-pastable DDL fixture the AC-3 schema-shape diff test diffs
|
||||
against. Includes every AZ-263 column / index / CHECK plus every
|
||||
additive item from this migration plus the two seeded
|
||||
`tile_freshness_rules` rows.
|
||||
|
||||
### Modified (tests + docs)
|
||||
|
||||
- `tests/unit/test_ac5_alembic.py` — head-revision assertion bumped
|
||||
from `0001_initial` to `0002_c6_tile_identity_and_lru`; function
|
||||
renamed `test_head_revision_is_0001_initial` →
|
||||
`test_head_revision_matches_latest_migration`.
|
||||
- `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md`
|
||||
— version bumped v1.1.0 → v1.2.0; new `TileMetadata.location_hash`
|
||||
section; `sector_boundaries` → `sector_classifications`
|
||||
corrections; Alembic migration path corrected from
|
||||
`c6_tile_cache/_alembic/` to `db/migrations/versions/`; Change Log
|
||||
v1.2.0 entry.
|
||||
- `_docs/02_document/module-layout.md` — `c6_tile_cache` component
|
||||
now explicitly `Owns` `db/migrations/**` (Alembic env was already
|
||||
living there since AZ-263 bootstrap, but not documented).
|
||||
`migrations.py` / `_uuid_namespace.py` listed under the component's
|
||||
internal modules.
|
||||
- `_docs/02_tasks/todo/AZ-304_c6_postgres_schema.md` → moved to
|
||||
`_docs/02_tasks/done/`. Spec was rewritten earlier this batch
|
||||
(Option A: strictly-additive 0002 + descope of
|
||||
`psycopg_pool`/`connection.py` to AZ-305 + replacement of
|
||||
`testcontainers` with the existing `@pytest.mark.docker` /
|
||||
`docker-compose.test.yml` infra).
|
||||
|
||||
## Design decisions
|
||||
|
||||
1. **Strictly additive 0002, no rename / retype / drop.** User picked
|
||||
Option A on the schema-reconciliation gate. Every AZ-263 column,
|
||||
table, index, and CHECK survives 0002 byte-identical. The single
|
||||
constraint mutation is the `ck_tiles_freshness_status` CHECK
|
||||
widening, which is a semantic loosening (more values accepted) and
|
||||
therefore additive per `data_model.md` § 6.1. Legacy AZ-263 values
|
||||
(`stale_warn`, `stale_reject`) remain valid until an ADR-gated
|
||||
future cycle deprecates them. DTO ↔ column mapping
|
||||
(`freshness_label` ↔ `freshness_status`, `quality_metadata` ↔
|
||||
`tile_quality_metadata`, `(lat, lon)` ↔ `(latitude, longitude)`)
|
||||
is owned by the future `PostgresFilesystemStore` task (AZ-305) —
|
||||
the schema does NOT pivot to DTO field names.
|
||||
|
||||
2. **UUIDv5 namespace pinned in code, not config.** `TILE_NAMESPACE_
|
||||
UUID = 5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` lives in the
|
||||
`_uuid_namespace` module as a `Final` constant — not in
|
||||
environment / config / DB — so cross-workspace coordination with
|
||||
the C# `satellite-provider` reduces to a code-level review check
|
||||
on either side. The runner logs the namespace value on every
|
||||
apply / no-op so production drift would surface immediately.
|
||||
|
||||
3. **`psycopg_pool` deferred to AZ-305.** The original spec assumed
|
||||
AZ-263 pinned `psycopg_pool`; reality check showed only
|
||||
`psycopg[binary]>=3.1` and `alembic>=1.13` are pinned. A new
|
||||
third-party dependency would inflate this task's scope and risk;
|
||||
moved to AZ-305 (`c6_postgres_filesystem_store`) where the
|
||||
runtime connection pool is genuinely needed. The migration runner
|
||||
uses Alembic's existing `NullPool` SQLAlchemy engine wired by
|
||||
AZ-263 — no pool needed for one-shot startup migrations.
|
||||
|
||||
4. **Retry-without-sleep on serialization failure.** The runner
|
||||
retries once on PG SQLSTATE `40001`, but without a `time.sleep`
|
||||
backoff. Component files are forbidden from calling `time.*`
|
||||
directly (`tests/_meta/test_no_direct_time_in_components.py`,
|
||||
Invariant 2 of the replay contract), and migrations run before the
|
||||
injected `Clock` is constructed, so `Clock.sleep` is unavailable
|
||||
too. Alembic's `NullPool` opens a fresh connection on retry, which
|
||||
already introduces natural jitter; a 0-50 ms backoff buys nothing
|
||||
measurable. Documented in `migrations.py` inline.
|
||||
|
||||
5. **`MigrationError` NOT a member of `TileCacheError` family.**
|
||||
Migrations run BEFORE the runtime error consumer (`c6_tile_cache.
|
||||
errors`) is constructed — making `MigrationError` a `TileCacheError`
|
||||
would create a forward-import cycle and conceptually mis-place a
|
||||
startup-phase failure inside a runtime-phase error tree. Documented
|
||||
in the runner module docstring + verified by
|
||||
`test_migration_error_is_not_tile_cache_error`.
|
||||
|
||||
6. **`module-layout.md` updated as scope-creep, user-approved.** Drift
|
||||
gate surfaced that `module-layout.md` referenced a `_alembic/`
|
||||
directory and `.sql` migration files; reality is `db/migrations/`
|
||||
`.py` files and the doc didn't claim ownership of any of it. User
|
||||
picked Option A to expand AZ-304 scope by one doc edit:
|
||||
`c6_tile_cache` now explicitly `Owns` `db/migrations/**`.
|
||||
|
||||
## AC coverage
|
||||
|
||||
| AC | Test name(s) | Status |
|
||||
|----|--------------|--------|
|
||||
| AC-1 | `test_ac1_apply_to_az263_baseline_advances_alembic_version` | Tier-2 (docker) |
|
||||
| AC-2 | `test_ac2_apply_at_head_is_no_op` | Tier-2 (docker) |
|
||||
| AC-3 | `test_ac3_schema_shape_diffs_clean_against_fixture` | Tier-2 (docker) |
|
||||
| AC-4 | `test_ac4_per_flight_separation_allowed` | Tier-2 (docker) |
|
||||
| AC-4b | `test_ac4b_duplicate_natural_key_rejected` + `..._googlemaps` | Tier-2 (docker) |
|
||||
| AC-5 | `test_ac5_freshness_check_widening_accepts_union_vocabulary` | Tier-2 (docker) |
|
||||
| AC-6 | `test_ac6_downgrade_reverses_cleanly` | Tier-2 (docker) |
|
||||
| AC-7 | `test_ac7_default_freshness_rules_seeded` | Tier-2 (docker) |
|
||||
| AC-8 | `test_ac8_runner_logs_info_on_apply_and_no_op` | Tier-2 (docker) |
|
||||
| AC-9 | `test_ac9_az263_columns_byte_identical_after_upgrade` | Tier-2 (docker) |
|
||||
| AC-10 | `test_ac10_namespace_locked_and_locked_vectors_match` + companions | Tier-1 passing |
|
||||
| AC-11 | `test_ac11_location_hash_invariant_across_source_and_flight_id` + companion | Tier-1 passing |
|
||||
| AC-12 | covered by AZ-303 `test_protocol_conformance.py` (DTO default still `None`) | Tier-1 passing |
|
||||
| NFR-perf-apply | `test_nfr_perf_apply_under_5s_on_baselined_empty_db` | Tier-2 (docker) |
|
||||
| NFR-perf-noop | `test_nfr_perf_noop_under_100ms` | Tier-2 (docker) |
|
||||
| NFR-reliability-retry | `test_nfr_reliability_retry_once_on_serialization_failure` + companion | Tier-1 passing |
|
||||
|
||||
The Tier-2 docker tests are explicitly skipped on Tier-1 by the
|
||||
project-wide `tests/conftest.py` mark-filter; they run locally via
|
||||
`docker compose -f docker-compose.test.yml up -d db && GPS_DENIED_TIER=2
|
||||
DB_URL=postgresql://gps_denied:dev@localhost:5432/gps_denied pytest
|
||||
tests/unit/c6_tile_cache/test_postgres_schema.py`.
|
||||
|
||||
## Test run
|
||||
|
||||
`python3 -m pytest tests/ -q` → **1180 passed, 32 skipped (Tier-2 /
|
||||
CUDA / cmake / actionlint / docker), 1 warning** in 21.77 s, no
|
||||
failures.
|
||||
|
||||
Mypy strict on production + test modules: clean.
|
||||
Ruff + ruff format on the modified set: clean.
|
||||
|
||||
## Self-review
|
||||
|
||||
- Production code: deterministic UUIDv5 with locked namespace +
|
||||
documented cross-repo invariants; no `time.*` calls in component
|
||||
files (Invariant 2 meta-test passes); no new third-party deps;
|
||||
migration is byte-additive on the AZ-263 baseline; runner emits
|
||||
structured INFO logs with `kind` + `kv` per the AZ-266 log schema;
|
||||
`MigrationError` correctly outside the `TileCacheError` family.
|
||||
- Tests: every AC has at least one named assertion (Tier-1 or Tier-2);
|
||||
cross-repo coordination vectors are locked with documented expected
|
||||
UUIDs; no hidden runtime dependencies in Tier-1 (mocked DBAPIError
|
||||
for the retry path).
|
||||
- Lint / type: ruff + mypy strict clean on the modified set.
|
||||
- Docs: AZ-304 spec moved to `done/`; contract bumped v1.2.0;
|
||||
`module-layout.md` reconciled with reality.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- Tier-2 Postgres tests need a real `db` service; rolled forward to
|
||||
the Tier-2 validation pass.
|
||||
- `psycopg_pool` connection helper deferred to AZ-305 — documented in
|
||||
both this report and the AZ-304 spec.
|
||||
- `target_metadata` wiring in `db/migrations/env.py` deferred until
|
||||
the first task introduces SQLAlchemy ORM models (none today,
|
||||
none in AZ-305 as currently scoped).
|
||||
- AZ-263 legacy CHECK values (`stale_warn`, `stale_reject`) remain
|
||||
valid in `ck_tiles_freshness_status`; deprecation requires a future
|
||||
ADR-gated cleanup migration.
|
||||
@@ -8,7 +8,7 @@ status: in_progress
|
||||
sub_step:
|
||||
phase: 3
|
||||
name: compute-batch
|
||||
detail: "batch 27/cycle1: AZ-304 spec rewritten per Option A (strict-additive 0002 on AZ-263)"
|
||||
detail: "batch 27/cycle1: AZ-304 complete (additive 0002 migration + UUIDv5 namespace + runner + v1.2.0 contract bump). Awaiting next batch selection."
|
||||
retry_count: 0
|
||||
cycle: 1
|
||||
tracker: jira
|
||||
|
||||
Reference in New Issue
Block a user