mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 14:21:14 +00:00
[AZ-304] C6 Postgres schema: additive 0002 migration + UUIDv5
Strictly additive Alembic migration on the AZ-263 baseline (data_model
.md § 6.1 / § 6.3): six new tiles columns (tile_uuid UNIQUE,
location_hash, content_sha256, disk_bytes, accessed_at, uploaded_at),
four new btree indices, one UNIQUE expression index over the
COALESCE-zero-uuid natural key, CHECK widening of
ck_tiles_freshness_status to the AZ-263 + AZ-303 vocabulary UNION,
four NULLable bbox columns on sector_classifications, and a new
tile_freshness_rules table seeded with the two default thresholds.
Pinned UUIDv5 namespace (TILE_NAMESPACE_UUID =
5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0) + derive_tile_id /
derive_location_hash helpers cross-coordinated with
satellite-provider. Migration runner apply_migrations(config) drives
Alembic command.upgrade("head") against the AZ-263 env with one
retry on PG SQLSTATE 40001 and structured INFO logs on apply / no-op.
Contract bump tile_metadata_store.md v1.1.0 -> v1.2.0 adds
TileMetadata.location_hash: UUID | None = None (non-breaking).
module-layout.md updated so c6_tile_cache explicitly Owns
db/migrations/**.
Tier-1 tests: UUIDv5 determinism + locked vectors + DSN resolution +
retry mocked DBAPIError -> 1180 passed, 32 skipped. Tier-2 docker
schema tests gated by @pytest.mark.docker run against the existing
docker-compose.test.yml db service.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+14
-16
@@ -40,7 +40,7 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- An Alembic migration script at `db/migrations/versions/0002_c6_tile_identity_and_lru.py` (forward `upgrade()` is purely additive; reverse `downgrade()` drops the additions and restores the original AZ-263 `freshness_status` CHECK). The migration is idempotent against a DB at AZ-263 head; Alembic rejects double-application via the standard `alembic_version` row.
|
||||
- The migration runner `apply_migrations(config) -> MigrationResult` at `src/gps_denied_onboard/components/c6_tile_cache/migrations.py`, invoked by the composition root at startup AFTER config load and BEFORE `PostgresFilesystemStore` construction. Returns `MigrationResult(applied: list[str], current_revision: str, no_op: bool)`. Logs INFO on every applied revision; logs INFO with `no_op=True` when the DB is already at head.
|
||||
- The pinned UUIDv5 namespace module at `src/gps_denied_onboard/components/c6_tile_cache/_uuid_namespace.py` exporting `TILE_NAMESPACE_UUID = UUID("5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0")`, `derive_tile_id(zoom_level, tile_x, tile_y, source, flight_id) -> UUID`, and `derive_location_hash(zoom_level, tile_x, tile_y) -> UUID`. The namespace value is cross-repo coordinated with `satellite-provider/SatelliteProvider.Common/Utils/Uuidv5.cs` per `AZ-TBD_tile_identity_uuidv5_bulk_list`; the same `uuidv5(NAMESPACE, name)` MUST produce byte-identical output on both sides.
|
||||
- The psycopg_pool connection helper at `src/gps_denied_onboard/components/c6_tile_cache/connection.py` (`psycopg_pool(config) -> psycopg_pool.ConnectionPool`), used by both this task's runner and the future `PostgresFilesystemStore`.
|
||||
- **No `psycopg_pool` helper ships in this task.** The migration runner relies on Alembic's existing SQLAlchemy engine (`engine_from_config(..., poolclass=pool.NullPool)`) already wired in `db/migrations/env.py` by AZ-263. `psycopg_pool` is NOT pinned by AZ-263 (only `psycopg[binary]>=3.1` is), so a runtime connection-pool helper (`PostgresFilesystemStore` use-case) introduces a new dependency and is the responsibility of AZ-305 (`c6_postgres_filesystem_store`).
|
||||
- After `apply_migrations(config)` on an AZ-263-baselined DB:
|
||||
- The `tiles` table has six additional columns (`tile_uuid`, `location_hash`, `content_sha256`, `disk_bytes`, `accessed_at`, `uploaded_at`); the AZ-263 columns are unchanged.
|
||||
- The `tiles` table has a new UNIQUE index `idx_tiles_natural_key` over the COALESCE-zero-uuid natural key; the AZ-263 indices are unchanged.
|
||||
@@ -50,7 +50,7 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- A new `tile_freshness_rules` table exists, seeded with the two default rows.
|
||||
- The AZ-303 contract `tile_metadata_store.md` is bumped v1.1.0 → v1.2.0 with the `location_hash: UUID | None` field added to the documented `TileMetadata` shape (non-breaking minor — Optional default `None`, populated by `PostgresFilesystemStore.insert_metadata` from `_uuid_namespace.derive_location_hash` when not supplied).
|
||||
- The DTO `TileMetadata` in `src/gps_denied_onboard/components/c6_tile_cache/_types.py` gains the same `location_hash: UUID | None = None` field (positional last, default value preserves existing constructor call sites).
|
||||
- A schema fixture `tests/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test.
|
||||
- A schema fixture `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the human-readable expected post-0002 DDL used by the schema-shape diff test (kept inside c6_tile_cache's owned test directory).
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -60,11 +60,10 @@ This task delivers the strictly-additive `0002` migration that closes those gaps
|
||||
- A `MigrationResult` dataclass `@dataclass(frozen=True)` at `c6_tile_cache.migrations`.
|
||||
- The `apply_migrations(config) -> MigrationResult` runner using the existing project-pinned Alembic env at `db/migrations/` (no new alembic.ini, no new env.py — AZ-263 bootstrap owns those; this task only wires `target_metadata` into `db/migrations/env.py` so future autogenerate diffs work).
|
||||
- The pinned UUIDv5 namespace module `_uuid_namespace.py` with `TILE_NAMESPACE_UUID`, `derive_tile_id`, `derive_location_hash`. No Postgres dependency; pure stdlib `uuid.uuid5`.
|
||||
- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/fixtures/c6_postgres_schema_v2.sql`. Test uses `testcontainers`-managed Postgres 16; no Python logic beyond `information_schema` queries.
|
||||
- The schema-shape diff test `tests/unit/c6_tile_cache/test_postgres_schema.py` that introspects a freshly-migrated test DB (Alembic upgraded to `0002` head) and asserts every column, index, CHECK constraint, and seed row matches `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql`. Test is `@pytest.mark.docker` (auto-skipped on Tier-1 per `tests/conftest.py`); it consumes the `db` service from `docker-compose.test.yml` via the `DB_URL` env var. No Python logic beyond `information_schema` queries.
|
||||
- The UUIDv5 determinism test `tests/unit/c6_tile_cache/test_uuid_namespace.py` that locks `TILE_NAMESPACE_UUID` and verifies ≥5 fixed `(z, x, y, source, flight_id)` input vectors produce the documented UUIDv5 outputs. These vectors are the cross-repo coordination evidence — the corresponding `satellite-provider` test MUST produce byte-identical UUIDs.
|
||||
- The connection helper `connection.psycopg_pool(config) -> psycopg_pool.ConnectionPool`.
|
||||
- Wiring of `db/migrations/env.py` `target_metadata` to a `c6_tile_cache.metadata` SQLAlchemy `MetaData` object that reflects both AZ-263 and AZ-304 schema (so future autogenerate diffs are mechanically comparable).
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against.
|
||||
- The `db/migrations/env.py` minimum-touch policy: AZ-263 already wires `engine_from_config` + DB_URL fallback for online mode; this task does NOT add `target_metadata` (we use Alembic `op.*` directly, never `autogenerate`). A `target_metadata` wiring lands when the first task adds SQLAlchemy ORM models (none in this task or AZ-305 as currently scoped).
|
||||
- The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` — copy-pastable DDL the test diffs against (lives inside c6_tile_cache's owned test glob per `module-layout.md`).
|
||||
- DTO extension in `_types.py`: `TileMetadata.location_hash: UUID | None = None` (positional last, default `None`).
|
||||
- Contract bump `_docs/02_document/contracts/c6_tile_cache/tile_metadata_store.md` v1.1.0 → v1.2.0 with a Change Log entry.
|
||||
|
||||
@@ -189,7 +188,7 @@ Then `MigrationResult.applied == []`; `MigrationResult.no_op == True`; no DDL is
|
||||
**AC-3: Schema shape matches the documented DDL**
|
||||
Given a DB upgraded through 0001 + 0002
|
||||
When the schema-shape diff test introspects `information_schema.columns` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` row contents
|
||||
Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/fixtures/c6_postgres_schema_v2.sql` is empty.
|
||||
Then every AZ-263 column / index / CHECK is present and unchanged; every additive column / index / CHECK from this task is present; the `ck_tiles_freshness_status` CHECK contains the UNION vocabulary; `tile_freshness_rules` has exactly two seeded rows with the documented values. Diff against `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is empty.
|
||||
|
||||
**AC-4: Natural-key uniqueness enforces per-flight separation**
|
||||
Given a `tiles` table after 0002
|
||||
@@ -250,8 +249,7 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
|
||||
**Compatibility**
|
||||
- Postgres 16.x.
|
||||
- `psycopg_pool` 3.x — pinned by AZ-263; this task adds no new third-party dependencies.
|
||||
- Alembic 1.13+ — pinned by AZ-263.
|
||||
- `psycopg[binary]>=3.1` and `alembic>=1.13` — both pinned by AZ-263. This task adds **no** new third-party dependencies. `psycopg_pool` is explicitly deferred to AZ-305 since AZ-263 did not pin it.
|
||||
- Cross-workspace UUIDv5 namespace: `TILE_NAMESPACE_UUID` MUST be byte-identical to the satellite-provider C# constant; any change requires a coordinated cross-repo release.
|
||||
|
||||
**Reliability**
|
||||
@@ -262,9 +260,9 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
|
||||
| AC Ref | What to Test | Required Outcome |
|
||||
|--------|-------------|-----------------|
|
||||
| AC-1 | `apply_migrations` against fresh `testcontainer` DB previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. |
|
||||
| AC-1 | `apply_migrations` against the `db` Postgres service (docker-compose.test.yml) previously upgraded to AZ-263 head | All additive columns / indices / `tile_freshness_rules` table exist; `alembic_version='0002_c6_tile_identity_and_lru'`; `result.applied=['0002_c6_tile_identity_and_lru']`. AZ-263 columns / indices / CHECKs byte-identical to pre-migration snapshot. |
|
||||
| AC-2 | `apply_migrations` against DB already at 0002 head | `result.applied=[]`; `result.no_op=True`; no DDL emitted. |
|
||||
| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/fixtures/c6_postgres_schema_v2.sql` | Zero diff. |
|
||||
| AC-3 | Introspect `information_schema` / `pg_indexes` / `pg_constraint` / `tile_freshness_rules` rows; diff against `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` | Zero diff. |
|
||||
| AC-4 | Two `onboard_ingest` INSERTs with same `(z, tile_x, tile_y, tile_size_meters)` and different `flight_id` | Both succeed; same `location_hash`; different `tile_uuid`. |
|
||||
| AC-4b | Two INSERTs with identical natural-key tuple (same `flight_id` or both NULL for `googlemaps`) | Second INSERT raises `psycopg.errors.UniqueViolation`. |
|
||||
| AC-5 | INSERT one row each for `'fresh'`, `'stale_warn'`, `'stale_reject'`, `'stale_active_conflict'`, `'stale_rear'`, `'downgraded'`; INSERT one row with `'bogus'` | First six succeed; last raises `psycopg.errors.CheckViolation`. |
|
||||
@@ -282,12 +280,12 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
## Constraints
|
||||
|
||||
- Postgres 16.x ONLY this cycle; no SQLite / no MySQL fallback.
|
||||
- Alembic + `psycopg_pool` are pinned by AZ-263; this task does NOT introduce new third-party dependencies.
|
||||
- Alembic + `psycopg[binary]` are pinned by AZ-263; this task does NOT introduce new third-party dependencies. `psycopg_pool` is NOT introduced here (deferred to AZ-305).
|
||||
- The migration MUST be reversible (`downgrade` drops the additions cleanly and restores the AZ-263 CHECK) — operator post-flight tooling depends on it for "drop-and-rebuild" flows.
|
||||
- The migration MUST be strictly additive on every AZ-263 column, table, index, and CHECK per `data_model.md` § 6.1 / § 6.3 and `coderule.mdc`. The single allowed constraint mutation is the `ck_tiles_freshness_status` CHECK widening, which is additive in semantic effect.
|
||||
- `pgcrypto` extension is NOT required by 0002 (no `gen_random_uuid()` use); `tile_freshness_rules.classification` is a TEXT PK and the seeded rows are static.
|
||||
- `MigrationError` is NOT a member of the `TileCacheError` family — migrations run before any `c6_tile_cache.errors` consumer is constructed.
|
||||
- The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time.
|
||||
- The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the diff target; updating it without a new migration revision is a Spec-Gap finding (High) at code-review time.
|
||||
- The pinned `TILE_NAMESPACE_UUID` MUST NOT be regenerated by this task. The value `5b8d0c2e-1a4f-4b3a-8c9d-e7f6a3b2c1d0` is locked here; subsequent edits require a coordinated cross-workspace release.
|
||||
- The `latitude` / `longitude` columns (AZ-263 names) remain advisory; `lat` / `lon` are NOT introduced. The DTO `TileId(zoom_level, lat, lon)` maps via `PostgresFilesystemStore` serialisation; the schema is NOT changed to match the DTO field name.
|
||||
- The `freshness_status` column name (AZ-263) is NOT renamed to `freshness_label`. DTO field `freshness_label: FreshnessLabel` maps via `PostgresFilesystemStore` serialisation.
|
||||
@@ -333,12 +331,12 @@ Then construction succeeds with `location_hash = None`; when constructors supply
|
||||
## Runtime Completeness
|
||||
|
||||
- **Named capability**: Postgres 16 deterministic-identity columns + per-flight natural-key UNIQUE + LRU/upload bookkeeping + sector-boundary geometry + per-classification freshness rules — all additive on the AZ-263 baseline (description.md / data_model.md / AC-NEW-3 / AC-NEW-6 / RESTRICT-SAT-2 / 2026-05-12 leftover § 4–5).
|
||||
- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner, real schema-fixture diff test, real `psycopg_pool` connection helper, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`.
|
||||
- **Allowed external stubs**: tests use `testcontainers`-managed Postgres 16 instances (already in the project's test infra per AZ-263); production wiring uses the operator's deployed Postgres.
|
||||
- **Production code that must exist**: real Alembic migration `0002_c6_tile_identity_and_lru.py`, real `apply_migrations` runner driving Alembic's `command.upgrade(cfg, "head")` against the AZ-263 env, real schema-fixture diff test, real `_uuid_namespace` module with `TILE_NAMESPACE_UUID` constant and `derive_tile_id` / `derive_location_hash` helpers, real DTO extension in `_types.py`, real contract bump in `tile_metadata_store.md`. **NOT included** in this task: a `psycopg_pool` connection helper (deferred to AZ-305).
|
||||
- **Allowed external stubs**: tests requiring a real Postgres are marked `@pytest.mark.docker` and run against the `db` service from `docker-compose.test.yml` (per the existing AZ-263 test-infra convention — see `tests/conftest.py` auto-skip logic). The schema-shape test reads `DB_URL` from the env at run time; on Tier-1 (`GPS_DENIED_TIER != 2`) the test auto-skips. Production wiring uses the operator's deployed Postgres.
|
||||
- **Unacceptable substitutes**: SQLite "for testing only" — production and test environments MUST both be Postgres 16; raw SQL DDL applied without Alembic (would defeat the version-tracking the runner depends on); a `tile_quality_metadata` validation at the DB layer (would lock the schema to the JSONB shape — the application-side validation is the single source of truth); a non-deterministic `tile_uuid` strategy (would defeat the cross-workspace coordination the namespace pin establishes); any operation that renames, retypes, or drops an AZ-263 column / table / index / CHECK (forbidden per `coderule.mdc` and `data_model.md` § 6.2 / § 6.3); a parallel Alembic env at `src/.../c6_tile_cache/_alembic/` (forbidden — the project uses one alembic env at `db/migrations/` per AZ-263 + `alembic.ini`).
|
||||
|
||||
## Contract
|
||||
|
||||
This task does NOT produce a new contract file — it implements the `tile_metadata_store.md` contract's persistence surface and bumps its version v1.1.0 → v1.2.0 with one non-breaking minor addition (`TileMetadata.location_hash: UUID | None = None`).
|
||||
|
||||
The schema-fixture file `tests/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract.
|
||||
The schema-fixture file `tests/unit/c6_tile_cache/fixtures/c6_postgres_schema_v2.sql` is the diff target referenced in `tile_metadata_store.md` § Test Cases (`schema-shape-fixture-diff`) — but the contract document of record stays the Protocol contract.
|
||||
Reference in New Issue
Block a user