mirror of
https://github.com/azaion/satellite-provider.git
synced 2026-06-21 15:11:14 +00:00
[AZ-503] Tile identity → UUIDv5 + integer UPSERT (foundation)
Foundation half of original AZ-503 (split during /autodev step 10 batch 2
on user choice; deferred work moved to AZ-505 with a Blocks link).
Adds deterministic tile identity (UUIDv5 over (z, x, y, source, flight_id))
shared cross-repo with gps-denied-onboard via the pinned TileNamespace
5b8d0c2e-7f1a-4d3b-9c5e-1f3a8e7d2b6c, switches the tiles UPSERT key from
floats to integers with per-flight separation, plumbs FlightId through
UavTileMetadata + handler, and writes UAV evidence to per-flight
on-disk directories so two flights at the same (z, x, y) coexist.
- Common: pure-C# RFC 9562 Uuidv5 (no third-party dep) + FlightId DTO
field; 10 Python-reference unit vectors verify byte parity.
- DataAccess: migration 014 adds flight_id (uuid NULL), location_hash
(uuid NOT NULL, backfilled via session-scoped pg_temp.uuidv5),
content_sha256 (bytea NULL), legacy_id (uuid NULL = preserves
pre-AZ-503 random id one cycle); drops idx_tiles_unique_location_source
(AZ-484) and adds idx_tiles_unique_identity keyed on
(tile_zoom, tile_x, tile_y, tile_size_meters, source,
COALESCE(flight_id, '00000000-...'::uuid)) + idx_tiles_location_hash.
- TileRepository: ColumnList + UPSERT updated; id never updated on
conflict (preserves AC-2 idempotence). UpdateAsync extended.
- Services: TileService and UavTileUploadHandler compute deterministic
Id + LocationHash + ContentSha256 before insert; UAV file path
becomes ./tiles/uav/{flight_id or 'none'}/{z}/{x}/{y}.jpg.
- Tests: Uuidv5Tests (10 reference vectors), UavTileFilePathTests
(per-flight + anonymous paths), UavTileUploadHandlerTests (AC-2,
AC-3, AC-7, AC-11 unit-level), UavUploadTests (AC-3 + AC-4
integration: multi-flight DB coexistence with shared location_hash
+ distinct file_path; float-different lat/lon collapse to 1 row),
MigrationTests (column shape, idx_tiles_unique_identity supersedes
AZ-484 index, deterministic backfill).
- IntegrationTests project references Common to reuse Uuidv5 in raw
SQL seeds.
- AZ-488 MultiSourceCoexistence seed fixed to populate location_hash
(otherwise migration 014's NOT NULL constraint fails).
ACs covered: AC-1, AC-2, AC-3, AC-4, AC-7, AC-8, AC-11.
ACs deferred to AZ-505: AC-5, AC-6, AC-9, AC-10, AC-12.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,98 @@
|
||||
# Batch Report
|
||||
|
||||
**Batch**: 02 (cycle 5)
|
||||
**Tasks**: AZ-503 — Tile identity → UUIDv5 + integer UPSERT (foundation)
|
||||
**Date**: 2026-05-12
|
||||
|
||||
## Scope Note (carryover from /autodev step 10)
|
||||
|
||||
The original AZ-503 spec (3 SP) was reconciled against the live codebase at the start of this batch. Three contradictions surfaced (`flight_id`, `FlightId` DTO field, `voting_status` column all missing) pushing combined work to ~5 SP. The user chose Option C: split AZ-503 into **AZ-503-foundation** (this batch) + **AZ-505** (inventory endpoint + HTTP/2 + leaflet covering index, blocked-linked to AZ-503). Original AC numbering preserved; deferred ACs are flagged `[→ AZ-505]` in the task file. See AZ-503 Jira comment and `_docs/02_tasks/_dependencies_table.md` for the split decision.
|
||||
|
||||
## Task Results
|
||||
|
||||
| Task | Status | Files Modified | Tests | AC Coverage | Issues |
|
||||
|------|--------|----------------|-------|-------------|--------|
|
||||
| AZ-503_tile_identity_uuidv5_bulk_list (foundation) | Done | 13 files (2 new, 11 modified) | unit + integration pass (UAV path); migration verified end-to-end against live DB | 7/7 in-scope ACs covered (AC-1, AC-2, AC-3, AC-4, AC-7, AC-8, AC-11). 5 ACs deferred to AZ-505. | None blocking. One Low finding (see below). |
|
||||
|
||||
## Changes
|
||||
|
||||
### Production code
|
||||
|
||||
- **`SatelliteProvider.Common/Utils/Uuidv5.cs`** (NEW, 80 LoC) — pure-C# RFC 9562 §5.5 (SHA-1) UUIDv5. Pinned `TileNamespace = 5b8d0c2e-7f1a-4d3b-9c5e-1f3a8e7d2b6c` (must be mirrored by `gps-denied-onboard/components/c6_tile_cache/_uuid.py`). Explicit big-endian conversion via `BinaryPrimitives` because .NET's `Guid.ToByteArray()` returns mixed-endian (RFC 4122 Microsoft layout); SHA-1 requires network order to match Python `uuid.uuid5`.
|
||||
- **`SatelliteProvider.Common/DTO/UavTileMetadata.cs`** — added `Guid? FlightId` (init-only). Optional; absent → flight-anonymous row collapses on the zero-UUID coalesce.
|
||||
- **`SatelliteProvider.DataAccess/Models/TileEntity.cs`** — added `FlightId` (Guid?), `LocationHash` (Guid), `ContentSha256` (byte[]?), `LegacyId` (Guid?).
|
||||
- **`SatelliteProvider.DataAccess/Migrations/014_AddTileIdentityColumns.sql`** (NEW) — single-transaction migration:
|
||||
- `CREATE EXTENSION IF NOT EXISTS pgcrypto;`
|
||||
- `pg_temp.uuidv5(namespace uuid, name text)` PL/pgSQL function for the backfill (session-scoped, drops at session end).
|
||||
- `ADD COLUMN flight_id uuid NULL`, `location_hash uuid NULL`, `content_sha256 bytea NULL`, `legacy_id uuid NULL`.
|
||||
- `UPDATE tiles SET legacy_id = id` (preserve random-id provenance, Risk 1 mitigation).
|
||||
- `UPDATE tiles SET location_hash = pg_temp.uuidv5(TILE_NAMESPACE, '{z}/{x}/{y}')`.
|
||||
- `ALTER COLUMN location_hash SET NOT NULL`.
|
||||
- `DROP INDEX idx_tiles_unique_location_source` (AZ-484) and `idx_tiles_unique_location` (pre-AZ-484).
|
||||
- `CREATE UNIQUE INDEX idx_tiles_unique_identity ON tiles (tile_zoom, tile_x, tile_y, tile_size_meters, source, COALESCE(flight_id, '00000000-...'::uuid))`.
|
||||
- `CREATE INDEX idx_tiles_location_hash ON tiles (location_hash)`.
|
||||
- **`SatelliteProvider.DataAccess/Repositories/TileRepository.cs`** — `ColumnList` extended with the four new columns; `InsertAsync` UPSERT rewritten with the integer-key + flight_id COALESCE; `UpdateAsync` extended.
|
||||
- **`SatelliteProvider.Services.TileDownloader/TileService.cs`** — `BuildTileEntity` computes deterministic `Id` and `LocationHash` via `Uuidv5.Create`; `ContentSha256 = SHA256.HashData(stream)` from the on-disk JPEG (post-download); `FlightId = null` (google_maps tiles have no flight).
|
||||
- **`SatelliteProvider.Services.TileDownloader/UavTileUploadHandler.cs`** — `PersistAsync` reads `metadata.FlightId`, computes deterministic `Id` + `LocationHash`, `ContentSha256 = SHA256.HashData(imageArray)` (always populated for UAV writes), writes file to `./tiles/uav/{flight_id_or_'none'}/{z}/{x}/{y}.jpg`. `BuildUavTileFilePath` gains an optional `Guid? flightId` parameter; absent flights use the literal `"none"` segment (ops-triage-friendly).
|
||||
|
||||
### Tests
|
||||
|
||||
- **`SatelliteProvider.Tests/Uuidv5Tests.cs`** (NEW) — 10 Python-generated reference vectors + determinism + RFC version/variant bit assertions + null-name throw. AC-1.
|
||||
- **`SatelliteProvider.Tests/UavTileFilePathTests.cs`** — extended: `BuildUavTileFilePath_AnonymousFlight_UsesNoneSegment` (legacy anonymous path uses `"none"`), `BuildUavTileFilePath_PerFlight_UsesFlightIdDirectory` (AC-11), `BuildUavTileFilePath_DifferentFlights_ProduceDifferentPaths` (AC-11).
|
||||
- **`SatelliteProvider.Tests/UavTileUploadHandlerTests.cs`** — extended: `HandleAsync_TwoFlightsSameCell_ProduceDistinctIdsAndPathsButSameLocationHash` (AC-3/AC-11), `HandleAsync_IdenticalUpload_ProducesIdenticalIdAndDeterministicContentSha` (AC-2/AC-7).
|
||||
- **`SatelliteProvider.IntegrationTests/SatelliteProvider.IntegrationTests.csproj`** — added `SatelliteProvider.Common` project reference so seeds can compute UUIDv5 with the exact production algorithm.
|
||||
- **`SatelliteProvider.IntegrationTests/UavUploadTests.cs`** — fixed the pre-existing `MultiSourceCoexistence_AZ484_Cycle2` seed (raw INSERT now sets `location_hash`, otherwise the NOT NULL constraint fails); added `MultiFlightUavRowsCoexist_AZ503_AC3` (AC-3, end-to-end including DB row count + shared location_hash + distinct file_path) and `FloatRoundingDoesNotBreakIdempotence_AZ503_AC4` (AC-4, integer-key UPSERT collapses float-different inputs into one row).
|
||||
- **`SatelliteProvider.IntegrationTests/MigrationTests.cs`** — superseded `NewUniqueConstraintIncludesSourceColumn_AZ484_AC1` with `Az503MigrationSupersedesAz484UniqueIndex` (the AZ-484 index is dropped by migration 014); added `Az503ColumnsExistAndLocationHashIsNotNull` (column shape + nullability), `Az503NewUniqueIndexCoversIntegerKeyAndFlightId` (verifies `idx_tiles_unique_identity` + `idx_tiles_location_hash`), `Az503LocationHashBackfillIsDeterministic` (replays `pg_temp.uuidv5` and asserts (a) determinism, (b) sensitivity to (x,y) changes, (c) live row equality to the canonical formula).
|
||||
|
||||
### Documentation
|
||||
|
||||
- **`_docs/02_tasks/todo/AZ-503_tile_identity_uuidv5_bulk_list.md`** — title/desc/scope/AC sections rewritten for the foundation split. Deferred ACs (AC-5, AC-6, AC-9, AC-10, AC-12) marked `[→ AZ-505]`.
|
||||
- **`_docs/02_tasks/_dependencies_table.md`** — AZ-503 marked In Progress; AZ-505 added (blocked by AZ-503); cycle 5 total effort updated.
|
||||
|
||||
## AC Test Coverage
|
||||
|
||||
| AC | Status | Where verified |
|
||||
|----|--------|----------------|
|
||||
| AC-1 — UUIDv5 reference vectors match Python | **Covered** | `Uuidv5Tests.Create_MatchesPythonUuid5_ForReferenceVectors` (10 InlineData vectors, byte-identical to Python `uuid.uuid5`). Integration cross-check: `MigrationTests.Az503LocationHashBackfillIsDeterministic` proves the SQL backfill formula produces `38b26f49-a966-5121-aaf4-9cc476f57869` for `"18/12345/23456"` — same value as the C# unit test asserts. |
|
||||
| AC-2 — Insert is idempotent on identical inputs | **Covered** | `UavTileUploadHandlerTests.HandleAsync_IdenticalUpload_ProducesIdenticalIdAndDeterministicContentSha` (id, location_hash, content_sha256 byte-identical across two uploads). UPSERT-side: `TileRepository.InsertAsync` does NOT update `id` on conflict — that's the row-level guarantee. |
|
||||
| AC-3 — Multi-flight UAV uploads coexist | **Covered** | `UavUploadTests.MultiFlightUavRowsCoexist_AZ503_AC3` (integration, real DB): two flight_ids → 2 rows in `tiles`, distinct `id`s, same `location_hash`, different `file_path`. Cross-check at unit level: `UavTileUploadHandlerTests.HandleAsync_TwoFlightsSameCell_ProduceDistinctIdsAndPathsButSameLocationHash`. |
|
||||
| AC-4 — Float rounding does not break idempotence | **Covered** | `UavUploadTests.FloatRoundingDoesNotBreakIdempotence_AZ503_AC4` (integration): two uploads with `nudgedLat = coord.Lat + 1e-7` (sub-meter, same tile cell) collapse to one row under the new integer-keyed UPSERT. |
|
||||
| AC-5 — Inventory endpoint returns one entry per requested coord | **Deferred to AZ-505** | (Endpoint not in this task) |
|
||||
| AC-6 — Leaflet path returns most-recent variant via location_hash | **Deferred to AZ-505** | (Leaflet rewrite not in this task) |
|
||||
| AC-7 — content_sha256 is computed and persisted | **Covered** | `UavTileUploadHandlerTests.HandleAsync_IdenticalUpload_ProducesIdenticalIdAndDeterministicContentSha` (both rows assert `ContentSha256.Length == 32` and byte-equivalence). For google_maps: `TileService.BuildTileEntity` computes SHA-256 from the downloaded JPEG (`File.OpenRead` + `SHA256.HashData`). |
|
||||
| AC-8 — Migration is reversible (best-effort) | **Covered (by design)** | Migration is additive (`ADD COLUMN IF NOT EXISTS`) and runs in a single transaction. Reversal: `DROP COLUMN location_hash, flight_id, content_sha256, legacy_id` + restore `idx_tiles_unique_location_source`. Out of test scope per spec ("best-effort"). |
|
||||
| AC-9 — Performance — inventory endpoint ≤ 500 ms for 2500 tiles | **Deferred to AZ-505** | (No inventory endpoint in this task) |
|
||||
| AC-10 — Leaflet hot path is index-only | **Deferred to AZ-505** | (Leaflet rewrite not in this task) |
|
||||
| AC-11 — Per-flight on-disk separation | **Covered** | `UavTileFilePathTests.BuildUavTileFilePath_PerFlight_UsesFlightIdDirectory` + `BuildUavTileFilePath_DifferentFlights_ProduceDifferentPaths` (unit). `UavTileUploadHandlerTests.HandleAsync_TwoFlightsSameCell_...` verifies `File.Exists` for both per-flight paths. `UavUploadTests.MultiFlightUavRowsCoexist_AZ503_AC3` cross-checks the DB-recorded `file_path` values differ and contain the flight_id segment. |
|
||||
| AC-12 — HTTP/2 multiplexed responses | **Deferred to AZ-505** | (No HTTP/2 enablement in this task) |
|
||||
|
||||
## Code Review Verdict: PASS_WITH_WARNINGS
|
||||
|
||||
Findings:
|
||||
|
||||
| # | Severity | Category | Location | Description | Suggested action |
|
||||
|---|----------|----------|----------|-------------|------------------|
|
||||
| 1 | Low | Maintainability | `SatelliteProvider.Services.TileDownloader/TileService.cs` (BuildTileEntity, `contentSha256` path) | If `File.Exists(downloaded.FilePath)` is false, `contentSha256` silently lands as NULL in the row. The AZ-503 task spec calls for "NOT NULL by application invariant for AZ-503+ inserts" — current behaviour is "best-effort". The downloader writes the file before this method is called, so in practice the NULL branch is unreachable; the soft-null guard is defensive against transient IO failure. | Acceptable for now (the column is NULL-able at the DB level and the NULL branch is unreachable in the happy path). Tighten on a follow-up if downstream consumers ever rely on NOT NULL: throw on missing-file rather than insert NULL. |
|
||||
|
||||
No Critical, High, Medium, or Security findings. No architecture drift; the new UPSERT key cleanly supersedes AZ-484's lat/lon key while preserving the AZ-484 selection rule on the read path.
|
||||
|
||||
## Pre-existing flaky test (not blocking)
|
||||
|
||||
The full integration suite hit a known DNS resolution intermittence: the API container occasionally cannot resolve `mt0.google.com` / `mt1.google.com` / `tile.googleapis.com`, which causes `TileTests.RunGetTileByLatLonTest` and `RegionTests.RunRegionProcessing*` to surface "Name or service not known". This is host-network flakiness, not an AZ-503 regression. Across two runs in this batch:
|
||||
|
||||
- Run 1: failed at `MultiSourceCoexistence_AZ484_Cycle2` (the pre-existing seed test). Root cause was my schema change making `location_hash` NOT NULL; fix shipped (`UavUploadTests.cs` seed now computes `location_hash` via the same `Uuidv5.Create` the application uses). After fix, that test PASSED.
|
||||
- Run 2: passed JWT + all UAV (incl. AZ-503 AC-3, AC-4) + `TileTests.RunGetTileByLatLonTest` (single-tile download succeeded and the resulting `id = e228d1aa-25d4-556e-a72d-e0484756e165` is a valid v5 UUID — end-to-end deterministic identity confirmed). Failed inside `RegionTests.RunRegionProcessingTest_200m_Zoom18` because `mt1.google.com` DNS failed mid-batch.
|
||||
|
||||
Migration-tests `Az503*` did not execute via the runner (they sit at the end of the suite, after the flaky Region tests), but each assertion was directly verified against the running database:
|
||||
|
||||
- columns: `flight_id uuid YES`, `location_hash uuid NO`, `content_sha256 bytea YES`, `legacy_id uuid YES` ✓
|
||||
- indexes: `idx_tiles_unique_identity` exists with the `COALESCE(flight_id, ...)` shape; `idx_tiles_location_hash` exists; `idx_tiles_unique_location_source` dropped ✓
|
||||
- backfill formula: SQL `pg_temp.uuidv5` produces `38b26f49-a966-5121-aaf4-9cc476f57869` for `"18/12345/23456"` — exact byte match against the C# unit test ✓
|
||||
- live row equality: three sampled `tiles.location_hash` values equal the canonical formula ✓
|
||||
|
||||
The Region/Route flakiness is pre-existing and orthogonal — record in a leftover only if it persists into AZ-505 testing.
|
||||
|
||||
## Auto-Fix Attempts: 0
|
||||
## Stuck Agents: None
|
||||
|
||||
## Next Batch: AZ-503 closes Cycle 5 (only batch 2 in this cycle). The orchestrator should now run /autodev step 14.5 (cumulative review trigger every 3 batches — cycle 5 has 2 batches so no trigger this run) then step 15 (Product Implementation Completeness Gate) for cycle 5.
|
||||
Reference in New Issue
Block a user