mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 19:21:12 +00:00
Decompose Step 6 snapshot: 140 task specs + contract docs
Closes out greenfield Step 6 (Decompose) for all 14 components (C1-C13 + cross-cutting helpers/replay). Covers tasks AZ-266..AZ-446 plus the _dependencies_table.md and component contract documents. State file updated to greenfield Step 7 (Implement), not_started. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,122 @@
|
||||
# Contract: DescriptorIndex Protocol
|
||||
|
||||
**Component**: c6_tile_cache
|
||||
**Producer task**: AZ-303 — `_docs/02_tasks/todo/AZ-303_c6_storage_interfaces.md`
|
||||
**Consumer tasks**:
|
||||
- AZ-TBD-c6-faiss-descriptor-index (implements: FAISS HNSW)
|
||||
- TBD at decompose time: E-C2 (AZ-255 — sole runtime consumer; per-frame top-K=10 retrieval), E-C10 (AZ-252 — F1 pre-flight index build via `rebuild_from_descriptors`)
|
||||
**Version**: 1.0.0
|
||||
**Status**: draft
|
||||
**Last Updated**: 2026-05-10
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the typed boundary to the per-flight descriptor retrieval index. C2 VPR queries the index per frame at 3 Hz (top-K=10) to nominate candidate tiles for C2.5 ReRanker. The concrete impl is FAISS HNSW (`FaissDescriptorIndex`), but consumers depend only on this Protocol so a future swap (e.g., ScaNN, custom index) does not ripple. C10 CacheProvisioner (`AZ-252`) is the F1 pre-flight write-side caller — it builds the `.index` file once per provisioning; in flight the index is read-only mmap.
|
||||
|
||||
## Shape
|
||||
|
||||
### Protocol surface
|
||||
|
||||
`typing.Protocol` (PEP 544) with `runtime_checkable=True`. All methods are sync; the index is held in memory-mapped form.
|
||||
|
||||
| Method | Signature | Throws / Errors | Blocking? |
|
||||
|--------|-----------|-----------------|-----------|
|
||||
| `search_topk` | `(query: np.ndarray, k: int) -> list[tuple[TileId, float]]` | `IndexUnavailableError` | sync (HNSW; ≤ 5 ms p95 warm; first call ≤ 1 s cold for mmap page-in) |
|
||||
| `descriptor_dim` | `() -> int` | — | sync; constant-time |
|
||||
| `mmap_handle` | `() -> Path` | `IndexUnavailableError` | sync; returns the `.index` file path (consumers needing custom mmap-aware tooling — e.g., operator post-flight inspection — call this) |
|
||||
| `rebuild_from_descriptors` | `(descriptors: np.ndarray, tile_ids: list[TileId], hnsw_params: HnswParams) -> None` | `IndexBuildError`, `TileFsError` | sync (offline; minutes for a full-area corpus). Atomic file replacement via the AZ-280 sidecar pattern. |
|
||||
| `index_metadata` | `() -> IndexMetadata` | `IndexUnavailableError` | sync; reads the sidecar metadata block |
|
||||
|
||||
### DTOs
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HnswParams:
|
||||
"""HNSW build hyperparameters. See description.md § 5; defaults from the
|
||||
FAISS team's HNSW32+M=32 / efConstruction=200 / efSearch=64 baseline."""
|
||||
m: int = 32 # # of connections per node
|
||||
ef_construction: int = 200 # build-time candidate list size
|
||||
ef_search: int = 64 # query-time candidate list size
|
||||
metric: str = "L2" # "L2" | "INNER_PRODUCT"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IndexMetadata:
|
||||
descriptor_dim: int # dimension of the indexed vectors
|
||||
n_vectors: int # number of indexed tiles
|
||||
backbone_label: str # producer backbone — e.g. "ultra_vpr_v0"
|
||||
backbone_sha256_hex: str # producer backbone weights hash (D-C10-3 chain)
|
||||
built_at: datetime # ISO 8601 UTC
|
||||
hnsw_params: HnswParams
|
||||
sidecar_sha256_hex: str # canonical content hash of the .index file
|
||||
file_path: Path # absolute path to the .index file
|
||||
```
|
||||
|
||||
### Numpy contract
|
||||
|
||||
- `query`: shape `(descriptor_dim,)`, dtype `float32`, C-contiguous. The Protocol does NOT auto-pad batches; per-frame is a single query (C2's per-frame call site).
|
||||
- `descriptors`: shape `(N, descriptor_dim)`, dtype `float32`, C-contiguous; `N == len(tile_ids)`. The Protocol does NOT validate shape mismatch — the impl raises `IndexBuildError` on dtype/shape violation.
|
||||
|
||||
### Errors
|
||||
|
||||
```
|
||||
TileCacheError (shared with TileStore / TileMetadataStore)
|
||||
└── IndexUnavailableError # mmap handle invalid, file missing, or sidecar mismatched
|
||||
|
||||
IndexBuildError # raised only by rebuild_from_descriptors; NOT in the read-side envelope
|
||||
```
|
||||
|
||||
`IndexBuildError` is intentionally NOT a subclass of `TileCacheError` — the build path is offline, lives in C10's pre-flight provisioning, and has different fault semantics than the runtime-read path. C2 (the only runtime consumer) catches `IndexUnavailableError`; C10 catches `IndexBuildError`.
|
||||
|
||||
## Invariants
|
||||
|
||||
- **I-1 (immutable in flight):** once an `.index` file is opened via the impl's loader, the file's content MUST NOT change for the lifetime of the impl instance. F1 pre-flight is the only legal write path; a mid-flight rebuild is forbidden (the impl raises `IndexUnavailableError` if it detects a content-hash mismatch on a periodic sidecar re-check — out-of-band tampering signal).
|
||||
- **I-2 (top-K is best-effort):** `search_topk(query, k=K)` MAY return fewer than K results when the corpus has fewer than K vectors. Consumers (C2) tolerate fewer-than-K results.
|
||||
- **I-3 (descriptor-dim is fixed at build):** `descriptor_dim()` returns the value baked into the `.index` file at build time; if a consumer's query vector dimension does not match, the impl raises `IndexUnavailableError` (NOT a separate `DimensionMismatchError` — keeps the read-side envelope to a single error type).
|
||||
- **I-4 (no GPU resident memory):** the impl MUST hold the index in CPU mmap'd memory only. FAISS GPU index variants are explicitly excluded — the F3 hot path's GPU is reserved for `c7_inference` engines (per NFT-LIM-01 / D-CROSS-LATENCY-1).
|
||||
- **I-5 (atomic rebuild):** `rebuild_from_descriptors` MUST write to a temporary path, sync to disk, atomically rename to the target path, write the sidecar `.sha256`, and only then return. A crash mid-rebuild leaves the prior index intact.
|
||||
- **I-6 (sidecar coherence):** `mmap_handle()` returns a path whose `.sha256` sidecar matches the file's actual content hash; if the sidecar is missing or mismatched, `IndexUnavailableError` is raised on the FIRST `search_topk` of the flight (not lazily on the read that hits the corrupted region). C10's pre-flight gate is the canonical place this is validated; this Protocol carries the runtime-side check as defence-in-depth.
|
||||
- **I-7 (frozen DTOs):** `HnswParams`, `IndexMetadata` are `@dataclass(frozen=True)`.
|
||||
- **I-8 (single-thread search):** `search_topk` is NOT re-entrant; the F3 hot path is single-threaded per the description.md assumption. Future multi-threaded callers MUST use a per-thread impl instance (out of scope this cycle).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Not covered: tile pixel I/O.** That's `TileStore`.
|
||||
- **Not covered: tile metadata bbox queries.** That's `TileMetadataStore`.
|
||||
- **Not covered: incremental updates / online learning.** F1 pre-flight is full-rebuild only. Future task if needed.
|
||||
- **Not covered: GPU FAISS variants.** I-4 forbids them this cycle.
|
||||
- **Not covered: cross-flight index sharing.** Each flight provisions its own per-area `.index`; cross-flight is a parent-suite concern (D-PROJ-2).
|
||||
- **Not covered: descriptor compression / PQ quantisation.** HNSW32 raw float32 is the only supported variant this cycle. Future task if AC-8.3 (10 GB cap) becomes binding.
|
||||
- **Not covered: backbone retraining.** This Protocol is consumer-facing; the producer side (C10's compile of an UltraVPR engine) lives in E-C7 / E-C10.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
Same rules as `tile_store.md` § Versioning Rules. Note that `IndexMetadata.backbone_sha256_hex` ties this contract's lifecycle to the C7 engine cache (AZ-298 / AZ-301 / AZ-281): a backbone weights bump invalidates every prior `.index` AND requires a coordinated update — recorded as a major version of THIS contract only when the field's shape changes; backbone-weight refreshes within the existing schema are non-breaking content updates handled by C10.
|
||||
|
||||
## Test Cases
|
||||
|
||||
| Case | Input | Expected | Notes |
|
||||
|------|-------|----------|-------|
|
||||
| protocol-conformance-full | A class implementing all 5 methods | `isinstance(impl, DescriptorIndex) == True` | Producer AC-1 |
|
||||
| protocol-conformance-partial | A class missing `index_metadata` | `isinstance == False` | CI drift gate |
|
||||
| search-topk-warm | Query vector of correct dim against a 10k-vector index, OS page cache warm | Returns `[(tile_id, distance), ...]` length ≤ k; p95 ≤ 5 ms | I-2 / Consumer C2-PT-01 |
|
||||
| search-topk-fewer-than-k | k=20 against a 10-vector index | Returns 10 results, ordered by distance ascending | I-2 |
|
||||
| search-topk-dim-mismatch | Query vector of wrong dim | `IndexUnavailableError` | I-3 |
|
||||
| search-topk-corrupted-sidecar | Index file present, sidecar missing | First `search_topk` raises `IndexUnavailableError`; subsequent calls also raise (no silent recovery) | I-6 |
|
||||
| descriptor-dim | After a rebuild with `descriptors.shape == (N, 512)` | `descriptor_dim() == 512` | I-3 |
|
||||
| rebuild-atomic-on-crash | Simulated `os._exit` mid-rebuild | The original `.index` file is intact and still loadable; partial temp file is cleaned up at next start | I-5 |
|
||||
| rebuild-sidecar-content-hash | Successful rebuild | `.sha256` sidecar matches `sha256(.index)` | I-6 / AZ-280 contract |
|
||||
| index-metadata | After rebuild | Returns `IndexMetadata` with matching `descriptor_dim`, `n_vectors`, `built_at` (within 1 s of call), `hnsw_params` (mirrors input), `sidecar_sha256_hex` (matches sidecar content) | I-7 |
|
||||
| frozen-dto-mutation | `HnswParams(m=32, ...).m = 64` | `FrozenInstanceError` | I-7 |
|
||||
|
||||
## Change Log
|
||||
|
||||
| Version | Date | Change | Author |
|
||||
|---------|------|--------|--------|
|
||||
| 1.0.0 | 2026-05-10 | Initial contract — 5-method Protocol + HNSW params DTO + IndexMetadata sidecar shape + immutable-in-flight + atomic-rebuild invariants. | autodev (decompose Step 2 of AZ-250 / E-C6) |
|
||||
@@ -0,0 +1,132 @@
|
||||
# Contract: TileMetadataStore Protocol
|
||||
|
||||
**Component**: c6_tile_cache
|
||||
**Producer task**: AZ-303 — `_docs/02_tasks/todo/AZ-303_c6_storage_interfaces.md`
|
||||
**Consumer tasks**:
|
||||
- AZ-TBD-c6-postgres-filesystem-store (implements)
|
||||
- AZ-TBD-c6-freshness-gate (insert hook + sector classification reader)
|
||||
- AZ-TBD-c6-cache-budget-eviction (LRU candidate enumeration + delete coordination)
|
||||
- TBD at decompose time: E-C10 (AZ-252 — manifest + provisioning), E-C11 (AZ-251 — both `TileDownloader` insert and `TileUploader` reader queries), E-C12 (AZ-253 — operator pre-flight tooling)
|
||||
**Version**: 1.0.0
|
||||
**Status**: draft
|
||||
**Last Updated**: 2026-05-10
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the typed boundary to the Postgres-backed spatial index over `TileMetadata`. Concrete impls (today only `PostgresFilesystemStore` — same class also implements `TileStore`) own row insert / bbox query / voting-state transitions. Pre-flight cache builders (C10 / C11 / C12), the F4 mid-flight orthorectifier path (via C5 → C6), and post-landing tooling (C11 `TileUploader`) all consume this surface.
|
||||
|
||||
## Shape
|
||||
|
||||
### Protocol surface
|
||||
|
||||
`typing.Protocol` (PEP 544) with `runtime_checkable=True`. All methods are sync; the Postgres connection pool is owned inside the impl.
|
||||
|
||||
| Method | Signature | Throws / Errors | Blocking? |
|
||||
|--------|-----------|-----------------|-----------|
|
||||
| `query_by_bbox` | `(bbox: Bbox, zoom: int, *, voting_filter: Optional[VotingStatus] = None, source_filter: Optional[TileSource] = None) -> list[TileMetadata]` | `TileMetadataError` | sync (btree index; ≤ 50 ms typical) |
|
||||
| `insert_metadata` | `(metadata: TileMetadata) -> None` | `TileMetadataError`, `FreshnessRejectionError` | sync (single-row insert) |
|
||||
| `update_voting_status` | `(tile_id: TileId, status: VotingStatus) -> None` | `TileMetadataError`, `TileNotFoundError` | sync |
|
||||
| `mark_uploaded` | `(tile_id: TileId, uploaded_at: datetime) -> None` | `TileMetadataError`, `TileNotFoundError` | sync |
|
||||
| `pending_uploads` | `() -> list[TileMetadata]` | `TileMetadataError` | sync (filtered query: `source = ONBOARD_INGEST AND uploaded_at IS NULL`) |
|
||||
| `record_lru_access` | `(tile_id: TileId, accessed_at: datetime) -> None` | `TileMetadataError` | sync (timestamp update only — no row-level read) |
|
||||
| `lru_candidates` | `(*, max_count: int) -> list[TileMetadata]` | `TileMetadataError` | sync (oldest-`accessed_at`-first; bounded result set) |
|
||||
| `total_disk_bytes` | `() -> int` | `TileMetadataError` | sync (sum of `disk_bytes` column; ≤ 100 ms even at 100k rows) |
|
||||
| `get_by_id` | `(tile_id: TileId) -> Optional[TileMetadata]` | `TileMetadataError` | sync; returns `None` if absent (NOT `TileNotFoundError`) |
|
||||
|
||||
### DTOs
|
||||
|
||||
Reuses `TileId`, `TileMetadata`, `TileQualityMetadata`, `TileSource`, `FreshnessLabel`, `VotingStatus` from `tile_store.md`. The same DTOs are shared across both Protocols by design (single source of truth in `c6_tile_cache._types`).
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Bbox:
|
||||
"""Axis-aligned WGS84 bounding box. Inclusive on min, exclusive on max."""
|
||||
min_lat: float
|
||||
min_lon: float
|
||||
max_lat: float
|
||||
max_lon: float
|
||||
```
|
||||
|
||||
In addition, `TileMetadata` is extended with two columns owned by the metadata store (NOT meaningful to `TileStore`; see Invariants):
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class TileMetadataPersistent:
|
||||
metadata: TileMetadata # the read-only DTO from tile_store.md
|
||||
accessed_at: datetime # LRU clock — last read time
|
||||
uploaded_at: Optional[datetime] # set when C11 TileUploader has confirmed upload
|
||||
disk_bytes: int # JPEG body size on disk; tracked for cache-budget enforcement
|
||||
```
|
||||
|
||||
The Protocol returns `TileMetadata` from queries. `TileMetadataPersistent` is the in-process view of LRU and disk-budget state, accessible only via `lru_candidates` / `record_lru_access` / `total_disk_bytes`.
|
||||
|
||||
### Sector classification (read-only input to the freshness gate)
|
||||
|
||||
```python
|
||||
class SectorClassification(str, Enum):
|
||||
ACTIVE_CONFLICT = "active_conflict"
|
||||
STABLE_REAR = "stable_rear"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SectorBoundary:
|
||||
bbox: Bbox
|
||||
classification: SectorClassification
|
||||
```
|
||||
|
||||
`SectorClassification` is set pre-flight by the operator via C12; the metadata store reads `SectorBoundary` rows from a sibling table (`sector_boundaries`) at insert-time to decide which freshness rule to apply. The Protocol does NOT expose insert-side methods for `SectorBoundary` rows — that surface lives in C12.
|
||||
|
||||
## Invariants
|
||||
|
||||
- **I-1 (composite key uniqueness):** `(zoom_level, lat, lon, source)` is the unique key in the `tiles` table. Re-inserting the same key with different content_sha256 raises `TileMetadataError` — no silent overwrite.
|
||||
- **I-2 (freshness gate at insert):** `insert_metadata` rejects (raises `FreshnessRejectionError`) iff the tile's `(lat, lon)` falls inside an `ACTIVE_CONFLICT` sector AND `capture_timestamp < now() - active_conflict_max_age`. The freshness rules table is configured per-flight (default 6 months for active_conflict; 12 months for stable_rear which downgrades rather than rejects).
|
||||
- **I-3 (downgrade marking):** when a tile in a `STABLE_REAR` sector is older than `stable_rear_max_age`, the row is inserted with `freshness_label=DOWNGRADED` (NOT rejected). `query_by_bbox` returns the downgrade flag intact so consumers (C2 / C3 spoof-rejection) can act on it.
|
||||
- **I-4 (LRU clock):** `record_lru_access` updates `accessed_at = max(current accessed_at, supplied timestamp)`; clock skew never sets `accessed_at` backward. `lru_candidates` returns oldest-first.
|
||||
- **I-5 (disk-budget invariant):** `total_disk_bytes` MUST equal `SUM(disk_bytes)` over all rows where `voting_status != REJECTED`. Rejected rows are tombstones — they keep the on-disk file deleted but retain the row for the manifest's content-hash check (D-C10-3).
|
||||
- **I-6 (frozen DTOs):** `Bbox`, `SectorBoundary`, `TileMetadataPersistent` are `@dataclass(frozen=True)`.
|
||||
- **I-7 (transactional writes):** `insert_metadata` is a single transaction over the `tiles` table; the freshness check + the row insert MUST be atomic (a parallel sector-boundary update MUST NOT race the gate).
|
||||
- **I-8 (no silent voting-status downgrade):** `update_voting_status` accepts only forward transitions (`PENDING → TRUSTED`, `PENDING → REJECTED`); a backward transition raises `TileMetadataError`. `TRUSTED → REJECTED` is allowed (covers the cache-poisoning recall path).
|
||||
- **I-9 (`pending_uploads` is the single source for C11 TileUploader):** the uploader MUST NOT scan the filesystem for pending tiles; it MUST drive its loop off `pending_uploads()`. The metadata store is the bookkeeping.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Not covered: filesystem JPEG I/O.** That's `TileStore`.
|
||||
- **Not covered: descriptor index queries.** That's `DescriptorIndex`.
|
||||
- **Not covered: sector boundary insert / update.** Owned by C12 operator-tooling against a sibling table; this Protocol is read-only on `SectorBoundary` and does NOT expose CRUD.
|
||||
- **Not covered: cross-flight aggregation / voting threshold computation.** That's `satellite-provider`'s D-PROJ-2 trust layer (parent suite); C6 just stamps the per-row `voting_status`.
|
||||
- **Not covered: full-text search / arbitrary-WHERE queries.** Only the methods above; ad-hoc queries go through DBA tooling, not this Protocol.
|
||||
- **Not covered: schema migrations.** Migration scripts live in `c6_tile_cache/_alembic/`; the Protocol is shape-only.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
Same rules as `tile_store.md` § Versioning Rules.
|
||||
|
||||
## Test Cases
|
||||
|
||||
| Case | Input | Expected | Notes |
|
||||
|------|-------|----------|-------|
|
||||
| protocol-conformance-full | A class implementing all 9 methods | `isinstance(impl, TileMetadataStore) == True` | Producer AC-1 |
|
||||
| query-by-bbox-basic | bbox covering 100 inserted tiles at zoom=18 | Returns exactly the 100 tiles; `voting_filter=None` returns all statuses | Smoke |
|
||||
| query-by-bbox-voting-filter | Same with `voting_filter=TRUSTED` | Returns only TRUSTED tiles in bbox | Used by C10 manifest builder |
|
||||
| insert-duplicate-key | Insert (z=18, lat, lon, src=GOOGLEMAPS) twice with different content_sha256 | First succeeds; second raises `TileMetadataError` | I-1 |
|
||||
| insert-active-conflict-stale | Insert into ACTIVE_CONFLICT sector, capture_timestamp = now - 7 months | `FreshnessRejectionError`; row not committed | I-2 / C6-IT-02 |
|
||||
| insert-stable-rear-stale | Insert into STABLE_REAR sector, capture_timestamp = now - 13 months | Row inserted with `freshness_label=DOWNGRADED` | I-3 |
|
||||
| update-voting-status-forward | PENDING → TRUSTED | Succeeds | I-8 |
|
||||
| update-voting-status-backward | TRUSTED → PENDING | `TileMetadataError` | I-8 |
|
||||
| update-voting-status-trusted-to-rejected | TRUSTED → REJECTED | Succeeds (recall path) | I-8 |
|
||||
| pending-uploads-empty | No ONBOARD_INGEST tiles | Returns `[]` | I-9 |
|
||||
| pending-uploads-after-mark | Insert + `mark_uploaded` for half | Returns the unmarked half | I-9 |
|
||||
| record-lru-access-monotonic | `record_lru_access(t, ts1)` then `record_lru_access(t, ts0 < ts1)` | `accessed_at` stays at `ts1` | I-4 |
|
||||
| lru-candidates-order | Mixed `accessed_at` for 100 rows; `lru_candidates(max_count=10)` | Returns the 10 oldest in ascending `accessed_at` order | I-4 |
|
||||
| total-disk-bytes-sum | Insert 5 tiles with known disk_bytes, mark 1 REJECTED | `total_disk_bytes()` excludes the rejected row | I-5 |
|
||||
| get-by-id-missing | Random tile_id never inserted | Returns `None` (not `TileNotFoundError`) | Documented null-return semantic |
|
||||
| frozen-dto-mutation | `Bbox(0, 0, 1, 1).min_lat = 5.0` | `FrozenInstanceError` | I-6 |
|
||||
|
||||
## Change Log
|
||||
|
||||
| Version | Date | Change | Author |
|
||||
|---------|------|--------|--------|
|
||||
| 1.0.0 | 2026-05-10 | Initial contract — 9-method Protocol + LRU/disk-budget extensions + freshness gate semantics + composite-key uniqueness invariant. | autodev (decompose Step 2 of AZ-250 / E-C6) |
|
||||
@@ -0,0 +1,166 @@
|
||||
# Contract: TileStore Protocol
|
||||
|
||||
**Component**: c6_tile_cache
|
||||
**Producer task**: AZ-303 — `_docs/02_tasks/todo/AZ-303_c6_storage_interfaces.md`
|
||||
**Consumer tasks**:
|
||||
- AZ-TBD-c6-postgres-filesystem-store (implements)
|
||||
- AZ-TBD-c6-freshness-gate (insert hook collaborator)
|
||||
- AZ-TBD-c6-cache-budget-eviction (uses `tile_exists` + `delete_tile`)
|
||||
- TBD at decompose time: E-C2.5 (AZ-256), E-C3 (AZ-257), E-C11 (AZ-251 — both `TileDownloader` and `TileUploader`)
|
||||
**Version**: 1.0.0
|
||||
**Status**: draft
|
||||
**Last Updated**: 2026-05-10
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the typed boundary between filesystem-resident tile pixel I/O and every component that produces or consumes JPEG tile bytes. Concrete impls (today only `PostgresFilesystemStore`) write JPEGs to a layout byte-identical to `satellite-provider`'s on-disk format so the C11 `TileUploader` post-landing upload (F10) is a straight copy.
|
||||
|
||||
## Shape
|
||||
|
||||
### Protocol surface
|
||||
|
||||
`typing.Protocol` (PEP 544 structural typing) with `runtime_checkable=True`.
|
||||
|
||||
| Method | Signature | Throws / Errors | Blocking? |
|
||||
|--------|-----------|-----------------|-----------|
|
||||
| `read_tile_pixels` | `(tile_id: TileId) -> TilePixelHandle` | `TileNotFoundError`, `TileFsError` | sync (mmap, ≤ 0.5 ms warm; ≤ 50 ms cold) |
|
||||
| `write_tile` | `(tile_blob: bytes, metadata: TileMetadata) -> None` | `TileFsError`, `TileMetadataError`, `ContentHashMismatchError`, `FreshnessRejectionError` | sync (atomic fs write + sidecar) |
|
||||
| `tile_exists` | `(tile_id: TileId) -> bool` | — | sync (page-cache lookup; ≤ 1 ms) |
|
||||
| `delete_tile` | `(tile_id: TileId) -> bool` | `TileFsError` | sync (returns `True` if a file was removed; `False` if missing — no-error path for the cache-eviction caller) |
|
||||
|
||||
### DTOs
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TileId:
|
||||
zoom_level: int # 0..21 — `satellite-provider` legal range
|
||||
lat: float # WGS84 centre latitude
|
||||
lon: float # WGS84 centre longitude
|
||||
|
||||
|
||||
class TileSource(str, Enum):
|
||||
GOOGLEMAPS = "googlemaps"
|
||||
ONBOARD_INGEST = "onboard_ingest"
|
||||
|
||||
|
||||
class FreshnessLabel(str, Enum):
|
||||
FRESH = "fresh"
|
||||
STALE_ACTIVE_CONFLICT = "stale_active_conflict"
|
||||
STALE_REAR = "stale_rear"
|
||||
DOWNGRADED = "downgraded"
|
||||
|
||||
|
||||
class VotingStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
TRUSTED = "trusted"
|
||||
REJECTED = "rejected"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TileQualityMetadata:
|
||||
estimator_label: str # "satellite_anchored" | "visual_propagated" | "dead_reckoned"
|
||||
covariance_2x2: tuple[tuple[float, float], tuple[float, float]]
|
||||
last_anchor_age_ms: int
|
||||
mre_px: float
|
||||
imu_bias_norm: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TileMetadata:
|
||||
tile_id: TileId
|
||||
tile_size_meters: float
|
||||
tile_size_pixels: int
|
||||
capture_timestamp: datetime # ISO 8601 UTC
|
||||
source: TileSource
|
||||
content_sha256_hex: str # canonical sha256 of the JPEG body
|
||||
freshness_label: FreshnessLabel
|
||||
flight_id: Optional[str] # uuid; set for ONBOARD_INGEST
|
||||
companion_id: Optional[str] # set for ONBOARD_INGEST
|
||||
quality_metadata: Optional[TileQualityMetadata] # set for ONBOARD_INGEST
|
||||
voting_status: VotingStatus # default PENDING for ONBOARD_INGEST
|
||||
|
||||
|
||||
class TilePixelHandle:
|
||||
"""Opaque handle: filesystem path + mmap pointer. Consumer MUST NOT copy the bytes
|
||||
or close the underlying mapping; the handle's lifetime is bounded by the caller's
|
||||
use-site `with` block."""
|
||||
|
||||
@property
|
||||
def filesystem_path(self) -> Path: ...
|
||||
def __enter__(self) -> memoryview: ...
|
||||
def __exit__(self, *exc) -> None: ...
|
||||
```
|
||||
|
||||
### Error types
|
||||
|
||||
All under `c6_tile_cache.errors`:
|
||||
|
||||
```
|
||||
TileCacheError (Exception subclass)
|
||||
├── TileNotFoundError # tile_id not present on disk
|
||||
├── TileFsError # I/O error on read/write/rename
|
||||
├── TileMetadataError # row missing despite file present, or vice-versa (consistency violation)
|
||||
├── ContentHashMismatchError # supplied JPEG bytes don't match declared content_sha256
|
||||
└── FreshnessRejectionError # rejected by the C6 freshness gate (raised on insert in active_conflict)
|
||||
```
|
||||
|
||||
`IndexUnavailableError` lives under the same package but is exclusively raised by `DescriptorIndex` — it is not part of `TileStore`'s envelope.
|
||||
|
||||
### Filesystem layout
|
||||
|
||||
JPEG body lands at `<root>/tiles/{zoom_level}/{x}/{y}.jpg` where `(x, y)` is derived from `(lat, lon, zoom_level)` per the same Web-Mercator tile-coordinate function `satellite-provider` uses (see `satellite-provider/README.md`). A sidecar file `<root>/tiles/{zoom_level}/{x}/{y}.jpg.sha256` carries the canonical content hash (produced by `helpers.sha256_sidecar.atomic_write_with_sidecar` per AZ-280 contract).
|
||||
|
||||
## Invariants
|
||||
|
||||
- **I-1 (byte-identity with satellite-provider):** for any `(zoom_level, lat, lon)`, the filesystem path computed by C6 `write_tile` MUST equal the path that `satellite-provider` would compute for the same coordinate; any deviation breaks AC-8.4 / F10 upload.
|
||||
- **I-2 (atomic write + sidecar invariant):** a successful `write_tile` returns only after BOTH the JPEG file AND its `.sha256` sidecar are durable on disk; partial states (file without sidecar or sidecar without file) MUST NOT be observable to readers.
|
||||
- **I-3 (content-hash gate):** `write_tile` rejects (raises `ContentHashMismatchError`) if `sha256(tile_blob) != metadata.content_sha256_hex`; the cache-poisoning safety budget (D-C10-3 + AC-NEW-7) is bound to this check.
|
||||
- **I-4 (read mmap is read-only):** `TilePixelHandle.__enter__()` returns a read-only `memoryview`; consumers MUST NOT mutate; a writer that mutates through the mmap is a `Reliability` finding (Critical) at code-review time.
|
||||
- **I-5 (race-free reads under concurrent F4 writes):** C2 / C2.5 / C3 readers see either the pre-write tile bytes or the post-write tile bytes — never partial bytes. Enforced by `atomicwrites` rename semantics on the writer side.
|
||||
- **I-6 (idempotent delete):** `delete_tile` returns `False` when the tile is missing; it does NOT raise. The cache-eviction caller relies on this no-error path because it deletes by LRU and may race with a concurrent eviction sweep.
|
||||
- **I-7 (frozen DTOs):** `TileId`, `TileMetadata`, `TileQualityMetadata` are `@dataclass(frozen=True)`. Mutation raises `FrozenInstanceError`.
|
||||
- **I-8 (fail-fast on consistency violation):** if a row exists in the metadata store but the JPEG file is missing (or vice-versa), `read_tile_pixels` raises `TileMetadataError` — NOT `TileNotFoundError`. The two errors are the operator's signal that the cache is in a degraded state and needs reprovisioning.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Not covered: tile descriptor index.** Descriptor mmap + HNSW search is `DescriptorIndex` — separate Protocol, separate contract.
|
||||
- **Not covered: spatial bbox queries.** `query_by_bbox` is on `TileMetadataStore` — separate Protocol.
|
||||
- **Not covered: HTTP transport to satellite-provider.** C11 `TileDownloader` / `TileUploader` own transport; they call `TileStore.write_tile` / `TileStore.read_tile_pixels` for the local-side persistence step.
|
||||
- **Not covered: eviction policy.** `delete_tile` is the eviction primitive; the LRU policy lives in the cache-budget enforcer (separate task).
|
||||
- **Not covered: multi-process readers writing concurrently.** Single-process producer/consumer per flight; multi-process scenarios are out of scope this cycle.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- **Breaking** (renamed method, removed field, type change, required→optional flip, error class removed from family) requires a major version bump and a coordinated update of every consumer task listed in this header. Producer task MUST surface the change to the user via Choose format before merging.
|
||||
- **Non-breaking additions** (new optional method via Protocol structural compatibility, new optional field on a DTO with a default, new error variant added to `TileCacheError`) require a minor version bump.
|
||||
- **Patch** (clarification only, no shape change) is documentation-only.
|
||||
|
||||
## Test Cases
|
||||
|
||||
| Case | Input | Expected | Notes |
|
||||
|------|-------|----------|-------|
|
||||
| protocol-conformance-full | A class implementing all four methods with matching signatures | `isinstance(impl, TileStore) == True` | I-1 / Producer AC-1 |
|
||||
| protocol-conformance-partial | A class missing `delete_tile` | `isinstance == False` | Drift detection at CI time |
|
||||
| frozen-dto-mutation | `TileId(zoom_level=18, lat=49.94, lon=36.31).lat = 0.0` | `FrozenInstanceError` | I-7 |
|
||||
| write-tile-byte-identical | `write_tile(blob, metadata)` for `(zoom=18, lat=49.94, lon=36.31)` | Filesystem path equals `satellite-provider`'s path for same coord; JPEG bytes equal `blob`; sidecar contains `sha256(blob)` | I-1 / I-2 / C6-IT-01 |
|
||||
| write-tile-content-hash-mismatch | `write_tile(blob, metadata.with(content_sha256_hex="0x00..."))` | `ContentHashMismatchError`; no file written; no sidecar written | I-3 / C6-ST-01 |
|
||||
| write-tile-freshness-reject | active_conflict sector + stale tile | `FreshnessRejectionError`; no file/row written | Hand-off to freshness-gate task |
|
||||
| read-tile-pixels-warm | `read_tile_pixels(tile_id)` after a prior write; OS page cache warm | `TilePixelHandle.__enter__()` returns within 0.5 ms; bytes equal the written JPEG body | C6-PT-01 |
|
||||
| read-tile-pixels-missing | `read_tile_pixels(tile_id)` for never-written tile | `TileNotFoundError` | I-8 (the row-missing-and-file-missing case) |
|
||||
| read-tile-pixels-row-without-file | metadata row exists; JPEG file deleted out-of-band | `TileMetadataError` (not `TileNotFoundError`) | I-8 |
|
||||
| concurrent-write-and-read | F4 writer + 9 Hz C2.5 reader on same tile | Reader sees either pre-write or post-write bytes — never partial | I-5 |
|
||||
| delete-tile-missing | `delete_tile` for a never-written tile | Returns `False`; no exception raised | I-6 |
|
||||
| delete-tile-existing | `delete_tile` after a prior write | Returns `True`; subsequent `tile_exists` returns `False`; sidecar also removed | I-6 |
|
||||
|
||||
## Change Log
|
||||
|
||||
| Version | Date | Change | Author |
|
||||
|---------|------|--------|--------|
|
||||
| 1.0.0 | 2026-05-10 | Initial contract — Protocol + DTOs + 5-error family + filesystem byte-identity invariant. | autodev (decompose Step 2 of AZ-250 / E-C6) |
|
||||
Reference in New Issue
Block a user