[AZ-777] Phase 1: wire e2e-runner to real satellite-provider + C11 contract adapt

Adapt C11 HttpTileDownloader to the AZ-505 v1.0.0 tile-inventory
contract (POST /api/satellite/tiles/inventory + GET /tiles/{z}/{x}/{y})
and wire the Jetson e2e harness against the real parent-suite
satellite-provider service. Closes Phase 1 of 5 for AZ-777; STOP
gate before Phase 2 (Derkachi catalog seed).

C11 changes:
- _LIST_PATH / _GET_PATH replaced with _INVENTORY_PATH + _TILES_PATH.
- _do_enumerate enumerates bbox tile coords client-side and posts
  chunked inventory requests (5000-entry cap per the contract).
- _download_one_tile parses tile_id_str into (z,x,y) and fetches
  the slippy-map URL.
- Common GET / POST retry+auth ladder consolidated into _send_request.
- New module helpers: _enumerate_bbox_tile_coords,
  _tile_center_latlon, _tile_size_meters_at, _format_tile_id_str,
  _parse_tile_id_str, _chunk_iter.
- _DEFAULT_ESTIMATED_TILE_BYTES (50 KiB) replaces the inventory-side
  estimatedBytes field the v1.0.0 contract dropped.

Tests:
- 14/14 unit tests in tests/unit/c11_tile_manager/test_tile_downloader.py
  rewritten for the new POST inventory + slippy-map GET handler.
  _StubTileWriter rekeyed by call-index (the downloader now derives
  lat/lon from the slippy-map coord, so fixtures can't fabricate
  arbitrary positions).
- New Tier-2 smoke at tests/e2e/satellite_provider/test_smoke.py:
  validates inventory POST schema + drives HttpTileDownloader against
  the real service. Gated by RUN_REPLAY_E2E=1 + tier2.

Compose / env:
- e2e-runner SATELLITE_PROVIDER_URL switched from mock-sat:5100 to
  https://satellite-provider:8080; TLS_INSECURE + Bearer JWT env +
  depends_on satellite-provider added.
- .env.test.example documents SATELLITE_PROVIDER_API_KEY + dev TLS
  bypass security note.
- scripts/mint_dev_jwt.py mints HS256 dev JWTs from env / .env.test.
- pyjwt added to dev extras.

Tracker hygiene:
- AZ-777 row in _dependencies_table.md bumped 5pt -> 8pt to match
  the 2026-05-21 override decision log.

Code review: PASS_WITH_WARNINGS (3 medium/low findings, all deferred
to later AZ-777 phases) -- see batch_104_review.md. Batch report at
batch_104_cycle3_report.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-21 14:52:39 +03:00
parent 544b37fdc9
commit 811b04e605
12 changed files with 1328 additions and 190 deletions
+8
View File
@@ -0,0 +1,8 @@
"""AZ-777 Phase 1 smoke tests against the parent-suite satellite-provider.
Tier-2 only: these tests assume the real `.NET` `satellite-provider`
service (and its Postgres) are running in the Jetson e2e compose graph
(`docker-compose.test.jetson.yml`). Each test is gated by
`RUN_REPLAY_E2E=1` (the env contract the rest of `tests/e2e/replay/`
already uses) and `@pytest.mark.tier2` so dev-laptop pytest runs auto-skip.
"""
+325
View File
@@ -0,0 +1,325 @@
"""AZ-777 Phase 1 smoke test against the real parent-suite satellite-provider.
Validates the wire C11 just plumbed through against the .NET service
running inside `docker-compose.test.jetson.yml`:
* TLS handshake works against the self-signed dev cert via
``SATELLITE_PROVIDER_TLS_INSECURE=1`` (development-only override —
production deploys MUST use a CA-issued cert and unset the flag).
* Bearer JWT minted from ``JWT_SECRET`` / ``JWT_ISSUER`` /
``JWT_AUDIENCE`` is accepted.
* ``POST /api/satellite/tiles/inventory`` (AZ-505) round-trips the
documented v1.0.0 schema for a 1-tile Derkachi-bbox query: response
is 200 with a ``results`` array of length == request.tiles.length
(Inv-2 from ``tile-inventory.md``) and per-entry shape matches the
contract regardless of seed state.
* The adapted C11 :class:`HttpTileDownloader` (Phase 1a) drives the
same wire against the real service. When the catalog is unseeded
(pre-Phase-2) the report is ``SUCCESS`` with zero downloads (every
entry comes back ``present=false``); when seeded, the stubbed C6
``write_tile_for_download`` receives one call per present tile and
the report counts match.
Phase 2 (Derkachi catalog seed via ``POST /api/satellite/request``)
turns the second test from a "wire works" check into a "tiles actually
land" check; the assertions here are written so both states (seeded /
unseeded) pass cleanly so the smoke runs green as soon as Phase 1
lands and stays green through Phase 2.
"""
from __future__ import annotations
import logging
import os
import ssl
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
import httpx
import pytest
from gps_denied_onboard.components.c11_tile_manager import (
C11Config,
DownloadOutcome,
DownloadRequest,
HttpTileDownloader,
SectorClassification,
)
from gps_denied_onboard.helpers.wgs_converter import WgsConverter
# ----------------------------------------------------------------------
# Skip gates
# ----------------------------------------------------------------------
def _heavy_skip_reason() -> str | None:
if os.environ.get("RUN_REPLAY_E2E", "").lower() not in {"1", "true", "yes", "on"}:
return "AZ-777 satellite-provider smoke gated by RUN_REPLAY_E2E=1"
return None
_HEAVY_SKIP = pytest.mark.skipif(
_heavy_skip_reason() is not None,
reason=_heavy_skip_reason() or "ok",
)
_INVENTORY_PATH = "/api/satellite/tiles/inventory"
# Derkachi bbox center used for the 1-tile inventory query. Zoom 15 is
# the upper bound of the AZ-777 Phase 2 catalog (zooms 15-18); picking
# the lowest zoom keeps the per-tile ground footprint largest, which
# makes a 1-tile query cover the most catalog.
_DERKACHI_TILE_ZOOM = 15
_DERKACHI_LAT = 50.10
_DERKACHI_LON = 36.10
# Tight bbox that surrounds the Derkachi tile center so the C11 bbox
# enumeration produces exactly one (z,x,y) coord at zoom 15.
_DERKACHI_BBOX = (50.099, 36.099, 50.101, 36.101)
# ----------------------------------------------------------------------
# Helpers
# ----------------------------------------------------------------------
def _mint_bearer_token_or_skip() -> str:
"""Return a valid Bearer JWT or `pytest.skip` with an actionable reason."""
token = os.environ.get("SATELLITE_PROVIDER_API_KEY", "").strip()
if token and token != "PASTE-MINTED-JWT-HERE":
return token
try:
import jwt # type: ignore[import-untyped]
except ImportError:
pytest.skip(
"SATELLITE_PROVIDER_API_KEY is unset and PyJWT is not "
"installed in-container; install dev extras "
"(`pip install -e .[dev]`) or set "
"SATELLITE_PROVIDER_API_KEY via .env.test."
)
secret = os.environ.get("JWT_SECRET", "").strip()
issuer = os.environ.get("JWT_ISSUER", "").strip()
audience = os.environ.get("JWT_AUDIENCE", "").strip()
missing = [
name
for name, value in [
("JWT_SECRET", secret),
("JWT_ISSUER", issuer),
("JWT_AUDIENCE", audience),
]
if not value
]
if missing:
pytest.skip(
"Cannot mint a fallback JWT — missing env: "
+ ", ".join(missing)
)
now = datetime.now(timezone.utc)
payload = {
"sub": "gps-denied-onboard-smoke",
"iss": issuer,
"aud": audience,
"jti": uuid.uuid4().hex,
"iat": int(now.timestamp()),
"nbf": int(now.timestamp()),
"exp": int((now + timedelta(hours=1)).timestamp()),
}
return str(jwt.encode(payload, secret, algorithm="HS256"))
def _make_http_client(base_url: str) -> httpx.Client:
"""Return an httpx.Client honouring SATELLITE_PROVIDER_TLS_INSECURE."""
insecure_raw = os.environ.get("SATELLITE_PROVIDER_TLS_INSECURE", "").strip()
insecure = insecure_raw.lower() in {"1", "true", "yes", "on"}
verify: bool | ssl.SSLContext
if insecure:
# Documented dev-only path; the audit-trail WARNING lives in
# `.env.test.example`. We do NOT silently disable verification
# unless the env var explicitly opts in.
verify = False
else:
verify = True
return httpx.Client(base_url=base_url, verify=verify, timeout=30.0)
def _resolve_satellite_provider_url() -> str:
url = os.environ.get("SATELLITE_PROVIDER_URL", "").strip()
if not url:
pytest.skip(
"SATELLITE_PROVIDER_URL is not set — the Jetson e2e compose "
"env block provides https://satellite-provider:8080; running "
"this smoke outside compose requires an explicit override."
)
return url
# ----------------------------------------------------------------------
# Test 1 — inventory POST contract (AZ-505 v1.0.0)
# ----------------------------------------------------------------------
@pytest.mark.tier2
@_HEAVY_SKIP
def test_smoke_satellite_provider_inventory_contract() -> None:
# Arrange
base_url = _resolve_satellite_provider_url()
bearer = _mint_bearer_token_or_skip()
tile_x, tile_y = WgsConverter.latlon_to_tile_xy(
_DERKACHI_TILE_ZOOM, _DERKACHI_LAT, _DERKACHI_LON
)
body = {
"tiles": [
{
"tileZoom": _DERKACHI_TILE_ZOOM,
"tileX": tile_x,
"tileY": tile_y,
}
]
}
# Act
with _make_http_client(base_url) as client:
response = client.post(
_INVENTORY_PATH,
json=body,
headers={"Authorization": f"Bearer {bearer}"},
)
# Assert — contract invariants from `tile-inventory.md` v1.0.0
assert response.status_code == 200, (
f"satellite-provider inventory POST returned "
f"{response.status_code}: {response.text!r}"
)
decoded = response.json()
assert isinstance(decoded, dict), f"expected JSON object, got {type(decoded)}"
assert "results" in decoded, f"missing 'results' key in {decoded!r}"
results = decoded["results"]
assert isinstance(results, list)
assert len(results) == len(body["tiles"]), (
# Inv-2: response order/length matches request order/length.
f"inventory response length {len(results)} != request length "
f"{len(body['tiles'])}"
)
entry = results[0]
assert entry["tileZoom"] == _DERKACHI_TILE_ZOOM
assert entry["tileX"] == tile_x
assert entry["tileY"] == tile_y
assert "present" in entry, f"missing 'present' in entry {entry!r}"
assert isinstance(entry["present"], bool)
if entry["present"]:
for required in ("id", "capturedAt", "source", "resolutionMPerPx"):
assert entry.get(required) is not None, (
f"present entry missing required field {required!r}: {entry!r}"
)
# ----------------------------------------------------------------------
# Test 2 — C11 HttpTileDownloader against the real service
# ----------------------------------------------------------------------
class _InMemoryC6Adapter:
"""Implements `_TileWriterLike` + `_BudgetEnforcerLike` in-memory.
The Phase 1 smoke does NOT exercise the real C6 store (that's
Phase 3 of AZ-777). It exercises the C11 wire end-to-end with a
stub C6 so the test stays scope-clean.
"""
def __init__(self) -> None:
self.write_calls: list[dict[str, Any]] = []
self.reserved_bytes: list[int] = []
self.exists_calls: list[tuple[int, float, float]] = []
def write_tile_for_download(
self,
*,
tile_blob: bytes,
zoom_level: int,
lat: float,
lon: float,
tile_size_meters: float,
tile_size_pixels: int,
capture_timestamp: datetime,
content_sha256_hex: str,
sector_class: str,
) -> str:
self.write_calls.append(
{
"zoom_level": zoom_level,
"lat": lat,
"lon": lon,
"tile_blob_len": len(tile_blob),
"sector_class": sector_class,
}
)
return "fresh"
def tile_already_present(
self, *, zoom_level: int, lat: float, lon: float
) -> bool:
self.exists_calls.append((zoom_level, lat, lon))
return False
def reserve_headroom(self, needed_bytes: int) -> object:
self.reserved_bytes.append(int(needed_bytes))
return object()
@pytest.mark.tier2
@_HEAVY_SKIP
def test_smoke_c11_download_via_http_pipeline(tmp_path: Path) -> None:
# Arrange
base_url = _resolve_satellite_provider_url()
bearer = _mint_bearer_token_or_skip()
adapter = _InMemoryC6Adapter()
cfg = C11Config(
satellite_provider_url=base_url,
service_api_key=bearer,
download_http_timeout_s=30.0,
download_max_5xx_retries=2,
download_max_retry_after_s=60,
download_resolution_floor_m_per_px=0.5,
)
logger = logging.getLogger("test_az777_smoke")
with _make_http_client(base_url) as http_client:
downloader = HttpTileDownloader(
http_client=http_client,
tile_writer=adapter, # type: ignore[arg-type]
budget_enforcer=adapter, # type: ignore[arg-type]
logger=logger,
config=cfg,
)
request = DownloadRequest(
flight_id=uuid.uuid4(),
bbox_min_lat=_DERKACHI_BBOX[0],
bbox_min_lon=_DERKACHI_BBOX[1],
bbox_max_lat=_DERKACHI_BBOX[2],
bbox_max_lon=_DERKACHI_BBOX[3],
zoom_levels=(_DERKACHI_TILE_ZOOM,),
sector_class=SectorClassification.STABLE_REAR,
cache_root=tmp_path,
)
# Act
report = downloader.download_tiles_for_area(request)
# Assert — Phase 1 smoke contract: the wire works regardless of
# catalog state. Pre-Phase-2 the catalog is empty → 0 downloads.
# Post-Phase-2 the catalog is seeded → ≥ 1 download. Both are valid
# outcomes for this test; the test fails only on a wire / contract
# break (any provider error would have raised before reaching this
# assertion block).
assert report.outcome == DownloadOutcome.SUCCESS
assert report.tiles_requested >= 0
assert report.tiles_downloaded == len(adapter.write_calls)
assert report.tiles_rejected_resolution >= 0
if report.tiles_downloaded > 0:
# Phase 2 catalog landed: verify the write actually carried the
# tile bytes we'd expect from a real GET /tiles/{z}/{x}/{y}.
assert all(call["tile_blob_len"] > 0 for call in adapter.write_calls)
assert all(call["zoom_level"] == _DERKACHI_TILE_ZOOM for call in adapter.write_calls)
@@ -1,16 +1,21 @@
"""AZ-316 ``HttpTileDownloader`` unit tests.
"""AZ-316 / AZ-777 ``HttpTileDownloader`` unit tests.
Covers AC-1 .. AC-12 plus the throughput NFR from
``_docs/02_tasks/todo/AZ-316_c11_tile_downloader.md``. Uses
:class:`httpx.MockTransport` for deterministic HTTP responses, a
list-backed log handler for log capture, and stub C6 stores so this
suite never depends on AZ-303 / AZ-305 / AZ-307 / AZ-308 internals.
``_docs/02_tasks/todo/AZ-316_c11_tile_downloader.md``, against the
AZ-777 contract update (POST ``/api/satellite/tiles/inventory`` for
bulk lookup + GET ``/tiles/{z}/{x}/{y}`` for body download — see
``../satellite-provider/_docs/02_document/contracts/api/tile-inventory.md``
v1.0.0). Uses :class:`httpx.MockTransport` for deterministic HTTP
responses, a list-backed log handler for log capture, and stub C6
stores so this suite never depends on AZ-303 / AZ-305 / AZ-307 /
AZ-308 internals.
"""
from __future__ import annotations
import json
import logging
import math
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
@@ -33,8 +38,14 @@ from gps_denied_onboard.components.c11_tile_manager import (
_BASE_URL = "https://parent-suite.test"
_LIST_PATH = "/api/satellite/tiles"
_INVENTORY_PATH = "/api/satellite/tiles/inventory"
_TILES_PATH_PREFIX = "/tiles/"
_API_KEY = "test-api-key-001"
# Mirror of c11.tile_downloader._DEFAULT_ESTIMATED_TILE_BYTES (AZ-777):
# the inventory contract no longer returns content-length hints, so the
# AZ-308 budget pre-check reserves this constant per `present=true`
# tile. Keep in sync with the production module.
_DEFAULT_ESTIMATED_TILE_BYTES = 50_000
# ----------------------------------------------------------------------
@@ -55,16 +66,22 @@ _StubFreshnessRejection.__name__ = "FreshnessRejectionError"
class _StubTileWriter:
"""Captures `write_tile_for_download` calls + scripts the freshness label."""
"""Captures `write_tile_for_download` calls + scripts the freshness label.
AZ-777: keyed by *call index* rather than `(z, lat, lon)` strings.
The downloader now derives (lat, lon) from the slippy-map coord, so
tests can no longer fabricate arbitrary lat/lons in their fixtures;
the call-order is the contract.
"""
def __init__(
self,
*,
labels: dict[str, str] | None = None,
rejected: set[str] | None = None,
labels_by_index: dict[int, str] | None = None,
rejected_indices: set[int] | None = None,
) -> None:
self.labels = labels or {}
self.rejected = rejected or set()
self.labels_by_index = labels_by_index or {}
self.rejected_indices = rejected_indices or set()
self.write_calls: list[dict[str, Any]] = []
self.exists_calls: list[tuple[int, float, float]] = []
@@ -81,18 +98,23 @@ class _StubTileWriter:
content_sha256_hex: str,
sector_class: str,
) -> str:
tid = _tid(zoom_level, lat, lon)
call_index = len(self.write_calls)
self.write_calls.append(
{
"tile_id": tid,
"call_index": call_index,
"zoom_level": zoom_level,
"lat": lat,
"lon": lon,
"tile_blob_len": len(tile_blob),
"content_sha256_hex": content_sha256_hex,
"sector_class": sector_class,
}
)
if tid in self.rejected:
raise _StubFreshnessRejection(f"freshness rejected {tid}")
return self.labels.get(tid, "fresh")
if call_index in self.rejected_indices:
raise _StubFreshnessRejection(
f"freshness rejected call_index={call_index}"
)
return self.labels_by_index.get(call_index, "fresh")
def tile_already_present(
self, *, zoom_level: int, lat: float, lon: float
@@ -115,10 +137,6 @@ class _StubBudgetEnforcer:
return object()
def _tid(zoom: int, lat: float, lon: float) -> str:
return f"z{int(zoom)}_{float(lat):.6f}_{float(lon):.6f}"
def _build_downloader(
*,
transport: httpx.MockTransport,
@@ -172,71 +190,135 @@ def _build_downloader(
return downloader, log_records, writer, enforcer, sleeps
_DEFAULT_TEST_BBOX = (45.0, -122.5, 45.5, -122.0)
def _make_request(
*,
flight_id: Any | None = None,
cache_root: Path,
zoom_levels: tuple[int, ...] = (14,),
bbox: tuple[float, float, float, float] = _DEFAULT_TEST_BBOX,
) -> DownloadRequest:
return DownloadRequest(
flight_id=flight_id or uuid4(),
bbox_min_lat=45.0,
bbox_min_lon=-122.5,
bbox_max_lat=45.5,
bbox_max_lon=-122.0,
bbox_min_lat=bbox[0],
bbox_min_lon=bbox[1],
bbox_max_lat=bbox[2],
bbox_max_lon=bbox[3],
zoom_levels=zoom_levels,
sector_class=SectorClassification.STABLE_REAR,
cache_root=cache_root,
)
def _list_response(
tiles: list[dict[str, Any]] | None = None,
) -> httpx.Response:
return httpx.Response(200, json={"tiles": tiles or []})
def _tile_center_latlon_from_zxy(zoom: int, x: int, y: int) -> tuple[float, float]:
"""Mirror of c11.tile_downloader._tile_center_latlon for test assertions.
Both sides compute lat/lon from the slippy-map (z, x, y) tuple, so
stub freshness/label maps can be keyed on the *expected* lat/lon
without depending on the production helper at import time.
"""
n = 1 << int(zoom)
lat_n = math.degrees(math.atan(math.sinh(math.pi * (1.0 - 2.0 * int(y) / n))))
lat_s = math.degrees(math.atan(math.sinh(math.pi * (1.0 - 2.0 * (int(y) + 1) / n))))
lon_w = int(x) / n * 360.0 - 180.0
lon_e = (int(x) + 1) / n * 360.0 - 180.0
return (lat_n + lat_s) / 2.0, (lon_w + lon_e) / 2.0
def _tile_entry(
def _inventory_entry_for_coord(
*,
zoom: int,
lat: float,
lon: float,
x: int,
y: int,
present: bool = True,
resolution_m_per_px: float = 0.5,
estimated_bytes: int = 4096,
produced_at: datetime | None = None,
captured_at: datetime | None = None,
) -> dict[str, Any]:
produced = produced_at or datetime(2026, 5, 13, 0, 0, 0, tzinfo=timezone.utc)
if not present:
return {
"tileZoom": int(zoom),
"tileX": int(x),
"tileY": int(y),
"locationHash": str(uuid4()),
"present": False,
"id": None,
"capturedAt": None,
"source": None,
"flightId": None,
"resolutionMPerPx": None,
}
captured = captured_at or datetime(2026, 5, 13, 0, 0, 0, tzinfo=timezone.utc)
return {
"tile_id": _tid(zoom, lat, lon),
"zoom_level": zoom,
"lat": lat,
"lon": lon,
"produced_at": produced.isoformat(),
"resolution_m_per_px": resolution_m_per_px,
"estimated_bytes": estimated_bytes,
"tile_size_meters": 100.0,
"tile_size_pixels": 256,
"tileZoom": int(zoom),
"tileX": int(x),
"tileY": int(y),
"locationHash": str(uuid4()),
"present": True,
"id": str(uuid4()),
"capturedAt": captured.isoformat(),
"source": "google_maps",
"flightId": None,
"resolutionMPerPx": float(resolution_m_per_px),
}
def _make_route_handler(
def _make_inventory_handler(
*,
list_response: httpx.Response | None = None,
present_count: int | None = None,
resolution_override_for_first_n: tuple[int, float] | None = None,
tile_response_factory: Any = None,
inventory_response_override: httpx.Response | None = None,
) -> Any:
"""Route GETs by URL path: list endpoint vs per-tile endpoint."""
"""Route requests by method+path: POST inventory vs GET /tiles/{z}/{x}/{y}.
The inventory handler echoes the request's tile coords in
response order (per contract invariant Inv-2 / Inv-3). The
`present_count` knob marks the FIRST N entries as
``present=true`` and the rest as ``present=false``; ``None`` means
"all present". `resolution_override_for_first_n=(K, RES)` overrides
the resolution of the first K present entries (used by the AZ-316
resolution-gate test).
"""
def _handler(request: httpx.Request) -> httpx.Response:
path = request.url.path
is_list = (
path.endswith(_LIST_PATH)
and request.url.params.get("list-only") == "true"
method = request.method.upper()
if method == "POST" and path.endswith(_INVENTORY_PATH):
if inventory_response_override is not None:
return inventory_response_override
body = json.loads(request.content.decode("utf-8"))
tiles_in = body["tiles"]
results: list[dict[str, Any]] = []
for i, t in enumerate(tiles_in):
is_present = present_count is None or i < int(present_count)
if (
is_present
and resolution_override_for_first_n is not None
and i < int(resolution_override_for_first_n[0])
):
resolution = float(resolution_override_for_first_n[1])
else:
resolution = 0.5
results.append(
_inventory_entry_for_coord(
zoom=int(t["tileZoom"]),
x=int(t["tileX"]),
y=int(t["tileY"]),
present=is_present,
resolution_m_per_px=resolution,
)
)
return httpx.Response(200, json={"results": results})
if method == "GET" and path.startswith(_TILES_PATH_PREFIX):
if tile_response_factory is None:
return httpx.Response(200, content=b"\xff\xd8\xff\xe0fake-jpeg")
return tile_response_factory(request)
return httpx.Response(
404,
json={"detail": f"test handler: unexpected {method} {path}"},
)
if is_list:
return list_response or _list_response()
if tile_response_factory is None:
return httpx.Response(200, content=b"\xff\xd8\xff\xe0fake-jpeg")
return tile_response_factory(request)
return _handler
@@ -247,13 +329,10 @@ def _make_route_handler(
def test_ac1_100_tile_happy_path_writes_all(tmp_path: Path) -> None:
# Arrange
tiles = [
_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0 - i * 0.001)
for i in range(100)
]
# Arrange — bbox at zoom 14 produces N coord candidates; the
# stub marks the first 100 as `present=true` and the rest absent.
transport = httpx.MockTransport(
_make_route_handler(list_response=_list_response(tiles))
_make_inventory_handler(present_count=100)
)
(downloader, _logs, writer, enforcer, _sleeps) = _build_downloader(
transport=transport
@@ -271,7 +350,7 @@ def test_ac1_100_tile_happy_path_writes_all(tmp_path: Path) -> None:
assert report.tiles_rejected_freshness == 0
assert report.tiles_downgraded == 0
assert len(writer.write_calls) == 100
assert enforcer.calls == [4096 * 100]
assert enforcer.calls == [_DEFAULT_ESTIMATED_TILE_BYTES * 100]
# ----------------------------------------------------------------------
@@ -280,15 +359,12 @@ def test_ac1_100_tile_happy_path_writes_all(tmp_path: Path) -> None:
def test_ac2_resolution_gate_rejects_sub_spec_tiles(tmp_path: Path) -> None:
# Arrange
tiles = []
for i in range(50):
res = 0.3 if i < 10 else 0.5
tiles.append(
_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0, resolution_m_per_px=res)
)
# Arrange — 50 present tiles; first 10 below the 0.5 m/px floor.
transport = httpx.MockTransport(
_make_route_handler(list_response=_list_response(tiles))
_make_inventory_handler(
present_count=50,
resolution_override_for_first_n=(10, 0.3),
)
)
(downloader, log_records, writer, _enforcer, _sleeps) = _build_downloader(
transport=transport
@@ -311,13 +387,9 @@ def test_ac2_resolution_gate_rejects_sub_spec_tiles(tmp_path: Path) -> None:
def test_ac3_freshness_rejections_counted_and_run_continues(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0) for i in range(10)]
rejected_ids = {_tid(14, 45.0 + i * 0.001, -122.0) for i in range(5)}
transport = httpx.MockTransport(
_make_route_handler(list_response=_list_response(tiles))
)
writer = _StubTileWriter(rejected=rejected_ids)
# Arrange — 10 present tiles; c6 rejects the first 5 writes.
transport = httpx.MockTransport(_make_inventory_handler(present_count=10))
writer = _StubTileWriter(rejected_indices={0, 1, 2, 3, 4})
(downloader, log_records, _writer, _enforcer, _sleeps) = _build_downloader(
transport=transport, tile_writer=writer
)
@@ -341,13 +413,9 @@ def test_ac3_freshness_rejections_counted_and_run_continues(tmp_path: Path) -> N
def test_ac4_downgraded_label_counted_and_persisted(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0) for i in range(5)]
labels = {_tid(14, 45.0 + i * 0.001, -122.0): "downgraded" for i in range(3)}
transport = httpx.MockTransport(
_make_route_handler(list_response=_list_response(tiles))
)
writer = _StubTileWriter(labels=labels)
# Arrange — 5 present tiles; c6 returns "downgraded" for the first 3 writes.
transport = httpx.MockTransport(_make_inventory_handler(present_count=5))
writer = _StubTileWriter(labels_by_index={0: "downgraded", 1: "downgraded", 2: "downgraded"})
(downloader, _logs, _writer, _enforcer, _sleeps) = _build_downloader(
transport=transport, tile_writer=writer
)
@@ -366,8 +434,7 @@ def test_ac4_downgraded_label_counted_and_persisted(tmp_path: Path) -> None:
def test_ac5_429_honours_retry_after(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
# Arrange — single present tile; tile GET returns 429 once then 200.
state = {"attempts": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -377,8 +444,8 @@ def test_ac5_429_honours_retry_after(tmp_path: Path) -> None:
return httpx.Response(200, content=b"\xff\xd8tile")
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -403,8 +470,7 @@ def test_ac5_429_honours_retry_after(tmp_path: Path) -> None:
def test_ac6_persistent_5xx_raises_satellite_provider_error(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
# Arrange — single present tile; tile GET always 503.
state = {"attempts": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -412,8 +478,8 @@ def test_ac6_persistent_5xx_raises_satellite_provider_error(tmp_path: Path) -> N
return httpx.Response(503)
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -433,8 +499,7 @@ def test_ac6_persistent_5xx_raises_satellite_provider_error(tmp_path: Path) -> N
def test_ac7_401_fails_fast_no_retry(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
# Arrange — single present tile; tile GET returns 401.
state = {"attempts": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -442,8 +507,8 @@ def test_ac7_401_fails_fast_no_retry(tmp_path: Path) -> None:
return httpx.Response(401)
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -463,8 +528,7 @@ def test_ac7_401_fails_fast_no_retry(tmp_path: Path) -> None:
def test_ac8_idempotent_rerun_yields_no_op(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0) for i in range(5)]
# Arrange — 5 present tiles; both runs reach the same handler.
state = {"tile_gets": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -472,8 +536,8 @@ def test_ac8_idempotent_rerun_yields_no_op(tmp_path: Path) -> None:
return httpx.Response(200, content=b"\xff\xd8tile")
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=5,
tile_response_factory=_factory,
)
)
@@ -501,8 +565,7 @@ def test_ac8_idempotent_rerun_yields_no_op(tmp_path: Path) -> None:
def test_ac9_cache_budget_pre_check_aborts(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0, estimated_bytes=10_000)]
# Arrange — single present tile; budget enforcer rejects up-front.
transport_state = {"tile_gets": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -510,8 +573,8 @@ def test_ac9_cache_budget_pre_check_aborts(tmp_path: Path) -> None:
return httpx.Response(200, content=b"\xff\xd8tile")
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -526,7 +589,7 @@ def test_ac9_cache_budget_pre_check_aborts(tmp_path: Path) -> None:
with pytest.raises(CacheBudgetExceededError):
downloader.download_tiles_for_area(_make_request(cache_root=tmp_path))
assert transport_state["tile_gets"] == 0
assert enforcer.calls == [10_000]
assert enforcer.calls == [_DEFAULT_ESTIMATED_TILE_BYTES]
# ----------------------------------------------------------------------
@@ -536,16 +599,11 @@ def test_ac9_cache_budget_pre_check_aborts(tmp_path: Path) -> None:
def test_ac11_service_api_key_never_appears_in_logs(tmp_path: Path) -> None:
# Arrange — exercise the failure path so the provider-failed ERROR
# log fires (the code that explicitly redacts the auth header).
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
def _factory(request: httpx.Request) -> httpx.Response:
return httpx.Response(401)
# log fires (the code that explicitly redacts the auth header). The
# inventory POST returns 401 → fast-fail before any tile GET.
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
tile_response_factory=_factory,
_make_inventory_handler(
inventory_response_override=httpx.Response(401),
)
)
(downloader, log_records, _writer, _enforcer, _sleeps) = _build_downloader(
@@ -568,12 +626,10 @@ def test_ac11_service_api_key_never_appears_in_logs(tmp_path: Path) -> None:
def test_ac12_partial_journal_resumed_on_rerun(tmp_path: Path) -> None:
# Arrange — 10 tiles; first run fetches all 10 successfully and
# leaves a complete journal. A second run with the SAME request
# must short-circuit (AC-8 covers that). To exercise AC-12 we
# MANUALLY truncate the journal between runs to simulate a crash
# AFTER 4 tile-writes, BEFORE the completed_at_iso stamp.
tiles = [_tile_entry(zoom=14, lat=45.0 + i * 0.001, lon=-122.0) for i in range(10)]
# Arrange — 10 present tiles; first run completes, then we truncate
# the journal to simulate a crash after 4 writes (clear
# `completed_at_iso`) so the second run must complete the remaining
# 6 tiles instead of short-circuiting on AC-8 idempotence.
state = {"tile_gets": 0}
def _factory(request: httpx.Request) -> httpx.Response:
@@ -581,8 +637,8 @@ def test_ac12_partial_journal_resumed_on_rerun(tmp_path: Path) -> None:
return httpx.Response(200, content=b"\xff\xd8tile")
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=10,
tile_response_factory=_factory,
)
)
@@ -637,8 +693,7 @@ def test_ac12_partial_journal_resumed_on_rerun(tmp_path: Path) -> None:
def test_429_retry_after_http_date_form_parses(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
# Arrange — single present tile; first tile GET 429 with HTTP-date Retry-After.
state = {"attempts": 0}
future = (datetime.now(timezone.utc) + timedelta(seconds=20)).strftime(
"%a, %d %b %Y %H:%M:%S GMT"
@@ -651,8 +706,8 @@ def test_429_retry_after_http_date_form_parses(tmp_path: Path) -> None:
return httpx.Response(200, content=b"\xff\xd8tile")
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -676,15 +731,15 @@ def test_429_retry_after_http_date_form_parses(tmp_path: Path) -> None:
def test_429_budget_exhaustion_raises_rate_limited_error(tmp_path: Path) -> None:
# Arrange
tiles = [_tile_entry(zoom=14, lat=45.0, lon=-122.0)]
# Arrange — single present tile; tile GET always 429 with very long
# Retry-After. After the configured retry budget is exhausted the
# client raises RateLimitedError.
def _factory(request: httpx.Request) -> httpx.Response:
return httpx.Response(429, headers={"Retry-After": "300"})
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
_make_inventory_handler(
present_count=1,
tile_response_factory=_factory,
)
)
@@ -711,25 +766,31 @@ def test_429_budget_exhaustion_raises_rate_limited_error(tmp_path: Path) -> None
def test_nfr_throughput_1000_tiles_under_budget(tmp_path: Path) -> None:
# Arrange
tiles = [
_tile_entry(zoom=14, lat=45.0 + i * 0.0001, lon=-122.0 + i * 0.0001)
for i in range(1000)
]
# Arrange — 1000 present tiles. Use a bbox wide enough to enumerate
# ≥1000 tile coords at zoom 14 (≈0.022° per tile at ~zoom 14).
transport = httpx.MockTransport(
_make_route_handler(
list_response=_list_response(tiles),
tile_response_factory=lambda r: httpx.Response(200, content=b"\xff\xd8tile"),
_make_inventory_handler(
present_count=1000,
tile_response_factory=lambda r: httpx.Response(
200, content=b"\xff\xd8tile"
),
)
)
(downloader, _logs, _writer, _enforcer, _sleeps) = _build_downloader(
transport=transport
)
# Big bbox at zoom 14: ~ 32x32 tile span on this latitude is enough.
# 1° ≈ 45 tiles at zoom 14 in latitude → 0.75° gives ≈ 33 tiles → ~1089 tiles.
request = _make_request(
cache_root=tmp_path,
bbox=(44.0, -123.0, 44.75, -122.25),
)
import time as _time
t0 = _time.perf_counter()
report = downloader.download_tiles_for_area(_make_request(cache_root=tmp_path))
report = downloader.download_tiles_for_area(request)
elapsed = _time.perf_counter() - t0
# Assert — budget is generous; the goal is to catch an O(n^2)