azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 18:01:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-777] Phase 1: wire e2e-runner to real satellite-provider + C11 contract adapt

Browse Source 
Adapt C11 HttpTileDownloader to the AZ-505 v1.0.0 tile-inventory
contract (POST /api/satellite/tiles/inventory + GET /tiles/{z}/{x}/{y})
and wire the Jetson e2e harness against the real parent-suite
satellite-provider service. Closes Phase 1 of 5 for AZ-777; STOP
gate before Phase 2 (Derkachi catalog seed).

C11 changes:
- _LIST_PATH / _GET_PATH replaced with _INVENTORY_PATH + _TILES_PATH.
- _do_enumerate enumerates bbox tile coords client-side and posts
  chunked inventory requests (5000-entry cap per the contract).
- _download_one_tile parses tile_id_str into (z,x,y) and fetches
  the slippy-map URL.
- Common GET / POST retry+auth ladder consolidated into _send_request.
- New module helpers: _enumerate_bbox_tile_coords,
  _tile_center_latlon, _tile_size_meters_at, _format_tile_id_str,
  _parse_tile_id_str, _chunk_iter.
- _DEFAULT_ESTIMATED_TILE_BYTES (50 KiB) replaces the inventory-side
  estimatedBytes field the v1.0.0 contract dropped.

Tests:
- 14/14 unit tests in tests/unit/c11_tile_manager/test_tile_downloader.py
  rewritten for the new POST inventory + slippy-map GET handler.
  _StubTileWriter rekeyed by call-index (the downloader now derives
  lat/lon from the slippy-map coord, so fixtures can't fabricate
  arbitrary positions).
- New Tier-2 smoke at tests/e2e/satellite_provider/test_smoke.py:
  validates inventory POST schema + drives HttpTileDownloader against
  the real service. Gated by RUN_REPLAY_E2E=1 + tier2.

Compose / env:
- e2e-runner SATELLITE_PROVIDER_URL switched from mock-sat:5100 to
  https://satellite-provider:8080; TLS_INSECURE + Bearer JWT env +
  depends_on satellite-provider added.
- .env.test.example documents SATELLITE_PROVIDER_API_KEY + dev TLS
  bypass security note.
- scripts/mint_dev_jwt.py mints HS256 dev JWTs from env / .env.test.
- pyjwt added to dev extras.

Tracker hygiene:
- AZ-777 row in _dependencies_table.md bumped 5pt -> 8pt to match
  the 2026-05-21 override decision log.

Code review: PASS_WITH_WARNINGS (3 medium/low findings, all deferred
to later AZ-777 phases) -- see batch_104_review.md. Batch report at
batch_104_cycle3_report.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-21 14:52:39 +03:00
parent 544b37fdc9
commit 811b04e605
12 changed files with 1328 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/gps_denied_onboard/components/c11_tile_manager/tile_downloader.py
+254 -42
View File
@@ -1,11 +1,23 @@
"""C11 ``HttpTileDownloader`` (AZ-316) — concrete :class:`TileDownloader`.
"""C11 ``HttpTileDownloader`` (AZ-316, AZ-777) — concrete :class:`TileDownloader`.
Operator-side pre-flight download path. Authenticated GETs against
``satellite-provider``, RESTRICT-SAT-4 enforcement at the C11 boundary,
c6 writes via the AZ-303 store + metadata Protocols (which run AZ-307's
freshness gate at insert), AZ-308 cache-headroom pre-check before any
GET fires, and a per-``(flight_id, request_hash)`` journal for
idempotent re-runs.
Operator-side pre-flight download path. Authenticated POST inventory
lookups + slippy-map GETs against ``satellite-provider``, RESTRICT-SAT-4
enforcement at the C11 boundary, c6 writes via the AZ-303 store +
metadata Protocols (which run AZ-307's freshness gate at insert),
AZ-308 cache-headroom pre-check before any GET fires (using a
configured per-tile bytes estimate, since the inventory contract does
not return content-length hints), and a per-``(flight_id,
request_hash)`` journal for idempotent re-runs.
Contract surface (AZ-777, against the parent-suite
``satellite-provider`` v1.0.0 inventory contract — see
``../satellite-provider/_docs/02_document/contracts/api/tile-inventory.md``):
* ``POST /api/satellite/tiles/inventory`` — bulk lookup of (z,x,y) tile
coords; returns one entry per request item with ``present: true|false``
and (when present) the metadata C11 needs to drive the resolution gate
and the c6 write.
* ``GET /tiles/{z}/{x}/{y}`` — slippy-map tile fetch by coords.
Architecture
------------
@@ -26,6 +38,7 @@ from __future__ import annotations
import hashlib
import json
import logging
import math
import os
import tempfile
from dataclasses import dataclass, field
@@ -50,6 +63,7 @@ from gps_denied_onboard.components.c11_tile_manager.errors import (
RateLimitedError,
SatelliteProviderError,
)
from gps_denied_onboard.helpers.wgs_converter import WgsConverter
__all__ = [
"DOWNLOAD_JOURNAL_DIRNAME",
@@ -58,9 +72,24 @@ __all__ = [
]
_LIST_PATH = "/api/satellite/tiles"
_GET_PATH = "/api/satellite/tiles"
_LIST_QUERY_LIST_ONLY = "list-only"
# AZ-777: parent-suite contract v1.0.0 (see module docstring).
_INVENTORY_PATH = "/api/satellite/tiles/inventory"
_TILES_PATH = "/tiles"
_INVENTORY_MAX_ENTRIES_PER_REQUEST = 5000
# Inventory response does not carry a content-length hint, but the
# AZ-308 budget pre-check needs a per-tile estimate before any GET
# fires. 50 KiB is conservative for 256x256 JPEG basemap tiles
# (typical CARTO Voyager tiles run 8-30 KiB; UAV-captured uploads
# run 30-80 KiB). Sized to over-reserve rather than under-reserve.
_DEFAULT_ESTIMATED_TILE_BYTES = 50_000
# Web-Mercator at zoom 0 covers the full equatorial circumference of
# the WGS-84 ellipsoid (≈40 075 016.686 m). Tile ground size at any
# (zoom, lat) follows: circumference * cos(lat_rad) / 2^zoom (the
# cos(lat) factor is the same projection-stretch correction the
# parent-suite uses to compute ``resolutionMPerPx``).
_EARTH_EQUATORIAL_CIRCUMFERENCE_M = 40_075_016.686
_TILE_SIZE_PIXELS = 256
DOWNLOAD_JOURNAL_DIRNAME = ".c11/journal"
_LOCKFILE_PATH = ".c11/lock"
_DEFAULT_BACKOFF_SCHEDULE_S: tuple[float, ...] = (1.0, 2.0, 4.0, 8.0)
@@ -295,6 +324,93 @@ def _default_sleep(seconds: float) -> None:
clock.sleep_until_ns(clock.monotonic_ns() + int(seconds * 1_000_000_000))
# ----------------------------------------------------------------------
# AZ-777 slippy-map helpers
#
# The parent-suite inventory contract (v1.0.0) is keyed by explicit
# (z, x, y) slippy-map coords. C11 enumerates the grid from the bbox
# locally and converts inventory hits back into lat/lon for c6 writes.
# Math matches the parent suite's Web-Mercator projection so the
# resolution / tile-size hints round-trip identically.
# ----------------------------------------------------------------------
def _enumerate_bbox_tile_coords(
bbox_min_lat: float,
bbox_min_lon: float,
bbox_max_lat: float,
bbox_max_lon: float,
zoom_levels: tuple[int, ...],
) -> tuple[tuple[int, int, int], ...]:
"""Return every (z, x, y) whose tile bounds intersect the bbox.
Slippy-map y grows southward, so the SW corner has (low x, high y)
and the NE corner has (high x, low y). The enumeration is inclusive
on both ends.
"""
coords: list[tuple[int, int, int]] = []
for zoom in zoom_levels:
x_sw, y_sw = WgsConverter.latlon_to_tile_xy(
int(zoom), bbox_min_lat, bbox_min_lon
)
x_ne, y_ne = WgsConverter.latlon_to_tile_xy(
int(zoom), bbox_max_lat, bbox_max_lon
)
x_lo, x_hi = (x_sw, x_ne) if x_sw <= x_ne else (x_ne, x_sw)
y_lo, y_hi = (y_ne, y_sw) if y_ne <= y_sw else (y_sw, y_ne)
for x in range(x_lo, x_hi + 1):
for y in range(y_lo, y_hi + 1):
coords.append((int(zoom), x, y))
return tuple(coords)
def _tile_center_latlon(zoom: int, x: int, y: int) -> tuple[float, float]:
bounds = WgsConverter.tile_xy_to_latlon_bounds(int(zoom), int(x), int(y))
lat = (bounds.min_lat_deg + bounds.max_lat_deg) / 2.0
lon = (bounds.min_lon_deg + bounds.max_lon_deg) / 2.0
return lat, lon
def _tile_size_meters_at(zoom: int, lat_deg: float) -> float:
return (
_EARTH_EQUATORIAL_CIRCUMFERENCE_M
* math.cos(math.radians(lat_deg))
/ (1 << int(zoom))
)
def _format_tile_id_str(zoom: int, x: int, y: int) -> str:
return f"{int(zoom)}_{int(x)}_{int(y)}"
def _parse_tile_id_str(tile_id_str: str) -> tuple[int, int, int]:
parts = tile_id_str.split("_")
if len(parts) != 3:
raise ValueError(
f"tile_id_str must be 'z_x_y'; got {tile_id_str!r}"
)
try:
return int(parts[0]), int(parts[1]), int(parts[2])
except ValueError as exc:
raise ValueError(
f"tile_id_str must contain three integers separated by '_'; "
f"got {tile_id_str!r}"
) from exc
def _chunk_iter(
seq: tuple[tuple[int, int, int], ...],
chunk_size: int,
) -> list[tuple[tuple[int, int, int], ...]]:
if chunk_size <= 0:
raise ValueError(f"chunk_size must be > 0; got {chunk_size}")
return [
tuple(seq[start : start + chunk_size])
for start in range(0, len(seq), chunk_size)
]
# ----------------------------------------------------------------------
# Internal session-state container
# ----------------------------------------------------------------------
@@ -546,58 +662,119 @@ class HttpTileDownloader:
bbox_max_lon: float,
zoom_levels: tuple[int, ...],
) -> list[TileSummary]:
params = {
"bbox": f"{bbox_min_lat},{bbox_min_lon},{bbox_max_lat},{bbox_max_lon}",
"zoom": ",".join(str(z) for z in zoom_levels),
_LIST_QUERY_LIST_ONLY: "true",
"""POST ``/api/satellite/tiles/inventory`` for every (z,x,y) in bbox.
AZ-777: the satellite-provider v1.0.0 inventory contract is
keyed by explicit slippy-map coords, NOT by a server-side
bbox query. This method enumerates the tile grid for the
bbox × zoom set, chunks into ≤5000-entry POSTs (the
``TileInventoryLimits.MaxEntriesPerRequest`` cap), and
returns one :class:`TileSummary` per ``present=true`` entry.
Absent tiles are silently dropped — they need to be seeded
via ``POST /api/satellite/request`` upstream before they
become downloadable.
"""
tile_coords = _enumerate_bbox_tile_coords(
bbox_min_lat,
bbox_min_lon,
bbox_max_lat,
bbox_max_lon,
zoom_levels,
)
if not tile_coords:
return []
summaries: list[TileSummary] = []
for chunk in _chunk_iter(tile_coords, _INVENTORY_MAX_ENTRIES_PER_REQUEST):
summaries.extend(self._fetch_inventory_chunk(chunk))
return summaries
def _fetch_inventory_chunk(
self, chunk: tuple[tuple[int, int, int], ...]
) -> list[TileSummary]:
body = {
"tiles": [
{"tileZoom": z, "tileX": x, "tileY": y}
for (z, x, y) in chunk
]
}
response = self._send_get(
self._config.satellite_provider_url.rstrip("/") + _LIST_PATH,
params=params,
response = self._send_post(
self._config.satellite_provider_url.rstrip("/") + _INVENTORY_PATH,
json_body=body,
session=None,
)
try:
body = response.json()
decoded = response.json()
except ValueError as exc:
self._log_provider_failure(
"list_not_json", response.status_code, str(exc)
"inventory_not_json", response.status_code, str(exc)
)
raise SatelliteProviderError(
"satellite-provider returned non-JSON list-only body"
"satellite-provider returned non-JSON inventory body"
) from exc
try:
entries = body["tiles"]
entries = decoded["results"]
except (KeyError, TypeError) as exc:
self._log_provider_failure(
"list_schema", response.status_code, str(exc)
"inventory_schema", response.status_code, str(exc)
)
raise SatelliteProviderError(
"satellite-provider list-only response missing 'tiles'"
"satellite-provider inventory response missing 'results'"
) from exc
if len(entries) != len(chunk):
self._log_provider_failure(
"inventory_order",
response.status_code,
f"results.len={len(entries)} request.tiles.len={len(chunk)}",
)
raise SatelliteProviderError(
f"satellite-provider inventory response broke order invariant: "
f"len(results)={len(entries)} != len(request.tiles)={len(chunk)}"
)
summaries: list[TileSummary] = []
for entry in entries:
try:
summaries.append(
TileSummary(
tile_id_str=str(entry["tile_id"]),
zoom_level=int(entry["zoom_level"]),
lat=float(entry["lat"]),
lon=float(entry["lon"]),
produced_at=_parse_iso(str(entry["produced_at"])),
resolution_m_per_px=float(entry["resolution_m_per_px"]),
estimated_bytes=int(entry["estimated_bytes"]),
tile_size_meters=float(entry.get("tile_size_meters", 100.0)),
tile_size_pixels=int(entry.get("tile_size_pixels", 256)),
)
)
except (KeyError, TypeError, ValueError) as exc:
present = bool(entry["present"])
except (KeyError, TypeError) as exc:
self._log_provider_failure(
"list_tile_schema", response.status_code, str(exc)
"inventory_entry_schema", response.status_code, str(exc)
)
raise SatelliteProviderError(
"satellite-provider list-only entry missing required fields"
"satellite-provider inventory entry missing 'present'"
) from exc
if not present:
continue
try:
zoom = int(entry["tileZoom"])
x = int(entry["tileX"])
y = int(entry["tileY"])
produced_at = _parse_iso(str(entry["capturedAt"]))
resolution_m_per_px = float(entry["resolutionMPerPx"])
except (KeyError, TypeError, ValueError) as exc:
self._log_provider_failure(
"inventory_entry_schema", response.status_code, str(exc)
)
raise SatelliteProviderError(
"satellite-provider inventory present-entry missing "
"required fields (tileZoom/tileX/tileY/capturedAt/"
"resolutionMPerPx)"
) from exc
lat, lon = _tile_center_latlon(zoom, x, y)
summaries.append(
TileSummary(
tile_id_str=_format_tile_id_str(zoom, x, y),
zoom_level=zoom,
lat=lat,
lon=lon,
produced_at=produced_at,
resolution_m_per_px=resolution_m_per_px,
estimated_bytes=_DEFAULT_ESTIMATED_TILE_BYTES,
tile_size_meters=_tile_size_meters_at(zoom, lat),
tile_size_pixels=_TILE_SIZE_PIXELS,
)
)
return summaries
def _reserve_budget(
@@ -648,10 +825,16 @@ class HttpTileDownloader:
)
return
try:
zoom, x, y = _parse_tile_id_str(summary.tile_id_str)
except ValueError as exc:
raise SatelliteProviderError(
f"internal: TileSummary.tile_id_str does not match the AZ-777 "
f"z_x_y format (got {summary.tile_id_str!r})"
) from exc
ingest_url = (
self._config.satellite_provider_url.rstrip("/")
+ _GET_PATH
+ f"/{summary.tile_id_str}"
+ f"{_TILES_PATH}/{zoom}/{x}/{y}"
)
response = self._send_get(ingest_url, params=None, session=session)
if not response.content:
@@ -717,15 +900,44 @@ class HttpTileDownloader:
) -> httpx.Response:
"""GET with auth header + 429 / 5xx handling."""
return self._send_request(
"GET", url, params=params, json_body=None, session=session
)
def _send_post(
self,
url: str,
json_body: Any,
session: _DownloadSession | None,
) -> httpx.Response:
"""POST with auth header + 429 / 5xx handling (AZ-777 inventory contract)."""
return self._send_request(
"POST", url, params=None, json_body=json_body, session=session
)
def _send_request(
self,
method: str,
url: str,
*,
params: dict[str, str] | None,
json_body: Any,
session: _DownloadSession | None,
) -> httpx.Response:
"""Auth header + 429 / 5xx handling for GET and POST."""
headers = {"Authorization": f"Bearer {self._config.service_api_key}"}
attempt = 0
last_error: str | None = None
while True:
attempt += 1
try:
response = self._http_client.get(
response = self._http_client.request(
method,
url,
params=params,
json=json_body,
headers=headers,
timeout=self._config.download_http_timeout_s,
)