mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 12:31:13 +00:00
Oleksandr Bezdieniezhnykh
b15454b9a9
Phase 1 hotfix:
- C11 HttpTileDownloader adapted to satellite-provider v2.0.0
z/x/y inventory contract (bulk POST keyed by slippy-map coords).
- Unit tests rewritten to exercise the new inventory schema.
- E2E smoke test updated to match the v2.0.0 wire.
Phase 2 (Derkachi seed + smoke-validated on Jetson):
- tests/fixtures/derkachi_c6/{README,bbox.yaml,seed_region.py}
drives POST /api/satellite/region against satellite-provider
with Google Maps as the imagery source. Smoke run produced
4 regions, 175 tiles, inventory 32/32.
- scripts/mint_dev_jwt.py + run-tests-jetson.sh auto-mint and
export SATELLITE_PROVIDER_API_KEY using JWT_SECRET / JWT_ISSUER
/ JWT_AUDIENCE env vars (no host port mappings; e2e-runner
reaches SP via internal docker network only).
Spec amendment: AZ-777 todo spec updated to record the
Google Maps imagery source decision and STOP-gate state.
AZ-777 Phase 3+ work is superseded by Epic AZ-835 (see next
commit).
Co-authored-by: Cursor <cursoragent@cursor.com>
121 lines
7.5 KiB
Markdown
121 lines
7.5 KiB
Markdown
# Derkachi reference C6 tile catalog — fixture seeding
|
||
|
||
**AZ-777 Phase 2 deliverable.** Seeds the parent-suite `satellite-provider` DB with the satellite tiles the C6 reference catalog needs for the Derkachi replay tests (`test_ac3_within_100m_80pct_of_ticks` on AC-4 + `test_az699_real_flight_validation_emits_verdict_and_report` on AC-5).
|
||
|
||
## What this folder contains
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `bbox.yaml` | bbox + zoom levels + actual flight extent + imagery source metadata + license attribution + chunking strategy |
|
||
| `seed_region.py` | Python script that submits `POST /api/satellite/request` to satellite-provider for each (zoom × chunk) and polls until completion |
|
||
| `README.md` | this file |
|
||
|
||
## Prerequisites
|
||
|
||
1. **Running satellite-provider.** Typically the Jetson e2e harness via `docker-compose.test.jetson.yml` (services `satellite-provider` + `satellite-provider-postgres`). Verify it's up and healthy:
|
||
|
||
```bash
|
||
ssh jetson-e2e "docker ps --filter name=satellite --format 'table {{.Names}}\t{{.Status}}'"
|
||
```
|
||
|
||
2. **`SATELLITE_PROVIDER_URL`** in `.env.test` (already covered by the AZ-777 Phase 1 wiring).
|
||
|
||
3. **`SATELLITE_PROVIDER_API_KEY`** — a valid HS256 JWT signed with the same `JWT_SECRET` the satellite-provider validates against. Mint with:
|
||
|
||
```bash
|
||
export SATELLITE_PROVIDER_API_KEY="$(python scripts/mint_dev_jwt.py)"
|
||
```
|
||
|
||
4. **Google Maps Platform API key** on the satellite-provider side (env `GOOGLE_MAPS_API_KEY` / config `MapConfig__ApiKey`). The satellite-provider uses this to actually download the tiles from Google Maps. The seed script does NOT need this key — it only triggers the producer's async download pipeline.
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
# from gps-denied-onboard repo root, with satellite-provider running on Jetson:
|
||
export SATELLITE_PROVIDER_API_KEY="$(python scripts/mint_dev_jwt.py)"
|
||
python tests/fixtures/derkachi_c6/seed_region.py
|
||
```
|
||
|
||
Expected runtime: ~5-15 minutes for the full spec bbox (8 region calls × ~30-60s each + inventory verification).
|
||
|
||
## Flags
|
||
|
||
```
|
||
--bbox-config PATH override bbox.yaml location
|
||
--env-file PATH override .env.test fallback location
|
||
--output-summary PATH write a JSON summary for downstream consumers
|
||
--dry-run validate config + plan without submitting
|
||
--right-sized-flight use the actual ~1 km^2 flight extent (98% fewer tiles)
|
||
--skip-poll submit + return; don't wait for terminal status
|
||
--skip-inventory-verification skip the final coverage check
|
||
```
|
||
|
||
## bbox sizing — important
|
||
|
||
`bbox.yaml` ships TWO bboxes:
|
||
|
||
* `bbox`: per AZ-777 spec — covers ~11.1 × 7.14 km (~80 km²) of the Derkachi village area. **~4570 tiles z15-z18 (~57 MB)**. Default seeding target.
|
||
* `actual_flight_extent`: the real Derkachi flight footprint per `data_imu.csv` — only ~254 × 457m (~0.12 km²) centered at (50.082, 36.110). **~60 tiles z15-z18 (~1 MB)** if seeded right-sized via `--right-sized-flight`.
|
||
|
||
The spec bbox is ~300× larger than the actual flight extent. The spec sizing is intentional generality — operators can fly any route within the box without re-seeding. The right-sized mode is appropriate when only the specific Derkachi clip needs coverage (e.g., CI test runs).
|
||
|
||
## Imagery source — IMPORTANT licensing note
|
||
|
||
AZ-777 was originally specced with **CARTO Voyager Basemap (CC-BY-3.0)** as the upstream imagery source. The 2026-05-22 black-box probe of the running satellite-provider revealed the actual upstream is **Google Maps satellite layer** (`mt0..mt3.google.com/vt/lyrs=s`). The AZ-777 spec was amended to reflect this reality (see Risk 4 in `_docs/02_tasks/todo/AZ-777_derkachi_c6_reference_fixture.md`).
|
||
|
||
**Operators MUST propagate the attribution string `"Imagery © Google"` to any end-user-visible context** that incorporates tiles seeded by this script. Per `bbox.yaml::license`.
|
||
|
||
**Dev/research use is approved. Production deploy requires either:**
|
||
|
||
1. Google Maps Platform licensing review for the offline-cache use case (the C6 reference dataset is a long-lived stored cache, which Google Maps ToS may restrict), OR
|
||
2. A parent-suite ticket to add a true CC-BY satellite imagery provider to satellite-provider (candidates: Esri World Imagery, Mapbox satellite, Sentinel-2 via Copernicus). TBD; not in scope for AZ-777.
|
||
|
||
## Re-seeding (after a satellite-provider DB wipe)
|
||
|
||
The script is **idempotent and safe to re-run**:
|
||
|
||
* Each invocation generates fresh region UUIDs, so each run creates a new set of region records on the producer side.
|
||
* The producer's tile-storage layer dedups via UPSERT on (zoom, x, y), so tiles already downloaded from Google Maps are NOT re-fetched — they're counted as `tilesReused` instead.
|
||
* Re-runs are cheap (just the region-tracking overhead) when the DB is warm.
|
||
|
||
To verify the catalog is populated without re-running the full seed, query inventory directly:
|
||
|
||
```bash
|
||
# inside the satellite-provider docker network:
|
||
docker run --rm --network gps-denied-onboard_default curlimages/curl:8.10.1 \
|
||
-sk -X POST -H "Authorization: Bearer $SATELLITE_PROVIDER_API_KEY" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"tiles":[{"z":18,"x":157497,"y":89000}]}' \
|
||
https://satellite-provider:8080/api/satellite/tiles/inventory
|
||
```
|
||
|
||
## Cost expectations
|
||
|
||
Per the 2026-05-22 probe baseline (200m @ z18 = 9 tiles, ~13 KB/tile, ~5s end-to-end):
|
||
|
||
| Mode | Tile count | DB size | Wall time (cold) | Wall time (warm DB) |
|
||
|------|------------|---------|------------------|---------------------|
|
||
| Spec bbox (~80 km²) | ~4570 | ~57 MB | ~5-15 min | ~30s (reuse) |
|
||
| Right-sized (~1 km²) | ~60 | ~1 MB | ~1-2 min | ~10s (reuse) |
|
||
|
||
Google Maps API cost per tile depends on the satellite-provider operator's Maps Platform pricing tier. The seed script does NOT bill — the producer's Google Maps account does.
|
||
|
||
## Failure modes
|
||
|
||
| Exit code | Meaning | Likely cause |
|
||
|-----------|---------|--------------|
|
||
| 71 | config file missing / malformed | `bbox.yaml` corrupted or wrong path |
|
||
| 72 | required env var missing | `SATELLITE_PROVIDER_URL` or `SATELLITE_PROVIDER_API_KEY` not set |
|
||
| 73 | satellite-provider unreachable | Service down, wrong URL, or TLS handshake failed (try `SATELLITE_PROVIDER_TLS_INSECURE=1`) |
|
||
| 74 | region request rejected | HTTP 4xx (auth, validation) or 5xx (producer crash); see stderr for HTTP body |
|
||
| 75 | one or more regions failed | Background processing failed — usually a Google Maps API quota / key issue on the producer side. Check `docker logs gps-denied-e2e-satellite-provider` |
|
||
| 76 | inventory verification mismatch | < 95% of expected tiles present; re-run to retry, or investigate producer logs |
|
||
|
||
## Cross-references
|
||
|
||
* AZ-777 spec: `_docs/02_tasks/todo/AZ-777_derkachi_c6_reference_fixture.md`
|
||
* AZ-777 Phase 1 (the wiring that makes this script callable): completed cycle 3 batch 105
|
||
* AZ-808 (parent-suite): strict validation for region-request endpoint — when this lands, malformed `seed_region.py` invocations will fail with RFC 7807 ValidationProblemDetails instead of silent zero-coercion; coordinate any consumer-side changes with that release
|
||
* AZ-812 (parent-suite): rename `RequestRegionRequest.{Latitude, Longitude}` → `{Lat, Lon}` for OSM consistency — when this lands, `seed_region.py` must be updated to send `lat`/`lon` instead of `latitude`/`longitude`
|
||
* satellite-provider Region API contract: today informally documented in `../../../../satellite-provider/_docs/02_document/modules/common_dtos.md::RegionRequest` + `system-flows.md` Flow F2; formal `region-request.md` contract will be published as part of AZ-808
|