[AZ-700] gps-denied-render-map: HTML map of estimated vs truth tracks

New operator-side console-script renders a self-contained HTML map
(folium / Leaflet) comparing the estimator's JSONL track against
the tlog ground-truth track. Pinned visual style: red truth + blue
estimated polylines, start/end markers per track, 100 m + 50 m
scale circles, optional AZ-699 accuracy-summary banner, and an
--offline-tiles mode (with optional local tile-URL template) for
Jetsons without internet.

folium is gated behind a new [operator-tools] optional-dep so the
airborne binary's cold-start NFR is unaffected (C12 binary doesn't
import the new module). 14 new unit tests pin polyline count,
marker count, scale-circle radii, summary embedding, offline-tile
behaviour, and full CLI smoke. Zero mypy --strict errors.

Refines the 2026-05-20 Jetson-only test policy: unit tests may run
locally, e2e/perf/resilience/security stay Jetson-only. Documented
in _docs/02_document/tests/environment.md (Where each tier runs)
and .cursor/rules/testing.mdc (Test environment for this project).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-20 17:04:01 +03:00
parent dcde602f61
commit b66b68ff76
8 changed files with 943 additions and 17 deletions
+48 -15
View File
@@ -1,17 +1,40 @@
# Test Environment
> **Active policy — 2026-05-20**: **all tests run on Jetson only.** The Jetson
> Orin Nano Super (or a Jetson-equivalent arm64 agent) is the single canonical
> test environment for every tier of testing — unit, integration, blackbox /
> e2e, performance, resilience, security, resource-limit. Workstation x86
> Docker (the historical "Tier-1" path) is **deprecated** and is not a
> supported test environment going forward; the Tier-1 sections below are
> retained as historical reference / traceability only. CI test pipelines
> target the colocated arm64 Jetson Woodpecker agent (see
> `_docs/04_deploy/ci_cd_pipeline.md`); local-development test runs SHOULD
> use `scripts/run-tests-jetson.sh` against the configured `jetson-e2e` SSH
> alias rather than `scripts/run-tests.sh`. This decision supersedes the
> 2026-05-09 "both" decision recorded in the § Test Execution section.
> **Active policy — 2026-05-20 (refined)**: the canonical CI / release-gate
> test environment is the Jetson Orin Nano Super (or a Jetson-equivalent
> arm64 agent). **Unit tests** (`pytest tests/unit/`) MAY be run on a local
> developer workstation for fast iteration — they are hardware-agnostic by
> construction, the suite is fully synthetic, and Jetson SSH round-trips add
> latency without adding signal. **Blackbox / e2e / performance / resilience
> / security / resource-limit tests** (`tests/e2e/`, `e2e/tests/`,
> `tests/perf/`, etc.) MUST run on the Jetson — never on a local workstation
> — because their pass criteria are tied to Jetson wall-clock latency,
> thermal envelope, and the real-camera + real-FC SITL loop. Workstation x86
> Docker (the historical "Tier-1" path) is **deprecated** as a supported
> e2e environment; the Tier-1 sections below are retained as historical
> reference / traceability only. CI e2e pipelines target the colocated
> arm64 Jetson Woodpecker agent (see `_docs/04_deploy/ci_cd_pipeline.md`);
> local-development e2e runs SHOULD use `scripts/run-tests-jetson.sh`
> against the configured `jetson-e2e` SSH alias rather than
> `scripts/run-tests.sh`. This refinement supersedes the 2026-05-20 "all
> tiers on Jetson" wording and the 2026-05-09 "both" decision recorded in
> the § Test Execution section.
## Where each tier runs (active policy)
| Tier | Local workstation | Jetson (canonical) | When local is the only option |
|------|--------------------|--------------------|-------------------------------|
| Unit (`tests/unit/`) | ✅ allowed and encouraged for dev iteration | ✅ also run as part of the Jetson CI lane | always |
| Blackbox / e2e (`tests/e2e/`, `e2e/tests/`) | ❌ forbidden — placeholder fixtures + missing hardware = false-negative runs | ✅ required for any merge / release decision | never — if Jetson is unreachable, the e2e verdict is "not run" rather than a local result |
| Performance / resilience / security / resource-limit | ❌ forbidden | ✅ required | never |
| Thermal chamber (AC-NEW-5) | ❌ forbidden | ✅ chamber Jetson only | never |
Practical consequences:
- A PR may merge on green local unit tests + green Jetson e2e tests.
- A PR MAY NOT merge on green local unit tests alone — the Jetson e2e lane is the binding signal.
- When the Jetson agent is offline, the e2e verdict is "pending Jetson" — record the gap (e.g. via `_docs/_process_leftovers/`) rather than substituting a local run.
- Tests in `tests/e2e/` that gate on `RUN_REPLAY_E2E` or `@pytest.mark.tier2` will SKIP locally; this is correct behaviour, not a failure to investigate.
## Overview
@@ -263,11 +286,21 @@ The captured-fixture builder framework (`e2e/fixtures/sitl_replay_builder/`) reg
## Test Execution
**Decision (2026-05-20)****Jetson only.** Supersedes the 2026-05-09 "both" decision below. All tests (unit, integration, blackbox / e2e, performance, resilience, security, resource-limit) run on the Jetson Orin Nano Super (or a Jetson-equivalent arm64 agent). The workstation x86 Docker path is deprecated. Rationale captured in `_docs/LESSONS.md` (2026-05-20 entry): repeated workstation-vs-Jetson environment divergences (Dockerfile build order, missing `libgl1`, gtsam wheel availability, venv symlink resolution, lazy-import side-effect registration) were producing false-negative test runs and consuming engineering time without ever exercising the production-equivalent hardware path.
**Decision (2026-05-20, refined later that day)****Jetson is the binding e2e environment; unit tests may run locally.** This refines the earlier "Jetson only for everything" wording. Rationale captured in `_docs/LESSONS.md` (2026-05-20 entries):
- The original "Jetson-only across all tiers" decision came from repeated workstation-vs-Jetson environment divergences in the e2e / build path (Dockerfile build order, missing `libgl1`, gtsam wheel availability, venv symlink resolution, lazy-import side-effect registration). Those divergences are real and continue to justify Jetson as the binding e2e environment.
- Forcing the unit-test suite over an SSH-orchestrated Jetson loop added 3090 s per iteration without producing any signal the local interpreter doesn't already produce. The unit suite is fully synthetic — no camera, no SITL, no Jetson-specific runtime — so a local PASS is equivalent to a Jetson PASS for that tier.
**Operational entry points**:
- Local-development: `scripts/run-tests-jetson.sh` against the configured `jetson-e2e` SSH alias (see `_docs/03_implementation/jetson_harness_setup.md` for one-time setup).
- CI: `.woodpecker/01-test.yml` on the colocated arm64 Jetson agent (see `_docs/04_deploy/ci_cd_pipeline.md`).
| Tier | Entry point | Where it runs |
|------|-------------|---------------|
| Unit (`tests/unit/`) | `pytest tests/unit/ -q` directly, or `scripts/run-tests.sh` | local workstation (Python 3.10+ venv) |
| Blackbox / e2e (`tests/e2e/`, `e2e/tests/`) | `scripts/run-tests-jetson.sh` (local dev) / `.woodpecker/01-test.yml` (CI) | colocated arm64 Jetson Woodpecker agent — see `_docs/04_deploy/ci_cd_pipeline.md` |
| Performance / resilience / security / resource-limit | same as e2e | Jetson only |
| AC-NEW-5 thermal chamber | quarterly + pre-release | `self-hosted-jetson-orin-chamber` |
A green local unit-test run is necessary-but-not-sufficient for merge; the Jetson e2e lane is the binding signal.
The remainder of this section preserves the original 2026-05-09 decision context for traceability.