mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 08:41:12 +00:00
[autodev] Update Jetson test environment and satellite-provider integration
ci/woodpecker/push/02-build-push Pipeline failed
ci/woodpecker/push/02-build-push Pipeline failed
- Added `.env.test` to `.gitignore` to exclude test environment variables. - Enhanced `docker-compose.test.jetson.yml` to include the real satellite-provider .NET service and its PostgreSQL database, replacing the mock service. - Updated test execution policy to mandate all tests run exclusively on Jetson hardware, deprecating the previous two-tier model. - Revised documentation in `_docs/LESSONS.md`, `_docs/02_document/tests/environment.md`, and `_docs/04_deploy/ci_cd_pipeline.md` to reflect the new testing strategy and environment setup. - Improved `run-tests-jetson.sh` script to ensure proper environment variable handling and satellite-provider integration. This commit aligns the testing framework with production environments, enhancing reliability and coverage.
This commit is contained in:
@@ -1,5 +1,18 @@
|
||||
# Test Environment
|
||||
|
||||
> **Active policy — 2026-05-20**: **all tests run on Jetson only.** The Jetson
|
||||
> Orin Nano Super (or a Jetson-equivalent arm64 agent) is the single canonical
|
||||
> test environment for every tier of testing — unit, integration, blackbox /
|
||||
> e2e, performance, resilience, security, resource-limit. Workstation x86
|
||||
> Docker (the historical "Tier-1" path) is **deprecated** and is not a
|
||||
> supported test environment going forward; the Tier-1 sections below are
|
||||
> retained as historical reference / traceability only. CI test pipelines
|
||||
> target the colocated arm64 Jetson Woodpecker agent (see
|
||||
> `_docs/04_deploy/ci_cd_pipeline.md`); local-development test runs SHOULD
|
||||
> use `scripts/run-tests-jetson.sh` against the configured `jetson-e2e` SSH
|
||||
> alias rather than `scripts/run-tests.sh`. This decision supersedes the
|
||||
> 2026-05-09 "both" decision recorded in the § Test Execution section.
|
||||
|
||||
## Overview
|
||||
|
||||
**System under test (SUT)**: `gps-denied-onboard` companion-PC service that produces WGS84 position estimates from nav-camera frames + FC IMU/attitude and emits them to the FC over its native external-positioning interface. Public boundaries (the only surfaces tests interact with):
|
||||
@@ -15,14 +28,19 @@
|
||||
|
||||
## Two-tier execution profile
|
||||
|
||||
This project requires two distinct test environments because the production target is Jetson hardware and AC-4.1/AC-4.2/AC-NEW-5 cannot be honestly validated on a generic x86 dev workstation.
|
||||
> **SUPERSEDED — 2026-05-20**: the two-tier model below is retained for
|
||||
> historical traceability. The active policy is **Jetson-only** (see banner
|
||||
> at the top of this doc). Tier-1 (workstation Docker) is deprecated; only
|
||||
> the Tier-2 row continues to describe a supported environment.
|
||||
|
||||
This project originally specified two distinct test environments because the production target is Jetson hardware and AC-4.1/AC-4.2/AC-NEW-5 cannot be honestly validated on a generic x86 dev workstation.
|
||||
|
||||
| Tier | Hardware | What it covers | What it skips |
|
||||
|------|----------|----------------|---------------|
|
||||
| **Tier-1 (workstation Docker)** | x86 dev workstation, optional NVIDIA dGPU for TensorRT validation | All `FT-*` correctness, schema, `NFT-RES-*` resilience scenarios, `NFT-SEC-*` security scenarios, `NFT-LIM-*` storage budgets | Any AC whose pass criterion is bound to Jetson Orin Nano Super wall-clock latency or thermal envelope: AC-4.1 / AC-4.2 / AC-NEW-1 / AC-NEW-5 |
|
||||
| **Tier-2 (Jetson hardware loop)** | Jetson Orin Nano Super (pinned hardware per `restrictions.md`), thermal chamber for AC-NEW-5 | AC-4.1 latency p95, AC-4.2 memory, AC-NEW-1 cold-start TTFF, AC-NEW-5 thermal envelope (chamber-only) | Iteration speed (manual hardware time) |
|
||||
| **Tier-1 (workstation Docker)** *(deprecated 2026-05-20)* | x86 dev workstation, optional NVIDIA dGPU for TensorRT validation | All `FT-*` correctness, schema, `NFT-RES-*` resilience scenarios, `NFT-SEC-*` security scenarios, `NFT-LIM-*` storage budgets | Any AC whose pass criterion is bound to Jetson Orin Nano Super wall-clock latency or thermal envelope: AC-4.1 / AC-4.2 / AC-NEW-1 / AC-NEW-5 |
|
||||
| **Jetson (canonical, 2026-05-20)** *(formerly "Tier-2")* | Jetson Orin Nano Super (pinned hardware per `restrictions.md`), thermal chamber for AC-NEW-5 | Everything: `FT-*` correctness, schema, `NFT-RES-*`, `NFT-SEC-*`, `NFT-LIM-*`, `NFT-PERF-*` (AC-4.1 latency p95), AC-4.2 memory, AC-NEW-1 cold-start TTFF, AC-NEW-5 thermal envelope (chamber-only) | Nothing — anything that doesn't run here doesn't run at all |
|
||||
|
||||
CI runs Tier-1 on every PR. Tier-2 runs on hardware-attached runners on a nightly cadence and pre-release gate; results are imported into the same CSV report format as Tier-1.
|
||||
CI runs the Jetson pipeline (`01-test.yml`) on the colocated arm64 Jetson agent. Chamber-only AC-NEW-5 runs on `self-hosted-jetson-orin-chamber` on the documented quarterly + pre-release cadence; results are recorded in the same CSV report format.
|
||||
|
||||
## Docker Environment (Tier-1)
|
||||
|
||||
@@ -213,20 +231,19 @@ The captured-fixture builder framework (`e2e/fixtures/sitl_replay_builder/`) reg
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
**When to run**:
|
||||
- Tier-1 (workstation Docker): on every PR to `dev` branch and nightly on `dev` HEAD.
|
||||
- Tier-2 (Jetson hardware loop): nightly on `dev`, and as a hard gate before any release tag.
|
||||
- AC-NEW-5 thermal envelope: monthly on chamber-attached Jetson runner; failures block release tags only.
|
||||
> **2026-05-20**: rewritten for the Jetson-only policy. Tier-1 references in the historical sub-sections below are no longer operative.
|
||||
|
||||
**Pipeline stage**:
|
||||
- Tier-1 fits in the standard CI matrix as a single job (~30-45 min wall-clock for the full suite at first cut).
|
||||
- Tier-2 is a separate workflow on `self-hosted-jetson-orin` runner.
|
||||
**When to run** (active policy):
|
||||
|
||||
**Gate behavior**: Tier-1 blocks PR merge on any test failure. Tier-2 blocks release tag on any test failure. Chamber tests are warning-only on PRs and blocking on release tags.
|
||||
- Jetson (colocated arm64 Woodpecker agent): on every PR to `dev` branch, nightly on `dev` HEAD, and as a hard gate before any release tag.
|
||||
- AC-NEW-5 thermal envelope: quarterly on the chamber-attached Jetson runner; failures block release tags only.
|
||||
|
||||
**Pipeline stage**: a single Jetson workflow (`.woodpecker/01-test.yml`) on the `self-hosted-jetson-orin` runner exercises the full suite — there is no longer a parallel x86 lane.
|
||||
|
||||
**Gate behavior**: Jetson blocks PR merge on any test failure and blocks release tags on any test failure. Chamber tests are warning-only on PRs and blocking on release tags.
|
||||
|
||||
**Timeout**:
|
||||
- Tier-1: 60 min per matrix entry.
|
||||
- Tier-2: 4 hr per matrix entry (allows for full Derkachi 8 min replay × ~10 scenarios + cold-boot loops).
|
||||
- Jetson: 4 hr per matrix entry (allows for full Derkachi 8 min replay × ~10 scenarios + cold-boot loops).
|
||||
- Thermal chamber AC-NEW-5: 9 hr (8 h hot-soak + setup/teardown).
|
||||
|
||||
## Reporting
|
||||
@@ -246,7 +263,17 @@ The captured-fixture builder framework (`e2e/fixtures/sitl_replay_builder/`) reg
|
||||
|
||||
## Test Execution
|
||||
|
||||
**Decision (2026-05-09)**: **both** — Tier-1 Docker + Tier-2 Jetson hardware loop. Confirmed at the Hardware-Dependency Assessment Step 4 gate.
|
||||
**Decision (2026-05-20)** — **Jetson only.** Supersedes the 2026-05-09 "both" decision below. All tests (unit, integration, blackbox / e2e, performance, resilience, security, resource-limit) run on the Jetson Orin Nano Super (or a Jetson-equivalent arm64 agent). The workstation x86 Docker path is deprecated. Rationale captured in `_docs/LESSONS.md` (2026-05-20 entry): repeated workstation-vs-Jetson environment divergences (Dockerfile build order, missing `libgl1`, gtsam wheel availability, venv symlink resolution, lazy-import side-effect registration) were producing false-negative test runs and consuming engineering time without ever exercising the production-equivalent hardware path.
|
||||
|
||||
**Operational entry points**:
|
||||
- Local-development: `scripts/run-tests-jetson.sh` against the configured `jetson-e2e` SSH alias (see `_docs/03_implementation/jetson_harness_setup.md` for one-time setup).
|
||||
- CI: `.woodpecker/01-test.yml` on the colocated arm64 Jetson agent (see `_docs/04_deploy/ci_cd_pipeline.md`).
|
||||
|
||||
The remainder of this section preserves the original 2026-05-09 decision context for traceability.
|
||||
|
||||
---
|
||||
|
||||
**Decision (2026-05-09, SUPERSEDED)**: **both** — Tier-1 Docker + Tier-2 Jetson hardware loop. Confirmed at the Hardware-Dependency Assessment Step 4 gate.
|
||||
|
||||
### Hardware dependencies found (Phase 3 → Hardware Assessment scan)
|
||||
|
||||
@@ -340,8 +367,13 @@ When invoked on a control host (typical), the script SSH-orchestrates the Jetson
|
||||
|
||||
### CI runner mapping
|
||||
|
||||
- `ubuntu-24.04` (GitHub-hosted) → Tier-1 Docker, every PR + nightly. ~30-45 min per matrix entry.
|
||||
- `self-hosted-jetson-orin` → Tier-2 Jetson, nightly on `dev` HEAD + pre-release gate. ~4 hr per matrix entry.
|
||||
**Active mapping (2026-05-20)**:
|
||||
|
||||
- `self-hosted-jetson-orin` (colocated arm64 Woodpecker agent) → all test runs, every PR + nightly + pre-release. ~4 hr per matrix entry. **This is the single canonical CI test runner.**
|
||||
- `self-hosted-jetson-orin-chamber` → AC-NEW-5 hot-soak. Quarterly + before any release tag. ~9 hr.
|
||||
|
||||
**Removed (2026-05-20)**:
|
||||
|
||||
- ~~`ubuntu-24.04` (GitHub-hosted) → Tier-1 Docker, every PR + nightly. ~30-45 min per matrix entry.~~ — Tier-1 workstation Docker is deprecated; no x86 CI agent participates in the test path. CI build-push lanes that ship images may still run on amd64 if/when that matrix dimension is uncommented in `02-build-push.yml`, but the test lane is Jetson-only.
|
||||
|
||||
**Matrix dimensions**: `FC_ADAPTER × VIO_STRATEGY × build_kind` where `build_kind ∈ {production, research}`. Production `vins_mono` is excluded (D-C1-1-SUB-A locked); research includes all three VioStrategy values.
|
||||
|
||||
Reference in New Issue
Block a user