mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 14:21:14 +00:00
[AZ-407] [AZ-444] [AZ-445] Batch 68: fixtures, Tier-2 harness, NFR reporter
Three blackbox-harness tasks landed together — all depend only on
AZ-406 and unblock the FT-* / NFT-* scenario tasks scheduled for
batches 69+.
AZ-407 — Static fixture builders (3pt):
* tile-cache-builder/{builder.py, Dockerfile, build.sh} produces a
deterministic tile-cache-fixture Docker volume from
_docs/00_problem/input_data/. Reproducibility primitives: sorted
iteration, frozen PIL JPEG settings, FAISS HNSW32 built single-
threaded with seeded stub descriptors.
* age-injector/{age_injector.py, inject.sh} clones the volume and
shifts capture_date by N×30.44 days; tile JPEG bytes preserved
bit-identical. Emits synth-age-7mo + synth-age-13mo volumes.
* cold-boot/cold_boot_fixture.json: frozen FC pose snapshot at
Derkachi sector centre, schema v1.
* secrets/mavlink-test-passkey.txt: 64-hex with required
`# TEST ONLY` header line per AC-5. Passkey-equality test now
compares the secret line after stripping the header.
* security/cve-2025-53644.jpg: synthetic 158-byte malformed JPEG
(truncated SOS marker). OpenCV 4.11.x rejects gracefully with
imdecode → None. AZ-439 will sharpen for ASan instrumentation.
* Top-level Makefile with `make fixtures` / `make fixtures-*` /
`make e2e-tier1*` / `make unit-tests` targets.
AZ-444 — Tier-2 Jetson harness wrapper (5pt):
* run-tier2.sh rewritten as orchestrator. Detects local
(aarch64 + TIER2_HOST=localhost) vs remote (ssh into TIER2_HOST).
New flags: -k/--selector, --build-kind production|asan,
--reflash (gated behind TIER2_REFLASH_ACK=1 two-key gate),
--dry-run.
* tier2-on-jetson.sh (new) — on-device delegate. Verifies
gps-denied-onboard{,-asan}.service health; restarts with 5s
tolerance; spawns tegrastats + jtop parallel samplers; tails
ASan unit's journal in asan mode; drives docker compose with
TIER=tier2-jetson; forwards SELECTOR to pytest -k.
* docker/run-tier1.sh (new) — selector-parity sibling.
* AC-1 (selector parity) and AC-6 (reflash gating) unit-tested via
--dry-run output assertions. AC-2/AC-3/AC-4/AC-5 are hardware-
loop ACs verified by the Tier-2 runtime smoke (no Jetson in the
unit-test layer).
AZ-445 — CSV reporter + evidence bundler refinements (2pt):
* reporting/nfr_recorder.py (new) — pytest plugin. Provides the
`nfr_recorder` fixture with record_metric(name, value, ac_id)
and partial(ac_id, reason). At session end emits:
- per-nfr/<scenario_id>.json (AC-1)
- traceability-status.json with every AC ID parsed from
traceability-matrix.md, classified Covered/PARTIAL/NOT
COVERED with source scenario IDs (AC-2)
- regression-baseline.json with all numeric metrics (AC-3)
* csv_reporter.py extended — `_outcome_to_result` consults the
aggregator; rows flip PASS → PARTIAL when an AC was marked
PARTIAL by nfr_recorder (AC-4). Graceful fallback when
aggregator isn't registered (unit-test contexts).
* conftest.py registers nfr_recorder in pytest_plugins.
* New --traceability-matrix CLI flag seeds the NOT COVERED rows.
Build / config:
* pyproject.toml dev extras: added Pillow>=10.4,<13.0 for the
tile-cache-builder unit test (broad enough to keep torchvision's
Pillow 12 pin happy; the production builder runs inside its own
Docker image with its own pin).
* Updated test_directory_layout.py to cover 10 new files + replaced
the byte-equal passkey assertion with the header-stripping
variant.
Test results:
* 157 focused tests pass (was 97 in batch 67; +60 new across this
batch). No regressions.
Module-layout / spec drift:
* AZ-407 spec text says `tests/fixtures/...`; module-layout
blackbox_tests entry (commit d7a17a8) authoritatively places the
harness under `e2e/`. Implementation followed the layout entry.
* AZ-444 spec mentions `e2e/tier2/run-tier2.sh`; AZ-406 placed it
at `e2e/jetson/run-tier2.sh`. Kept at `e2e/jetson/` for
consistency.
* Cold-boot README ownership: corrected from AZ-419 to AZ-407 per
AZ-419's own Dependencies field.
Specs archived to _docs/02_tasks/done/. Jira tickets transitioned to
In Testing on commit.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
+169
-80
@@ -1,31 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tier-2 Jetson hardware-loop entrypoint.
|
||||
# Tier-2 Jetson hardware-loop entrypoint (orchestrator).
|
||||
#
|
||||
# This script runs FROM a control host (typically x86) and ssh-orchestrates
|
||||
# the on-Jetson half (`tier2-on-jetson.sh`). When invoked on the Jetson
|
||||
# itself (uname -m == aarch64 AND TIER2_HOST=localhost), it delegates
|
||||
# directly without going through ssh.
|
||||
#
|
||||
# Usage:
|
||||
# ./run-tier2.sh --fc-adapter <ardupilot|inav> --vio-strategy <okvis2|klt_ransac> [--duration <5min|8h>] [--enable-chamber]
|
||||
# ./run-tier2.sh \
|
||||
# --fc-adapter <ardupilot|inav> \
|
||||
# --vio-strategy <okvis2|klt_ransac|vins_mono> \
|
||||
# [-k <pytest selector>] \
|
||||
# [--build-kind <production|asan>] \
|
||||
# [--duration <5min|8h>] \
|
||||
# [--enable-chamber] \
|
||||
# [--reflash] \
|
||||
# [--dry-run]
|
||||
#
|
||||
# Pre-requisites (verified at startup):
|
||||
# Required env vars (when TIER2_HOST != localhost):
|
||||
# TIER2_HOST Jetson hostname or IP
|
||||
# TIER2_USER SSH user on the Jetson
|
||||
# TIER2_KEY_PATH Path to the SSH private key
|
||||
#
|
||||
# Pre-requisites verified at startup:
|
||||
# * The Jetson is provisioned per `_docs/02_document/tests/environment.md`
|
||||
# § Execution instructions — Tier-2 (JetPack 6.2, CUDA, TensorRT 10.3, cuDNN).
|
||||
# * `gps-denied-onboard.service` is installed via systemd
|
||||
# (`tier2.service` is the template; operator copies it to /etc/systemd/system).
|
||||
# § Execution instructions — Tier-2 (JetPack 6.2, CUDA, TensorRT 10.3,
|
||||
# cuDNN).
|
||||
# * `gps-denied-onboard.service` (or `gps-denied-onboard-asan.service`
|
||||
# for --build-kind=asan) is installed via systemd. `tier2.service` is
|
||||
# the template.
|
||||
# * SITLs + mock + listener + runner reachable on the same network via
|
||||
# `docker compose -f e2e/docker/docker-compose.test.yml -f e2e/docker/docker-compose.tier2-bridge.yml up ...`
|
||||
# on a paired x86 host. (Same-Jetson SITL is also supported — set JETSON_HOST=localhost.)
|
||||
# `docker compose -f e2e/docker/docker-compose.test.yml
|
||||
# -f e2e/docker/docker-compose.tier2-bridge.yml up ...`
|
||||
# on a paired x86 host (same as Tier-1's `docker-compose.test.yml`
|
||||
# network).
|
||||
#
|
||||
# Outputs the same CSV format as Tier-1 to ./e2e-results/run-${RUN_ID}/report.csv
|
||||
# Outputs the same CSV format as Tier-1 to
|
||||
# ./e2e-results/run-${RUN_ID}/report.csv
|
||||
# plus the per-sample tegrastats + jtop CSVs in the evidence bundle.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
FC_ADAPTER=""
|
||||
VIO_STRATEGY=""
|
||||
SELECTOR=""
|
||||
BUILD_KIND="production"
|
||||
DURATION="5min"
|
||||
ENABLE_CHAMBER=0
|
||||
JETSON_HOST_OVERRIDE=""
|
||||
RUN_REFLASH=0
|
||||
DRY_RUN=0
|
||||
|
||||
usage() {
|
||||
grep -E '^# ' "$0" | sed 's/^# //'
|
||||
grep -E '^# ' "$0" | sed 's/^# //' >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
@@ -33,9 +59,12 @@ while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--fc-adapter) FC_ADAPTER="$2"; shift 2 ;;
|
||||
--vio-strategy) VIO_STRATEGY="$2"; shift 2 ;;
|
||||
-k|--selector) SELECTOR="$2"; shift 2 ;;
|
||||
--build-kind) BUILD_KIND="$2"; shift 2 ;;
|
||||
--duration) DURATION="$2"; shift 2 ;;
|
||||
--enable-chamber) ENABLE_CHAMBER=1; shift ;;
|
||||
--jetson-host) JETSON_HOST_OVERRIDE="$2"; shift 2 ;;
|
||||
--reflash) RUN_REFLASH=1; shift ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo "Unknown arg: $1" >&2; usage ;;
|
||||
esac
|
||||
@@ -56,93 +85,153 @@ case "$VIO_STRATEGY" in
|
||||
*) echo "ERROR: --vio-strategy must be okvis2 | klt_ransac | vins_mono (got: $VIO_STRATEGY)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
case "$BUILD_KIND" in
|
||||
production|asan) ;;
|
||||
*) echo "ERROR: --build-kind must be production or asan (got: $BUILD_KIND)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
# AC-6 (image-flash gating). Even when --reflash is requested, refuse to
|
||||
# proceed unless the operator has acknowledged via TIER2_REFLASH_ACK=1.
|
||||
# This is a two-key gate so a stray flag flip in CI cannot accidentally
|
||||
# re-provision a development board.
|
||||
if [[ "${RUN_REFLASH}" -eq 1 ]]; then
|
||||
if [[ "${TIER2_REFLASH_ACK:-0}" != "1" ]]; then
|
||||
echo "ERROR: --reflash requires TIER2_REFLASH_ACK=1 in the env" >&2
|
||||
echo " This is a destructive operation; set the ack to" >&2
|
||||
echo " confirm you intend to re-flash the Jetson via" >&2
|
||||
echo " nvidia-sdkmanager-cli." >&2
|
||||
exit 4
|
||||
fi
|
||||
fi
|
||||
|
||||
# RUN_ID — caller may set; default is utc-stamp + adapter pair.
|
||||
: "${RUN_ID:=tier2-$(date -u +%Y%m%dT%H%M%SZ)-${FC_ADAPTER}-${VIO_STRATEGY}}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
RESULTS_DIR="${REPO_ROOT}/e2e-results/run-${RUN_ID}"
|
||||
EVIDENCE_DIR="${RESULTS_DIR}/evidence"
|
||||
|
||||
mkdir -p "${EVIDENCE_DIR}"
|
||||
|
||||
echo "[tier2] RUN_ID=${RUN_ID}"
|
||||
echo "[tier2] FC_ADAPTER=${FC_ADAPTER} VIO_STRATEGY=${VIO_STRATEGY} DURATION=${DURATION}"
|
||||
echo "[tier2] RESULTS_DIR=${RESULTS_DIR}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-flight: confirm the SUT systemd unit is healthy.
|
||||
# Determine mode:
|
||||
# * local mode — run on the Jetson itself; no ssh wrapper.
|
||||
# Triggered when TIER2_HOST=localhost OR is unset on an aarch64 host.
|
||||
# * remote mode — orchestrator: ssh into TIER2_HOST and execute the
|
||||
# on-Jetson delegate there.
|
||||
# ---------------------------------------------------------------------------
|
||||
if ! systemctl is-active --quiet gps-denied-onboard.service; then
|
||||
echo "[tier2] gps-denied-onboard.service is not active — attempting restart..." >&2
|
||||
sudo systemctl restart gps-denied-onboard.service
|
||||
sleep 3
|
||||
if ! systemctl is-active --quiet gps-denied-onboard.service; then
|
||||
echo "[tier2] FATAL: gps-denied-onboard.service failed to start" >&2
|
||||
sudo systemctl status gps-denied-onboard.service --no-pager || true
|
||||
exit 3
|
||||
TIER2_HOST="${TIER2_HOST:-}"
|
||||
if [[ -z "${TIER2_HOST}" ]]; then
|
||||
if [[ "$(uname -m)" == "aarch64" ]]; then
|
||||
TIER2_HOST="localhost"
|
||||
else
|
||||
echo "ERROR: TIER2_HOST must be set when running from a non-Jetson host" >&2
|
||||
echo " (uname -m is $(uname -m); this script is not running on a Jetson)" >&2
|
||||
exit 5
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Start tegrastats + jtop background samplers (evidence bundle inputs).
|
||||
# ---------------------------------------------------------------------------
|
||||
TEGRA_CSV="${EVIDENCE_DIR}/tegrastats.csv"
|
||||
JTOP_CSV="${EVIDENCE_DIR}/jtop.csv"
|
||||
echo "[tier2] RUN_ID=${RUN_ID}"
|
||||
echo "[tier2] FC_ADAPTER=${FC_ADAPTER} VIO_STRATEGY=${VIO_STRATEGY} BUILD_KIND=${BUILD_KIND}"
|
||||
echo "[tier2] SELECTOR='${SELECTOR}' DURATION=${DURATION} ENABLE_CHAMBER=${ENABLE_CHAMBER}"
|
||||
echo "[tier2] TIER2_HOST=${TIER2_HOST}"
|
||||
|
||||
# tegrastats emits at 5 Hz by default; parser converts to per-sample CSV rows.
|
||||
if command -v tegrastats >/dev/null 2>&1; then
|
||||
tegrastats --interval 200 \
|
||||
| python3 "${SCRIPT_DIR}/tegrastats_parser.py" --out "${TEGRA_CSV}" &
|
||||
TEGRA_PID=$!
|
||||
else
|
||||
echo "[tier2] WARNING: tegrastats not in PATH — skipping that evidence channel." >&2
|
||||
TEGRA_PID=
|
||||
# ---------------------------------------------------------------------------
|
||||
# Build the ssh command prefix for the orchestrator mode.
|
||||
# ---------------------------------------------------------------------------
|
||||
SSH_CMD=""
|
||||
if [[ "${TIER2_HOST}" != "localhost" ]]; then
|
||||
: "${TIER2_USER:?TIER2_USER must be set for remote orchestrator mode}"
|
||||
: "${TIER2_KEY_PATH:?TIER2_KEY_PATH must be set for remote orchestrator mode}"
|
||||
if [[ ! -f "${TIER2_KEY_PATH}" ]]; then
|
||||
echo "ERROR: TIER2_KEY_PATH does not point at a real file: ${TIER2_KEY_PATH}" >&2
|
||||
exit 6
|
||||
fi
|
||||
SSH_CMD="ssh -o StrictHostKeyChecking=accept-new -i ${TIER2_KEY_PATH} ${TIER2_USER}@${TIER2_HOST}"
|
||||
fi
|
||||
|
||||
if command -v jtop >/dev/null 2>&1; then
|
||||
python3 "${SCRIPT_DIR}/jtop_parser.py" --out "${JTOP_CSV}" --interval 1.0 &
|
||||
JTOP_PID=$!
|
||||
else
|
||||
echo "[tier2] WARNING: jtop not in PATH — skipping that evidence channel." >&2
|
||||
JTOP_PID=
|
||||
fi
|
||||
# ---------------------------------------------------------------------------
|
||||
# AC-2: idempotent provisioning. apt update + install is idempotent on
|
||||
# its own; we just gate it behind a `--reflash` flag because re-running
|
||||
# it on every test invocation is needlessly slow.
|
||||
# ---------------------------------------------------------------------------
|
||||
provision_jetson() {
|
||||
local PROVISION_CMD
|
||||
PROVISION_CMD="set -eu;
|
||||
if ! dpkg -s python3-pip >/dev/null 2>&1; then
|
||||
sudo apt-get update;
|
||||
sudo apt-get install -y --no-install-recommends \
|
||||
python3-pip docker.io openssh-client iproute2;
|
||||
fi"
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
[[ -n "${TEGRA_PID:-}" ]] && kill "${TEGRA_PID}" 2>/dev/null || true
|
||||
[[ -n "${JTOP_PID:-}" ]] && kill "${JTOP_PID}" 2>/dev/null || true
|
||||
echo "[tier2] cleanup complete (rc=${rc})"
|
||||
exit "${rc}"
|
||||
if [[ "${TIER2_HOST}" == "localhost" ]]; then
|
||||
bash -c "${PROVISION_CMD}"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
${SSH_CMD} "${PROVISION_CMD}"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Run the e2e suite — the runner image is the SAME as Tier-1; only TIER differs.
|
||||
# AC-6: reflash via NVIDIA's sdkmanager-cli. This is the destructive
|
||||
# path; only runs when --reflash AND TIER2_REFLASH_ACK=1 are BOTH set.
|
||||
# ---------------------------------------------------------------------------
|
||||
JETSON_HOST_ARG="${JETSON_HOST_OVERRIDE:-localhost}"
|
||||
CHAMBER_ARG=()
|
||||
[[ "${ENABLE_CHAMBER}" -eq 1 ]] && CHAMBER_ARG=("--enable-chamber")
|
||||
reflash_jetson() {
|
||||
local FLASH_CMD
|
||||
FLASH_CMD="set -eu;
|
||||
if ! command -v nvidia-sdkmanager-cli >/dev/null 2>&1; then
|
||||
echo 'ERROR: nvidia-sdkmanager-cli not installed on Jetson' >&2
|
||||
exit 7
|
||||
fi
|
||||
echo '[tier2] re-flashing JetPack image via nvidia-sdkmanager-cli...' >&2
|
||||
nvidia-sdkmanager-cli flash --target-spec jetson-orin-nano-super"
|
||||
|
||||
(
|
||||
cd "${REPO_ROOT}/e2e/docker"
|
||||
RUN_ID="${RUN_ID}" \
|
||||
FC_ADAPTER="${FC_ADAPTER}" \
|
||||
VIO_STRATEGY="${VIO_STRATEGY}" \
|
||||
TIER="tier2-jetson" \
|
||||
JETSON_HOST="${JETSON_HOST_ARG}" \
|
||||
docker compose \
|
||||
-f docker-compose.test.yml \
|
||||
-f docker-compose.tier2-bridge.yml \
|
||||
run --rm \
|
||||
-e TIER=tier2-jetson \
|
||||
e2e-runner \
|
||||
pytest /test-suite \
|
||||
--csv="/e2e-results/run-${RUN_ID}/report.csv" \
|
||||
--csv-columns="test_id,test_name,traces_to,fc_adapter,vio_strategy,tier,started_at_utc,execution_time_ms,result,error_message,evidence_paths" \
|
||||
--evidence-out="/e2e-results/run-${RUN_ID}/evidence" \
|
||||
--build-kind=production \
|
||||
"${CHAMBER_ARG[@]}"
|
||||
if [[ "${TIER2_HOST}" == "localhost" ]]; then
|
||||
bash -c "${FLASH_CMD}"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
${SSH_CMD} "${FLASH_CMD}"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Execute the on-Jetson delegate.
|
||||
# ---------------------------------------------------------------------------
|
||||
ENV_PREFIX=(
|
||||
"RUN_ID=${RUN_ID}"
|
||||
"FC_ADAPTER=${FC_ADAPTER}"
|
||||
"VIO_STRATEGY=${VIO_STRATEGY}"
|
||||
"BUILD_KIND=${BUILD_KIND}"
|
||||
"SELECTOR=${SELECTOR}"
|
||||
"ENABLE_CHAMBER=${ENABLE_CHAMBER}"
|
||||
"JETSON_HOST=${TIER2_HOST}"
|
||||
)
|
||||
|
||||
echo "[tier2] Suite complete. Report: ${RESULTS_DIR}/report.csv"
|
||||
if [[ "${TIER2_HOST}" == "localhost" ]]; then
|
||||
DELEGATE_CMD=(env "${ENV_PREFIX[@]}" "${SCRIPT_DIR}/tier2-on-jetson.sh")
|
||||
else
|
||||
# Remote mode: rsync the e2e/ tree onto the Jetson and run the
|
||||
# delegate over ssh. We mirror the repo to /opt/azaion-e2e/ on the
|
||||
# Jetson; subsequent invocations are incremental via rsync's default
|
||||
# delta-transfer.
|
||||
REMOTE_REPO="/opt/azaion-e2e"
|
||||
RSYNC_CMD="rsync -az --delete -e 'ssh -o StrictHostKeyChecking=accept-new -i ${TIER2_KEY_PATH}' ${REPO_ROOT}/e2e/ ${TIER2_USER}@${TIER2_HOST}:${REMOTE_REPO}/e2e/"
|
||||
DELEGATE_CMD=(
|
||||
bash -c
|
||||
"${RSYNC_CMD} && ${SSH_CMD} \"env $(printf '%q ' "${ENV_PREFIX[@]}")${REMOTE_REPO}/e2e/jetson/tier2-on-jetson.sh\""
|
||||
)
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" -eq 1 ]]; then
|
||||
echo "[tier2] --dry-run: showing actions that would execute, then exiting."
|
||||
echo "[tier2] provision: ${SSH_CMD:-(local)} apt-get install -y python3-pip docker.io openssh-client iproute2"
|
||||
if [[ "${RUN_REFLASH}" -eq 1 ]]; then
|
||||
echo "[tier2] reflash: ${SSH_CMD:-(local)} nvidia-sdkmanager-cli flash --target-spec jetson-orin-nano-super"
|
||||
fi
|
||||
echo "[tier2] delegate: ${DELEGATE_CMD[*]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
provision_jetson
|
||||
[[ "${RUN_REFLASH}" -eq 1 ]] && reflash_jetson
|
||||
|
||||
"${DELEGATE_CMD[@]}"
|
||||
|
||||
echo "[tier2] Suite complete. RUN_ID=${RUN_ID}"
|
||||
|
||||
Executable
+149
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tier-2 ON-JETSON delegate. NOT invoked directly by humans — `run-tier2.sh`
|
||||
# ssh-orchestrates this script onto the configured Jetson host.
|
||||
#
|
||||
# Responsibilities:
|
||||
# * Verify `gps-denied-onboard.service` (or the `*-asan` variant) is healthy.
|
||||
# * Spawn tegrastats + jtop parallel samplers; route their output into the
|
||||
# evidence bundle.
|
||||
# * Drive the e2e-runner image via docker compose against
|
||||
# `docker-compose.test.yml + docker-compose.tier2-bridge.yml`.
|
||||
# * Tear down samplers cleanly on EXIT / INT / TERM.
|
||||
#
|
||||
# Required env vars (set by run-tier2.sh):
|
||||
# RUN_ID Run identifier (utc-stamp).
|
||||
# FC_ADAPTER ardupilot | inav
|
||||
# VIO_STRATEGY okvis2 | klt_ransac | vins_mono
|
||||
# BUILD_KIND production | asan
|
||||
# SELECTOR pytest -k expression (may be empty)
|
||||
# ENABLE_CHAMBER 0 | 1
|
||||
# JETSON_HOST host alias used by the test for SUT identification
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
: "${RUN_ID:?RUN_ID must be set by run-tier2.sh}"
|
||||
: "${FC_ADAPTER:?FC_ADAPTER must be set}"
|
||||
: "${VIO_STRATEGY:?VIO_STRATEGY must be set}"
|
||||
: "${BUILD_KIND:=production}"
|
||||
: "${SELECTOR:=}"
|
||||
: "${ENABLE_CHAMBER:=0}"
|
||||
: "${JETSON_HOST:=localhost}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
RESULTS_DIR="${REPO_ROOT}/e2e-results/run-${RUN_ID}"
|
||||
EVIDENCE_DIR="${RESULTS_DIR}/evidence"
|
||||
|
||||
mkdir -p "${EVIDENCE_DIR}"
|
||||
|
||||
# AC-5: the asan build is a separate systemd unit so it can run alongside
|
||||
# the production one for control/treatment comparisons.
|
||||
case "${BUILD_KIND}" in
|
||||
production)
|
||||
SUT_UNIT="gps-denied-onboard.service"
|
||||
;;
|
||||
asan)
|
||||
SUT_UNIT="gps-denied-onboard-asan.service"
|
||||
# ASan stderr stream is captured into the evidence bundle (see
|
||||
# AC-5: "stderr captured into asan-fuzz-${test_id}.log"). We tail
|
||||
# the unit's journal into the evidence file via journalctl.
|
||||
ASAN_LOG="${EVIDENCE_DIR}/asan-fuzz.log"
|
||||
;;
|
||||
*)
|
||||
echo "[tier2-on-jetson] FATAL: unknown BUILD_KIND=${BUILD_KIND}" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
# AC-3: systemd lifecycle. Restart on demand; fail loud if it doesn't
|
||||
# come back up.
|
||||
echo "[tier2-on-jetson] verifying ${SUT_UNIT} is active..."
|
||||
if ! systemctl is-active --quiet "${SUT_UNIT}"; then
|
||||
echo "[tier2-on-jetson] ${SUT_UNIT} is not active — restarting..." >&2
|
||||
sudo systemctl restart "${SUT_UNIT}"
|
||||
# AC-3 says "restart within ≤5 s"; we poll up to 5s + 1s safety
|
||||
# margin.
|
||||
for _ in 1 2 3 4 5 6; do
|
||||
sleep 1
|
||||
if systemctl is-active --quiet "${SUT_UNIT}"; then
|
||||
break
|
||||
fi
|
||||
done
|
||||
if ! systemctl is-active --quiet "${SUT_UNIT}"; then
|
||||
echo "[tier2-on-jetson] FATAL: ${SUT_UNIT} failed to start" >&2
|
||||
sudo systemctl status "${SUT_UNIT}" --no-pager || true
|
||||
exit 3
|
||||
fi
|
||||
fi
|
||||
|
||||
# AC-4: tegrastats + jtop parallel capture. Output streams into the
|
||||
# evidence bundle.
|
||||
TEGRA_CSV="${EVIDENCE_DIR}/tegrastats-${JETSON_HOST}-${RUN_ID}.csv"
|
||||
JTOP_CSV="${EVIDENCE_DIR}/jtop-${JETSON_HOST}-${RUN_ID}.csv"
|
||||
TEGRA_PID=""
|
||||
JTOP_PID=""
|
||||
ASAN_TAIL_PID=""
|
||||
|
||||
if command -v tegrastats >/dev/null 2>&1; then
|
||||
# 5 Hz sampling matches the parser's expected cadence.
|
||||
tegrastats --interval 200 \
|
||||
| python3 "${SCRIPT_DIR}/tegrastats_parser.py" --out "${TEGRA_CSV}" &
|
||||
TEGRA_PID=$!
|
||||
echo "[tier2-on-jetson] tegrastats sampler pid=${TEGRA_PID} → ${TEGRA_CSV}"
|
||||
else
|
||||
echo "[tier2-on-jetson] WARNING: tegrastats not in PATH — skipping that evidence channel." >&2
|
||||
fi
|
||||
|
||||
if command -v jtop >/dev/null 2>&1; then
|
||||
python3 "${SCRIPT_DIR}/jtop_parser.py" --out "${JTOP_CSV}" --interval 1.0 &
|
||||
JTOP_PID=$!
|
||||
echo "[tier2-on-jetson] jtop sampler pid=${JTOP_PID} → ${JTOP_CSV}"
|
||||
else
|
||||
echo "[tier2-on-jetson] WARNING: jtop not in PATH — skipping that evidence channel." >&2
|
||||
fi
|
||||
|
||||
if [[ "${BUILD_KIND}" == "asan" ]]; then
|
||||
journalctl -u "${SUT_UNIT}" -f --no-pager > "${ASAN_LOG}" 2>&1 &
|
||||
ASAN_TAIL_PID=$!
|
||||
echo "[tier2-on-jetson] asan journal tail pid=${ASAN_TAIL_PID} → ${ASAN_LOG}"
|
||||
fi
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
[[ -n "${TEGRA_PID}" ]] && kill "${TEGRA_PID}" 2>/dev/null || true
|
||||
[[ -n "${JTOP_PID}" ]] && kill "${JTOP_PID}" 2>/dev/null || true
|
||||
[[ -n "${ASAN_TAIL_PID}" ]] && kill "${ASAN_TAIL_PID}" 2>/dev/null || true
|
||||
echo "[tier2-on-jetson] cleanup complete (rc=${rc})"
|
||||
exit "${rc}"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# AC-1: selector parity. SELECTOR is forwarded as `-k "<expr>"` to the
|
||||
# pytest inside the runner image; empty SELECTOR means "all tests".
|
||||
PYTEST_ARGS=("/test-suite")
|
||||
PYTEST_ARGS+=("--csv=/e2e-results/run-${RUN_ID}/report.csv")
|
||||
PYTEST_ARGS+=("--csv-columns=test_id,test_name,traces_to,fc_adapter,vio_strategy,tier,started_at_utc,execution_time_ms,result,error_message,evidence_paths")
|
||||
PYTEST_ARGS+=("--evidence-out=/e2e-results/run-${RUN_ID}/evidence")
|
||||
PYTEST_ARGS+=("--build-kind=${BUILD_KIND}")
|
||||
[[ "${ENABLE_CHAMBER}" -eq 1 ]] && PYTEST_ARGS+=("--enable-chamber")
|
||||
[[ -n "${SELECTOR}" ]] && PYTEST_ARGS+=("-k" "${SELECTOR}")
|
||||
|
||||
(
|
||||
cd "${REPO_ROOT}/e2e/docker"
|
||||
RUN_ID="${RUN_ID}" \
|
||||
FC_ADAPTER="${FC_ADAPTER}" \
|
||||
VIO_STRATEGY="${VIO_STRATEGY}" \
|
||||
TIER="tier2-jetson" \
|
||||
JETSON_HOST="${JETSON_HOST}" \
|
||||
BUILD_KIND="${BUILD_KIND}" \
|
||||
docker compose \
|
||||
-f docker-compose.test.yml \
|
||||
-f docker-compose.tier2-bridge.yml \
|
||||
run --rm \
|
||||
-e TIER=tier2-jetson \
|
||||
-e BUILD_KIND="${BUILD_KIND}" \
|
||||
e2e-runner \
|
||||
pytest "${PYTEST_ARGS[@]}"
|
||||
)
|
||||
|
||||
echo "[tier2-on-jetson] Suite complete. Report: ${RESULTS_DIR}/report.csv"
|
||||
Reference in New Issue
Block a user