mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-23 08:01:13 +00:00
Update demo replay validation and testing documentation
ci/woodpecker/push/02-build-push Pipeline failed
ci/woodpecker/push/02-build-push Pipeline failed
- Modified the autodev state to reflect the current testing phase and details of the new `jetson-e2e` tests. - Enhanced the "How to Test" documentation to provide clearer instructions on the demo replay validation process, including video and tlog alignment steps. - Updated architectural documentation to include the new demo replay operator flow and its dependencies. - Documented the removal of deprecated auto-sync features and clarified the operator-facing UI for replay validation. - Added new entries in the dependencies table for upcoming tasks related to the demo replay flow. These changes improve clarity and usability for operators and developers working with the demo replay system.
This commit is contained in:
@@ -0,0 +1,109 @@
|
||||
# Test Runner Script Structure
|
||||
|
||||
Reference for generating `scripts/run-tests.sh` and `scripts/run-performance-tests.sh`.
|
||||
|
||||
## When to generate a local `run-tests.sh`
|
||||
|
||||
A local shell script is needed **only** for hardware-dependent projects that require real hardware (GPU, CoreML, TPU, sensors, etc.) to exercise the actual code paths. If the Hardware-Dependency Assessment (Phase 4 prerequisite) determined **local** or **both** execution, generate this script.
|
||||
|
||||
For all other projects, **use Docker** (`docker-compose.test.yml` / `Dockerfile.test`). Docker is the default — it provides reproducibility, isolation, and CI parity. Do not generate a local `run-tests.sh` when Docker is sufficient.
|
||||
|
||||
## `scripts/run-tests.sh` (local / hardware-dependent only)
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
UNIT_ONLY=false
|
||||
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
||||
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--unit-only) UNIT_ONLY=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
cleanup() {
|
||||
# tear down services started by this script
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
|
||||
# --- Install Dependencies ---
|
||||
# MANDATORY: install all project + test dependencies before building or running.
|
||||
# A fresh clone or CI runner may have nothing installed.
|
||||
# Python: pip install -q -r requirements.txt -r e2e/requirements.txt
|
||||
# .NET: dotnet restore
|
||||
# Rust: cargo fetch
|
||||
# Node: npm ci
|
||||
|
||||
# --- Build (if needed) ---
|
||||
# [e.g. Cython: python setup.py build_ext --inplace]
|
||||
|
||||
# --- Unit Tests ---
|
||||
# [detect runner: pytest / dotnet test / cargo test / npm test]
|
||||
# [run and capture exit code]
|
||||
|
||||
# --- Blackbox Tests (skip if --unit-only) ---
|
||||
# if ! $UNIT_ONLY; then
|
||||
# [start mock services]
|
||||
# [start system under test]
|
||||
# [wait for health checks]
|
||||
# [run blackbox test suite]
|
||||
# fi
|
||||
|
||||
# --- Summary ---
|
||||
# [print passed / failed / skipped counts]
|
||||
# [exit 0 if all passed, exit 1 otherwise]
|
||||
```
|
||||
|
||||
## `scripts/run-performance-tests.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
||||
|
||||
cleanup() {
|
||||
# tear down test environment if started
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
|
||||
# --- Install Dependencies ---
|
||||
# [same as above — always install first]
|
||||
|
||||
# --- Start System Under Test ---
|
||||
# [docker compose up -d or start local server]
|
||||
# [wait for health checks]
|
||||
|
||||
# --- Run Performance Scenarios ---
|
||||
# [detect tool: k6 / locust / artillery / wrk / built-in]
|
||||
# [run each scenario from performance-tests.md]
|
||||
# [capture metrics: latency P50/P95/P99, throughput, error rate]
|
||||
|
||||
# --- Compare Against Thresholds ---
|
||||
# [read thresholds from test spec or CLI args]
|
||||
# [print per-scenario pass/fail]
|
||||
|
||||
# --- Summary ---
|
||||
# [exit 0 if all thresholds met, exit 1 otherwise]
|
||||
```
|
||||
|
||||
## Key Requirements
|
||||
|
||||
- **Docker is the default**: only generate a local `run-tests.sh` for hardware-dependent projects. Otherwise use `docker-compose.test.yml`.
|
||||
- **Always install dependencies first**: the script must install all project and test dependencies before building or running tests. A fresh clone or CI runner may have nothing installed. Missing a single dependency causes collection errors that abort the entire test run.
|
||||
- Both scripts must be idempotent (safe to run multiple times)
|
||||
- Both scripts must work in CI (no interactive prompts, no GUI)
|
||||
- Use `trap cleanup EXIT` to ensure teardown even on failure
|
||||
- Exit codes: 0 = all pass, 1 = failures detected
|
||||
- Write results to `test-results/` directory (add to `.gitignore` if not already present)
|
||||
- The actual commands depend on the detected tech stack — fill them in during Phase 4 of the test-spec skill
|
||||
Reference in New Issue
Block a user