mirror of
https://github.com/azaion/detections-semantic.git
synced 2026-04-22 22:46:37 +00:00
Oleksandr Bezdieniezhnykh
110 lines
3.7 KiB
Markdown
110 lines
3.7 KiB
Markdown
# Test Runner Script Structure
|
|
|
|
Reference for generating `scripts/run-tests.sh` and `scripts/run-performance-tests.sh`.
|
|
|
|
## When to generate a local `run-tests.sh`
|
|
|
|
A local shell script is needed **only** for hardware-dependent projects that require real hardware (GPU, CoreML, TPU, sensors, etc.) to exercise the actual code paths. If the Hardware-Dependency Assessment (Phase 4 prerequisite) determined **local** or **both** execution, generate this script.
|
|
|
|
For all other projects, **use Docker** (`docker-compose.test.yml` / `Dockerfile.test`). Docker is the default — it provides reproducibility, isolation, and CI parity. Do not generate a local `run-tests.sh` when Docker is sufficient.
|
|
|
|
## `scripts/run-tests.sh` (local / hardware-dependent only)
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
UNIT_ONLY=false
|
|
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
|
|
|
for arg in "$@"; do
|
|
case $arg in
|
|
--unit-only) UNIT_ONLY=true ;;
|
|
esac
|
|
done
|
|
|
|
cleanup() {
|
|
# tear down services started by this script
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
mkdir -p "$RESULTS_DIR"
|
|
|
|
# --- Install Dependencies ---
|
|
# MANDATORY: install all project + test dependencies before building or running.
|
|
# A fresh clone or CI runner may have nothing installed.
|
|
# Python: pip install -q -r requirements.txt -r e2e/requirements.txt
|
|
# .NET: dotnet restore
|
|
# Rust: cargo fetch
|
|
# Node: npm ci
|
|
|
|
# --- Build (if needed) ---
|
|
# [e.g. Cython: python setup.py build_ext --inplace]
|
|
|
|
# --- Unit Tests ---
|
|
# [detect runner: pytest / dotnet test / cargo test / npm test]
|
|
# [run and capture exit code]
|
|
|
|
# --- Blackbox Tests (skip if --unit-only) ---
|
|
# if ! $UNIT_ONLY; then
|
|
# [start mock services]
|
|
# [start system under test]
|
|
# [wait for health checks]
|
|
# [run blackbox test suite]
|
|
# fi
|
|
|
|
# --- Summary ---
|
|
# [print passed / failed / skipped counts]
|
|
# [exit 0 if all passed, exit 1 otherwise]
|
|
```
|
|
|
|
## `scripts/run-performance-tests.sh`
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
|
|
|
cleanup() {
|
|
# tear down test environment if started
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
mkdir -p "$RESULTS_DIR"
|
|
|
|
# --- Install Dependencies ---
|
|
# [same as above — always install first]
|
|
|
|
# --- Start System Under Test ---
|
|
# [docker compose up -d or start local server]
|
|
# [wait for health checks]
|
|
|
|
# --- Run Performance Scenarios ---
|
|
# [detect tool: k6 / locust / artillery / wrk / built-in]
|
|
# [run each scenario from performance-tests.md]
|
|
# [capture metrics: latency P50/P95/P99, throughput, error rate]
|
|
|
|
# --- Compare Against Thresholds ---
|
|
# [read thresholds from test spec or CLI args]
|
|
# [print per-scenario pass/fail]
|
|
|
|
# --- Summary ---
|
|
# [exit 0 if all thresholds met, exit 1 otherwise]
|
|
```
|
|
|
|
## Key Requirements
|
|
|
|
- **Docker is the default**: only generate a local `run-tests.sh` for hardware-dependent projects. Otherwise use `docker-compose.test.yml`.
|
|
- **Always install dependencies first**: the script must install all project and test dependencies before building or running tests. A fresh clone or CI runner may have nothing installed. Missing a single dependency causes collection errors that abort the entire test run.
|
|
- Both scripts must be idempotent (safe to run multiple times)
|
|
- Both scripts must work in CI (no interactive prompts, no GUI)
|
|
- Use `trap cleanup EXIT` to ensure teardown even on failure
|
|
- Exit codes: 0 = all pass, 1 = failures detected
|
|
- Write results to `test-results/` directory (add to `.gitignore` if not already present)
|
|
- The actual commands depend on the detected tech stack — fill them in during Phase 4 of the test-spec skill
|