mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-23 08:41:13 +00:00
Update demo replay validation and testing documentation
ci/woodpecker/push/02-build-push Pipeline failed
ci/woodpecker/push/02-build-push Pipeline failed
- Modified the autodev state to reflect the current testing phase and details of the new `jetson-e2e` tests. - Enhanced the "How to Test" documentation to provide clearer instructions on the demo replay validation process, including video and tlog alignment steps. - Updated architectural documentation to include the new demo replay operator flow and its dependencies. - Documented the removal of deprecated auto-sync features and clarified the operator-facing UI for replay validation. - Added new entries in the dependencies table for upcoming tasks related to the demo replay flow. These changes improve clarity and usability for operators and developers working with the demo replay system.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,135 @@
|
||||
# Expected Results Template
|
||||
|
||||
Save as `_docs/00_problem/input_data/expected_results/results_report.md`.
|
||||
For complex expected outputs, place reference CSV files alongside it in `_docs/00_problem/input_data/expected_results/`.
|
||||
Referenced by the test-spec skill (`.cursor/skills/test-spec/SKILL.md`).
|
||||
|
||||
---
|
||||
|
||||
```markdown
|
||||
# Expected Results
|
||||
|
||||
Maps every input data item to its quantifiable expected result.
|
||||
Tests use this mapping to compare actual system output against known-correct answers.
|
||||
|
||||
## Result Format Legend
|
||||
|
||||
| Result Type | When to Use | Example |
|
||||
|-------------|-------------|---------|
|
||||
| Exact value | Output must match precisely | `status_code: 200`, `detection_count: 3` |
|
||||
| Tolerance range | Numeric output with acceptable variance | `confidence: 0.92 ± 0.05`, `bbox_x: 120 ± 10px` |
|
||||
| Threshold | Output must exceed or stay below a limit | `latency < 500ms`, `confidence ≥ 0.85` |
|
||||
| Pattern match | Output must match a string/regex pattern | `error_message contains "invalid format"` |
|
||||
| File reference | Complex output compared against a reference file | `match expected_results/case_01.json` |
|
||||
| Schema match | Output structure must conform to a schema | `response matches DetectionResultSchema` |
|
||||
| Set/count | Output must contain specific items or counts | `classes ⊇ {"car", "person"}`, `detections.length == 5` |
|
||||
|
||||
## Comparison Methods
|
||||
|
||||
| Method | Description | Tolerance Syntax |
|
||||
|--------|-------------|-----------------|
|
||||
| `exact` | Actual == Expected | N/A |
|
||||
| `numeric_tolerance` | abs(actual - expected) ≤ tolerance | `± <value>` or `± <percent>%` |
|
||||
| `range` | min ≤ actual ≤ max | `[min, max]` |
|
||||
| `threshold_min` | actual ≥ threshold | `≥ <value>` |
|
||||
| `threshold_max` | actual ≤ threshold | `≤ <value>` |
|
||||
| `regex` | actual matches regex pattern | regex string |
|
||||
| `substring` | actual contains substring | substring |
|
||||
| `json_diff` | structural comparison against reference JSON | diff tolerance per field |
|
||||
| `set_contains` | actual output set contains expected items | subset notation |
|
||||
| `file_reference` | compare against reference file in expected_results/ | file path |
|
||||
|
||||
## Input → Expected Result Mapping
|
||||
|
||||
### [Scenario Group Name, e.g. "Single Image Detection"]
|
||||
|
||||
| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
|
||||
|---|-------|-------------------|-----------------|------------|-----------|---------------|
|
||||
| 1 | `[file or parameters]` | [what this input represents] | [quantifiable expected output] | [method from table above] | [± value, range, or N/A] | [path in expected_results/ or N/A] |
|
||||
|
||||
#### Example — Object Detection
|
||||
|
||||
| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
|
||||
|---|-------|-------------------|-----------------|------------|-----------|---------------|
|
||||
| 1 | `image_01.jpg` | Aerial photo, 3 vehicles visible | `detection_count: 3`, classes: `["ArmorVehicle", "ArmorVehicle", "Truck"]` | exact (count), set_contains (classes) | N/A | N/A |
|
||||
| 2 | `image_01.jpg` | Same image, bbox positions | bboxes: `[(120,80,340,290), (400,150,580,310), (50,400,200,520)]` | numeric_tolerance | ± 15px per coordinate | `expected_results/image_01_detections.json` |
|
||||
| 3 | `image_01.jpg` | Same image, confidence scores | confidences: `[0.94, 0.88, 0.91]` | threshold_min | each ≥ 0.85 | N/A |
|
||||
| 4 | `empty_scene.jpg` | Aerial photo, no objects | `detection_count: 0`, empty detections array | exact | N/A | N/A |
|
||||
| 5 | `corrupted.dat` | Invalid file format | HTTP 400, body contains `"error"` key | exact (status), substring (body) | N/A | N/A |
|
||||
|
||||
#### Example — Performance
|
||||
|
||||
| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
|
||||
|---|-------|-------------------|-----------------|------------|-----------|---------------|
|
||||
| 1 | `standard_image.jpg` | 1920x1080 single image | Response time | threshold_max | ≤ 2000ms | N/A |
|
||||
| 2 | `large_image.jpg` | 8000x6000 tiled image | Response time | threshold_max | ≤ 10000ms | N/A |
|
||||
|
||||
#### Example — Error Handling
|
||||
|
||||
| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
|
||||
|---|-------|-------------------|-----------------|------------|-----------|---------------|
|
||||
| 1 | `POST /detect` with no file | Missing required input | HTTP 422, message matches `"file.*required"` | exact (status), regex (message) | N/A | N/A |
|
||||
| 2 | `POST /detect` with `probability_threshold: 5.0` | Out-of-range config | HTTP 422 or clamped to valid range | exact (status) or range [0.0, 1.0] | N/A | N/A |
|
||||
|
||||
## Expected Result Reference Files
|
||||
|
||||
When the expected output is too complex for an inline table cell (e.g., full JSON response with nested objects), place a reference file in `_docs/00_problem/input_data/expected_results/`.
|
||||
|
||||
### File Naming Convention
|
||||
|
||||
`<input_name>_expected.<format>`
|
||||
|
||||
Examples:
|
||||
- `image_01_detections.json`
|
||||
- `batch_A_results.csv`
|
||||
- `video_01_annotations.json`
|
||||
|
||||
### Reference File Requirements
|
||||
|
||||
- Must be machine-readable (JSON, CSV, YAML — not prose)
|
||||
- Must contain only the expected output structure and values
|
||||
- Must include tolerance annotations where applicable (as metadata fields or comments)
|
||||
- Must be valid and parseable by standard libraries
|
||||
|
||||
### Reference File Example (JSON)
|
||||
|
||||
File: `expected_results/image_01_detections.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"input": "image_01.jpg",
|
||||
"expected": {
|
||||
"detection_count": 3,
|
||||
"detections": [
|
||||
{
|
||||
"class": "ArmorVehicle",
|
||||
"confidence": { "min": 0.85 },
|
||||
"bbox": { "x1": 120, "y1": 80, "x2": 340, "y2": 290, "tolerance_px": 15 }
|
||||
},
|
||||
{
|
||||
"class": "ArmorVehicle",
|
||||
"confidence": { "min": 0.85 },
|
||||
"bbox": { "x1": 400, "y1": 150, "x2": 580, "y2": 310, "tolerance_px": 15 }
|
||||
},
|
||||
{
|
||||
"class": "Truck",
|
||||
"confidence": { "min": 0.85 },
|
||||
"bbox": { "x1": 50, "y1": 400, "x2": 200, "y2": 520, "tolerance_px": 15 }
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Guidance Notes
|
||||
|
||||
- Every row in the mapping table must have at least one quantifiable comparison — no row should say only "should work" or "returns result".
|
||||
- Use `exact` comparison for counts, status codes, and discrete values.
|
||||
- Use `numeric_tolerance` for floating-point values and spatial coordinates where minor variance is expected.
|
||||
- Use `threshold_min`/`threshold_max` for performance metrics and confidence scores.
|
||||
- Use `file_reference` when the expected output has more than ~3 fields or nested structures.
|
||||
- Reference files must be committed alongside input data — they are part of the test specification.
|
||||
- When the system has non-deterministic behavior (e.g., model inference variance across hardware), document the expected tolerance explicitly and justify it.
|
||||
@@ -0,0 +1,109 @@
|
||||
# Test Runner Script Structure
|
||||
|
||||
Reference for generating `scripts/run-tests.sh` and `scripts/run-performance-tests.sh`.
|
||||
|
||||
## When to generate a local `run-tests.sh`
|
||||
|
||||
A local shell script is needed **only** for hardware-dependent projects that require real hardware (GPU, CoreML, TPU, sensors, etc.) to exercise the actual code paths. If the Hardware-Dependency Assessment (Phase 4 prerequisite) determined **local** or **both** execution, generate this script.
|
||||
|
||||
For all other projects, **use Docker** (`docker-compose.test.yml` / `Dockerfile.test`). Docker is the default — it provides reproducibility, isolation, and CI parity. Do not generate a local `run-tests.sh` when Docker is sufficient.
|
||||
|
||||
## `scripts/run-tests.sh` (local / hardware-dependent only)
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
UNIT_ONLY=false
|
||||
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
||||
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--unit-only) UNIT_ONLY=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
cleanup() {
|
||||
# tear down services started by this script
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
|
||||
# --- Install Dependencies ---
|
||||
# MANDATORY: install all project + test dependencies before building or running.
|
||||
# A fresh clone or CI runner may have nothing installed.
|
||||
# Python: pip install -q -r requirements.txt -r e2e/requirements.txt
|
||||
# .NET: dotnet restore
|
||||
# Rust: cargo fetch
|
||||
# Node: npm ci
|
||||
|
||||
# --- Build (if needed) ---
|
||||
# [e.g. Cython: python setup.py build_ext --inplace]
|
||||
|
||||
# --- Unit Tests ---
|
||||
# [detect runner: pytest / dotnet test / cargo test / npm test]
|
||||
# [run and capture exit code]
|
||||
|
||||
# --- Blackbox Tests (skip if --unit-only) ---
|
||||
# if ! $UNIT_ONLY; then
|
||||
# [start mock services]
|
||||
# [start system under test]
|
||||
# [wait for health checks]
|
||||
# [run blackbox test suite]
|
||||
# fi
|
||||
|
||||
# --- Summary ---
|
||||
# [print passed / failed / skipped counts]
|
||||
# [exit 0 if all passed, exit 1 otherwise]
|
||||
```
|
||||
|
||||
## `scripts/run-performance-tests.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
RESULTS_DIR="$PROJECT_ROOT/test-results"
|
||||
|
||||
cleanup() {
|
||||
# tear down test environment if started
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
|
||||
# --- Install Dependencies ---
|
||||
# [same as above — always install first]
|
||||
|
||||
# --- Start System Under Test ---
|
||||
# [docker compose up -d or start local server]
|
||||
# [wait for health checks]
|
||||
|
||||
# --- Run Performance Scenarios ---
|
||||
# [detect tool: k6 / locust / artillery / wrk / built-in]
|
||||
# [run each scenario from performance-tests.md]
|
||||
# [capture metrics: latency P50/P95/P99, throughput, error rate]
|
||||
|
||||
# --- Compare Against Thresholds ---
|
||||
# [read thresholds from test spec or CLI args]
|
||||
# [print per-scenario pass/fail]
|
||||
|
||||
# --- Summary ---
|
||||
# [exit 0 if all thresholds met, exit 1 otherwise]
|
||||
```
|
||||
|
||||
## Key Requirements
|
||||
|
||||
- **Docker is the default**: only generate a local `run-tests.sh` for hardware-dependent projects. Otherwise use `docker-compose.test.yml`.
|
||||
- **Always install dependencies first**: the script must install all project and test dependencies before building or running tests. A fresh clone or CI runner may have nothing installed. Missing a single dependency causes collection errors that abort the entire test run.
|
||||
- Both scripts must be idempotent (safe to run multiple times)
|
||||
- Both scripts must work in CI (no interactive prompts, no GUI)
|
||||
- Use `trap cleanup EXIT` to ensure teardown even on failure
|
||||
- Exit codes: 0 = all pass, 1 = failures detected
|
||||
- Write results to `test-results/` directory (add to `.gitignore` if not already present)
|
||||
- The actual commands depend on the detected tech stack — fill them in during Phase 4 of the test-spec skill
|
||||
Reference in New Issue
Block a user