azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-23 12:21:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update demo replay validation and testing documentation
ci/woodpecker/push/02-build-push Pipeline failed
Details

Browse Source 
- Modified the autodev state to reflect the current testing phase and details of the new `jetson-e2e` tests.
- Enhanced the "How to Test" documentation to provide clearer instructions on the demo replay validation process, including video and tlog alignment steps.
- Updated architectural documentation to include the new demo replay operator flow and its dependencies.
- Documented the removal of deprecated auto-sync features and clarified the operator-facing UI for replay validation.
- Added new entries in the dependencies table for upcoming tasks related to the demo replay flow.

These changes improve clarity and usability for operators and developers working with the demo replay system.
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-06-20 11:24:43 +03:00
parent 12d0008763
commit 1f634c2604
175 changed files with 20701 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cursor/skills/test-spec/SKILL.md
+273
View File
@@ -0,0 +1,273 @@
---
name: test-spec
description: |
Test specification skill. Analyzes input data and expected results completeness,
then produces detailed test scenarios (blackbox, performance, resilience, security, resource limits)
that treat the system as a black box. Every test pairs input data with quantifiable expected results
so tests can verify correctness, not just execution.
4-phase workflow: input data + expected results analysis, test scenario specification, data + results validation gate,
test runner script generation. Produces 8 artifacts under tests/ and 2 shell scripts under scripts/.
Trigger phrases:
- "test spec", "test specification", "test scenarios"
- "blackbox test spec", "black box tests", "blackbox tests"
- "performance tests", "resilience tests", "security tests"
category: build
tags: [testing, black-box, blackbox-tests, test-specification, qa]
disable-model-invocation: true
---
# Test Scenario Specification
Analyze input data completeness and produce detailed black-box test specifications. Tests describe what the system should do given specific inputs — they never reference internals.
## Core Principles
- **Black-box only**: tests describe observable behavior through public interfaces; no internal implementation details
- **Traceability**: every test traces to at least one acceptance criterion or restriction
- **Save immediately**: write artifacts to disk after each phase; never accumulate unsaved work
- **Ask, don't assume**: when requirements are ambiguous, ask the user before proceeding
- **Spec, don't code**: this workflow produces test specifications, never test implementation code
- **Every test must have a pass/fail criterion**. Two acceptable shapes:
- **Input/output shape**: concrete input data paired with a quantifiable expected result (exact value, tolerance, threshold, pattern, reference file). Typical for functional blackbox tests, performance tests with load data, data-processing pipelines.
- **Behavioral shape**: a trigger condition + observable system behavior + quantifiable pass/fail criterion, with no input data required. Typical for startup/shutdown tests, retry/backoff policies, state transitions, logging/metrics emission, resilience scenarios. Example criteria: "startup logs `service ready` within 5s", "retry emits 3 attempts with exponential backoff (base 100ms ± 20ms)", "on SIGTERM, service drains in-flight requests within 30s grace period", "health endpoint returns 503 while migrations run".
- For behavioral tests the observable (log line, metric value, state transition, emitted event, elapsed time) must still be quantifiable — the test must programmatically decide pass/fail.
- A test that cannot produce a pass/fail verdict through either shape is not verifiable and must be removed.
## Context Resolution
Fixed paths:
- PROBLEM_DIR: `_docs/00_problem/`
- SOLUTION_DIR: `_docs/01_solution/`
- DOCUMENT_DIR: `_docs/02_document/`
- TESTS_OUTPUT_DIR: `_docs/02_document/tests/`
Announce the resolved paths and the detected invocation mode (below) to the user before proceeding.
### Invocation Modes
- **full** (default): runs all 4 phases against the whole `PROBLEM_DIR` + `DOCUMENT_DIR`. Used in greenfield Plan Step 1 and existing-code Step 3.
- **cycle-update**: runs only a scoped refresh of the existing test-spec artifacts against the current feature cycle's completed tasks. Used by the existing-code flow's per-cycle sync step. See `modes/cycle-update.md` for the narrowed workflow.
## Input Specification
### Required Files
| File | Purpose |
|------|---------|
| `_docs/00_problem/problem.md` | Problem description and context |
| `_docs/00_problem/acceptance_criteria.md` | Measurable acceptance criteria |
| `_docs/00_problem/restrictions.md` | Constraints and limitations |
| `_docs/00_problem/input_data/` | Reference data examples, expected results, and optional reference files |
| `_docs/01_solution/solution.md` | Finalized solution |
### Expected Results Specification
Every input data item MUST have a corresponding expected result that defines what the system should produce. Expected results MUST be **quantifiable** — the test must be able to programmatically compare actual system output against the expected result and produce a pass/fail verdict.
Expected results live inside `_docs/00_problem/input_data/` in one or both of:
1. **Mapping file** (`input_data/expected_results/results_report.md`): a table pairing each input with its quantifiable expected output, using the format defined in `templates/expected-results.md`
2. **Reference files folder** (`input_data/expected_results/`): machine-readable files (JSON, CSV, etc.) containing full expected outputs for complex cases, referenced from the mapping file
```
input_data/
├── expected_results/ ← required: expected results folder
│ ├── results_report.md ← required: input→expected result mapping
│ ├── image_01_expected.csv ← per-file expected detections
│ └── video_01_expected.csv
├── image_01.jpg
├── empty_scene.jpg
└── data_parameters.md
```
**Quantifiability requirements** (see `templates/expected-results.md` for full format and examples):
- Numeric values: exact value or value ± tolerance (e.g., `confidence ≥ 0.85`, `position ± 10px`)
- Structured data: exact JSON/CSV values, or a reference file in `expected_results/`
- Counts: exact counts (e.g., "3 detections", "0 errors")
- Text/patterns: exact string or regex pattern to match
- Timing: threshold (e.g., "response ≤ 500ms")
- Error cases: expected error code, message pattern, or HTTP status
### Optional Files (used when available)
| File | Purpose |
|------|---------|
| `DOCUMENT_DIR/architecture.md` | System architecture for environment design |
| `DOCUMENT_DIR/system-flows.md` | System flows for test scenario coverage |
| `DOCUMENT_DIR/components/` | Component specs for interface identification |
### Prerequisite Checks (BLOCKING)
1. `acceptance_criteria.md` exists and is non-empty — **STOP if missing**
2. `restrictions.md` exists and is non-empty — **STOP if missing**
3. `input_data/` exists and contains at least one file — **STOP if missing**
4. `input_data/expected_results/results_report.md` exists and is non-empty — **STOP if missing**. Prompt the user: *"Expected results mapping is required. Please create `_docs/00_problem/input_data/expected_results/results_report.md` pairing each input with its quantifiable expected output. Use `templates/expected-results.md` as the format reference."*
5. `problem.md` exists and is non-empty — **STOP if missing**
6. `solution.md` exists and is non-empty — **STOP if missing**
7. Create TESTS_OUTPUT_DIR if it does not exist
8. If TESTS_OUTPUT_DIR already contains files, ask user: **resume from last checkpoint or start fresh?**
## Artifact Management
### Directory Structure
```
TESTS_OUTPUT_DIR/
├── environment.md
├── test-data.md
├── blackbox-tests.md
├── performance-tests.md
├── resilience-tests.md
├── security-tests.md
├── resource-limit-tests.md
└── traceability-matrix.md
```
### Save Timing
| Phase | Save immediately after | Filename |
|-------|------------------------|----------|
| Phase 1 | Input data analysis (no file — findings feed Phase 2) | — |
| Phase 2 | Environment spec | `environment.md` |
| Phase 2 | Test data spec | `test-data.md` |
| Phase 2 | Blackbox tests | `blackbox-tests.md` |
| Phase 2 | Performance tests | `performance-tests.md` |
| Phase 2 | Resilience tests | `resilience-tests.md` |
| Phase 2 | Security tests | `security-tests.md` |
| Phase 2 | Resource limit tests | `resource-limit-tests.md` |
| Phase 2 | Traceability matrix | `traceability-matrix.md` |
| Phase 3 | Updated test data spec (if data added) | `test-data.md` |
| Phase 3 | Updated test files (if tests removed) | respective test file |
| Phase 3 | Updated traceability matrix (if tests removed) | `traceability-matrix.md` |
| Hardware Assessment | Test Execution section | `environment.md` (updated) |
| Phase 4 | Test runner script | `scripts/run-tests.sh` |
| Phase 4 | Performance test runner script | `scripts/run-performance-tests.sh` |
### Resumability
If TESTS_OUTPUT_DIR already contains files:
1. List existing files and match them to the save timing table above
2. Identify which phase/artifacts are complete
3. Resume from the next incomplete artifact
4. Inform the user which artifacts are being skipped
## Progress Tracking
At the start of execution, create a TodoWrite with all four phases (plus the hardware assessment between Phase 3 and Phase 4). Update status as each phase completes.
## Workflow
### Phase 1: Input Data & Expected Results Completeness Analysis
Read and follow `phases/01-input-data-analysis.md`.
---
### Phase 2: Test Scenario Specification
Read and follow `phases/02-test-scenarios.md`.
---
### Phase 3: Test Data Validation Gate (HARD GATE)
Read and follow `phases/03-data-validation-gate.md`.
---
### Hardware-Dependency & Execution Environment Assessment (BLOCKING — runs between Phase 3 and Phase 4)
Read and follow `phases/hardware-assessment.md`.
---
### Phase 4: Test Runner Script Generation
Read and follow `phases/04-runner-scripts.md`.
---
### cycle-update mode
If invoked in `cycle-update` mode (see "Invocation Modes" above), read and follow `modes/cycle-update.md` instead of the full 4-phase workflow.
## Escalation Rules
| Situation | Action |
|-----------|--------|
| Missing acceptance_criteria.md, restrictions.md, or input_data/ | **STOP** — specification cannot proceed |
| Missing input_data/expected_results/results_report.md | **STOP** — ask user to provide expected results mapping using the template |
| Ambiguous requirements | ASK user |
| Input data coverage below the canonical threshold (Phase 1) | Search internet for supplementary data, ASK user to validate. See `.cursor/rules/cursor-meta.mdc` Quality Thresholds for the canonical 75% number — do not hardcode a different threshold here. |
| Expected results missing or not quantifiable (Phase 1) | ASK user to provide quantifiable expected results before proceeding |
| Test scenario conflicts with restrictions | ASK user to clarify intent |
| System interfaces unclear (no architecture.md) | ASK user or derive from solution.md |
| Test data or expected result not provided for a test scenario (Phase 3) | WARN user and REMOVE the test |
| Final coverage below the canonical threshold after removals (Phase 3) | BLOCK — require user to supply data or accept reduced spec (see `cursor-meta.mdc` Quality Thresholds) |
## Common Mistakes
- **Referencing internals**: tests must be black-box — no internal module names, no direct DB queries against the system under test
- **Vague expected outcomes**: "works correctly" is not a test outcome; use specific measurable values
- **Missing pass/fail criterion**: input/output tests without an expected result, OR behavioral tests without a measurable observable — both are unverifiable and must be removed
- **Non-quantifiable criteria**: "should return good results", "works correctly", "behaves properly" — not verifiable. Use exact values, tolerances, thresholds, pattern matches, or timing bounds that code can evaluate.
- **Forcing the wrong shape**: do not invent fake input data for a behavioral test (e.g., "input: SIGTERM signal") just to fit the input/output shape. Classify the test correctly and use the matching checklist.
- **Missing negative scenarios**: every positive scenario category should have corresponding negative/edge-case tests
- **Untraceable tests**: every test should trace to at least one AC or restriction
- **Writing test code**: this skill produces specifications, never implementation code
## Trigger Conditions
When the user wants to:
- Specify blackbox tests before implementation or refactoring
- Analyze input data completeness for test coverage
- Produce test scenarios from acceptance criteria
**Keywords**: "test spec", "test specification", "blackbox test spec", "black box tests", "blackbox tests", "test scenarios"
## Methodology Quick Reference
```
┌──────────────────────────────────────────────────────────────────────┐
│ Test Scenario Specification (4-Phase) │
├──────────────────────────────────────────────────────────────────────┤
│ PREREQ: Data Gate (BLOCKING) │
│ → verify AC, restrictions, input_data (incl. expected_results.md) │
│ │
│ Phase 1: Input Data & Expected Results Completeness Analysis │
│ → phases/01-input-data-analysis.md │
│ [BLOCKING: user confirms input data + expected results coverage] │
│ │
│ Phase 2: Test Scenario Specification │
│ → phases/02-test-scenarios.md │
│ → environment.md · test-data.md · blackbox-tests.md │
│ → performance-tests.md · resilience-tests.md · security-tests.md │
│ → resource-limit-tests.md · traceability-matrix.md │
│ [BLOCKING: user confirms test coverage] │
│ │
│ Phase 3: Test Data & Expected Results Validation Gate (HARD GATE) │
│ → phases/03-data-validation-gate.md │
│ [BLOCKING: coverage ≥ canonical threshold required to pass — │
│ see cursor-meta.mdc Quality Thresholds (75%)] │
│ │
│ Hardware-Dependency Assessment (BLOCKING, pre-Phase-4) │
│ → phases/hardware-assessment.md │
│ │
│ Phase 4: Test Runner Script Generation │
│ → phases/04-runner-scripts.md │
│ → scripts/run-tests.sh (unit + blackbox) │
│ → scripts/run-performance-tests.sh (load/perf scenarios) │
│ │
│ cycle-update mode (scoped refresh) │
│ → modes/cycle-update.md │
├──────────────────────────────────────────────────────────────────────┤
│ Principles: Black-box only · Traceability · Save immediately │
│ Ask don't assume · Spec don't code │
│ No test without data · No test without expected result │
└──────────────────────────────────────────────────────────────────────┘
```