azaion/detections
1
0
Fork 0
mirror of https://github.com/azaion/detections.git synced 2026-04-22 09:06:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-137] [AZ-138] Decompose test tasks and scaffold E2E test infrastructure

Browse Source 
Made-with: Cursor
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-03-23 14:07:54 +02:00
parent 091d9a8fb0
commit 86d8e7e22d
47 changed files with 1883 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cursor/skills/autopilot/SKILL.md
+11 -7
View File
@@ -62,22 +62,26 @@ Every invocation follows this sequence:
6. Present Status Summary (format in protocols.md)
7. Execute:
a. Delegate to current skill (see Skill Delegation below)
b. When skill completes → update state file (rules in state.md)
c. Re-detect next step from the active flow's detection rules
d. If next skill is ready → auto-chain (go to 7a with next skill)
e. If session boundary reached → update state, suggest new conversation (rules in state.md)
f. If all steps done → update state → report completion
b. If skill returns FAILED → apply Skill Failure Retry Protocol (see protocols.md):
- Auto-retry the same skill (failure may be caused by missing user input or environment issue)
- If 3 consecutive auto-retries fail → record in state file Blockers, warn user, stop auto-retry
c. When skill completes successfully → reset retry counter, update state file (rules in state.md)
d. Re-detect next step from the active flow's detection rules
e. If next skill is ready → auto-chain (go to 7a with next skill)
f. If session boundary reached → update state, suggest new conversation (rules in state.md)
g. If all steps done → update state → report completion
```
## Skill Delegation
For each step, the delegation pattern is:
1. Update state file: set `step` to the autopilot step number, status to `in_progress`, set `sub_step` to the sub-skill's current internal step/phase
1. Update state file: set `step` to the autopilot step number, status to `in_progress`, set `sub_step` to the sub-skill's current internal step/phase, reset `retry_count: 0`
2. Announce: "Starting [Skill Name]..."
3. Read the skill file: `.cursor/skills/[name]/SKILL.md`
4. Execute the skill's workflow exactly as written, including all BLOCKING gates, self-verification checklists, save actions, and escalation rules. Update `sub_step` in state each time the sub-skill advances.
5. When complete: mark step `completed`, record date + key outcome, add key decisions to state file, return to auto-chain rules (from active flow file)
5. If the skill **fails**: follow the Skill Failure Retry Protocol in `protocols.md` — increment `retry_count`, auto-retry up to 3 times, then escalate.
6. When complete (success): reset `retry_count: 0`, mark step `completed`, record date + key outcome, add key decisions to state file, return to auto-chain rules (from active flow file)
Do NOT modify, skip, or abbreviate any part of the sub-skill's workflow. The autopilot is a sequencer, not an optimizer.
.cursor/skills/autopilot/protocols.md
+90 -18
View File
@@ -106,6 +106,76 @@ All error situations that require user input MUST use the **Choose A / B / C / D
| User wants to go back to a previous step | Use Choose format: A) re-run (with overwrite warning), B) stay on current step |
| User asks "where am I?" without wanting to continue | Show Status Summary only, do not start execution |
## Skill Failure Retry Protocol
Sub-skills can return a **failed** result. Failures are often caused by missing user input, environment issues, or transient errors that resolve on retry. The autopilot auto-retries before escalating.
### Retry Flow
```
Skill execution → FAILED
│
├─ retry_count < 3 ?
│ YES → increment retry_count in state file
│ → log failure reason in state file (Retry Log section)
│ → re-read the sub-skill's SKILL.md
│ → re-execute from the current sub_step
│ → (loop back to check result)
│
│ NO (retry_count = 3) →
│ → set status: failed in Current Step
│ → add entry to Blockers section:
│ "[Skill Name] failed 3 consecutive times at sub_step [M].
│ Last failure: [reason]. Auto-retry exhausted."
│ → present warning to user (see Escalation below)
│ → do NOT auto-retry again until user intervenes
```
### Retry Rules
1. **Auto-retry immediately**: when a skill fails, retry it without asking the user — the failure is often transient (missing user confirmation in a prior step, docker not running, file lock, etc.)
2. **Preserve sub_step**: retry from the last recorded `sub_step`, not from the beginning of the skill — unless the failure indicates corruption, in which case restart from sub_step 1
3. **Increment `retry_count`**: update `retry_count` in the state file's `Current Step` section on each retry attempt
4. **Log each failure**: append the failure reason and timestamp to the state file's `Retry Log` section
5. **Reset on success**: when the skill eventually succeeds, reset `retry_count: 0` and clear the `Retry Log` for that step
### Escalation (after 3 consecutive failures)
After 3 failed auto-retries of the same skill, the failure is likely not user-related. Stop retrying and escalate:
1. Update the state file:
- Set `status: failed` in `Current Step`
- Set `retry_count: 3`
- Add a blocker entry describing the repeated failure
2. Play notification sound (per `human-input-sound.mdc`)
3. Present using Choose format:
```
══════════════════════════════════════
SKILL FAILED: [Skill Name] — 3 consecutive failures
══════════════════════════════════════
Step: [N] — [Name]
SubStep: [M] — [sub-step name]
Last failure reason: [reason]
══════════════════════════════════════
A) Retry with fresh context (new conversation)
B) Skip this step with warning
C) Abort — investigate and fix manually
══════════════════════════════════════
Recommendation: A — fresh context often resolves
persistent failures
══════════════════════════════════════
```
### Re-Entry After Failure
On the next autopilot invocation (new conversation), if the state file shows `status: failed` and `retry_count: 3`:
- Present the blocker to the user before attempting execution
- If the user chooses to retry → reset `retry_count: 0`, set `status: in_progress`, and re-execute
- If the user chooses to skip → mark step as `skipped`, proceed to next step
- Do NOT silently auto-retry — the user must acknowledge the persistent failure first
## Error Recovery Protocol
### Stuck Detection
@@ -211,17 +281,18 @@ On every invocation, before executing any skill, present a status summary built
═══════════════════════════════════════════════════
AUTOPILOT STATUS (greenfield)
═══════════════════════════════════════════════════
Step 0 Problem [DONE / IN PROGRESS / NOT STARTED]
Step 1 Research [DONE (N drafts) / IN PROGRESS / NOT STARTED]
Step 2 Plan [DONE / IN PROGRESS / NOT STARTED]
Step 3 Decompose [DONE (N tasks) / IN PROGRESS / NOT STARTED]
Step 4 Implement [DONE / IN PROGRESS (batch M of ~N) / NOT STARTED]
Step 5 Run Tests [DONE (N passed, M failed) / IN PROGRESS / NOT STARTED]
Step 5b Security Audit [DONE / SKIPPED / IN PROGRESS / NOT STARTED]
Step 6 Deploy [DONE / IN PROGRESS / NOT STARTED]
Step 0 Problem [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 1 Research [DONE (N drafts) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2 Plan [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 3 Decompose [DONE (N tasks) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 4 Implement [DONE / IN PROGRESS (batch M of ~N) / NOT STARTED / FAILED (retry N/3)]
Step 5 Run Tests [DONE (N passed, M failed) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 5b Security Audit [DONE / SKIPPED / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 6 Deploy [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
═══════════════════════════════════════════════════
Current: Step N — Name
SubStep: M — [sub-skill internal step name]
Retry: [N/3 if retrying, omit if 0]
Action: [what will happen next]
═══════════════════════════════════════════════════
```
@@ -232,19 +303,20 @@ On every invocation, before executing any skill, present a status summary built
═══════════════════════════════════════════════════
AUTOPILOT STATUS (existing-code)
═══════════════════════════════════════════════════
Pre Document [DONE / IN PROGRESS / NOT STARTED]
Step 2b Blackbox Test Spec [DONE / IN PROGRESS / NOT STARTED]
Step 2c Decompose Tests [DONE (N tasks) / IN PROGRESS / NOT STARTED]
Step 2d Implement Tests [DONE / IN PROGRESS (batch M) / NOT STARTED]
Step 2e Refactor [DONE / IN PROGRESS (phase N) / NOT STARTED]
Step 2f New Task [DONE (N tasks) / IN PROGRESS / NOT STARTED]
Step 2g Implement [DONE / IN PROGRESS (batch M of ~N) / NOT STARTED]
Step 2h Run Tests [DONE (N passed, M failed) / IN PROGRESS / NOT STARTED]
Step 2hb Security Audit [DONE / SKIPPED / IN PROGRESS / NOT STARTED]
Step 2i Deploy [DONE / IN PROGRESS / NOT STARTED]
Pre Document [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2b Blackbox Test Spec [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2c Decompose Tests [DONE (N tasks) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2d Implement Tests [DONE / IN PROGRESS (batch M) / NOT STARTED / FAILED (retry N/3)]
Step 2e Refactor [DONE / IN PROGRESS (phase N) / NOT STARTED / FAILED (retry N/3)]
Step 2f New Task [DONE (N tasks) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2g Implement [DONE / IN PROGRESS (batch M of ~N) / NOT STARTED / FAILED (retry N/3)]
Step 2h Run Tests [DONE (N passed, M failed) / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2hb Security Audit [DONE / SKIPPED / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
Step 2i Deploy [DONE / IN PROGRESS / NOT STARTED / FAILED (retry N/3)]
═══════════════════════════════════════════════════
Current: Step N — Name
SubStep: M — [sub-skill internal step name]
Retry: [N/3 if retrying, omit if 0]
Action: [what will happen next]
═══════════════════════════════════════════════════
```
.cursor/skills/autopilot/state.md
+21 -2
View File
@@ -12,8 +12,9 @@ The autopilot persists its state to `_docs/_autopilot_state.md`. This file is th
## Current Step
step: [0-6 or "2b" / "2c" / "2d" / "2e" / "2f" / "2g" / "2h" / "2hb" / "2i" or "5b" or "done"]
name: [Problem / Research / Plan / Blackbox Test Spec / Decompose Tests / Implement Tests / Refactor / New Task / Implement / Run Tests / Security Audit / Deploy / Decompose / Done]
status: [not_started / in_progress / completed / skipped]
status: [not_started / in_progress / completed / skipped / failed]
sub_step: [optional — sub-skill internal step number + name if interrupted mid-step]
retry_count: [0-3 — number of consecutive auto-retry attempts for current step, reset to 0 on success]
## Step ↔ SubStep Reference
(include the step reference table from the active flow file)
@@ -21,11 +22,19 @@ sub_step: [optional — sub-skill internal step number + name if interrupted mid
When updating `Current Step`, always write it as:
step: N ← autopilot step (0–6 or 2b/2c/2d/2e/2f/2g/2h/2hb/2i or 5b)
sub_step: M ← sub-skill's own internal step/phase number + name
retry_count: 0 ← reset on new step or success; increment on each failed retry
Example:
step: 2
name: Plan
status: in_progress
sub_step: 4 — Architecture Review & Risk Assessment
retry_count: 0
Example (failed after 3 retries):
step: 2b
name: Blackbox Test Spec
status: failed
sub_step: 1b — Test Case Generation
retry_count: 3
## Completed Steps
@@ -45,6 +54,14 @@ ended_at: Step [N] [Name] — SubStep [M] [sub-step name]
reason: [completed step / session boundary / user paused / context limit]
notes: [any context for next session]
## Retry Log
| Attempt | Step | Name | SubStep | Failure Reason | Timestamp |
|---------|------|------|---------|----------------|-----------|
| 1 | [step] | [name] | [sub_step] | [reason] | [date-time] |
| ... | ... | ... | ... | ... | ... |
(Clear this table when the step succeeds or user resets. Append a row on each failed auto-retry.)
## Blockers
- [blocker 1, if any]
- [none]
@@ -53,10 +70,12 @@ notes: [any context for next session]
### State File Rules
1. **Create** the state file on the very first autopilot invocation (after state detection determines Step 0)
2. **Update** the state file after every step completion, every session boundary, and every BLOCKING gate confirmation
2. **Update** the state file after every step completion, every session boundary, every BLOCKING gate confirmation, and every failed retry attempt
3. **Read** the state file as the first action on every invocation — before folder scanning
4. **Cross-check**: after reading the state file, verify against actual `_docs/` folder contents. If they disagree (e.g., state file says Step 2 but `_docs/02_document/architecture.md` already exists), trust the folder structure and update the state file to match
5. **Never delete** the state file. It accumulates history across the entire project lifecycle
6. **Retry tracking**: increment `retry_count` on each failed auto-retry; reset to `0` when the step succeeds or the user manually resets. If `retry_count` reaches 3, set `status: failed` and add an entry to `Blockers`
7. **Failed state on re-entry**: if the state file shows `status: failed` with `retry_count: 3`, do NOT auto-retry — present the blocker to the user and wait for their decision before proceeding
## State Detection
.cursor/skills/blackbox-test-spec/SKILL.md
+125 -22
View File
@@ -3,7 +3,7 @@ name: blackbox-test-spec
description: |
Black-box integration test specification skill. Analyzes input data completeness and produces
detailed E2E test scenarios (functional + non-functional) that treat the system as a black box.
2-phase workflow: input data completeness analysis, then test scenario specification.
3-phase workflow: input data completeness analysis, test scenario specification, test data validation gate.
Produces 5 artifacts under integration_tests/.
Trigger phrases:
- "blackbox test spec", "black box tests", "integration test spec"
@@ -25,6 +25,7 @@ Analyze input data completeness and produce detailed black-box integration test
- **Save immediately**: write artifacts to disk after each phase; never accumulate unsaved work
- **Ask, don't assume**: when requirements are ambiguous, ask the user before proceeding
- **Spec, don't code**: this workflow produces test specifications, never test implementation code
- **No test without data**: every test scenario MUST have concrete test data; tests without data are removed
## Context Resolution
@@ -84,12 +85,16 @@ TESTS_OUTPUT_DIR/
| Phase | Save immediately after | Filename |
|-------|------------------------|----------|
| Phase 1a | Input data analysis (no file — findings feed Phase 1b) | — |
| Phase 1b | Environment spec | `environment.md` |
| Phase 1b | Test data spec | `test_data.md` |
| Phase 1b | Functional tests | `functional_tests.md` |
| Phase 1b | Non-functional tests | `non_functional_tests.md` |
| Phase 1b | Traceability matrix | `traceability_matrix.md` |
| Phase 1 | Input data analysis (no file — findings feed Phase 2) | — |
| Phase 2 | Environment spec | `environment.md` |
| Phase 2 | Test data spec | `test_data.md` |
| Phase 2 | Functional tests | `functional_tests.md` |
| Phase 2 | Non-functional tests | `non_functional_tests.md` |
| Phase 2 | Traceability matrix | `traceability_matrix.md` |
| Phase 3 | Updated test data spec (if data added) | `test_data.md` |
| Phase 3 | Updated functional tests (if tests removed) | `functional_tests.md` |
| Phase 3 | Updated non-functional tests (if tests removed) | `non_functional_tests.md` |
| Phase 3 | Updated traceability matrix (if tests removed) | `traceability_matrix.md` |
### Resumability
@@ -102,11 +107,11 @@ If TESTS_OUTPUT_DIR already contains files:
## Progress Tracking
At the start of execution, create a TodoWrite with both phases. Update status as each phase completes.
At the start of execution, create a TodoWrite with all three phases. Update status as each phase completes.
## Workflow
### Phase 1a: Input Data Completeness Analysis
### Phase 1: Input Data Completeness Analysis
**Role**: Professional Quality Assurance Engineer
**Goal**: Assess whether the available input data is sufficient to build comprehensive test scenarios
@@ -128,7 +133,7 @@ At the start of execution, create a TodoWrite with both phases. Update status as
---
### Phase 1b: Black-Box Test Scenario Specification
### Phase 2: Black-Box Test Scenario Specification
**Role**: Professional Quality Assurance Engineer
**Goal**: Produce detailed black-box test specifications covering functional and non-functional scenarios
@@ -164,15 +169,103 @@ Capture any new questions, findings, or insights that arise during test specific
---
### Phase 3: Test Data Validation Gate (HARD GATE)
**Role**: Professional Quality Assurance Engineer
**Goal**: Ensure every test scenario produced in Phase 2 has concrete, sufficient test data. Remove tests that lack data. Verify final coverage stays above 70%.
**Constraints**: This phase is MANDATORY and cannot be skipped.
#### Step 1 — Build the test-data requirements checklist
Scan `functional_tests.md` and `non_functional_tests.md`. For every test scenario, extract:
| # | Test Scenario ID | Test Name | Required Data Description | Required Data Quality | Required Data Quantity | Data Provided? |
|---|-----------------|-----------|---------------------------|----------------------|----------------------|----------------|
Present this table to the user.
#### Step 2 — Ask user to provide test data
For each row where **Data Provided?** is **No**, ask the user:
> **Option A — Provide the data**: Supply the necessary test data files (with required quality and quantity as described in the table). Place them in `_docs/00_problem/input_data/` or indicate the location.
>
> **Option B — Skip this test**: If you cannot provide the data, this test scenario will be **removed** from the specification.
**BLOCKING**: Wait for the user's response for every missing data item.
#### Step 3 — Validate provided data
For each item where the user chose **Option A**:
1. Verify the data file(s) exist at the indicated location
2. Verify **quality**: data matches the format, schema, and constraints described in the test scenario (e.g., correct image resolution, valid JSON structure, expected value ranges)
3. Verify **quantity**: enough data samples to cover the scenario (e.g., at least N images for a batch test, multiple edge-case variants)
4. If validation fails, report the specific issue and loop back to Step 2 for that item
#### Step 4 — Remove tests without data
For each item where the user chose **Option B**:
1. Warn the user: `⚠️ Test scenario [ID] "[Name]" will be REMOVED from the specification due to missing test data.`
2. Remove the test scenario from `functional_tests.md` or `non_functional_tests.md`
3. Remove corresponding rows from `traceability_matrix.md`
4. Update `test_data.md` to reflect the removal
**Save action**: Write updated files under TESTS_OUTPUT_DIR:
- `test_data.md`
- `functional_tests.md` (if tests removed)
- `non_functional_tests.md` (if tests removed)
- `traceability_matrix.md` (if tests removed)
#### Step 5 — Final coverage check
After all removals, recalculate coverage:
1. Count remaining test scenarios that trace to acceptance criteria
2. Count total acceptance criteria + restrictions
3. Calculate coverage percentage: `covered_items / total_items * 100`
| Metric | Value |
|--------|-------|
| Total AC + Restrictions | ? |
| Covered by remaining tests | ? |
| **Coverage %** | **?%** |
**Decision**:
- **Coverage ≥ 70%** → Phase 3 **PASSED**. Present final summary to user.
- **Coverage < 70%** → Phase 3 **FAILED**. Report:
> ❌ Test coverage dropped to **X%** (minimum 70% required). The removed test scenarios left gaps in the following acceptance criteria / restrictions:
>
> | Uncovered Item | Type (AC/Restriction) | Missing Test Data Needed |
> |---|---|---|
>
> **Action required**: Provide the missing test data for the items above, or add alternative test scenarios that cover these items with data you can supply.
**BLOCKING**: Loop back to Step 2 with the uncovered items. Do NOT finalize until coverage ≥ 70%.
#### Phase 3 Completion
When coverage ≥ 70% and all remaining tests have validated data:
1. Present the final coverage report
2. List all removed tests (if any) with reasons
3. Confirm all artifacts are saved and consistent
---
## Escalation Rules
| Situation | Action |
|-----------|--------|
| Missing acceptance_criteria.md, restrictions.md, or input_data/ | **STOP** — specification cannot proceed |
| Ambiguous requirements | ASK user |
| Input data coverage below 70% | Search internet for supplementary data, ASK user to validate |
| Input data coverage below 70% (Phase 1) | Search internet for supplementary data, ASK user to validate |
| Test scenario conflicts with restrictions | ASK user to clarify intent |
| System interfaces unclear (no architecture.md) | ASK user or derive from solution.md |
| Test data not provided for a test scenario (Phase 3) | WARN user and REMOVE the test |
| Final coverage below 70% after removals (Phase 3) | BLOCK — require user to supply data or accept reduced spec |
## Common Mistakes
@@ -181,6 +274,7 @@ Capture any new questions, findings, or insights that arise during test specific
- **Missing negative scenarios**: every positive scenario category should have corresponding negative/edge-case tests
- **Untraceable tests**: every test should trace to at least one AC or restriction
- **Writing test code**: this skill produces specifications, never implementation code
- **Tests without data**: every test scenario MUST have concrete test data; a test spec without data is not executable and must be removed
## Trigger Conditions
@@ -194,25 +288,34 @@ When the user wants to:
## Methodology Quick Reference
```
┌────────────────────────────────────────────────────────────────┐
│ Black-Box Test Scenario Specification (2-Phase) │
├────────────────────────────────────────────────────────────────┤
┌─────────────────────────────────────────────────────────────────┐
│ Black-Box Test Scenario Specification (3-Phase) │
├─────────────────────────────────────────────────────────────────┤
│ PREREQ: Data Gate (BLOCKING) │
│ → verify AC, restrictions, input_data, solution exist │
│ │
│ Phase 1a: Input Data Completeness Analysis │
│ │
│ Phase 1: Input Data Completeness Analysis │
│ → assess input_data/ coverage vs AC scenarios (≥70%) │
│ [BLOCKING: user confirms input data coverage] │
│ │
│ Phase 1b: Black-Box Test Scenario Specification │
│ [BLOCKING: user confirms input data coverage] │
│ │
│ Phase 2: Black-Box Test Scenario Specification │
│ → environment.md │
│ → test_data.md │
│ → functional_tests.md (positive + negative) │
│ → non_functional_tests.md (perf, resilience, security, limits)│
│ → traceability_matrix.md │
│ [BLOCKING: user confirms test coverage] │
├────────────────────────────────────────────────────────────────┤
│ [BLOCKING: user confirms test coverage] │
│ │
│ Phase 3: Test Data Validation Gate (HARD GATE) │
│ → build test-data requirements checklist │
│ → ask user: provide data (Option A) or remove test (Option B) │
│ → validate provided data (quality + quantity) │
│ → remove tests without data, warn user │
│ → final coverage check (≥70% or FAIL + loop back) │
│ [BLOCKING: coverage ≥ 70% required to pass] │
├─────────────────────────────────────────────────────────────────┤
│ Principles: Black-box only · Traceability · Save immediately │
│ Ask don't assume · Spec don't code │
└────────────────────────────────────────────────────────────────┘
│ No test without data │
└─────────────────────────────────────────────────────────────────┘
```