mirror of
https://github.com/azaion/detections.git
synced 2026-04-22 11:06:32 +00:00
Refactor testing framework to replace integration tests with blackbox tests across various skills and documentation. Update related workflows, templates, and task specifications to align with the new blackbox testing approach. Remove obsolete integration test files and enhance clarity in task management and reporting structures.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -2,7 +2,7 @@
|
||||
name: decompose
|
||||
description: |
|
||||
Decompose planned components into atomic implementable tasks with bootstrap structure plan.
|
||||
4-step workflow: bootstrap structure plan, component task decomposition, integration test task decomposition, and cross-task verification.
|
||||
4-step workflow: bootstrap structure plan, component task decomposition, blackbox test task decomposition, and cross-task verification.
|
||||
Supports full decomposition (_docs/ structure), single component mode, and tests-only mode.
|
||||
Trigger phrases:
|
||||
- "decompose", "decompose features", "feature decomposition"
|
||||
@@ -36,7 +36,7 @@ Determine the operating mode based on invocation before any other logic runs.
|
||||
- DOCUMENT_DIR: `_docs/02_document/`
|
||||
- TASKS_DIR: `_docs/02_tasks/`
|
||||
- Reads from: `_docs/00_problem/`, `_docs/01_solution/`, DOCUMENT_DIR
|
||||
- Runs Step 1 (bootstrap) + Step 2 (all components) + Step 3 (integration tests) + Step 4 (cross-verification)
|
||||
- Runs Step 1 (bootstrap) + Step 2 (all components) + Step 3 (blackbox tests) + Step 4 (cross-verification)
|
||||
|
||||
**Single component mode** (provided file is within `_docs/02_document/` and inside a `components/` subdirectory):
|
||||
- DOCUMENT_DIR: `_docs/02_document/`
|
||||
@@ -45,12 +45,12 @@ Determine the operating mode based on invocation before any other logic runs.
|
||||
- Ask user for the parent Epic ID
|
||||
- Runs Step 2 (that component only, appending to existing task numbering)
|
||||
|
||||
**Tests-only mode** (provided file/directory is within `integration_tests/`, or `DOCUMENT_DIR/integration_tests/` exists and input explicitly requests test decomposition):
|
||||
**Tests-only mode** (provided file/directory is within `tests/`, or `DOCUMENT_DIR/tests/` exists and input explicitly requests test decomposition):
|
||||
- DOCUMENT_DIR: `_docs/02_document/`
|
||||
- TASKS_DIR: `_docs/02_tasks/`
|
||||
- TESTS_DIR: `DOCUMENT_DIR/integration_tests/`
|
||||
- TESTS_DIR: `DOCUMENT_DIR/tests/`
|
||||
- Reads from: `_docs/00_problem/`, `_docs/01_solution/`, TESTS_DIR
|
||||
- Runs Step 1t (test infrastructure bootstrap) + Step 3 (integration test decomposition) + Step 4 (cross-verification against test coverage)
|
||||
- Runs Step 1t (test infrastructure bootstrap) + Step 3 (blackbox test decomposition) + Step 4 (cross-verification against test coverage)
|
||||
- Skips Step 1 (project bootstrap) and Step 2 (component decomposition) — the codebase already exists
|
||||
|
||||
Announce the detected mode and resolved paths to the user before proceeding.
|
||||
@@ -70,7 +70,7 @@ Announce the detected mode and resolved paths to the user before proceeding.
|
||||
| `DOCUMENT_DIR/architecture.md` | Architecture from plan skill |
|
||||
| `DOCUMENT_DIR/system-flows.md` | System flows from plan skill |
|
||||
| `DOCUMENT_DIR/components/[##]_[name]/description.md` | Component specs from plan skill |
|
||||
| `DOCUMENT_DIR/integration_tests/` | Integration test specs from plan skill |
|
||||
| `DOCUMENT_DIR/tests/` | Blackbox test specs from plan skill |
|
||||
|
||||
**Single component mode:**
|
||||
|
||||
@@ -84,10 +84,13 @@ Announce the detected mode and resolved paths to the user before proceeding.
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `TESTS_DIR/environment.md` | Test environment specification (Docker services, networks, volumes) |
|
||||
| `TESTS_DIR/test_data.md` | Test data management (seed data, mocks, isolation) |
|
||||
| `TESTS_DIR/functional_tests.md` | Functional test scenarios (positive + negative) |
|
||||
| `TESTS_DIR/non_functional_tests.md` | Non-functional test scenarios (perf, resilience, security, limits) |
|
||||
| `TESTS_DIR/traceability_matrix.md` | AC/restriction coverage mapping |
|
||||
| `TESTS_DIR/test-data.md` | Test data management (seed data, mocks, isolation) |
|
||||
| `TESTS_DIR/blackbox-tests.md` | Blackbox functional scenarios (positive + negative) |
|
||||
| `TESTS_DIR/performance-tests.md` | Performance test scenarios |
|
||||
| `TESTS_DIR/resilience-tests.md` | Resilience test scenarios |
|
||||
| `TESTS_DIR/security-tests.md` | Security test scenarios |
|
||||
| `TESTS_DIR/resource-limit-tests.md` | Resource limit test scenarios |
|
||||
| `TESTS_DIR/traceability-matrix.md` | AC/restriction coverage mapping |
|
||||
| `_docs/00_problem/problem.md` | Problem context |
|
||||
| `_docs/00_problem/restrictions.md` | Constraints for test design |
|
||||
| `_docs/00_problem/acceptance_criteria.md` | Acceptance criteria being verified |
|
||||
@@ -103,7 +106,7 @@ Announce the detected mode and resolved paths to the user before proceeding.
|
||||
1. The provided component file exists and is non-empty — **STOP if missing**
|
||||
|
||||
**Tests-only mode:**
|
||||
1. `TESTS_DIR/functional_tests.md` exists and is non-empty — **STOP if missing**
|
||||
1. `TESTS_DIR/blackbox-tests.md` exists and is non-empty — **STOP if missing**
|
||||
2. `TESTS_DIR/environment.md` exists — **STOP if missing**
|
||||
3. Create TASKS_DIR if it does not exist
|
||||
4. If TASKS_DIR already contains task files, ask user: **resume from last checkpoint or start fresh?**
|
||||
@@ -130,7 +133,7 @@ TASKS_DIR/
|
||||
| Step 1 | Bootstrap structure plan complete + Jira ticket created + file renamed | `[JIRA-ID]_initial_structure.md` |
|
||||
| Step 1t | Test infrastructure bootstrap complete + Jira ticket created + file renamed | `[JIRA-ID]_test_infrastructure.md` |
|
||||
| Step 2 | Each component task decomposed + Jira ticket created + file renamed | `[JIRA-ID]_[short_name].md` |
|
||||
| Step 3 | Each integration test task decomposed + Jira ticket created + file renamed | `[JIRA-ID]_[short_name].md` |
|
||||
| Step 3 | Each blackbox test task decomposed + Jira ticket created + file renamed | `[JIRA-ID]_[short_name].md` |
|
||||
| Step 4 | Cross-task verification complete | `_dependencies_table.md` |
|
||||
|
||||
### Resumability
|
||||
@@ -153,7 +156,7 @@ At the start of execution, create a TodoWrite with all applicable steps. Update
|
||||
**Goal**: Produce `01_test_infrastructure.md` — the first task describing the test project scaffold
|
||||
**Constraints**: This is a plan document, not code. The `/implement` skill executes it.
|
||||
|
||||
1. Read `TESTS_DIR/environment.md` and `TESTS_DIR/test_data.md`
|
||||
1. Read `TESTS_DIR/environment.md` and `TESTS_DIR/test-data.md`
|
||||
2. Read problem.md, restrictions.md, acceptance_criteria.md for domain context
|
||||
3. Document the test infrastructure plan using `templates/test-infrastructure-task.md`
|
||||
|
||||
@@ -162,20 +165,20 @@ The test infrastructure bootstrap must include:
|
||||
- Mock/stub service definitions for each external dependency
|
||||
- `docker-compose.test.yml` structure from environment.md
|
||||
- Test runner configuration (framework, plugins, fixtures)
|
||||
- Test data fixture setup from test_data.md seed data sets
|
||||
- Test data fixture setup from test-data.md seed data sets
|
||||
- Test reporting configuration (format, output path)
|
||||
- Data isolation strategy
|
||||
|
||||
**Self-verification**:
|
||||
- [ ] Every external dependency from environment.md has a mock service defined
|
||||
- [ ] Docker Compose structure covers all services from environment.md
|
||||
- [ ] Test data fixtures cover all seed data sets from test_data.md
|
||||
- [ ] Test data fixtures cover all seed data sets from test-data.md
|
||||
- [ ] Test runner configuration matches the consumer app tech stack from environment.md
|
||||
- [ ] Data isolation strategy is defined
|
||||
|
||||
**Save action**: Write `01_test_infrastructure.md` (temporary numeric name)
|
||||
|
||||
**Jira action**: Create a Jira ticket for this task under the "Integration Tests" epic. Write the Jira ticket ID and Epic ID back into the task header.
|
||||
**Jira action**: Create a Jira ticket for this task under the "Blackbox Tests" epic. Write the Jira ticket ID and Epic ID back into the task header.
|
||||
|
||||
**Rename action**: Rename the file from `01_test_infrastructure.md` to `[JIRA-ID]_test_infrastructure.md`. Update the **Task** field inside the file to match the new filename.
|
||||
|
||||
@@ -199,27 +202,27 @@ The bootstrap structure plan must include:
|
||||
- Shared models, interfaces, and DTOs
|
||||
- Dockerfile per component (multi-stage, non-root, health checks, pinned base images)
|
||||
- `docker-compose.yml` for local development (all components + database + dependencies)
|
||||
- `docker-compose.test.yml` for integration test environment (black-box test runner)
|
||||
- `docker-compose.test.yml` for blackbox test environment (blackbox test runner)
|
||||
- `.dockerignore`
|
||||
- CI/CD pipeline file (`.github/workflows/ci.yml` or `azure-pipelines.yml`) with stages from `deployment/ci_cd_pipeline.md`
|
||||
- Database migration setup and initial seed data scripts
|
||||
- Observability configuration: structured logging setup, health check endpoints (`/health/live`, `/health/ready`), metrics endpoint (`/metrics`)
|
||||
- Environment variable documentation (`.env.example`)
|
||||
- Test structure with unit and integration test locations
|
||||
- Test structure with unit and blackbox test locations
|
||||
|
||||
**Self-verification**:
|
||||
- [ ] All components have corresponding folders in the layout
|
||||
- [ ] All inter-component interfaces have DTOs defined
|
||||
- [ ] Dockerfile defined for each component
|
||||
- [ ] `docker-compose.yml` covers all components and dependencies
|
||||
- [ ] `docker-compose.test.yml` enables black-box integration testing
|
||||
- [ ] `docker-compose.test.yml` enables blackbox testing
|
||||
- [ ] CI/CD pipeline file defined with lint, test, security, build, deploy stages
|
||||
- [ ] Database migration setup included
|
||||
- [ ] Health check endpoints specified for each service
|
||||
- [ ] Structured logging configuration included
|
||||
- [ ] `.env.example` with all required environment variables
|
||||
- [ ] Environment strategy covers dev, staging, production
|
||||
- [ ] Test structure includes unit and integration test locations
|
||||
- [ ] Test structure includes unit and blackbox test locations
|
||||
|
||||
**Save action**: Write `01_initial_structure.md` (temporary numeric name)
|
||||
|
||||
@@ -265,33 +268,33 @@ For each component (or the single provided component):
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Integration Test Task Decomposition (default and tests-only modes)
|
||||
### Step 3: Blackbox Test Task Decomposition (default and tests-only modes)
|
||||
|
||||
**Role**: Professional Quality Assurance Engineer
|
||||
**Goal**: Decompose integration test specs into atomic, implementable task specs
|
||||
**Goal**: Decompose blackbox test specs into atomic, implementable task specs
|
||||
**Constraints**: Behavioral specs only — describe what, not how. No test code.
|
||||
|
||||
**Numbering**:
|
||||
- In default mode: continue sequential numbering from where Step 2 left off.
|
||||
- In tests-only mode: start from 02 (01 is the test infrastructure bootstrap from Step 1t).
|
||||
|
||||
1. Read all test specs from `DOCUMENT_DIR/integration_tests/` (functional_tests.md, non_functional_tests.md)
|
||||
1. Read all test specs from `DOCUMENT_DIR/tests/` (`blackbox-tests.md`, `performance-tests.md`, `resilience-tests.md`, `security-tests.md`, `resource-limit-tests.md`)
|
||||
2. Group related test scenarios into atomic tasks (e.g., one task per test category or per component under test)
|
||||
3. Each task should reference the specific test scenarios it implements and the environment/test_data specs
|
||||
3. Each task should reference the specific test scenarios it implements and the environment/test-data specs
|
||||
4. Dependencies:
|
||||
- In default mode: integration test tasks depend on the component implementation tasks they exercise
|
||||
- In tests-only mode: integration test tasks depend on the test infrastructure bootstrap task (Step 1t)
|
||||
- In default mode: blackbox test tasks depend on the component implementation tasks they exercise
|
||||
- In tests-only mode: blackbox test tasks depend on the test infrastructure bootstrap task (Step 1t)
|
||||
5. Write each task spec using `templates/task.md`
|
||||
6. Estimate complexity per task (1, 2, 3, 5 points); no task should exceed 5 points — split if it does
|
||||
7. Note task dependencies (referencing Jira IDs of already-created dependency tasks)
|
||||
8. **Immediately after writing each task file**: create a Jira ticket under the "Integration Tests" epic, write the Jira ticket ID and Epic ID back into the task header, then rename the file from `[##]_[short_name].md` to `[JIRA-ID]_[short_name].md`.
|
||||
8. **Immediately after writing each task file**: create a Jira ticket under the "Blackbox Tests" epic, write the Jira ticket ID and Epic ID back into the task header, then rename the file from `[##]_[short_name].md` to `[JIRA-ID]_[short_name].md`.
|
||||
|
||||
**Self-verification**:
|
||||
- [ ] Every functional test scenario from `integration_tests/functional_tests.md` is covered by a task
|
||||
- [ ] Every non-functional test scenario from `integration_tests/non_functional_tests.md` is covered by a task
|
||||
- [ ] Every scenario from `tests/blackbox-tests.md` is covered by a task
|
||||
- [ ] Every scenario from `tests/performance-tests.md`, `tests/resilience-tests.md`, `tests/security-tests.md`, and `tests/resource-limit-tests.md` is covered by a task
|
||||
- [ ] No task exceeds 5 complexity points
|
||||
- [ ] Dependencies correctly reference the dependency tasks (component tasks in default mode, test infrastructure in tests-only mode)
|
||||
- [ ] Every task has a Jira ticket linked to the "Integration Tests" epic
|
||||
- [ ] Every task has a Jira ticket linked to the "Blackbox Tests" epic
|
||||
|
||||
**Save action**: Write each `[##]_[short_name].md` (temporary numeric name), create Jira ticket inline, then rename to `[JIRA-ID]_[short_name].md`.
|
||||
|
||||
@@ -306,7 +309,7 @@ For each component (or the single provided component):
|
||||
1. Verify task dependencies across all tasks are consistent
|
||||
2. Check no gaps:
|
||||
- In default mode: every interface in architecture.md has tasks covering it
|
||||
- In tests-only mode: every test scenario in `traceability_matrix.md` is covered by a task
|
||||
- In tests-only mode: every test scenario in `traceability-matrix.md` is covered by a task
|
||||
3. Check no overlaps: tasks don't duplicate work
|
||||
4. Check no circular dependencies in the task graph
|
||||
5. Produce `_dependencies_table.md` using `templates/dependencies-table.md`
|
||||
@@ -320,7 +323,7 @@ Default mode:
|
||||
- [ ] `_dependencies_table.md` contains every task with correct dependencies
|
||||
|
||||
Tests-only mode:
|
||||
- [ ] Every test scenario from traceability_matrix.md "Covered" entries has a corresponding task
|
||||
- [ ] Every test scenario from traceability-matrix.md "Covered" entries has a corresponding task
|
||||
- [ ] No circular dependencies in the task graph
|
||||
- [ ] Test task dependencies reference the test infrastructure bootstrap
|
||||
- [ ] `_dependencies_table.md` contains every task with correct dependencies
|
||||
@@ -366,14 +369,14 @@ Tests-only mode:
|
||||
│ 1. Bootstrap Structure → [JIRA-ID]_initial_structure.md │
|
||||
│ [BLOCKING: user confirms structure] │
|
||||
│ 2. Component Tasks → [JIRA-ID]_[short_name].md each │
|
||||
│ 3. Integration Tests → [JIRA-ID]_[short_name].md each │
|
||||
│ 3. Blackbox Tests → [JIRA-ID]_[short_name].md each │
|
||||
│ 4. Cross-Verification → _dependencies_table.md │
|
||||
│ [BLOCKING: user confirms dependencies] │
|
||||
│ │
|
||||
│ TESTS-ONLY MODE: │
|
||||
│ 1t. Test Infrastructure → [JIRA-ID]_test_infrastructure.md │
|
||||
│ [BLOCKING: user confirms test scaffold] │
|
||||
│ 3. Integration Tests → [JIRA-ID]_[short_name].md each │
|
||||
│ 3. Blackbox Tests → [JIRA-ID]_[short_name].md each │
|
||||
│ 4. Cross-Verification → _dependencies_table.md │
|
||||
│ [BLOCKING: user confirms dependencies] │
|
||||
│ │
|
||||
|
||||
@@ -49,7 +49,7 @@ project-root/
|
||||
| Build | Compile/bundle the application | Every push |
|
||||
| Lint / Static Analysis | Code quality and style checks | Every push |
|
||||
| Unit Tests | Run unit test suite | Every push |
|
||||
| Integration Tests | Run integration test suite | Every push |
|
||||
| Blackbox Tests | Run blackbox test suite | Every push |
|
||||
| Security Scan | SAST / dependency check | Every push |
|
||||
| Deploy to Staging | Deploy to staging environment | Merge to staging branch |
|
||||
|
||||
|
||||
@@ -64,7 +64,7 @@ Then [expected result]
|
||||
|--------|-------------|-----------------|
|
||||
| AC-1 | [test subject] | [expected result] |
|
||||
|
||||
## Integration Tests
|
||||
## Blackbox Tests
|
||||
|
||||
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||
|--------|------------------------|-------------|-------------------|----------------|
|
||||
|
||||
@@ -9,10 +9,10 @@ Use this template for the test infrastructure bootstrap (Step 1t in tests-only m
|
||||
|
||||
**Task**: [JIRA-ID]_test_infrastructure
|
||||
**Name**: Test Infrastructure
|
||||
**Description**: Scaffold the E2E test project — test runner, mock services, Docker test environment, test data fixtures, reporting
|
||||
**Description**: Scaffold the Blackbox test project — test runner, mock services, Docker test environment, test data fixtures, reporting
|
||||
**Complexity**: [3|5] points
|
||||
**Dependencies**: None
|
||||
**Component**: Integration Tests
|
||||
**Component**: Blackbox Tests
|
||||
**Jira**: [TASK-ID]
|
||||
**Epic**: [EPIC-ID]
|
||||
|
||||
@@ -124,6 +124,6 @@ Then a report file exists at the configured output path with correct columns
|
||||
|
||||
- This is a PLAN document, not code. The `/implement` skill executes it.
|
||||
- Focus on test infrastructure decisions, not individual test implementations.
|
||||
- Reference environment.md and test_data.md from the test specs — don't repeat everything.
|
||||
- Reference environment.md and test-data.md from the test specs — don't repeat everything.
|
||||
- Mock services must be deterministic: same input always produces same output.
|
||||
- The Docker environment must be self-contained: `docker compose up` sufficient.
|
||||
|
||||
Reference in New Issue
Block a user