mirror of
https://github.com/azaion/detections.git
synced 2026-04-23 00:26:31 +00:00
Refactor testing framework to replace integration tests with blackbox tests across various skills and documentation. Update related workflows, templates, and task specifications to align with the new blackbox testing approach. Remove obsolete integration test files and enhance clarity in task management and reporting structures.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -6,12 +6,15 @@ All artifacts are written directly under DOCUMENT_DIR:
|
||||
|
||||
```
|
||||
DOCUMENT_DIR/
|
||||
├── integration_tests/
|
||||
│ ├── environment.md
|
||||
│ ├── test_data.md
|
||||
│ ├── functional_tests.md
|
||||
│ ├── non_functional_tests.md
|
||||
│ └── traceability_matrix.md
|
||||
├── tests/
|
||||
│ ├── test-environment.md
|
||||
│ ├── test-data.md
|
||||
│ ├── blackbox-tests.md
|
||||
│ ├── performance-tests.md
|
||||
│ ├── resilience-tests.md
|
||||
│ ├── security-tests.md
|
||||
│ ├── resource-limit-tests.md
|
||||
│ └── traceability-matrix.md
|
||||
├── architecture.md
|
||||
├── system-flows.md
|
||||
├── data_model.md
|
||||
@@ -47,11 +50,14 @@ DOCUMENT_DIR/
|
||||
|
||||
| Step | Save immediately after | Filename |
|
||||
|------|------------------------|----------|
|
||||
| Step 1 | Integration test environment spec | `integration_tests/environment.md` |
|
||||
| Step 1 | Integration test data spec | `integration_tests/test_data.md` |
|
||||
| Step 1 | Integration functional tests | `integration_tests/functional_tests.md` |
|
||||
| Step 1 | Integration non-functional tests | `integration_tests/non_functional_tests.md` |
|
||||
| Step 1 | Integration traceability matrix | `integration_tests/traceability_matrix.md` |
|
||||
| Step 1 | Blackbox test environment spec | `tests/test-environment.md` |
|
||||
| Step 1 | Blackbox test data spec | `tests/test-data.md` |
|
||||
| Step 1 | Blackbox tests | `tests/blackbox-tests.md` |
|
||||
| Step 1 | Blackbox performance tests | `tests/performance-tests.md` |
|
||||
| Step 1 | Blackbox resilience tests | `tests/resilience-tests.md` |
|
||||
| Step 1 | Blackbox security tests | `tests/security-tests.md` |
|
||||
| Step 1 | Blackbox resource limit tests | `tests/resource-limit-tests.md` |
|
||||
| Step 1 | Blackbox traceability matrix | `tests/traceability-matrix.md` |
|
||||
| Step 2 | Architecture analysis complete | `architecture.md` |
|
||||
| Step 2 | System flows documented | `system-flows.md` |
|
||||
| Step 2 | Data model documented | `data_model.md` |
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
### Phase 2a: Architecture & Flows
|
||||
|
||||
1. Read all input files thoroughly
|
||||
2. Incorporate findings, questions, and insights discovered during Step 1 (integration tests)
|
||||
2. Incorporate findings, questions, and insights discovered during Step 1 (blackbox tests)
|
||||
3. Research unknown or questionable topics via internet; ask user about ambiguities
|
||||
4. Document architecture using `templates/architecture.md` as structure
|
||||
5. Document system flows using `templates/system-flows.md` as structure
|
||||
@@ -17,7 +17,7 @@
|
||||
- [ ] System flows cover all main user/system interactions
|
||||
- [ ] No contradictions with problem.md or restrictions.md
|
||||
- [ ] Technology choices are justified
|
||||
- [ ] Integration test findings are reflected in architecture decisions
|
||||
- [ ] Blackbox test findings are reflected in architecture decisions
|
||||
|
||||
**Save action**: Write `architecture.md` and `system-flows.md`
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
**Constraints**: No code; only names, interfaces, inputs/outputs. Follow SRP strictly.
|
||||
|
||||
1. Identify components from the architecture; think about separation, reusability, and communication patterns
|
||||
2. Use integration test scenarios from Step 1 to validate component boundaries
|
||||
2. Use blackbox test scenarios from Step 1 to validate component boundaries
|
||||
3. If additional components are needed (data preparation, shared helpers), create them
|
||||
4. For each component, write a spec using `templates/component-spec.md` as structure
|
||||
5. Generate diagrams:
|
||||
@@ -19,7 +19,7 @@
|
||||
- [ ] All inter-component interfaces are defined (who calls whom, with what)
|
||||
- [ ] Component dependency graph has no circular dependencies
|
||||
- [ ] All components from architecture.md are accounted for
|
||||
- [ ] Every integration test scenario can be traced through component interactions
|
||||
- [ ] Every blackbox test scenario can be traced through component interactions
|
||||
|
||||
**Save action**: Write:
|
||||
- each component `components/[##]_[name]/description.md`
|
||||
|
||||
@@ -35,7 +35,7 @@ Do NOT create minimal epics with just a summary and short description. The Jira
|
||||
|
||||
**Self-verification**:
|
||||
- [ ] "Bootstrap & Initial Structure" epic exists and is first in order
|
||||
- [ ] "Integration Tests" epic exists
|
||||
- [ ] "Blackbox Tests" epic exists
|
||||
- [ ] Every component maps to exactly one epic
|
||||
- [ ] Dependency order is respected (no epic depends on a later one)
|
||||
- [ ] Acceptance criteria are measurable
|
||||
@@ -43,6 +43,6 @@ Do NOT create minimal epics with just a summary and short description. The Jira
|
||||
- [ ] Every epic description includes architecture diagram, interface spec, data flow, risks, and NFRs
|
||||
- [ ] Epic descriptions are self-contained — readable without opening other files
|
||||
|
||||
7. **Create "Integration Tests" epic** — this epic will parent the integration test tasks created by the `/decompose` skill. It covers implementing the test scenarios defined in `integration_tests/`.
|
||||
7. **Create "Blackbox Tests" epic** — this epic will parent the blackbox test tasks created by the `/decompose` skill. It covers implementing the test scenarios defined in `tests/`.
|
||||
|
||||
**Save action**: Epics created in Jira via MCP. Also saved locally in `epics.md` with Jira IDs.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
Before writing the final report, verify ALL of the following:
|
||||
|
||||
### Integration Tests
|
||||
- [ ] Every acceptance criterion is covered in traceability_matrix.md
|
||||
### Blackbox Tests
|
||||
- [ ] Every acceptance criterion is covered in traceability-matrix.md
|
||||
- [ ] Every restriction is verified by at least one test
|
||||
- [ ] Positive and negative scenarios are balanced
|
||||
- [ ] Docker environment is self-contained
|
||||
@@ -14,7 +14,7 @@ Before writing the final report, verify ALL of the following:
|
||||
- [ ] Covers all capabilities from solution.md
|
||||
- [ ] Technology choices are justified
|
||||
- [ ] Deployment model is defined
|
||||
- [ ] Integration test findings are reflected in architecture decisions
|
||||
- [ ] Blackbox test findings are reflected in architecture decisions
|
||||
|
||||
### Data Model
|
||||
- [ ] Every entity from architecture.md is defined
|
||||
@@ -35,7 +35,7 @@ Before writing the final report, verify ALL of the following:
|
||||
- [ ] No circular dependencies
|
||||
- [ ] All inter-component interfaces are defined and consistent
|
||||
- [ ] No orphan components (unused by any flow)
|
||||
- [ ] Every integration test scenario can be traced through component interactions
|
||||
- [ ] Every blackbox test scenario can be traced through component interactions
|
||||
|
||||
### Risks
|
||||
- [ ] All High/Critical risks have mitigations
|
||||
@@ -49,7 +49,7 @@ Before writing the final report, verify ALL of the following:
|
||||
|
||||
### Epics
|
||||
- [ ] "Bootstrap & Initial Structure" epic exists
|
||||
- [ ] "Integration Tests" epic exists
|
||||
- [ ] "Blackbox Tests" epic exists
|
||||
- [ ] Every component maps to an epic
|
||||
- [ ] Dependency order is correct
|
||||
- [ ] Acceptance criteria are measurable
|
||||
|
||||
Reference in New Issue
Block a user