mirror of
https://github.com/azaion/annotations.git
synced 2026-06-21 10:21:07 +00:00
Oleksandr Bezdieniezhnykh
221 lines
11 KiB
Markdown
221 lines
11 KiB
Markdown
## Research Engine — Analysis Phase (Steps 4–8)
|
||
|
||
### Step 4: Build Comparison/Analysis Framework
|
||
|
||
Based on the question type, select fixed analysis dimensions. **For dimension lists** (General, Concept Comparison, Decision Support): Read `references/comparison-frameworks.md`
|
||
|
||
**Save action**:
|
||
Write to `03_comparison_framework.md`:
|
||
```markdown
|
||
# Comparison Framework
|
||
|
||
## Selected Framework Type
|
||
[Concept Comparison / Decision Support / ...]
|
||
|
||
## Selected Dimensions
|
||
1. [Dimension 1]
|
||
2. [Dimension 2]
|
||
...
|
||
|
||
## Initial Population
|
||
| Dimension | X | Y | Factual Basis |
|
||
|-----------|---|---|---------------|
|
||
| [Dimension 1] | [description] | [description] | Fact #1, #3 |
|
||
| ... | | | |
|
||
```
|
||
|
||
**Required exact-fit dimensions for component/tool decisions**:
|
||
When the output selects or recommends a component, tool, library, service, architecture pattern, or algorithm, the framework MUST include these dimensions unless explicitly not applicable:
|
||
- Option family (`Simple baseline`, `Established production`, `Open-source`, `Commercial/vendor`, `Current SOTA`, `Adjacent-domain`, `No-build/defer`, `Known bad`)
|
||
- Required inputs/outputs and ownership boundaries
|
||
- Operating context and lifecycle fit
|
||
- Non-functional envelope fit
|
||
- Implementation assumptions and hard disqualifiers
|
||
- Evidence quality and source tier
|
||
- Selection status (`Selected`, `Rejected`, `Experimental only`, `Needs user decision`)
|
||
|
||
For each component area, include multiple candidates in the initial population. Do not present only the preferred option unless the investigation found no realistic alternatives; if so, state the searches that proved the narrow landscape.
|
||
|
||
---
|
||
|
||
### Step 5: Reference Point Baseline Alignment
|
||
|
||
Ensure all compared parties have clear, consistent definitions:
|
||
|
||
**Checklist**:
|
||
- [ ] Is the reference point's definition stable/widely accepted?
|
||
- [ ] Does it need verification, or can domain common knowledge be used?
|
||
- [ ] Does the reader's understanding of the reference point match mine?
|
||
- [ ] Are there ambiguities that need to be clarified first?
|
||
|
||
---
|
||
|
||
### Step 6: Fact-to-Conclusion Reasoning Chain
|
||
|
||
Explicitly write out the "fact → comparison → conclusion" reasoning process:
|
||
|
||
```markdown
|
||
## Reasoning Process
|
||
|
||
### Regarding [Dimension Name]
|
||
|
||
1. **Fact confirmation**: According to [source], X's mechanism is...
|
||
2. **Compare with reference**: While Y's mechanism is...
|
||
3. **Conclusion**: Therefore, the difference between X and Y on this dimension is...
|
||
```
|
||
|
||
**Key discipline**:
|
||
- Conclusions come from mechanism comparison, not "gut feelings"
|
||
- Every conclusion must be traceable to specific facts
|
||
- Uncertain conclusions must be annotated
|
||
|
||
**Save action**:
|
||
Write to `04_reasoning_chain.md`:
|
||
```markdown
|
||
# Reasoning Chain
|
||
|
||
## Dimension 1: [Dimension Name]
|
||
|
||
### Fact Confirmation
|
||
According to [Fact #X], X's mechanism is...
|
||
|
||
### Reference Comparison
|
||
While Y's mechanism is... (Source: [Fact #Y])
|
||
|
||
### Conclusion
|
||
Therefore, the difference between X and Y on this dimension is...
|
||
|
||
### Confidence
|
||
✅/⚠️/❓ + rationale
|
||
|
||
---
|
||
## Dimension 2: [Dimension Name]
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
### Step 7: Use-Case Validation (Sanity Check)
|
||
|
||
Validate conclusions against a typical scenario:
|
||
|
||
**Validation questions**:
|
||
- Based on my conclusions, how should this scenario be handled?
|
||
- Is that actually the case?
|
||
- Are there counterexamples that need to be addressed?
|
||
|
||
**Review checklist**:
|
||
- [ ] Are draft conclusions consistent with Step 3 fact cards?
|
||
- [ ] Are there any important dimensions missed?
|
||
- [ ] Is there any over-extrapolation?
|
||
- [ ] Are conclusions actionable/verifiable?
|
||
- [ ] Does every selected component/tool/pattern match the Project Constraint Matrix?
|
||
- [ ] Are mismatches marked as disqualifiers instead of hidden as generic "limitations"?
|
||
|
||
**Save action**:
|
||
Write to `05_validation_log.md`:
|
||
```markdown
|
||
# Validation Log
|
||
|
||
## Validation Scenario
|
||
[Scenario description]
|
||
|
||
## Expected Based on Conclusions
|
||
If using X: [expected behavior]
|
||
If using Y: [expected behavior]
|
||
|
||
## Actual Validation Results
|
||
[actual situation]
|
||
|
||
## Counterexamples
|
||
[yes/no, describe if yes]
|
||
|
||
## Review Checklist
|
||
- [x] Draft conclusions consistent with fact cards
|
||
- [x] No important dimensions missed
|
||
- [x] No over-extrapolation
|
||
- [ ] Issue found: [if any]
|
||
|
||
## Conclusions Requiring Revision
|
||
[if any]
|
||
```
|
||
|
||
---
|
||
|
||
### Step 7.5: Component Applicability Gate (BLOCKING)
|
||
|
||
**Applicability**: this gate applies only when the run is classified as **Technical-component selection** in the SKILL's Research Output Class section. For non-technical research (concept comparison, market/policy investigation, root-cause analysis without tooling, knowledge organization), skip this entire step and proceed to Step 8 — there are no components to gate. State the skip once in `05_validation_log.md`: `Step 7.5 (Component Applicability Gate): not applicable — Non-technical investigation`. For mixed runs (some component areas technical, some not), apply this gate only to the technical component areas; the non-technical ones do not produce 7.5 rows.
|
||
|
||
Before finalizing the solution draft, build an exact-fit matrix for every component/tool/library/service/pattern/algorithm that is selected, recommended, rejected, or treated as a fallback. Free-form prose in a "Project Constraints Checked" column is **not sufficient** — mismatches hide inside rationale text. The matrix must be structured per restriction and per acceptance criterion.
|
||
|
||
#### 7.5.1 Top-level Component Fit Matrix
|
||
|
||
```markdown
|
||
# Component Fit Matrix
|
||
|
||
| Component Area | Candidate | Pinned Mode/Config | Option Family | Intended Role | API Capability Evidence | Mismatches / Disqualifiers | Status | Decision Rationale |
|
||
|----------------|-----------|--------------------|---------------|---------------|-------------------------|----------------------------|--------|--------------------|
|
||
| [area] | [name] | [exact mode/config the project will use, copied verbatim from the MVE block in Step 2] | [family] | [role] | MVE: [link to MVE block in `02_fact_cards.md` / `02_fact_cards/` or `02_mve_evidence.md`]; docs: [Source #] | [none / list] | Selected / Rejected / Experimental only / Needs user decision | [why] |
|
||
```
|
||
|
||
The new **Pinned Mode/Config** column is mandatory. A row without a pinned mode is incomplete. The new **API Capability Evidence** column links to the Minimum Viable Example saved during Step 2's API Capability Verification — without an MVE link the candidate cannot be `Selected`.
|
||
|
||
#### 7.5.2 Restrictions × Candidate-Modes Sub-Matrix (MANDATORY)
|
||
|
||
For each lead candidate row in the top-level matrix, append a structured cross-check that walks every numbered line of `restrictions.md` and `acceptance_criteria.md` against the candidate's **pinned mode/config**.
|
||
|
||
```markdown
|
||
## Sub-Matrix — <Candidate Name> in <Pinned Mode>
|
||
|
||
| Restriction / AC | Candidate-mode behavior | Result | Evidence |
|
||
|------------------|-------------------------|--------|----------|
|
||
| R1: <verbatim line from restrictions.md> | <how the pinned mode behaves under this restriction> | ✅ Pass / ❌ Fail / ❓ Verify / N/A | [Fact # / Source # / MVE link] |
|
||
| R2: ... | ... | ... | ... |
|
||
| ... | ... | ... | ... |
|
||
| AC-1.1: <verbatim line from acceptance_criteria.md> | <how the pinned mode satisfies (or contradicts) this AC's measurable target> | ✅ / ❌ / ❓ / N/A | [Fact # / Source # / MVE link] |
|
||
| AC-1.2: ... | ... | ... | ... |
|
||
| ... | ... | ... | ... |
|
||
```
|
||
|
||
Cell semantics:
|
||
- ✅ **Pass** — the candidate's pinned mode satisfies this line, with cited official-doc or MVE evidence.
|
||
- ❌ **Fail** — the candidate's pinned mode contradicts this line, with cited evidence. Even one ❌ disqualifies the candidate from `Selected` status.
|
||
- ❓ **Verify** — no evidence yet either way; further investigation required (loops back to Step 2 / Step 3.5). A row left ❓ at the end of analysis blocks the candidate.
|
||
- **N/A** — the line is irrelevant to this component area (state why in one phrase).
|
||
|
||
A candidate row may not be marked `Selected` while any cell is ❌ or ❓.
|
||
|
||
#### 7.5.3 Decision Rules
|
||
|
||
- `Selected` is allowed only when (a) the top-level row has an MVE link, (b) the sub-matrix has zero ❌, (c) the sub-matrix has zero ❓, and (d) the candidate's documented implementation assumptions match the project's explicit constraints and acceptance criteria.
|
||
- `Experimental only` is required when a candidate might work but lacks proof for the exact operating context (e.g., MVE exists for a similar configuration but not the exact one).
|
||
- `Rejected` is required when documented assumptions conflict with project constraints (any sub-matrix row is ❌ with cited evidence).
|
||
- `Needs user decision` is required when a mismatch changes scope, cost, safety, product behavior, or acceptance criteria — and the user has not yet been consulted.
|
||
- Each component area must include at least one selected or fallback-safe option, plus the most credible rejected/experimental alternatives discovered during web research.
|
||
- A component area with only one candidate is incomplete unless `00_question_decomposition.md` documents the broader searches and why they yielded no realistic alternatives.
|
||
- A candidate may not appear as the lead solution in Step 8 unless this gate marks it `Selected`.
|
||
- "Validation gate required" footnotes are not equivalent to `Selected`. If the validation gate concerns API capability (does the mode produce the required output?), that is a Step-2 / Step-7.5 question and must be resolved here, not deferred to runtime. Only validation gates concerning *runtime quality* (e.g., "does this VO converge on this terrain class?") may be carried forward as `Selected with runtime gate`.
|
||
|
||
**Save action**: Write `06_component_fit_matrix.md` (or, when split, the equivalent files under `06_component_fit_matrix/` — typically `00_summary.md` for the top-level matrix plus per-component sub-matrix files) containing both 7.5.1 (top-level) and 7.5.2 (per-candidate sub-matrices).
|
||
|
||
**BLOCKING**: If any lead candidate has ❌, ❓, `Experimental only`, `Rejected`, or `Needs user decision` status, do not silently proceed. Ask the user or choose a different selected candidate.
|
||
|
||
---
|
||
|
||
### Step 8: Deliverable Formatting
|
||
|
||
Make the output **readable, traceable, and actionable**.
|
||
|
||
**Save action**:
|
||
Integrate all intermediate artifacts. Write to `OUTPUT_DIR/solution_draft##.md` using the appropriate output template based on active mode:
|
||
- Mode A: `templates/solution_draft_mode_a.md`
|
||
- Mode B: `templates/solution_draft_mode_b.md`
|
||
|
||
Sources to integrate:
|
||
- Extract background from `00_question_decomposition.md`
|
||
- Reference key facts from `02_fact_cards.md` (or files under `02_fact_cards/` if split)
|
||
- Organize conclusions from `04_reasoning_chain.md`
|
||
- Generate references from `01_source_registry.md` (or files under `01_source_registry/` if split)
|
||
- Supplement with use cases from `05_validation_log.md`
|
||
- For Mode A: include AC assessment from `00_ac_assessment.md`
|