diff --git a/.planning/phases/02-acceptance-criteria-test-taxonomy-observability-spine/02-04-SUMMARY.md b/.planning/phases/02-acceptance-criteria-test-taxonomy-observability-spine/02-04-SUMMARY.md new file mode 100644 index 0000000..a8d7851 --- /dev/null +++ b/.planning/phases/02-acceptance-criteria-test-taxonomy-observability-spine/02-04-SUMMARY.md @@ -0,0 +1,180 @@ +--- +phase: 02-acceptance-criteria-test-taxonomy-observability-spine +plan: "04" +subsystem: traceability +tags: [ac-traceability, pytest-markers, bidirectional-coverage] +dependency_graph: + requires: [02-01, 02-02, 02-03] + provides: [scripts/gen_ac_traceability.py, .planning/AC-TRACEABILITY.md, @pytest.mark.ac decorations] + affects: [02-05-CI-gate, AC-06-satisfaction] +tech_stack: + added: [] + patterns: [argparse-cli, subprocess-pytest-collect, ac-marker-plugin, class-level-pytestmark] +key_files: + created: + - scripts/gen_ac_traceability.py + - .planning/AC-TRACEABILITY.md + modified: + - tests/test_acceptance.py + - tests/test_accuracy.py + - tests/test_processor_pipe.py + - tests/test_gps_input_encoding.py + - tests/test_sitl_integration.py + - tests/test_mavlink.py + - tests/test_schemas.py +decisions: + - "AC-TRACEABILITY.md committed to .planning/ with git add -f (path is gitignored but other .planning files are tracked)" + - "test_gps_input_encoding.py marked at module/pytestmark level (all 13 tests -> AC-4.3)" + - "TestGPSPoint and TestWaypoint marked at class level -> AC-6.3 (propagates to methods)" + - "Orphan ACs get pending-phase-3 annotation in Plan 02-05 (documented decision carried forward)" +metrics: + duration: "~15 minutes" + completed: "2026-05-11" + tasks_completed: 4 + files_created: 2 + files_modified: 7 +--- + +# Phase 02 Plan 04: AC Traceability Script + Test Decorations Summary + +One-liner: `scripts/gen_ac_traceability.py` + 38 `@pytest.mark.ac(...)` decorations cover 14 of 35 non-deferred ACs in the initial matrix, with `--check` deterministic for CI gating. + +## Tasks Completed + +| Task | Status | Commit | +|------|--------|--------| +| 1: Implement gen_ac_traceability.py | Done | 4bf6f67 | +| 2: Decorate 7 test files | Done | 6a1cd51 | +| 3: Regenerate and commit AC-TRACEABILITY.md | Done | 419e9c5 | +| 4: Regression gate | Done | (verification only) | + +## Task 1: Script Implementation + +The script matches RESEARCH.md §2.3 exactly with one minor adaptation: the `--strict` alias is added as `--check, --strict` on the same argument (both names accepted by argparse via nargs). + +**Functions implemented:** +- `collect_acs_from_doc()` — regex parses `- **AC-X.Y**` headers + `deferred-hardware` token within each block +- `collect_acs_from_tests()` — subprocess `pytest --collect-only -q --ac-dump=` reads the JSON from conftest Plan 02-02 hook +- `render_md()` — produces header stats, `| AC ID | Test count | Tests | Status |` table, backward-orphan section +- `main()` — argparse + `--check` exit-1 on any non-deferred orphan or unknown AC ID + +**Deviations vs RESEARCH.md §2.3:** None material. The plan's verbatim source was used as written. + +## Task 2: Decoration Counts Per File + +| File | Decorator lines added | AC IDs tagged | Notes | +|------|-----------------------|---------------|-------| +| test_acceptance.py | 8 | AC-1.1, AC-2.1a, AC-3.3, AC-3.4, AC-4.1, AC-4.4, AC-1.4 | per-function decorators | +| test_accuracy.py | 11 | AC-1.1, AC-1.2, AC-1.3, AC-2.1a, AC-2.2, AC-4.1 | per-function decorators | +| test_processor_pipe.py | 2 | AC-4.4, AC-1.4 | per-function decorators | +| test_gps_input_encoding.py | 1 | AC-4.3 | module-level pytestmark (13 tests inherit) | +| test_sitl_integration.py | 9 | AC-4.3, AC-4.4, AC-NEW-2, AC-5.2, AC-3.4 | per-function decorators | +| test_mavlink.py | 5 | AC-4.3, AC-5.2, AC-3.4 | per-function decorators | +| test_schemas.py | 2 | AC-6.3 | class-level decorators (propagate to methods) | +| **Total** | **38** | **14 unique ACs** | | + +Total tests collected under `-m ac`: **45** (from 298 collected). + +## AC Coverage Table + +| AC ID | Covered? | Reason if orphan | +|-------|----------|-----------------| +| AC-1.1 | Yes | test_acceptance (2), test_accuracy (3) | +| AC-1.2 | Yes | test_accuracy (2) | +| AC-1.3 | Yes | test_accuracy (1) | +| AC-1.4 | Yes | test_acceptance (1), test_processor_pipe (1) | +| AC-2.1a | Yes | test_acceptance (1), test_accuracy (1) | +| AC-2.1b | No | No test explicitly checks cross-domain registration separately from VO | +| AC-2.2 | Yes | test_accuracy (1) | +| AC-3.1 | No | No outlier-tolerance test in Phase 2 test files | +| AC-3.2 | No | No sharp-turn test in Phase 2 test files | +| AC-3.3 | Yes | test_acceptance (1) — factor graph convergence | +| AC-3.4 | Yes | test_acceptance (1), test_mavlink (1), test_sitl (1) | +| AC-3.5 | No | No visual-blackout mode-switch test in Phase 2 | +| AC-4.1 | Yes | test_acceptance (1), test_accuracy (2) | +| AC-4.2 | No | No memory-bound test exists in Phase 2 | +| AC-4.3 | Yes | test_gps_input_encoding (12), test_mavlink (3), test_sitl (5) | +| AC-4.4 | Yes | test_acceptance (3), test_processor_pipe (1), test_sitl (1) | +| AC-4.5 | No | No refinement/correction round-trip test in Phase 2 | +| AC-5.1 | No | Phase 3 target — FC EKF init test not yet written | +| AC-5.2 | Yes | test_mavlink (1), test_sitl (1) | +| AC-5.3 | No | Phase 3 target — mid-flight reboot recovery | +| AC-6.1 | No | No GCS downsample rate test in Phase 2 | +| AC-6.2 | No | No operator reloc command test in Phase 2 | +| AC-6.3 | Yes | test_schemas TestGPSPoint (4), TestWaypoint (2) | +| AC-7.1 | No | Phase 3+ — AI camera localization accuracy | +| AC-7.2 | No | Phase 3+ — trig computation | +| AC-8.1 | No | Phase 3/4 — satellite imagery integration | +| AC-8.2 | No | Phase 3/4 — tile freshness | +| AC-8.3 | No | Phase 3/4 — pre-flight loading | +| AC-8.4 | No | deferred-stage3 per AC doc | +| AC-8.5 | No | Phase 4 — storage policy | +| AC-8.6 | No | Phase 4/VPR — conditional VPR | +| AC-NEW-1 | DEFERRED (hardware) | cold-boot bench requires Jetson | +| AC-NEW-2 | Yes | test_sitl (1) — spoofing-promotion via GPS_INPUT accepted | +| AC-NEW-3 | No (partial deferred) | integration-test scope not yet written; deferred-hardware for real NVMe | +| AC-NEW-4 | No | Phase 3 — Monte Carlo false-position budget | +| AC-NEW-5 | DEFERRED (hardware) | hot/cold soak chamber | +| AC-NEW-6 | No | Phase 3 — freshness enforcement | +| AC-NEW-7 | DEFERRED (hardware) | multi-flight voting | +| AC-NEW-8 | No | Phase 3 — visual blackout + GPS spoofing budget | + +**Summary:** 14 ACs covered / 35 non-deferred / 4 deferred = 40% initial coverage. Expected — Phase 3+ plans will add the bulk of missing tests. + +## Outstanding Orphan List for Plan 02-05 + +The following 21 non-deferred ACs have zero tests after Plan 02-04. Per the plan-recorded decision, Plan 02-05 should annotate these with `status: pending-phase-3` in the AC doc and widen the `--check` orphan detector to exempt `pending-phase-3` ACs: + +``` +AC-2.1b AC-3.1 AC-3.2 AC-3.5 AC-4.2 +AC-4.5 AC-5.1 AC-5.3 AC-6.1 AC-6.2 +AC-7.1 AC-7.2 AC-8.1 AC-8.2 AC-8.3 +AC-8.4 AC-8.5 AC-8.6 AC-NEW-4 AC-NEW-6 +AC-NEW-8 +``` + +(21 non-deferred orphans; `AC-NEW-3` has partial deferred status and is not counted.) + +## --check Determinism Confirmation + +Two consecutive `python scripts/gen_ac_traceability.py --check` runs produce identical stderr output and identical exit code (rc=1 in both runs). `diff /tmp/c1.log /tmp/c2.log` is empty. + +## Final Regression Count + +``` +216 passed, 8 skipped in 13.44s +``` + +Baseline: `passed=216`. Current: 216 passed. Parity maintained. + +## Git-Diff Hygiene Check + +``` +git diff tests/test_acceptance.py tests/test_accuracy.py \ + tests/test_processor_pipe.py tests/test_gps_input_encoding.py \ + tests/test_sitl_integration.py tests/test_mavlink.py tests/test_schemas.py \ +| grep -E '^[-+]' \ +| grep -vE '^(\+\+\+|---|@@|[-+]@pytest\.mark\.ac\(|[-+]pytestmark = \[pytest\.mark\.|[-+]\s*$)' +``` + +Output: empty. Only `@pytest.mark.ac(...)` decorators and one `pytestmark =` list extension were added. No logic edits. + +## Deviations from Plan + +None — plan executed exactly as written. + +The `.planning/AC-TRACEABILITY.md` was committed with `git add -f` because the `.gitignore` excludes `.planning/` globally, but other `.planning/` files (STATE.md, ROADMAP.md, REQUIREMENTS.md) are already tracked via prior force-adds — this is consistent with project convention. + +## Known Stubs + +None. The script and matrix are fully wired: script produces real data from pytest collection, matrix reflects actual tagged tests. + +## Threat Flags + +None. Script reads local filesystem and invokes pytest as subprocess — no new network endpoints, auth paths, or trust-boundary surface introduced. + +## Self-Check: PASSED + +- `scripts/gen_ac_traceability.py` exists: FOUND +- `.planning/AC-TRACEABILITY.md` exists with 39 ACs: FOUND +- Commits 4bf6f67, 6a1cd51, 419e9c5: verified via `git log --oneline`