Refactor testing framework to replace integration tests with blackbox tests across various skills and documentation. Update related workflows, templates, and task specifications to align with the new blackbox testing approach. Remove obsolete integration test files and enhance clarity in task management and reporting structures.

2026-04-22 12:56:31 +00:00 · 2026-03-24 03:38:36 +02:00
parent ae3ad50b9e
commit e609586c7c
49 changed files with 2222 additions and 872 deletions
@@ -0,0 +1,94 @@
+# Resilience Tests
+
+### NFT-RES-01: Loader service outage after engine initialization
+
+**Summary**: Verify that detections continue working when the Loader service goes down after the engine is already loaded.
+**Traces to**: RESTRICT-ENV-1
+
+**Preconditions**:
+- Engine is initialized (model already downloaded)
+
+**Fault injection**:
+- Stop mock-loader service
+
+**Steps**:
+
+| Step | Action | Expected Behavior |
+|------|--------|------------------|
+| 1 | Stop mock-loader | — |
+| 2 | `POST /detect` with small-image | 200 OK — detection succeeds (engine already in memory) |
+| 3 | `GET /health` | `aiAvailability` remains "Enabled" |
+
+**Pass criteria**: Detection continues to work. Health status remains stable. No errors from loader unavailability.
+
+---
+
+### NFT-RES-02: Annotations service outage during async detection
+
+**Summary**: Verify that async detection completes and delivers SSE events even when Annotations service is down.
+**Traces to**: RESTRICT-ENV-2
+
+**Preconditions**:
+- Engine is initialized
+- SSE client connected
+
+**Fault injection**:
+- Stop mock-annotations mid-processing
+
+**Steps**:
+
+| Step | Action | Expected Behavior |
+|------|--------|------------------|
+| 1 | Start async detection: `POST /detect/test-media-res01` | `{"status": "started"}` |
+| 2 | After first few SSE events, stop mock-annotations | — |
+| 3 | Continue listening to SSE | Events continue arriving. Annotations POST failures are silently caught |
+| 4 | Wait for completion | Final `AIProcessed` event received |
+
+**Pass criteria**: Detection pipeline completes fully. SSE delivery is unaffected. No crash or 500 errors.
+
+---
+
+### NFT-RES-03: Engine initialization retry after transient loader failure
+
+**Summary**: Verify that if model download fails on first attempt, a subsequent detection request retries initialization.
+**Traces to**: AC-EL-2
+
+**Preconditions**:
+- Fresh service (engine not initialized)
+
+**Fault injection**:
+- Mock-loader returns 503 on first model request, then recovers
+
+**Steps**:
+
+| Step | Action | Expected Behavior |
+|------|--------|------------------|
+| 1 | Configure mock-loader to fail first request | — |
+| 2 | `POST /detect` with small-image | Error (503 or 422) |
+| 3 | Configure mock-loader to succeed | — |
+| 4 | `POST /detect` with small-image | 200 OK — engine initializes on retry |
+
+**Pass criteria**: Second detection succeeds after loader recovers. System does not permanently lock into error state.
+
+---
+
+### NFT-RES-04: Service restart with in-memory state loss
+
+**Summary**: Verify that after a service restart, all in-memory state (_active_detections, _event_queues) is cleanly reset.
+**Traces to**: RESTRICT-OP-5, RESTRICT-OP-6
+
+**Preconditions**:
+- Previous detection may have been in progress
+
+**Fault injection**:
+- Restart detections container
+
+**Steps**:
+
+| Step | Action | Expected Behavior |
+|------|--------|------------------|
+| 1 | Restart detections container | — |
+| 2 | `GET /health` | Returns `aiAvailability: "None"` (fresh start) |
+| 3 | `POST /detect/any-media-id` | Accepted (no stale _active_detections blocking it) |
+
+**Pass criteria**: No stale state from previous session. All endpoints functional after restart.