# Resilience Tests ### NFT-RES-01: Loader service outage after engine initialization **Summary**: Verify that detections continue working when the Loader service goes down after the engine is already loaded. **Traces to**: RESTRICT-ENV-1 **Preconditions**: - Engine is initialized (model already downloaded) **Fault injection**: - Stop mock-loader service **Steps**: | Step | Action | Expected Behavior | |------|--------|------------------| | 1 | Stop mock-loader | — | | 2 | `POST /detect` with small-image | 200 OK — detection succeeds (engine already in memory) | | 3 | `GET /health` | `aiAvailability` remains "Enabled" | **Pass criteria**: Detection continues to work. Health status remains stable. No errors from loader unavailability. --- ### NFT-RES-02: Annotations service outage during async detection **Summary**: Verify that async detection completes and delivers SSE events even when Annotations service is down. **Traces to**: RESTRICT-ENV-2 **Preconditions**: - Engine is initialized - SSE client connected **Fault injection**: - Stop mock-annotations mid-processing **Steps**: | Step | Action | Expected Behavior | |------|--------|------------------| | 1 | Start async detection: `POST /detect/test-media-res01` | `{"status": "started"}` | | 2 | After first few SSE events, stop mock-annotations | — | | 3 | Continue listening to SSE | Events continue arriving. Annotations POST failures are silently caught | | 4 | Wait for completion | Final `AIProcessed` event received | **Pass criteria**: Detection pipeline completes fully. SSE delivery is unaffected. No crash or 500 errors. --- ### NFT-RES-03: Engine initialization retry after transient loader failure **Summary**: Verify that if model download fails on first attempt, a subsequent detection request retries initialization. **Traces to**: AC-EL-2 **Preconditions**: - Fresh service (engine not initialized) **Fault injection**: - Mock-loader returns 503 on first model request, then recovers **Steps**: | Step | Action | Expected Behavior | |------|--------|------------------| | 1 | Configure mock-loader to fail first request | — | | 2 | `POST /detect` with small-image | Error (503 or 422) | | 3 | Configure mock-loader to succeed | — | | 4 | `POST /detect` with small-image | 200 OK — engine initializes on retry | **Pass criteria**: Second detection succeeds after loader recovers. System does not permanently lock into error state. --- ### NFT-RES-04: Service restart with in-memory state loss **Summary**: Verify that after a service restart, all in-memory state (_active_detections, _event_queues) is cleanly reset. **Traces to**: RESTRICT-OP-5, RESTRICT-OP-6 **Preconditions**: - Previous detection may have been in progress **Fault injection**: - Restart detections container **Steps**: | Step | Action | Expected Behavior | |------|--------|------------------| | 1 | Restart detections container | — | | 2 | `GET /health` | Returns `aiAvailability: "None"` (fresh start) | | 3 | `POST /detect/any-media-id` | Accepted (no stale _active_detections blocking it) | **Pass criteria**: No stale state from previous session. All endpoints functional after restart.