# Resilience Tests

### NFT-RES-01: RabbitMQ broker outage during create

**Summary**: `POST /annotations` succeeds (HTTP 200) when the RabbitMQ broker is unreachable; the outbox row is preserved; `FailsafeProducer` does not crash; on broker recovery the message is delivered.
**Traces to**: AC-F-12, OP-02 (single-instance baseline)

**Preconditions**: SUT healthy; broker initially reachable; clean outbox.

**Fault injection**:
- `docker exec rabbitmq rabbitmqctl stop_app` mid-test (stops AMQP/streams listeners; container stays up).

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | Stop RabbitMQ app | broker unreachable on 5552 |
| 2 | `POST /annotations` once | HTTP 200; outbox row inserted |
| 3 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | `count == 1` (row not deleted because drain failed) |
| 4 | `GET /health` | HTTP 200 (SUT not crashed) |
| 5 | `docker exec rabbitmq rabbitmqctl start_app` | broker recovers |
| 6 | Wait `drain_interval × 3` | drainer publishes the queued message |
| 7 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | `count == 0` (drained) |
| 8 | Stream consumer (started before step 5 at offset `next`) reads one message | message body matches the documented schema |

**Pass criteria**: zero 5xx during outage; outbox preserves the row; recovery delivers the deferred message; total recovery time ≤ 60s after broker comes back.
**Duration**: ~2 minutes.

---

### NFT-RES-02: Postgres restart between writes

**Summary**: Killing and restarting Postgres during a quiet period does not corrupt state; subsequent writes succeed.
**Traces to**: AC-N-02 (idempotent migrator), implicit data-integrity NFR

**Fault injection**: `docker compose restart postgres` while no in-flight requests.

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | `POST /annotations` once (FT-P-01-shape) | HTTP 200; row in DB |
| 2 | `docker compose restart postgres` | DB up after ~5s |
| 3 | Wait for SUT `/health` to return 200 | SUT recovers connection pool (or restarts itself) |
| 4 | `POST /annotations` again | HTTP 200; row in DB |
| 5 | `GET /annotations/<id from step 1>` | HTTP 200; original row intact |

**Pass criteria**: original row intact after restart; new write succeeds within 30s of DB recovery; zero data loss.
**Duration**: ~2 minutes.

---

### NFT-RES-03: Postgres unreachable during create

**Summary**: When DB is unreachable mid-request, the SUT returns a structured error envelope (no 500 with stack trace); the SUT recovers when DB returns.
**Traces to**: AC-N-04 (zero unhandled exceptions to clients)

**Fault injection**: `docker pause postgres` between request start and request end (race-y; use a delay-injecting test proxy if needed).

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | `docker pause postgres` | DB connections hang |
| 2 | `POST /annotations` once with timeout 30s | HTTP 5xx OR HTTP 503; **error envelope present**; **no raw exception text in body** |
| 3 | `docker unpause postgres` | DB responsive |
| 4 | `POST /annotations` again | HTTP 200; SUT recovered |

**Pass criteria**: under-DB-outage response uses the error envelope; SUT recovers within 30s of DB recovery.
**Duration**: ~2 minutes.

---

### NFT-RES-04: SSE subscriber disconnect mid-stream

**Summary**: A subscriber that disconnects mid-stream does not crash the SUT or block other subscribers.
**Traces to**: AC-F-10, OP-01 (per-instance SSE state)

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | Open 3 SSE connections to `/annotations/events?missionId=<m>` | all 3 alive |
| 2 | Abruptly close subscriber #2 (TCP RST) | SUT cleans up its channel slot |
| 3 | `POST /annotations` for mission `<m>` | HTTP 200 |
| 4 | Subscribers #1 and #3 each receive the event | both receive within 1000ms |
| 5 | `GET /health` | HTTP 200 |

**Pass criteria**: surviving subscribers still receive events; no SUT memory growth visible (channel slots reclaimed); `/health` stays green.
**Duration**: ~1 minute.

---

### NFT-RES-05: Repeated FailsafeProducer empty-catch path

**Summary**: When the image referenced by an outbox row no longer exists on disk, the drainer logs and proceeds (post RB-05). Tests today's behavior (empty catch) AND, after RB-05 lands, asserts the logged failure path.
**Traces to**: RB-05

**Fault injection**: insert an outbox row whose `annotation_id` references a missing image (manually delete the file after `POST /annotations` returned 200, before the drain interval fires).

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | `POST /annotations` (FT-P-01) | HTTP 200; outbox row + image file present |
| 2 | Delete `<images_dir>/<id>.jpg` | image gone |
| 3 | Wait `drain_interval × 2` | drainer runs |
| 4 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | today's behavior: row may be deleted or stuck (empty catch swallows IOException) — **document actual behavior here** |
| 5 `[after RB-05]` | Inspect SUT logs for an `ERROR` entry mentioning the missing image | one log entry present; metric counter `failsafe_drain_errors` incremented |

**Pass criteria today**: SUT does not crash; `/health` stays 200.
**Pass criteria after RB-05**: as above + the logged failure path is exercised.
**Duration**: ~1 minute.

---

### NFT-RES-06: Stream consumer reconnect

**Summary**: A stream consumer that drops and reconnects with offset `last_committed` reads only post-disconnect messages.
**Traces to**: implicit (consumer-side concern, but documents the contract Annotations producer expects)

**Steps**:

| Step | Action | Expected Behavior |
|------|--------|------------------|
| 1 | Start consumer at offset `next`; record current end-of-stream offset `O0` | consumer up |
| 2 | `POST /annotations` 5 times | 5 outbox rows; 5 stream messages produced shortly after |
| 3 | Consumer reads all 5; commits offset after each | consumer offset = `O0 + 5` |
| 4 | Disconnect consumer | done |
| 5 | `POST /annotations` 3 more times | 3 more stream messages |
| 6 | Reconnect consumer at `last_committed = O0 + 5` | consumer reads only messages 6..8 |

**Pass criteria**: re-attached consumer sees no duplicates and no gaps.
**Duration**: ~1 minute.