missions/_docs/02_document/tests/resilience-tests.md

Resilience Tests

Status: produced by autodev /test-spec Phase 2 (2026-05-14). Naming: post-rename target. Resilience scenarios use the side-channel + container-orchestration tools to inject faults; the system-under-test is treated as a black box. Critical scenarios: AC-3.3 (cascade NOT transaction-wrapped) and AC-3.4 (orphan-row race) are the highest-impact resilience invariants — they intentionally encode current (sub-optimal) behavior so tests catch any silent change. Several other resilience tests verify documented operational behaviors (idempotent migrator, DB-down crash, JWT secret rotation).

---

NFT-RES-01: Cascade is NOT transaction-wrapped — partial deletes survive mid-walk failure

Summary: Verifies the documented ADR-006 carry-forward — when the cascade walk fails mid-way (e.g., media table absent), already-committed deletes remain. Traces to: AC-3.3, AC-3.4 (related), AC-10.2

Preconditions:

• fixture_cascade_F3 applied (chain rooted at mid)
• missions running

Fault injection: side-channel DROP TABLE media CASCADE; BEFORE the request — this turns the second sub-step of the cascade walk into a relation does not exist failure.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP TABLE media CASCADE	succeeds
2	DELETE /missions/{mid} (JWT FL)	500; envelope { statusCode:500, message:"Internal server error" }
3	Side-channel SELECT COUNT(*) FROM map_objects WHERE mission_id={mid}	count == 0 (work BEFORE the failure point committed — non-zero pre-fault, zero post-fault)
4	Side-channel SELECT COUNT(*) FROM missions WHERE id={mid}	count == 1 (work AFTER the failure point did NOT run — row remains)
5	docker logs missions | grep "Unhandled exception"	At least one matching log line containing relation and media

Pass criteria: map_objects count is 0 (deleted before failure) AND missions count is 1 (not deleted because failure short-circuited the walk) AND the response is 500 with the redacted body. Note: this test will FAIL once a transaction wrap is added (ADR-006 closure) — at that point ALL deletes will roll back and map_objects count will be >0. When the transaction wrap lands, update this test. Max execution time: 10s.

---

NFT-RES-02: Waypoint cascade is NOT transaction-wrapped (same invariant as F3)

Summary: Same invariant as NFT-RES-01, scoped to F4 (waypoint cascade). Traces to: AC-4.6, AC-3.3 (same root cause), AC-10.2

Preconditions:

• fixture_cascade_F4 applied (waypoint wp1 with chain)

Fault injection: side-channel DROP TABLE media CASCADE BEFORE the request.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP TABLE media CASCADE	succeeds
2	DELETE /missions/{mid}/waypoints/{wp1} (JWT FL)	500
3	Side-channel SELECT COUNT(*) FROM detection WHERE annotation_id IN (wp1 chain)	count == 0 (deleted before failure)
4	Side-channel SELECT COUNT(*) FROM waypoints WHERE id={wp1}	count == 1 (not deleted)

Pass criteria: same shape as NFT-RES-01. Max execution time: 10s.

---

NFT-RES-03: Idempotent migrator — second startup is a no-op

Summary: Verifies the migrator can run twice in a row without relation already exists errors. Traces to: AC-6.6, AC-6.4

Preconditions:

• seed_empty (schema migrated once via first missions startup)

Fault injection: container restart (NOT volume reset).

Steps:

Step	Action	Expected Behavior
1	docker compose restart missions	container exits cleanly + restarts
2	Wait for GET /health to return 200	within ≤ 30s
3	docker logs missions | grep -E "(error|Error|exception)"	no NEW error / exception lines after the restart timestamp
4	Side-channel \d+ vehicles	schema unchanged from after the first migrate

Pass criteria: second start completes; no new error log lines; schema unchanged. Max execution time: 60s.

---

NFT-RES-04: B9 one-shot legacy table drop runs once and is idempotent

Summary: Verifies that the post-B9 DROP TABLE IF EXISTS orthophotos / gps_corrections block in the migrator is destructive on the FIRST run against a legacy device, and a no-op on subsequent runs. Traces to: AC-6.5, AC-10.5

Preconditions:

• seed_legacy_gps_tables (schema includes orthophotos + gps_corrections with 1 row each)
• missions NOT yet started for this scenario

Fault injection: none — purely observe migrator behavior.

Steps:

Step	Action	Expected Behavior
1	Side-channel SELECT to_regclass('orthophotos'), to_regclass('gps_corrections')	both NON-NULL (legacy tables present)
2	docker compose up -d missions	container starts
3	Wait for GET /health to return 200
4	Side-channel re-query	both NULL (dropped)
5	docker compose restart missions
6	Side-channel re-query	both still NULL (idempotent — no error)
7	docker logs missions | grep -i "does not exist"	NO log line (because of IF EXISTS)

Pass criteria: legacy tables absent after first start; subsequent restarts produce no errors and leave them absent. Note: this test only meaningfully runs on a post-B9 build. Before B9 lands, the migrator has no DROP block; gate this scenario on a build-time flag or by inspecting the migrator source. Max execution time: 60s.

---

NFT-RES-05: DB unreachable at startup — process exits non-zero

Summary: Verifies AC-6.7 — DB unreachability causes process exit, NOT silent retry-forever. Traces to: AC-6.7

Preconditions:

• missions NOT running

Fault injection: stop postgres-test (docker compose stop postgres-test) then start missions.

Steps:

Step	Action	Expected Behavior
1	docker compose stop postgres-test
2	docker compose up -d missions
3	Poll docker inspect --format '{{.State.ExitCode}}' missions every 1s for ≤ 30s	At some point within 30s, the container has exited with non-zero exit code
4	docker logs missions	Contains an Npgsql connection error message (e.g., Connection refused)

Pass criteria: container exits with non-zero code within 30s; logs contain a recognisable Npgsql error. Max execution time: 60s.

---

NFT-RES-06: DB missing (database does not exist) — process exits with Npgsql 3D000

Summary: Verifies AC-6.8 — when the azaion database does not exist, process exits with the documented PostgreSQL error code. Traces to: AC-6.8

Preconditions:

• postgres-test running with the azaion database NOT yet created (use POSTGRES_DB=postgres instead, or DROP DATABASE azaion)

Fault injection: same as preconditions.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP DATABASE IF EXISTS azaion
2	docker compose up -d missions
3	Poll exit code for ≤ 30s	non-zero
4	docker logs missions | grep "3D000"	at least one match

Pass criteria: container exits non-zero within 30s; logs contain 3D000. Max execution time: 60s.

---

NFT-RES-07: JWT_SECRET rotation invalidates existing tokens

Summary: Verifies AC-5.7 — restarting the service with a different JWT_SECRET causes previously-valid tokens to fail validation. Traces to: AC-5.7

Preconditions:

• missions running with JWT_SECRET=test-secret-32-chars-min!!!!!!!!!
• Token T1 minted with the same secret, valid for 1h

Fault injection: restart missions with JWT_SECRET=rotated-secret-32-chars-min!!!!!.

Steps:

Step	Action	Expected Behavior
1	GET /vehicles with Authorization: Bearer T1	200
2	docker compose stop missions
3	docker compose run -e JWT_SECRET=rotated-secret-32-chars-min!!!!! -d missions
4	Wait for GET /health 200
5	GET /vehicles with Authorization: Bearer T1 (same token as step 1)	401
6	Mint token T2 with the new secret, GET /vehicles with T2	200

Pass criteria: T1 works pre-rotation, fails post-rotation; T2 works post-rotation. Max execution time: 90s.

---

NFT-RES-08: TOCTOU on default-vehicle exclusivity (race window)

Summary: Verifies AC-1.4 — the clear-then-set is NOT transaction-wrapped, so a concurrent INSERT can leave 2+ defaults. Traces to: AC-1.4 (carry-forward)

Preconditions:

• seed_one_default_vehicle (default P1)

Fault injection: a second concurrent client issues INSERT INTO vehicles (..., is_default=true) directly to the side-channel DB at the same moment as the API client issues POST /vehicles { IsDefault:true }. Synchronisation: the test orchestrator pauses at pg_advisory_lock(1) after the service's UPDATE vehicles SET is_default=FALSE and BEFORE the service's INSERT — implemented via a Postgres function that the test installs and the service path traverses (this requires an instrumented test build; if not available, the test is best-effort by issuing 100 parallel INSERTs).

Steps:

Step	Action	Expected Behavior
1	Run 100 parallel iterations of (POST /vehicles { IsDefault:true } + side-channel INSERT (..., is_default=true))	each iteration completes
2	Side-channel SELECT COUNT(*) FROM vehicles WHERE is_default=true	count is ≥ 2 in at least one iteration

Pass criteria: at least one iteration produces default_count ≥ 2. If 0 iterations produce the race, the test FAILS — either the race window has been closed (good news; rewrite the test to assert default_count == 1 and update AC-1.4 to remove the race carry-forward), OR the test concurrency primitive is wrong (investigate). Note: this test is intentionally PROBABILISTIC — it asserts the RACE EXISTS, not that the system is broken. A PASS here is bad news for the system but means the test is correctly observing the documented behavior. Max execution time: 30s.

---

Notes

• Tests that drop tables (NFT-RES-01, NFT-RES-02, NFT-RES-08) run in a per-class IClassFixture<DbResetFixture> that recreates the schema after each scenario.
• NFT-RES-03 through NFT-RES-07 require container-orchestration via docker compose from inside the test runner. The e2e-consumer container needs the docker CLI + a mounted Docker socket — alternatively, the test runs from the host with docker compose available there. Hardware-assessment will lock this preference.
• NFT-RES-08 is intentionally probabilistic and may be flaky on slow runners. CI marks this as [Trait("Stability","probabilistic")] and tolerates ≤ 1 failed run per 5; deterministic implementation (advisory lock + instrumented build) is a follow-up.