missions/_docs/02_document/tests/resilience-tests.md

Resilience Tests

Status: produced by autodev /test-spec Phase 2 (2026-05-14). Naming: post-rename target. Resilience scenarios use the side-channel + container-orchestration tools to inject faults; the system-under-test is treated as a black box. Critical scenarios: AC-3.3 (cascade NOT transaction-wrapped) and AC-3.4 (orphan-row race) are the highest-impact resilience invariants — they intentionally encode current (sub-optimal) behavior so tests catch any silent change. Several other resilience tests verify documented operational behaviors (idempotent migrator, DB-down crash, JWT secret rotation).

---

NFT-RES-01: Cascade is NOT transaction-wrapped — partial deletes survive mid-walk failure

Summary: Verifies the documented ADR-006 carry-forward — when the cascade walk fails mid-way (e.g., media table absent), already-committed deletes remain. Traces to: AC-3.3, AC-3.4 (related), AC-10.2

Preconditions:

• fixture_cascade_F3 applied (chain rooted at mid)
• missions running

Fault injection: side-channel DROP TABLE media CASCADE; BEFORE the request — this turns the second sub-step of the cascade walk into a relation does not exist failure.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP TABLE media CASCADE	succeeds
2	DELETE /missions/{mid} (JWT FL)	500; envelope { statusCode:500, message:"Internal server error" }
3	Side-channel SELECT COUNT(*) FROM map_objects WHERE mission_id={mid}	count == 0 (work BEFORE the failure point committed — non-zero pre-fault, zero post-fault)
4	Side-channel SELECT COUNT(*) FROM missions WHERE id={mid}	count == 1 (work AFTER the failure point did NOT run — row remains)
5	docker logs missions | grep "Unhandled exception"	At least one matching log line containing relation and media

Pass criteria: map_objects count is 0 (deleted before failure) AND missions count is 1 (not deleted because failure short-circuited the walk) AND the response is 500 with the redacted body. Note: this test will FAIL once a transaction wrap is added (ADR-006 closure) — at that point ALL deletes will roll back and map_objects count will be >0. When the transaction wrap lands, update this test. Max execution time: 10s.

---

NFT-RES-02: Waypoint cascade is NOT transaction-wrapped (same invariant as F3)

Summary: Same invariant as NFT-RES-01, scoped to F4 (waypoint cascade). Traces to: AC-4.6, AC-3.3 (same root cause), AC-10.2

Preconditions:

• fixture_cascade_F4 applied (waypoint wp1 with chain)

Fault injection: side-channel DROP TABLE media CASCADE BEFORE the request.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP TABLE media CASCADE	succeeds
2	DELETE /missions/{mid}/waypoints/{wp1} (JWT FL)	500
3	Side-channel SELECT COUNT(*) FROM detection WHERE annotation_id IN (wp1 chain)	count == 0 (deleted before failure)
4	Side-channel SELECT COUNT(*) FROM waypoints WHERE id={wp1}	count == 1 (not deleted)

Pass criteria: same shape as NFT-RES-01. Max execution time: 10s.

---

NFT-RES-03: Idempotent migrator — second startup is a no-op

Summary: Verifies the migrator can run twice in a row without relation already exists errors. Traces to: AC-6.6, AC-6.4

Preconditions:

• seed_empty (schema migrated once via first missions startup)

Fault injection: container restart (NOT volume reset).

Steps:

Step	Action	Expected Behavior
1	docker compose restart missions	container exits cleanly + restarts
2	Wait for GET /health to return 200	within ≤ 30s
3	docker logs missions | grep -E "(error|Error|exception)"	no NEW error / exception lines after the restart timestamp
4	Side-channel \d+ vehicles	schema unchanged from after the first migrate

Pass criteria: second start completes; no new error log lines; schema unchanged. Max execution time: 60s.

---

NFT-RES-04: B9 one-shot legacy table drop runs once and is idempotent

Summary: Verifies that the post-B9 DROP TABLE IF EXISTS orthophotos / gps_corrections block in the migrator is destructive on the FIRST run against a legacy device, and a no-op on subsequent runs. Traces to: AC-6.5, AC-10.5

Preconditions:

• seed_legacy_gps_tables (schema includes orthophotos + gps_corrections with 1 row each)
• missions NOT yet started for this scenario

Fault injection: none — purely observe migrator behavior.

Steps:

Step	Action	Expected Behavior
1	Side-channel SELECT to_regclass('orthophotos'), to_regclass('gps_corrections')	both NON-NULL (legacy tables present)
2	docker compose up -d missions	container starts
3	Wait for GET /health to return 200
4	Side-channel re-query	both NULL (dropped)
5	docker compose restart missions
6	Side-channel re-query	both still NULL (idempotent — no error)
7	docker logs missions | grep -i "does not exist"	NO log line (because of IF EXISTS)

Pass criteria: legacy tables absent after first start; subsequent restarts produce no errors and leave them absent. Note: this test only meaningfully runs on a post-B9 build. Before B9 lands, the migrator has no DROP block; gate this scenario on a build-time flag or by inspecting the migrator source. Max execution time: 60s.

---

NFT-RES-05: Required configuration missing → fail-fast at startup

Summary: Verifies AC-6.1 / AC-6.2 / E3 — Infrastructure/ConfigurationResolver.ResolveRequiredOrThrow throws InvalidOperationException when any of the four required env vars (DATABASE_URL, JWT_ISSUER, JWT_AUDIENCE, JWT_JWKS_URL) is missing or whitespace-only. Also verifies AC-6.7 — DB unreachability (after config resolution succeeds) still causes process exit. The legacy "silent dev fallback boot" failure mode is structurally eliminated. Traces to: AC-6.1, AC-6.2, AC-6.7, E3, E4

Preconditions:

• missions NOT running.
• This scenario uses docker run outside the main compose to isolate env-var manipulation.

Steps (each row is a separate docker run invocation; each times out at 30s):

Step	Action	Expected Behavior
1	docker run --rm azaion/missions:test with ALL four required env vars unset	container exits non-zero within 5s; logs contain InvalidOperationException; logs mention at least one of the four required keys
2	docker run with DATABASE_URL unset; the three JWT vars set correctly	same shape; logs mention DATABASE_URL or Database:Url
3	docker run with JWT_ISSUER="" (whitespace-only); other three set	same shape; logs mention JWT_ISSUER or Jwt:Issuer
4	docker run with JWT_AUDIENCE unset; others set	same shape; logs mention JWT_AUDIENCE or Jwt:Audience
5	docker run with JWT_JWKS_URL unset; others set	same shape; logs mention JWT_JWKS_URL or Jwt:JwksUrl
6	docker compose stop postgres-test, then start missions with all four env vars set correctly — config resolution succeeds, then DB-connect fails	container exits non-zero within 30s; logs contain a recognisable Npgsql connection error (e.g., Connection refused) — NOT an InvalidOperationException from the resolver (this differentiates "config missing" from "config valid but DB down")

Pass criteria: rows 1–5 → fail-fast at config resolution; row 6 → fail at DB-connect AFTER config resolution succeeded. Note: this test now exercises BOTH the fail-fast resolver (rows 1–5) AND the DB-unreachable case (row 6). Pre-revision, only row 6 was tested under the assumption of hardcoded dev fallbacks. Max execution time: 180s (6 docker-run cycles).

---

NFT-RES-06: DB missing (database does not exist) — process exits with Npgsql 3D000

Summary: Verifies AC-6.8 — when the azaion database does not exist, process exits with the documented PostgreSQL error code. Traces to: AC-6.8

Preconditions:

• postgres-test running with the azaion database NOT yet created (use POSTGRES_DB=postgres instead, or DROP DATABASE azaion)

Fault injection: same as preconditions.

Steps:

Step	Action	Expected Behavior
1	Side-channel DROP DATABASE IF EXISTS azaion
2	docker compose up -d missions
3	Poll exit code for ≤ 30s	non-zero
4	docker logs missions | grep "3D000"	at least one match

Pass criteria: container exits non-zero within 30s; logs contain 3D000. Max execution time: 60s.

---

NFT-RES-07: JWKS key rotation — no missions restart required

Summary: Verifies AC-5.7 — rotating the signing key on admin (via jwks-mock POST /rotate-key) propagates to missions on the JWKS cache refresh tick without restarting missions. This is the primary operational win over the legacy shared-HMAC model, which required coordinated re-deploy across every backend on the device. Traces to: AC-5.7

Preconditions:

• missions running with warm JWKS cache (any previous protected request succeeded).
• jwks-mock running with Cache-Control: max-age=60 and OldKeyGraceSeconds=5.
• Token T1 requested via POST /sign with the CURRENT kid (kid_v1), valid for 1h.

Fault injection: POST https://jwks-mock:8443/rotate-key {} — generates kid_v2, retains kid_v1 in the JWKS for OldKeyGraceSeconds, then evicts kid_v1.

Steps:

Step	Action	Expected Behavior
1	GET /vehicles with Authorization: Bearer T1	200 (cached JWKS knows kid_v1)
2	POST https://jwks-mock:8443/rotate-key {} → returns kid_v2	jwks-mock now publishes BOTH kid_v1 and kid_v2 in its JWKS for OldKeyGraceSeconds=5
3	Immediately request token T2 signed with kid_v2 via POST /sign {}	mock returns JWT with header kid: kid_v2
4	Immediately GET /vehicles with Authorization: Bearer T2 (BEFORE missions JWKS cache refresh)	401 (cache still only has kid_v1)
5	Wait up to 90s for missions's ConfigurationManager<JsonWebKeySet> to refresh against the new JWKS (the mock's max-age=60 triggers a refresh on the next request after that interval)	—
6	GET /vehicles with Authorization: Bearer T2 again	200 (cache now contains kid_v2)
7	GET /vehicles with Authorization: Bearer T1 (still has unexpired lifetime, signed with kid_v1)	200 IF the JWKS refresh happened BEFORE the mock's OldKeyGraceSeconds=5 window closed (the JWKS still had kid_v1); 401 AFTER the grace window when missions refreshes and kid_v1 is no longer in the JWKS. Test asserts the eventual 401
8	Verify missions was NEVER restarted during this scenario (docker inspect --format '{{.State.StartedAt}}' missions is unchanged from before step 1)	startup timestamp unchanged

Pass criteria: rotation propagates without restart; T2 (new kid) eventually accepted; T1 (old kid) eventually rejected; missions startup timestamp unchanged. Note: this test replaces the pre-revision "shared-secret rotation requires coordinated redeploy" scenario. The pre-revision test asserted that ALL services on the device had to restart together; the post-revision test asserts the opposite — they do NOT have to restart. Max execution time: 180s (longest wait is the JWKS refresh tick).

---

NFT-RES-08: TOCTOU on default-vehicle exclusivity (race window)

Summary: Verifies AC-1.4 — the clear-then-set is NOT transaction-wrapped, so a concurrent INSERT can leave 2+ defaults. Traces to: AC-1.4 (carry-forward)

Preconditions:

• seed_one_default_vehicle (default P1)

Fault injection: a second concurrent client issues INSERT INTO vehicles (..., is_default=true) directly to the side-channel DB at the same moment as the API client issues POST /vehicles { IsDefault:true }. Synchronisation: the test orchestrator pauses at pg_advisory_lock(1) after the service's UPDATE vehicles SET is_default=FALSE and BEFORE the service's INSERT — implemented via a Postgres function that the test installs and the service path traverses (this requires an instrumented test build; if not available, the test is best-effort by issuing 100 parallel INSERTs).

Steps:

Step	Action	Expected Behavior
1	Run 100 parallel iterations of (POST /vehicles { IsDefault:true } + side-channel INSERT (..., is_default=true))	each iteration completes
2	Side-channel SELECT COUNT(*) FROM vehicles WHERE is_default=true	count is ≥ 2 in at least one iteration

Pass criteria: at least one iteration produces default_count ≥ 2. If 0 iterations produce the race, the test FAILS — either the race window has been closed (good news; rewrite the test to assert default_count == 1 and update AC-1.4 to remove the race carry-forward), OR the test concurrency primitive is wrong (investigate). Note: this test is intentionally PROBABILISTIC — it asserts the RACE EXISTS, not that the system is broken. A PASS here is bad news for the system but means the test is correctly observing the documented behavior. Max execution time: 30s.

---

Notes

• Tests that drop tables (NFT-RES-01, NFT-RES-02, NFT-RES-08) run in a per-class IClassFixture<DbResetFixture> that recreates the schema after each scenario.
• NFT-RES-03 through NFT-RES-07 require container-orchestration via docker compose from inside the test runner. The e2e-consumer container needs the docker CLI + a mounted Docker socket — alternatively, the test runs from the host with docker compose available there. Hardware-assessment will lock this preference.
• NFT-RES-08 is intentionally probabilistic and may be flaky on slow runners. CI marks this as [Trait("Stability","probabilistic")] and tolerates ≤ 1 failed run per 5; deterministic implementation (advisory lock + instrumented build) is a follow-up.