From b0c71328898b879a003b612c3a5030d3ee652e4e Mon Sep 17 00:00:00 2001 From: Oleksandr Bezdieniezhnykh Date: Fri, 15 May 2026 06:37:00 +0300 Subject: [PATCH] [AZ-575] Add 11 blackbox test task specs from decompose Step 5 Decompose Step 5 (tests-only mode) produced the test-task ladder for the Blackbox Tests epic. Test infrastructure (AZ-576) blocks the rest; all 10 blackbox child tasks fan out from it. Tasks (epic AZ-575): - AZ-576 test_infrastructure (5 SP) - AZ-577 test_vehicles_positive (5 SP) - AZ-578 test_missions_positive (5 SP) - AZ-579 test_waypoints_health_positive (5 SP) - AZ-580 test_validation_authz_negative (3 SP) - AZ-581 test_security_auth_claims (5 SP) - AZ-582 test_security_alg_rotation_cors (5 SP) - AZ-583 test_resilience_cascade_migrator (3 SP) - AZ-584 test_resilience_config_db_rotation_race (5 SP) - AZ-585 test_resource_limits (3 SP) - AZ-586 test_performance (3 SP) Total: 45 SP across 11 tasks. Coverage verified against blackbox/security/resilience/resource-limit/performance test specs (56 scenarios). _docs/_autodev_state.md advanced to Step 6 (Implement Tests). Co-authored-by: Cursor --- _docs/_autodev_state.md | 8 +- _docs/tasks/_dependencies_table.md | 71 ++++++ .../tasks/todo/AZ-576_test_infrastructure.md | 227 ++++++++++++++++++ .../todo/AZ-577_test_vehicles_positive.md | 114 +++++++++ .../todo/AZ-578_test_missions_positive.md | 121 ++++++++++ .../AZ-579_test_waypoints_health_positive.md | 120 +++++++++ .../AZ-580_test_validation_authz_negative.md | 134 +++++++++++ .../todo/AZ-581_test_security_auth_claims.md | 125 ++++++++++ .../AZ-582_test_security_alg_rotation_cors.md | 140 +++++++++++ ...AZ-583_test_resilience_cascade_migrator.md | 115 +++++++++ ...test_resilience_config_db_rotation_race.md | 120 +++++++++ .../tasks/todo/AZ-585_test_resource_limits.md | 116 +++++++++ _docs/tasks/todo/AZ-586_test_performance.md | 117 +++++++++ 13 files changed, 1524 insertions(+), 4 deletions(-) create mode 100644 _docs/tasks/_dependencies_table.md create mode 100644 _docs/tasks/todo/AZ-576_test_infrastructure.md create mode 100644 _docs/tasks/todo/AZ-577_test_vehicles_positive.md create mode 100644 _docs/tasks/todo/AZ-578_test_missions_positive.md create mode 100644 _docs/tasks/todo/AZ-579_test_waypoints_health_positive.md create mode 100644 _docs/tasks/todo/AZ-580_test_validation_authz_negative.md create mode 100644 _docs/tasks/todo/AZ-581_test_security_auth_claims.md create mode 100644 _docs/tasks/todo/AZ-582_test_security_alg_rotation_cors.md create mode 100644 _docs/tasks/todo/AZ-583_test_resilience_cascade_migrator.md create mode 100644 _docs/tasks/todo/AZ-584_test_resilience_config_db_rotation_race.md create mode 100644 _docs/tasks/todo/AZ-585_test_resource_limits.md create mode 100644 _docs/tasks/todo/AZ-586_test_performance.md diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 779987f..7a2a16b 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -2,8 +2,8 @@ ## Current Step flow: existing-code -step: 5 -name: Decompose Tests +step: 6 +name: Implement Tests status: not_started sub_step: phase: 0 @@ -13,5 +13,5 @@ retry_count: 0 cycle: 1 tracker: jira -## Rename tracking (Jira AZ-EPIC + child stories B1-B12) -See `_docs/_process_leftovers/2026-05-14_rename-flights-to-missions.md`. Local code work for B5, B6, B7, B8, B9, B12 landed 2026-05-15; .woodpecker tag rename done. Cross-repo work pending: B4 (suite), B10-suite, B11 (autopilot + ui), B12 spec catch-up in suite. Leftover stays until those land. +## Last Updated +2026-05-15 diff --git a/_docs/tasks/_dependencies_table.md b/_docs/tasks/_dependencies_table.md new file mode 100644 index 0000000..d70eb7c --- /dev/null +++ b/_docs/tasks/_dependencies_table.md @@ -0,0 +1,71 @@ +# Dependencies Table + +**Date**: 2026-05-15 +**Mode**: tests-only decomposition (Step 5 of `existing-code` autodev flow) +**Epic**: AZ-575 — Blackbox Tests — Missions +**Total Tasks**: 11 +**Total Complexity Points**: 45 (5 + 5 + 5 + 5 + 3 + 5 + 5 + 3 + 5 + 3 + 3) + +| Task | Name | Complexity | Dependencies | Epic | +|------|------|-----------|-------------|------| +| AZ-576 | test_infrastructure | 5 | None | AZ-575 | +| AZ-577 | test_vehicles_positive | 5 | AZ-576 | AZ-575 | +| AZ-578 | test_missions_positive | 5 | AZ-576 | AZ-575 | +| AZ-579 | test_waypoints_health_positive | 5 | AZ-576 | AZ-575 | +| AZ-580 | test_validation_authz_negative | 3 | AZ-576 | AZ-575 | +| AZ-581 | test_security_auth_claims | 5 | AZ-576 | AZ-575 | +| AZ-582 | test_security_alg_rotation_cors | 5 | AZ-576 | AZ-575 | +| AZ-583 | test_resilience_cascade_migrator | 3 | AZ-576 | AZ-575 | +| AZ-584 | test_resilience_config_db_rotation_race | 5 | AZ-576 | AZ-575 | +| AZ-585 | test_resource_limits | 3 | AZ-576 | AZ-575 | +| AZ-586 | test_performance | 3 | AZ-576 | AZ-575 | + +## Coverage Verification + +| Spec file | Scenarios | Covered by | +|-----------|-----------|------------| +| `tests/blackbox-tests.md` § Positive | FT-P-01..06 (Vehicles) | AZ-577 | +| `tests/blackbox-tests.md` § Positive | FT-P-07..12 (Missions) | AZ-578 | +| `tests/blackbox-tests.md` § Positive | FT-P-13..18 (Waypoints + Health) | AZ-579 | +| `tests/blackbox-tests.md` § Negative | FT-N-01..08 | AZ-580 | +| `tests/security-tests.md` | NFT-SEC-01..06 + 04b | AZ-581 | +| `tests/security-tests.md` | NFT-SEC-07..13 | AZ-582 | +| `tests/resilience-tests.md` | NFT-RES-01..04 | AZ-583 | +| `tests/resilience-tests.md` | NFT-RES-05..08 | AZ-584 | +| `tests/resource-limit-tests.md` | NFT-RES-LIM-01..04 | AZ-585 | +| `tests/performance-tests.md` | NFT-PERF-01..04 | AZ-586 | + +**Total scenarios covered**: 56 (18 FT-P + 8 FT-N + 14 NFT-SEC + 8 NFT-RES + 4 NFT-RES-LIM + 4 NFT-PERF). + +## Cross-Task Consistency Checks + +| Check | Result | +|-------|--------| +| Every scenario from `blackbox-tests.md` § Positive (FT-P-01..18) is covered | PASS | +| Every scenario from `blackbox-tests.md` § Negative (FT-N-01..08) is covered | PASS | +| Every scenario from `security-tests.md` (NFT-SEC-01..13 + 04b) is covered | PASS | +| Every scenario from `resilience-tests.md` (NFT-RES-01..08) is covered | PASS | +| Every scenario from `resource-limit-tests.md` (NFT-RES-LIM-01..04) is covered | PASS | +| Every scenario from `performance-tests.md` (NFT-PERF-01..04) is covered | PASS | +| No task exceeds 5 complexity points | PASS | +| Every blackbox test task depends on the test-infrastructure task (AZ-576) | PASS | +| Test-infrastructure task (AZ-576) has no upstream test dependencies | PASS | +| No circular dependencies in the task graph | PASS — graph is a fan-out: AZ-576 → {AZ-577..AZ-586} | +| Every e2e/blackbox task has a System Under Test Boundary section | PASS — all 10 child tasks include the section | +| System Under Test Boundary forbids stubbing internal product modules | PASS — verified in each task spec | +| System Under Test Boundary requires comparison to expected-results artifacts | PASS — every task references `_docs/00_problem/input_data/expected_results/results_report.md` and/or the relevant machine-readable expected-result JSON | + +## Overlap & Shared-Concern Notes + +- **NFT-SEC-08 (Task 15) ↔ FT-N-08 (Task 13)** both exercise the 500 error envelope. FT-N-08 owns the destructive `DROP TABLE vehicles` fault injection and asserts redaction + log line presence; NFT-SEC-08 additionally asserts the body has NO key matching `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/type-name. No work duplication — the two tests share the fixture but assert distinct invariants. +- **NFT-SEC-11 (Task 15) ↔ NFT-RES-07 (Task 17)** both exercise JWKS rotation. NFT-SEC-11 focuses on the `kid`-cache mechanics + grace-window timing; NFT-RES-07 additionally asserts the `docker inspect StartedAt` invariant (no restart). Sharing the same primitive via the `JwksRotateFixture` from AZ-576. +- **NFT-SEC-12 (Task 15) ↔ NFT-RES-05 (Task 17)** both exercise startup fail-fast on missing required env vars. NFT-SEC-12 covers 4 missing-env cases + HTTP-JWKS-URL path. NFT-RES-05 covers the same 4 missing-env cases + an additional whitespace-only case + the DB-down-after-config-resolution differentiator (proves config resolution succeeded before Npgsql failed). Tasks share the `MissionsContainerHelper` docker-run primitive from AZ-576. + +## Execution Order Hint + +Recommended dependency-aware batches for `/implement`: + +1. **Batch 1 (sequential, blocking the rest)**: AZ-576 — test_infrastructure +2. **Batch 2 (parallel, fan-out from AZ-576)**: AZ-577..AZ-586 in any order. Independent test classes within a single xUnit assembly; no inter-task ordering needed. + +CSV report sorting at suite end: by `Category` (Blackbox / Sec / Res / ResLim / Perf), then by test ID within category. diff --git a/_docs/tasks/todo/AZ-576_test_infrastructure.md b/_docs/tasks/todo/AZ-576_test_infrastructure.md new file mode 100644 index 0000000..f483907 --- /dev/null +++ b/_docs/tasks/todo/AZ-576_test_infrastructure.md @@ -0,0 +1,227 @@ +# Test Infrastructure + +**Task**: AZ-576_test_infrastructure +**Name**: Test Infrastructure (Missions e2e) +**Description**: Scaffold the Blackbox test project — xUnit runner, JWKS mock service, Docker test environment wiring, test data fixtures, reporting. Compose file already exists at repo root and references not-yet-built build contexts; this task fills in those contexts. +**Complexity**: 5 points +**Dependencies**: None (C01 + C02 testability refactor already landed; see `_docs/04_refactoring/01-testability-refactoring/testability_changes_summary.md`) +**Component**: Blackbox Tests +**Tracker**: AZ-576 +**Epic**: AZ-575 + +## Scope + +Two artifacts that the existing `docker-compose.test.yml` references but does not yet build, plus the run script the suite expects: + +1. `tests/Azaion.Missions.JwksMock/` — minimal HTTPS service holding an ECDSA P-256 keypair in memory, serving JWKS + `POST /sign` + `POST /rotate-key`. Image tag `azaion/jwks-mock:test`. +2. `tests/Azaion.Missions.E2E.Tests/` — xUnit 2.x test project that drives the running `missions` service over HTTP, mints tokens via `https://jwks-mock:8443/sign`, asserts DB side-effects through a side-channel Npgsql connection, and produces `report.csv`. +3. `tests/jwks-mock-ca.crt` — the self-signed CA cert that both `missions` and `e2e-consumer` mount and `update-ca-certificates --fresh` adds to the OS trust bundle (per `docker-entrypoint.sh` from C02). +4. `scripts/run-tests.sh` — wraps `docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer`, collects `report.csv`, then `down -v`. +5. `scripts/run-performance-tests.sh` — same compose stack with the `[Trait("Category","Perf")]` filter and the perf seed. + +The `missions` and `postgres-test` services already exist in `docker-compose.test.yml`; the `jwks-mock` and `e2e-consumer` services exist but point at build contexts that this task creates. + +## Test Project Folder Layout + +``` +tests/ +├── jwks-mock-ca.crt # self-signed CA (mounted into missions + e2e-consumer) +├── Azaion.Missions.JwksMock/ +│ ├── Azaion.Missions.JwksMock.csproj +│ ├── Dockerfile # builds azaion/jwks-mock:test, exposes 8443/tcp +│ ├── Program.cs # ASP.NET Core minimal API +│ ├── Endpoints/ +│ │ ├── JwksEndpoint.cs # GET /.well-known/jwks.json +│ │ ├── SignEndpoint.cs # POST /sign +│ │ └── RotateKeyEndpoint.cs # POST /rotate-key +│ ├── Services/ +│ │ ├── KeyStore.cs # in-memory ECDSA P-256 keypair + old-key grace window +│ │ └── TokenSigner.cs # ECDSA signing with alg_override/kid_override support +│ └── appsettings.json # JWT_ISSUER, JWT_AUDIENCE, OLD_KEY_GRACE_SECONDS +└── Azaion.Missions.E2E.Tests/ + ├── Azaion.Missions.E2E.Tests.csproj # xUnit 2.x + Bogus 35.x + Npgsql 10.x + ├── Dockerfile # runs `dotnet test --logger trx` + trx→csv post-step + ├── TestBase.cs # HttpClient factory, default JWT, shared MissionsBaseUrl + ├── TokenMinter.cs # POST jwks-mock:8443/sign with overrides + ├── Fixtures/ + │ ├── DbResetFixture.cs # IClassFixture<>: TRUNCATE between classes + │ ├── DbSeedFixture.cs # base for the named seed sets in test-data.md + │ ├── ComposeRestartFixture.cs # docker compose down -v && up -d for bootstrap-sensitive tests + │ └── JwksRotateFixture.cs # POST /rotate-key + wait for missions to refresh JWKS cache + ├── Helpers/ + │ ├── DbAssertions.cs # Npgsql side-channel asserts (row counts, default-vehicle invariants) + │ ├── HttpAssertions.cs # PascalCase shape, error-envelope shape, ordering, pagination + │ └── FixtureSql.cs # loads fixture_cascade_F3.sql / fixture_cascade_F4.sql + ├── Tests/ + │ ├── Vehicles/ # FT-P-01..06, FT-N-01..03 + │ ├── Missions/ # FT-P-07..12, FT-N-04..06 + │ ├── Waypoints/ # FT-P-13..15, FT-P-18, FT-N-07 + │ ├── Health/ # FT-P-16..17, FT-N-08 + │ ├── Security/ # NFT-SEC-01..13, 04b + │ ├── Resilience/ # NFT-RES-01..08 + │ ├── ResourceLimits/ # NFT-RES-LIM-01..04 + │ └── Performance/ # NFT-PERF-01..04 + └── Reporting/ + ├── TrxToCsvPostProcessor.cs # produces /app/results/report.csv per environment.md § Reporting + └── ResultRow.cs # TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage +``` + +### Layout Rationale + +- **Per-feature test folders** (`Vehicles/`, `Missions/`, etc.) match the natural decomposition surface in `blackbox-tests.md` and let `dotnet test --filter` slice the suite per Jira child ticket. +- **`Fixtures/` separate from `Tests/`** so xUnit `IClassFixture<>` lifetime is explicit (class-scoped DB reset) and not entangled with test cases. +- **`Helpers/` named for the assertion family** (DB, HTTP, FixtureSql) so each test reads as a single `// Arrange` + `// Act` + `// Assert` block per `coderule.mdc`. +- **JwksMock is a SEPARATE csproj**, not nested inside the e2e tests, because the build context is mounted as a service in `docker-compose.test.yml` (`tests/Azaion.Missions.JwksMock/`). Sharing a project would force the e2e runner to ship JWKS code into its container. +- **CA cert lives at `tests/jwks-mock-ca.crt`** rather than inside a project so both consumers (missions + e2e-consumer) mount the same file. The cert is regenerated only when the keypair changes — committed to the repo for deterministic test runs. + +## Mock Services + +| Mock Service | Replaces | Endpoints | Behavior | +|-------------|----------|-----------|----------| +| `jwks-mock` | `admin` JWT issuer + JWKS endpoint | `GET https://jwks-mock:8443/.well-known/jwks.json`; `POST https://jwks-mock:8443/sign`; `POST https://jwks-mock:8443/rotate-key` | Holds one ECDSA P-256 keypair in memory; serves the public half as JWKS with `Cache-Control: public, max-age=60`; signs ECDSA-SHA256 JWTs on `/sign` honoring optional `iss`/`aud`/`exp_offset_seconds`/`permissions`/`alg_override`/`kid_override`; rotates keypair on `/rotate-key` while retaining the old public key for `OLD_KEY_GRACE_SECONDS` (5s in tests). Private key never leaves the container. | + +DB-only stubs (no service running, side-channel SQL inserts only): `annotations`, `detection`, `media`, `map_objects` — see `_docs/02_document/tests/test-data.md` § External Dependency Mocks. + +### Mock Control API + +`jwks-mock` exposes `POST /sign` and `POST /rotate-key` as its full control surface. The `/sign` body shape is documented in `test-data.md` § "JWKS mock token-minting contract": + +```http +POST https://jwks-mock:8443/sign +{ + "iss": "https://admin-test.azaion.local", # optional + "aud": "azaion-edge", # optional + "exp_offset_seconds": 3600, # optional; negative for expired + "permissions": "FL", # optional; "" / "ADMIN" / "fl" / "FLight" for claim-mismatch + "alg_override": null, # "HS256" to test alg-confusion (NFT-SEC-10) + "kid_override": null # non-existent kid for unknown-key tests (NFT-SEC-11) +} +``` + +Response: `{ "token": "", "kid": "" }`. + +## Docker Test Environment + +### docker-compose.test.yml Structure + +| Service | Image / Build | Purpose | Depends On | +|---------|--------------|---------|------------| +| `postgres-test` | `postgres:16-alpine` | Owned test PostgreSQL; `tmpfs:/var/lib/postgresql/data` for `down -v` isolation | — | +| `jwks-mock` | build `tests/Azaion.Missions.JwksMock/` → `azaion/jwks-mock:test` | Mock JWKS issuer | — | +| `missions` | build `.` (repo root `Dockerfile`) → `azaion/missions:test` | System under test | `postgres-test` (healthy), `jwks-mock` (healthy) | +| `e2e-consumer` | build `tests/Azaion.Missions.E2E.Tests/` | xUnit runner; emits `report.csv` to host-mounted `./test-results/` | `missions` (healthy), `jwks-mock` (healthy) | + +The compose file is already authored at the repo root. This task does NOT modify it — the file IS the contract; the task fills in the two missing build contexts so the references resolve. + +### Networks and Volumes + +| Resource | Purpose | +|----------|---------| +| `e2e-net` (bridge) | Isolated test network; no host network access. All four services attach. | +| `tmpfs:/var/lib/postgresql/data` | Ephemeral PG data; recreated per `docker compose down -v`. | +| `./test-results:/app/results` | `e2e-consumer` mounts this for `report.csv` output to the host. | +| `./tests/jwks-mock-ca.crt:/usr/local/share/ca-certificates/jwks-mock-ca.crt:ro` | Mounted into `missions` AND `e2e-consumer` so both trust the mock's HTTPS cert after `update-ca-certificates --fresh` runs in `docker-entrypoint.sh`. | + +## Test Runner Configuration + +**Framework**: xUnit 2.x +**Plugins**: `Microsoft.NET.Test.Sdk`, `xunit.runner.visualstudio`, `Bogus 35.x` (synthetic data), `Npgsql 10.x` (side-channel only — NO `Azaion.Missions.*` project reference) +**Entry point**: `dotnet test tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj --logger "trx;LogFileName=results.trx"` followed by `TrxToCsvPostProcessor` converting `results.trx` → `report.csv` +**AAA convention**: every test method has `// Arrange` / `// Act` / `// Assert` comments per `.cursor/rules/coderule.mdc`. + +### Fixture Strategy + +| Fixture | Scope | Purpose | +|---------|-------|---------| +| `DbResetFixture` | Class (`IClassFixture<>`) | `TRUNCATE TABLE` for all schema tables between classes; cheap reset for read-path tests (AC-1, AC-2, AC-4) | +| `DbSeedFixture` | Class | Applies the named seed sets from `test-data.md` (`seed_empty`, `seed_one_default_vehicle`, `seed_3_vehicles_2_default`, `seed_25_missions`, `fixture_cascade_F3`, `fixture_cascade_F4`, `seed_5_waypoints_unordered`, `seed_legacy_gps_tables`) via Npgsql side-channel | +| `ComposeRestartFixture` | Collection | `docker compose -f docker-compose.test.yml down -v && up -d` between scenarios that assert startup-time behavior (AC-6.3..6.7, AC-5.7) | +| `JwksRotateFixture` | Scenario | `POST jwks-mock:8443/rotate-key` then waits for missions to refresh its JWKS cache (≤ 30s in tests, capped by `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS`) | +| `JwksMockReverseFixture` | Scenario | Boots `missions` outside compose via `docker run` with `ASPNETCORE_ENVIRONMENT=Production` + empty `CorsConfig:AllowedOrigins` to test E9 lock (NFT-SEC-13) | + +### xUnit traits + +Every test method MUST set `[Trait("Category", "Blackbox" | "Sec" | "Res" | "ResLim" | "Perf")]`. The CSV `Category` column reads from this trait. Traceability IDs go into a second `[Trait("Traces", "AC-1.2,AC-1.4")]` trait, comma-separated. + +## Test Data Fixtures + +Loaded entirely from `_docs/02_document/tests/test-data.md` § Seed Data Sets. The fixtures bind the named seeds to the AC IDs that consume them: + +| Data Set | Source | Format | Used By | +|----------|--------|--------|---------| +| `seed_empty` | `down -v` + `missions` startup migrator | Schema only, no rows | bootstrap, unauth, 404 scenarios | +| `seed_one_default_vehicle` | Side-channel `INSERT INTO vehicles ...` | Inline SQL string | AC-1.2 default-clear, AC-1.3 TOCTOU, AC-1.4 setDefault, AC-2.1 mission-create | +| `seed_3_vehicles_2_default` | Side-channel SQL | Inline | AC-1.5 list, AC-1.6 filter | +| `seed_25_missions` | Side-channel SQL with deterministic UUIDs | Inline | AC-2.3..2.5 pagination + date filter | +| `fixture_cascade_F3` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql` | SQL file | AC-3.1, 3.3, 3.4, 10.2 | +| `fixture_cascade_F4` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql` | SQL file | AC-4.5, 4.6 | +| `seed_5_waypoints_unordered` | Side-channel SQL with `order_num [3,1,2,5,4]` | Inline | AC-4.3 unpaginated ordering | +| `seed_legacy_gps_tables` | `CREATE TABLE orthophotos / gps_corrections` + `INSERT` | Inline | AC-3.5 absence, AC-6.5 one-shot drop, AC-10.5 legacy migration | + +### Data Isolation + +Three tiers, by scenario type (per `test-data.md` § Data Isolation Strategy): + +- **Class-scoped DB reset** (`IClassFixture`): for scenarios that share a seed within a class but must not leak across classes. Used for AC-1, AC-2, AC-4 read paths. +- **Scenario-scoped container restart** (`docker compose down -v && up -d`): for scenarios that assert startup-time behavior or migrator side-effects (AC-6.3..6.7, AC-6.11, AC-5.7). +- **No per-test transaction rollback** — the system under test is a separate process; its `DataConnection` is not in the test transaction. + +## Test Reporting + +**Format**: CSV +**Columns**: `TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage` +**Output path**: `/app/results/report.csv` inside `e2e-consumer`, mounted to `./test-results/report.csv` on the host +**Source**: post-processor reads `results.trx` (xUnit logger output), joins each test's `[Trait("Category",...)]` and `[Trait("Traces",...)]` into the CSV columns. `Result` is `pass` / `fail` / `skip`. `ErrorMessage` is the first line of the failure message (CRs stripped). + +## Acceptance Criteria + +**AC-1: Test environment starts** +Given the `docker-compose.test.yml` at repo root +When `docker compose -f docker-compose.test.yml up --build` runs +Then `postgres-test`, `jwks-mock`, and `missions` all reach `healthy`, and `e2e-consumer` starts after them + +**AC-2: Mock JWKS service responds** +Given the test environment is running +When `GET https://jwks-mock:8443/.well-known/jwks.json` is issued from inside `e2e-net` +Then the response is `200 OK` with a JWKS body containing exactly one ECDSA P-256 public key +And `POST https://jwks-mock:8443/sign` with body `{}` returns a valid ECDSA-SHA256 JWT whose `iss` / `aud` match the mock's env vars + +**AC-3: Test runner executes** +Given the test environment is running +When `e2e-consumer` starts and `dotnet test` runs +Then the runner discovers ≥ 1 test in each of the eight test folders (`Vehicles/`, `Missions/`, `Waypoints/`, `Health/`, `Security/`, `Resilience/`, `ResourceLimits/`, `Performance/`) + +**AC-4: Test report generated** +Given tests have been executed +When `e2e-consumer` exits +Then `./test-results/report.csv` exists on the host +And the first line is the documented column header `TestId,TestName,Category,Traces,ExecutionTimeMs,Result,ErrorMessage` +And every executed test has exactly one CSV row + +**AC-5: CA trust works end-to-end** +Given `tests/jwks-mock-ca.crt` is mounted into both `missions` and `e2e-consumer` +When `docker-entrypoint.sh` runs `update-ca-certificates --fresh` and `missions` issues `GET https://jwks-mock:8443/.well-known/jwks.json` to populate its JWKS cache +Then the TLS handshake succeeds (no `RemoteCertificateNotAvailable` / `RemoteCertificateNameMismatch`) +And the cached JWKS contains the public key the consumer-issued tokens are signed with + +**AC-6: JWKS rotation observable inside the 15-minute CI gate** +Given the test compose sets `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30` and `JWT_JWKS_REFRESH_INTERVAL_SECONDS=10` (per C01) +When `POST https://jwks-mock:8443/rotate-key` is called +Then within 30s `missions` refreshes its JWKS cache and accepts tokens signed with the new `kid` +And during the 5s `OLD_KEY_GRACE_SECONDS` window tokens signed with the old `kid` are still accepted + +**AC-7: AAA pattern enforced** +Given the xUnit test project compiles +When `dotnet build` runs +Then every `[Fact]` / `[Theory]` method in `tests/Azaion.Missions.E2E.Tests/Tests/` contains the literal comment lines `// Arrange` (when setup exists), `// Act`, and `// Assert` in that order — verified by a Roslyn analyzer test or a single integration assertion that greps the source files + +## Constraints + +- `restrictions.md` SW-01: target framework .NET 10 (matches `Azaion.Missions.csproj`) +- `restrictions.md` HW-01: ARM64 + AMD64 (multi-arch base images on both projects) +- `restrictions.md` ENV-01: HTTPS-only for the JWKS endpoint (HTTP would short-circuit AC-6.12) +- `coderule.mdc`: AAA pattern with `// Arrange` / `// Act` / `// Assert` comments, no narrative comments otherwise +- No project reference from `Azaion.Missions.E2E.Tests` → `Azaion.Missions.csproj` (consumer must remain blackbox; assertions only via HTTP and Npgsql side-channel) +- Side-channel DB access limited to fixture seeding + post-call assertions; marked with `[Trait("db_access","seed-or-assert-only")]` where used +- Token signing happens ONLY inside `jwks-mock`; the consumer never imports a JWT signing library +- `report.csv` lives in `./test-results/` (host-mounted); this directory MUST be in `.gitignore` diff --git a/_docs/tasks/todo/AZ-577_test_vehicles_positive.md b/_docs/tasks/todo/AZ-577_test_vehicles_positive.md new file mode 100644 index 0000000..a8965f0 --- /dev/null +++ b/_docs/tasks/todo/AZ-577_test_vehicles_positive.md @@ -0,0 +1,114 @@ +# Vehicles Positive Flow Tests + +**Task**: AZ-577_test_vehicles_positive +**Name**: Vehicles positive tests (FT-P-01..06) +**Description**: Implement xUnit blackbox tests for the 6 happy-path Vehicle CRUD scenarios — create non-default, create default (demotes prior), setDefault, list (no-pagination + Name ASC), filter (case-INSENSITIVE name + exact isDefault), delete with no references. +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-577 +**Epic**: AZ-575 + +## Problem + +The `/vehicles` surface implements two non-obvious invariants that documentation alone cannot guarantee: (1) creating a default vehicle clears any prior default in the same logical step, and (2) the list filter is case-INSENSITIVE on `name` (the docs said case-sensitive until 2026-05-14 — drift now corrected, but only an executable test can pin the actual code path). Without these tests, a future refactor of `VehicleService` could silently re-introduce two default rows or a case-sensitive filter and break consumers (`autopilot` reads the default vehicle on boot). + +## Outcome + +- All six FT-P-01..06 scenarios run against the dockerised `missions` service via HTTP + Npgsql side-channel and pass. +- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-1.x`, `Result=pass`, and an `ExecutionTimeMs` under the documented `Max execution time` (5s for create paths, 2s for read/delete). +- The list test asserts both shape (`array` not `PaginatedResponse`) and ordering (`Name ASC`). +- The filter test asserts case-INSENSITIVE matching for two casings (`BR` and `br`). +- The default-clear invariant is verified via DB count (`is_default=true` count == 1 after every default-creating action). + +## Scope + +### Included + +- FT-P-01 Create non-default — `POST /vehicles` body shape + PascalCase response + DB row count. +- FT-P-02 Create default demotes prior default — `seed_one_default_vehicle` precondition; assert exactly one default after. +- FT-P-03 setDefault promotes existing vehicle — `POST /vehicles/{id}/setDefault`; assert clear-then-set via side-channel. +- FT-P-04 List unpaginated + Name ASC — assert body is JSON array (not `{Items,Page,…}`), assert length and ordering. +- FT-P-05 Filter `name=BR&isDefault=true` then `name=br&…` — assert case-INSENSITIVE substring match against `seed_3_vehicles_2_default`. +- FT-P-06 Delete with no references — `204` + DB count 0. + +### Excluded + +- FT-N-03 "delete vehicle in use returns 409" lives in Task 13 (negative tests). +- Validation-of-input scenarios (empty `Name`, negative `BatteryCapacity`, unknown `Type` int) are carry-forwards documented in `test-data.md` § Data Validation Rules; they are NOT tested here because the spec marks them as "accepted today" — they belong to the Refactor Backlog, not this task. +- TOCTOU race on default-vehicle exclusivity (NFT-RES-08) lives in Task 17. + +## Acceptance Criteria + +**AC-1: FT-P-01 returns 201 with PascalCase body** +Given `seed_empty` and a JWT with `permissions=FL` +When `POST /vehicles` is issued with the documented body +Then response is `201 Created`, body parses as `Vehicle` with PascalCase keys, `Id` parses as UUID, side-channel `SELECT COUNT(*) FROM vehicles WHERE id=` returns 1 + +**AC-2: FT-P-02 demotes prior default** +Given `seed_one_default_vehicle` (prior row `P1.is_default=true`) +When `POST /vehicles { …, IsDefault:true }` is issued +Then response is `201`, side-channel shows new row `is_default=true`, row `P1.is_default=false`, and `SELECT COUNT(*) WHERE is_default=true` == 1 + +**AC-3: FT-P-03 setDefault clears prior** +Given `seed_one_default_vehicle` plus a non-default row `P2` +When `POST /vehicles/{P2}/setDefault { IsDefault:true }` is issued +Then response is `200` with `Id==P2, IsDefault==true`, and side-channel shows `P2.is_default=true`, `P1.is_default=false`, count==1 + +**AC-4: FT-P-04 list is unpaginated and ordered** +Given `seed_3_vehicles_2_default` containing `BR-01, BR-02, MQ-9` in any insert order +When `GET /vehicles` is issued +Then response is `200`, body parses as a JSON array (NOT an object with `Items`), `body.length == 3`, and `[v.Name for v in body] == ["BR-01","BR-02","MQ-9"]` + +**AC-5: FT-P-05 filter is case-INSENSITIVE** +Given `seed_3_vehicles_2_default` +When `GET /vehicles?name=BR&isDefault=true` AND `GET /vehicles?name=br&isDefault=true` are issued +Then both responses are `200` with `body.length == 1` and `body[0].Name == "BR-01"` + +**AC-6: FT-P-06 delete is 204 + row gone** +Given one vehicle row with no missions referencing it +When `DELETE /vehicles/{id}` is issued +Then response is `204 No Content` with empty body, and side-channel shows `count == 0` for that id + +## Non-Functional Requirements + +**Performance** +- Each test must complete inside the documented `Max execution time` from `blackbox-tests.md` (5s for FT-P-01..03, 5s for FT-P-07-style writes, 2s for FT-P-04..06). The xUnit `[Trait("max_ms", "5000")]` or per-test `Timeout` must reflect this. + +**Reliability** +- Tests share a `[Collection("Vehicles")]` xUnit collection and use `IClassFixture` to TRUNCATE between scenarios. No state must leak between FT-P-01 and FT-P-04. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `seed_empty`, JWT permissions=FL | `POST /vehicles` non-default body | `201` + PascalCase `Vehicle` + DB count 1 | — | +| AC-2 | `seed_one_default_vehicle` (P1) | `POST /vehicles { IsDefault:true }` | `201` + DB shows count==1 default after | AC-1.2 invariant | +| AC-3 | `seed_one_default_vehicle` + extra P2 | `POST /vehicles/{P2}/setDefault` | `200` + DB count==1 default; P1 cleared | AC-1.2 / AC-1.4 | +| AC-4 | `seed_3_vehicles_2_default` (`BR-01,BR-02,MQ-9`) | `GET /vehicles` shape + order | `200` + array + Name ASC | AC-1.5 | +| AC-5 | `seed_3_vehicles_2_default` | `GET /vehicles?name=BR…` + `?name=br…` | `200` + len 1 + `BR-01` for both casings | AC-1.6 | +| AC-6 | One row, zero missions | `DELETE /vehicles/{id}` | `204` + DB count 0 | AC-1.10 | + +## Constraints + +- HTTP only against `http://missions:8080` (no project reference to `Azaion.Missions.csproj`). +- Bearer token minted via `https://jwks-mock:8443/sign` with `permissions=FL`. +- DB assertions through the Npgsql side-channel only; marked `[Trait("db_access","seed-or-assert-only")]`. +- AAA pattern with `// Arrange` / `// Act` / `// Assert` comments per `coderule.mdc`. +- PascalCase JSON contract (`PropertyNamingPolicy = null`) is part of the SUT contract; the test must NOT silently accept camelCase. + +## Risks & Mitigation + +**Risk 1: Tests depend on side-channel SQL that drifts from the SUT migrator** +- *Risk*: If the migrator changes the `vehicles` column set, hand-rolled `INSERT` in the seed fixture breaks. +- *Mitigation*: Seed fixtures use the schema produced by the SUT's own startup migrator — `docker compose up` runs first, then the fixture inserts into the already-migrated tables. + +**Risk 2: Ordering test (AC-4) is flaky if insert order accidentally matches alphabetic order** +- *Risk*: A non-deterministic seed insert could mask a missing `OrderBy`. +- *Mitigation*: Seed fixture inserts rows in `[MQ-9, BR-02, BR-01]` order (reverse alphabetic) so the test fails if the SUT omits the `OrderBy(a => a.Name)`. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080/vehicles*`) plus the documented DB side-channel for fixture seeding and post-call assertions; expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.1, 1.2, 1.4, 1.5, 1.6, 1.10. +- Stubs are allowed ONLY for the external `admin` JWT issuer (the `jwks-mock` container per `tests/Azaion.Missions.JwksMock/`). +- Stubs, fakes, monkeypatches, deterministic fallbacks, or direct imports are NOT allowed for any internal product module — including `VehicleService`, `VehiclesController`, `AppDataConnection`, `DatabaseMigrator`, `JwtExtensions`, or `ErrorHandlingMiddleware`. If any of these is not implemented (e.g., the SUT image hasn't been built), the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-578_test_missions_positive.md b/_docs/tasks/todo/AZ-578_test_missions_positive.md new file mode 100644 index 0000000..0489e0a --- /dev/null +++ b/_docs/tasks/todo/AZ-578_test_missions_positive.md @@ -0,0 +1,121 @@ +# Missions Positive Flow Tests + +**Task**: AZ-578_test_missions_positive +**Name**: Missions positive tests (FT-P-07..12) +**Description**: Implement xUnit blackbox tests for the 6 happy-path Mission scenarios — create with default CreatedDate, paginated list (PageSize=20, CreatedDate DESC, case-INSENSITIVE name filter), page 2, date-range filter, partial update preserving null fields, and full cascade delete across map_objects/detection/annotations/media/waypoints/missions. +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-578 +**Epic**: AZ-575 + +## Problem + +The `/missions` surface is the project's most consequential read+write path. Three behaviours are easy to silently break: (1) the default `CreatedDate = UtcNow` when the body omits it (AC-2.1), (2) `PaginatedResponse` envelope with `Page,PageSize,TotalCount,Items` PascalCase keys + `CreatedDate DESC` ordering (AC-2.3), and (3) the cascade delete walking every dependency table including DB-only stub tables `map_objects`, `detection`, `annotations`, `media` (AC-3.1). The cascade is **not** transaction-wrapped (NFT-RES-01 in Task 16 pins that invariant); the positive scenario here verifies the happy-path walk completes. + +## Outcome + +- All six FT-P-07..12 scenarios run against the dockerised `missions` service and pass. +- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-2.x` or `AC-3.1`, `Result=pass`, within the documented `Max execution time` (5s for create, 2s for list/update, 10s for cascade delete). +- The pagination test asserts both the envelope shape (`Items, TotalCount, Page, PageSize` PascalCase) AND `CreatedDate` DESC ordering across all 20 items. +- The cascade test compares per-table delete counts against `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` via `json_diff`. + +## Scope + +### Included + +- FT-P-07 Mission create with default CreatedDate — assert `|body.CreatedDate - t0| ≤ 5s`. +- FT-P-08 Mission list default page — envelope shape, `Page==1`, `PageSize==20`, `TotalCount==25`, `Items.length==20`, `CreatedDate` DESC ordering, plus case-INSENSITIVE `?name=re` filter. +- FT-P-09 Mission list page 2 — `Page==2`, `Items.length==5`, UUID-set disjoint from page 1. +- FT-P-10 Mission list date range — `?fromDate=&toDate=` inclusivity (January 2026 returns 5 of 25). +- FT-P-11 Mission partial update — `PUT /missions/{id}` with `VehicleId:null` preserves prior `VehicleId`. +- FT-P-12 Mission cascade delete (F3) — `DELETE /missions/{id}` walks every dependency table; per-table counts compared against `cascade_F3_walk.json`. + +### Excluded + +- FT-N-04 "create mission with non-existent VehicleId returns 400" lives in Task 13. +- FT-N-05 "GET mission 404" lives in Task 13. +- FT-N-06 "cascade delete short-circuits on missing mission (no DELETE issued against dependency tables)" lives in Task 13. +- Cascade NOT-transaction-wrapped invariant (NFT-RES-01) lives in Task 16. + +## Acceptance Criteria + +**AC-1: FT-P-07 mission create defaults CreatedDate to UtcNow** +Given `seed_one_default_vehicle` and a JWT with `permissions=FL` +When the consumer captures `t0 = UtcNow` then issues `POST /missions { Name:"Recon-01", VehicleId:, CreatedDate:null }` +Then response is `201`, `body.CreatedDate` parses as UTC, and `abs(body.CreatedDate - t0) ≤ 5s` + +**AC-2: FT-P-08 list returns PaginatedResponse with DESC ordering and case-INSENSITIVE name filter** +Given `seed_25_missions` (5 January, 20 February 2026, mix of `Recon-*` names) +When `GET /missions` is issued +Then response is `200` with `Page==1, PageSize==20, TotalCount==25, Items.length==20`, all PascalCase keys, AND for every `i ∈ [0..18]` `Items[i].CreatedDate >= Items[i+1].CreatedDate` (strictly DESC ordering) +And when `GET /missions?name=re` (lowercase) is issued, `body.TotalCount > 0` (case-INSENSITIVE substring match against `Recon-*`) + +**AC-3: FT-P-09 page 2 returns the remaining 5 items, disjoint from page 1** +Given `seed_25_missions` +When `GET /missions?page=2&pageSize=20` is issued +Then response is `200`, `Page==2`, `Items.length==5`, AND the set of `Items[*].Id` is disjoint from the page-1 response + +**AC-4: FT-P-10 date range filter is inclusive of bounds** +Given `seed_25_missions` (5 in January 2026, 20 in February 2026) +When `GET /missions?fromDate=2026-01-01T00:00:00Z&toDate=2026-01-31T23:59:59Z` is issued +Then response is `200`, `TotalCount==5`, and every `Items[i].CreatedDate` is within January 2026 UTC + +**AC-5: FT-P-11 partial update preserves null fields** +Given one mission row with known `Name="Original"` and `VehicleId=V1` +When `PUT /missions/{id} { Name:"Renamed", VehicleId:null }` is issued +Then response is `200`, `body.Name == "Renamed"`, AND `body.VehicleId == V1` (preserved) + +**AC-6: FT-P-12 cascade delete walks every dependency table** +Given `fixture_cascade_F3` applied (one mission with 2 waypoints → 2 media → 2 annotations → 2 detection rows + 3 map_objects) +When `DELETE /missions/{mid}` is issued +Then response is `204`, AND side-channel `SELECT COUNT(*)` returns 0 for `map_objects`, `detection`, `annotations`, `media`, `waypoints`, `missions` rows in the seeded chain +And the per-table counts after deletion match `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` via deep JSON diff + +## Non-Functional Requirements + +**Performance** +- FT-P-07: ≤ 5s. FT-P-08..11: ≤ 2s each. FT-P-12: ≤ 10s (cascade through 5 tables). + +**Reliability** +- FT-P-12 must use `IClassFixture` that recreates `fixture_cascade_F3` fresh per scenario (the fixture is destructive). FT-P-08..10 share `seed_25_missions` across the same class. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `seed_one_default_vehicle` | `POST /missions { CreatedDate:null }` | `201` + `\|body.CreatedDate - t0\| ≤ 5s` | AC-2.1 | +| AC-2 | `seed_25_missions` | `GET /missions` then `GET /missions?name=re` | `200` + envelope + DESC + case-INSENSITIVE match | AC-2.3, AC-8.7 | +| AC-3 | `seed_25_missions` | `GET /missions?page=2&pageSize=20` | `200` + `Page=2` + len 5 + disjoint UUIDs | AC-2.3 | +| AC-4 | `seed_25_missions` | `GET /missions?fromDate=…&toDate=…` (January window) | `200` + `TotalCount=5` + all in window | AC-2.3 | +| AC-5 | One row with `Name=Original, VehicleId=V1` | `PUT /missions/{id} { Name:"Renamed", VehicleId:null }` | `200` + Name updated + VehicleId preserved | AC-2.5 | +| AC-6 | `fixture_cascade_F3` | `DELETE /missions/{mid}` | `204` + DB counts 0 across 6 tables + `cascade_F3_walk.json` match | AC-3.1 | + +## Constraints + +- HTTP only against `http://missions:8080/missions*` (no project reference to `Azaion.Missions.csproj`). +- Bearer token minted via `https://jwks-mock:8443/sign` with `permissions=FL`. +- FT-P-12 fixture uses the SQL file at `_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql` (NOT a hand-rolled INSERT — the SQL file is the contract). +- Per-table count comparison in FT-P-12 uses `json_diff` against `cascade_F3_walk.json`; if the file is missing, the test must fail (not silently pass). +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. +- `seed_25_missions` MUST use deterministic UUIDs and deterministic `CreatedDate` values so the disjoint-set assertion in AC-3 and the date-range assertion in AC-4 are reproducible. + +## Risks & Mitigation + +**Risk 1: cascade_F3_walk.json drifts from fixture_cascade_F3.sql** +- *Risk*: Updating the seed SQL without updating the walk JSON makes AC-6 silently pass with wrong counts. +- *Mitigation*: Both files live under the same `expected_results/` directory; the test loads the walk JSON at runtime and verifies BOTH that pre-delete counts match the walk's `before` values AND post-delete counts match the walk's `after` values. A drift fails the "before" assertion first. + +**Risk 2: AC-2 ordering assertion is flaky if seed CreatedDate values collide** +- *Risk*: Two missions with identical `CreatedDate` produce a tie-breaker-dependent order; the DESC assertion would be deterministic only if the comparator is stable. +- *Mitigation*: `seed_25_missions` SQL assigns distinct `CreatedDate` values spaced ≥ 1 second apart; any future seed change must preserve this invariant. + +**Risk 3: cascade test pollutes neighbour scenarios** +- *Risk*: F3 fixture deletes rows across 6 tables; if FT-P-12 runs in the same xUnit class as a read-path test, that test sees an empty DB. +- *Mitigation*: FT-P-12 lives in its own xUnit `[Collection("CascadeF3")]` and uses `IClassFixture` to reset between every scenario in the class. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080/missions*`) plus the documented DB side-channel for fixture seeding and post-call assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-2 2.1, 2.3, 2.4, 2.5, 2.7 and AC-3 row 3.1, and against the machine-readable file `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` for the cascade walk. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL because the owning services are out of scope per `environment.md`). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `MissionService`, `MissionsController`, `WaypointService`, `AppDataConnection`, `DatabaseMigrator`, `JwtExtensions`, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-579_test_waypoints_health_positive.md b/_docs/tasks/todo/AZ-579_test_waypoints_health_positive.md new file mode 100644 index 0000000..e422df7 --- /dev/null +++ b/_docs/tasks/todo/AZ-579_test_waypoints_health_positive.md @@ -0,0 +1,120 @@ +# Waypoints + Health Positive Flow Tests + +**Task**: AZ-579_test_waypoints_health_positive +**Name**: Waypoints + Health positive tests (FT-P-13..18) +**Description**: Implement xUnit blackbox tests for the 6 happy-path Waypoint + Health scenarios — waypoint list ordered by OrderNum ASC, waypoint create echoes geo fields (no auto-conversion), waypoint update is full overwrite, health 200 anonymous, health 200 with Postgres stopped (no DB ping), and waypoint cascade delete scoped to one waypoint (sibling chain intact). +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-579 +**Epic**: AZ-575 + +## Problem + +Waypoints carry two non-obvious behaviors: (1) the list endpoint orders by `OrderNum` ASC regardless of insert order (AC-4.3), and (2) `PUT /missions/{id}/waypoints/{wpId}` is a FULL overwrite even though the DTO looks "partial" (non-nullable enums + numerics) — passing `Height:0` overwrites the previous `Height:120` (AC-4.4). The waypoint cascade delete (AC-4.5) is the tighter sibling of the mission cascade — it must remove the target waypoint's chain (`media → annotations → detection`) without touching a sibling waypoint's chain. The health endpoint (AC-7.1, AC-7.2) is the suite's probe contract: it MUST return 200 anonymously AND MUST NOT ping the database, because the suite reverse proxy uses `/health` to decide whether to route traffic — a DB outage must not depool a healthy process. + +## Outcome + +- All six FT-P-13..18 scenarios run against the dockerised `missions` service and pass. +- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-4.x` or `AC-7.x`, `Result=pass`, within the documented `Max execution time` (2s for FT-P-13..16, 5s for FT-P-17 to allow PG stop, 10s for FT-P-18 cascade). +- The list test asserts both shape (JSON array) and ordering (`[1,2,3,4,5]` ASC from a `[3,1,2,5,4]` insert order). +- The update test asserts the FULL overwrite by passing `Height:0` and checking the new value is 0 (not the preserved 120). +- The "PG stopped" health test asserts the process answers `200` even with `postgres-test` stopped — proving the probe does not ping the DB. +- The cascade test (F4) asserts target-waypoint chain deleted AND sibling-waypoint chain preserved, with per-table counts compared against `cascade_F4_walk.json`. + +## Scope + +### Included + +- FT-P-13 Waypoint list ordered by `OrderNum` ASC — `seed_5_waypoints_unordered` inserts in `[3,1,2,5,4]` order. +- FT-P-14 Waypoint create echoes `GeoPoint` fields (no auto lat/lon ↔ MGRS conversion today — preserves the documented divergence from spec). +- FT-P-15 Waypoint update is full overwrite — `Height:0` overwrites `Height:120`, `OrderNum` changes, `GeoPoint:null` clears. +- FT-P-16 Health 200 anonymous — no `Authorization` header, exact JSON `{ "status": "healthy" }`. +- FT-P-17 Health 200 with PG stopped — proves process-liveness only, no DB ping. +- FT-P-18 Waypoint cascade delete (F4) — `DELETE /missions/{mid}/waypoints/{wp1}`; per-table counts on `wp1` chain go to 0; sibling `wp2` chain intact. + +### Excluded + +- FT-N-07 "waypoint operation against missing mission returns 404" lives in Task 13. +- Waypoint nested existence check (single composite-FK predicate per `state.json` drift entry) is implementation detail; the blackbox test only asserts the observable 404 in FT-N-07. + +## Acceptance Criteria + +**AC-1: FT-P-13 waypoint list is ordered by OrderNum ASC** +Given `seed_5_waypoints_unordered` under one mission, with `order_num` values `[3,1,2,5,4]` inserted in that order +When `GET /missions/{id}/waypoints` is issued with a valid JWT +Then response is `200`, body parses as JSON array, `body.length == 5`, AND `[w.OrderNum for w in body] == [1,2,3,4,5]` + +**AC-2: FT-P-14 waypoint create echoes geo fields, no MGRS conversion** +Given one mission row +When `POST /missions/{id}/waypoints { GeoPoint:{Lat:50.45, Lon:30.52, Mgrs:null}, WaypointSource:0, WaypointObjective:0, OrderNum:1, Height:120 }` is issued +Then response is `201`, `body.GeoPoint.Lat == 50.45`, `body.GeoPoint.Lon == 30.52`, AND `body.GeoPoint.Mgrs == null` (NO auto-conversion) + +**AC-3: FT-P-15 waypoint update is full overwrite** +Given one waypoint with `Height=120, OrderNum=1, GeoPoint=(Lat:50.45, …)` +When `PUT /missions/{id}/waypoints/{wpId} { GeoPoint:null, WaypointSource:1, WaypointObjective:1, OrderNum:2, Height:0 }` is issued +Then response is `200`, `body.Height == 0` (overwritten from 120), `body.OrderNum == 2`, AND `body.GeoPoint == null` + +**AC-4: FT-P-16 health is 200 anonymous** +Given a running `missions` container +When `GET /health` is issued with NO `Authorization` header +Then response is `200`, body is exactly `{ "status": "healthy" }` with case-sensitive key + +**AC-5: FT-P-17 health is 200 with PG stopped** +Given `missions` is running AND `docker compose stop postgres-test` has succeeded +When `GET /health` is issued +Then response is `200`, body is exactly `{ "status": "healthy" }` — proving the probe does NOT ping the DB + +**AC-6: FT-P-18 waypoint cascade scope is one waypoint** +Given `fixture_cascade_F4` (target waypoint `wp1` with chain `media → annotations → detection`; sibling waypoint `wp2` with its own chain) +When `DELETE /missions/{mid}/waypoints/{wp1}` is issued +Then response is `204`, AND side-channel `SELECT COUNT(*)` returns 0 for the `wp1` chain rows in `detection`, `annotations`, `media`, AND for `wp1` itself in `waypoints` +And side-channel returns `1` for `wp2` in `waypoints` AND `> 0` for the `wp2` chain rows in `media, annotations, detection` +And the per-table counts after deletion match `_docs/00_problem/input_data/expected_results/cascade_F4_walk.json` via deep JSON diff + +## Non-Functional Requirements + +**Performance** +- FT-P-13..16: ≤ 2s each. FT-P-17: ≤ 5s (allow PG stop time). FT-P-18: ≤ 10s (cascade through 4 tables). + +**Reliability** +- FT-P-17 must restore `postgres-test` to `Up` before exiting (try/finally with `docker compose start postgres-test` in the fixture teardown) — otherwise subsequent tests fail with `ConnectionRefused`. +- FT-P-18 uses `IClassFixture` with the F4 fixture recreated per scenario. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `seed_5_waypoints_unordered` ([3,1,2,5,4]) | `GET /missions/{id}/waypoints` | `200` + array + OrderNum ASC | AC-4.3 | +| AC-2 | One mission row | `POST /missions/{id}/waypoints { GeoPoint:{Lat,Lon,Mgrs:null} }` | `201` + GeoPoint echoed + Mgrs null (no conversion) | AC-4 (data_parameters § 2.3) | +| AC-3 | One waypoint Height=120 | `PUT … { Height:0, GeoPoint:null }` | `200` + Height=0 + GeoPoint=null (full overwrite) | AC-4.4 | +| AC-4 | Running container | `GET /health` no auth | `200` + exact `{"status":"healthy"}` | AC-7.1 | +| AC-5 | PG stopped | `GET /health` | `200` + exact `{"status":"healthy"}` | AC-7.2, AC-7.3 | +| AC-6 | `fixture_cascade_F4` | `DELETE /missions/{mid}/waypoints/{wp1}` | `204` + wp1 chain 0 + wp2 chain intact + `cascade_F4_walk.json` match | AC-4.5 | + +## Constraints + +- HTTP only against `http://missions:8080`; bearer token via `https://jwks-mock:8443/sign` with `permissions=FL` (for waypoint endpoints); FT-P-16 and FT-P-17 explicitly send no `Authorization` header. +- FT-P-17 uses `ComposeRestartFixture`-style helper that runs `docker compose -f docker-compose.test.yml stop postgres-test` then `docker compose -f docker-compose.test.yml start postgres-test` in teardown. +- FT-P-18 fixture uses `_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql` (NOT a hand-rolled INSERT). +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: FT-P-15 silently passes if SUT exposes a "partial" update path** +- *Risk*: If a future refactor adds a JSON-merge update mode, sending `Height:0` might be interpreted as "leave Height unchanged" rather than overwrite. +- *Mitigation*: The test ALSO sets `GeoPoint:null` and asserts the value is null after — proving the path is full-overwrite, not patch. + +**Risk 2: FT-P-17 PG-stop leaks to other tests** +- *Risk*: If the test fails before teardown, subsequent tests run against a dead DB. +- *Mitigation*: The fixture uses `try/finally`; the teardown waits for `postgres-test` to reach `healthy` (poll `pg_isready`) before yielding control back to xUnit. + +**Risk 3: FT-P-18 sibling-intact assertion gives false-pass if F4 fixture is empty** +- *Risk*: If `fixture_cascade_F4.sql` failed to insert `wp2`'s chain, the post-delete assertion `wp2 chain > 0` fails trivially — but with a misleading message. +- *Mitigation*: The test asserts pre-delete counts FIRST (`wp1` chain > 0 AND `wp2` chain > 0); fixture failure is caught in the Arrange phase, not the Assert phase. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080/missions/{id}/waypoints*` and `http://missions:8080/health`) plus the documented DB side-channel for fixture seeding and post-call assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-4 4.2, 4.3, 4.4, 4.5 and AC-7 rows 7.1, 7.2, and against the machine-readable file `_docs/00_problem/input_data/expected_results/cascade_F4_walk.json`. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection` (seeded via side-channel SQL). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `WaypointService`, `MissionsController` (health route), `AppDataConnection`, or `Program.cs`'s health middleware. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-580_test_validation_authz_negative.md b/_docs/tasks/todo/AZ-580_test_validation_authz_negative.md new file mode 100644 index 0000000..8fdf21f --- /dev/null +++ b/_docs/tasks/todo/AZ-580_test_validation_authz_negative.md @@ -0,0 +1,134 @@ +# Validation + 404 + Authz Negative Tests + +**Task**: AZ-580_test_validation_authz_negative +**Name**: Functional negative tests (FT-N-01..08) +**Description**: Implement xUnit blackbox tests for the 8 negative scenarios — case-insensitive filter no-match, 404 for missing GET vehicle/mission/waypoint-parent, 409 for delete-vehicle-in-use, 400 for create-mission-with-bogus-VehicleId (carry-forward divergence), cascade short-circuit on missing mission (no dependency DELETEs issued), and the generic 500 redacted-body + stacktrace-in-log contract. +**Complexity**: 3 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-580 +**Epic**: AZ-575 + +## Problem + +The negative-path contract is what protects clients from undefined behaviour: every documented failure must produce a predictable status code + `{ statusCode, message }` envelope, and no failure mode may silently mutate state. Three behaviors are especially load-bearing: (1) `DELETE /missions/{missing}` must 404 *before* any dependency-table DELETE issues — otherwise a typo'd UUID could remove rows from `map_objects` belonging to a different mission (AC-3.2); (2) `DELETE /vehicles/{used}` must 409 and leave the row in place (AC-1.8); (3) the generic 500 must redact internals — `Internal server error` body, full stack only in container logs (AC-8.6, AC-10.3). + +## Outcome + +- All eight FT-N-01..08 scenarios run against the dockerised `missions` service and pass. +- Each test produces a CSV row with `Category=Blackbox` (negative subset; `Traces=AC-1.6, AC-1.7, AC-1.8, AC-2.2, AC-2.4, AC-3.2, AC-4.2, AC-8.6, AC-10.3`), `Result=pass`. +- The 500 test asserts BOTH that the body is exactly `{ "statusCode":500, "message":"Internal server error" }` AND that the container log emitted an `"Unhandled exception"` line within 2s. +- FT-N-06 asserts via `pg_stat_statements` (or post-request log scrape) that NO `DELETE FROM map_objects/waypoints/media/annotations/detection` SQL ran during the 404 request — the existence check short-circuits before the cascade. +- FT-N-04 explicitly pins the documented spec-divergence (returns 400 today, spec wants 404); test must include a comment marking it as a carry-forward to revisit when the divergence is closed. + +## Scope + +### Included + +- FT-N-01 Vehicle name filter no-match — `?name=ZZ` and `?name=zz` against `seed_3_vehicles_2_default` both return `body.length == 0`. +- FT-N-02 GET vehicle 404 — random UUID returns `{ statusCode:404, message:… }`. +- FT-N-03 Delete vehicle in use 409 — row not deleted afterwards. +- FT-N-04 Create mission with bogus VehicleId returns 400 today (CARRY-FORWARD comment). +- FT-N-05 GET mission 404 — envelope shape. +- FT-N-06 Cascade short-circuit — 404 + zero DELETE SQL issued. +- FT-N-07 Waypoint operation against missing mission — 404. +- FT-N-08 Generic 500 — redacted body + stacktrace in log. + +### Excluded + +- 401 / 403 auth-failure paths (NFT-SEC-01..06) live in Task 14. +- 400/422 spec-divergence carry-forwards that are NOT executable today (input validation for empty `Name`, negative `BatteryCapacity`, unknown `Type` int) are documented as Refactor Backlog items in `tests/blackbox-tests.md` and are NOT in scope here. + +## Acceptance Criteria + +**AC-1: FT-N-01 vehicle filter no-match returns empty array for both casings** +Given `seed_3_vehicles_2_default` (`BR-01, BR-02, MQ-9`) +When `GET /vehicles?name=ZZ` then `GET /vehicles?name=zz` are issued +Then both responses are `200` with `body.length == 0` + +**AC-2: FT-N-02 GET vehicle 404 returns the standard envelope** +Given any DB state and a valid JWT +When `GET /vehicles/{random uuid}` is issued +Then response is `404` with body parsing to JSON object having EXACTLY the keys `statusCode` and `message`, and `statusCode == 404` + +**AC-3: FT-N-03 delete in-use vehicle returns 409 and leaves row** +Given one vehicle and ≥ 1 mission referencing it +When `DELETE /vehicles/{id}` is issued +Then response is `409` with envelope `{ statusCode:409, message: }`, and side-channel `SELECT COUNT(*) FROM vehicles WHERE id={id}` returns `1` + +**AC-4: FT-N-04 create mission with bogus VehicleId returns 400 today (carry-forward)** +Given `seed_empty` +When `POST /missions { Name:"x", VehicleId:, CreatedDate:null }` is issued +Then response is `400` with envelope (carry-forward: spec wants 404; the test must include a `// CARRY-FORWARD: expected to flip to 404 when AC-2.2 divergence is closed` comment) +And side-channel `SELECT COUNT(*) FROM missions` returns `0` + +**AC-5: FT-N-05 GET mission 404 returns the standard envelope** +Given any DB state and a valid JWT +When `GET /missions/{random uuid}` is issued +Then response is `404` with envelope `{ statusCode:404, message: }` + +**AC-6: FT-N-06 cascade short-circuit issues zero dependency-table DELETEs** +Given `fixture_cascade_F3` (seeded chain rooted at `mid`) and a `postgres-test` started with `log_statement=all` +When `DELETE /missions/{mid'}` (random UUID, not `mid`) is issued +Then response is `404`, side-channel `SELECT COUNT(*) FROM map_objects` is unchanged, AND the `postgres-test` log (or `pg_stat_statements`) shows NO `DELETE FROM map_objects/waypoints/media/annotations/detection` SQL emitted by the request connection + +**AC-7: FT-N-07 waypoint operation against missing mission returns 404** +Given any DB state and a valid JWT +When `GET /missions/{random uuid}/waypoints` is issued +Then response is `404` with envelope `{ statusCode:404, message: }` + +**AC-8: FT-N-08 generic 500 redacts body, stacktrace lands in log** +Given side-channel has executed `DROP TABLE vehicles CASCADE` +When `GET /vehicles/{any uuid}` is issued with JWT `FL` +Then response is `500` with body EXACTLY `{ "statusCode":500, "message":"Internal server error" }` +And `docker logs missions-sut` contains an `"Unhandled exception"` line emitted ≤ 2s after the request timestamp, containing the exception type name (`PostgresException` or similar) + +## Non-Functional Requirements + +**Performance** +- FT-N-01..05, FT-N-07: ≤ 2s each. FT-N-06: ≤ 5s. FT-N-08: ≤ 5s (allow log scrape). + +**Reliability** +- FT-N-06 requires `postgres-test` to be started with `log_statement=all` (`command: ["postgres", "-c", "log_statement=all"]` overlay in `docker-compose.test.yml`, OR `ALTER SYSTEM SET` via side-channel in the fixture). The test must FAIL if logging is not enabled — not silently pass. +- FT-N-08 is destructive (drops the `vehicles` table). It MUST run in its own xUnit `[Collection("ErrorEnvelope500")]` with `ComposeRestartFixture` teardown (full `down -v && up -d`). + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `seed_3_vehicles_2_default` | `?name=ZZ` then `?name=zz` | `200` + `body.length == 0` for both | AC-1.6 | +| AC-2 | any | `GET /vehicles/{random}` | `404` + envelope | AC-1.7, AC-8.2 | +| AC-3 | Vehicle + mission referencing it | `DELETE /vehicles/{id}` | `409` + row preserved | AC-1.8, AC-8.5 | +| AC-4 | `seed_empty` | `POST /missions { VehicleId: }` | `400` (today) + no row written + carry-forward comment | AC-2.2 | +| AC-5 | any | `GET /missions/{random}` | `404` + envelope | AC-2.4, AC-8.2 | +| AC-6 | `fixture_cascade_F3` + PG logging on | `DELETE /missions/{random}` | `404` + zero dependency-table DELETE SQL | AC-3.2 | +| AC-7 | any | `GET /missions/{random}/waypoints` | `404` + envelope | AC-4.2 | +| AC-8 | side-channel DROPped vehicles | `GET /vehicles/{any}` | `500` + redacted body + stacktrace logged within 2s | AC-8.6, AC-10.3 | + +## Constraints + +- HTTP only against `http://missions:8080`; bearer token via `https://jwks-mock:8443/sign` with `permissions=FL`. +- FT-N-06 requires Postgres logging mode `log_statement=all`; the fixture must verify (via `SHOW log_statement`) that logging is on BEFORE running the test — fail in Arrange if not. +- FT-N-08 fixture teardown must restart the compose stack (`down -v && up -d`); subsequent tests would otherwise hit a missing table. +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. +- Carry-forward comments (FT-N-04) are required so future spec-vs-code work knows where to update. + +## Risks & Mitigation + +**Risk 1: FT-N-06 false-pass when PG logging is off** +- *Risk*: If `postgres-test` runs without `log_statement=all`, the "no DELETE issued" assertion trivially passes — the log is empty. +- *Mitigation*: Arrange phase runs `SHOW log_statement` via side-channel and fails fast if the result is not `"all"`. The compose overlay setting this MUST be loaded. + +**Risk 2: FT-N-08 leaves the SUT in a broken state** +- *Risk*: After `DROP TABLE vehicles CASCADE`, every subsequent test against `/vehicles` returns 500 until the migrator re-creates the table on next startup. +- *Mitigation*: Fixture runs `docker compose -f docker-compose.test.yml down -v && up -d` in teardown; subsequent tests wait for `missions` to reach `healthy`. + +**Risk 3: FT-N-04 expectation flips silently when spec divergence closes** +- *Risk*: When the spec-aligned 404 lands, this test will fail with a status mismatch — and the test author needs context to know it's intentional. +- *Mitigation*: The test includes a `// CARRY-FORWARD: AC-2.2 — expected to flip to 404 when bogus-VehicleId divergence is closed` source-level comment AND `[Trait("carry_forward", "AC-2.2")]` so a future filter can find it. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080/{vehicles,missions}*`) plus the documented DB side-channel for fixture seeding, post-call assertions, and (for FT-N-06) reading `pg_stat_statements` / Postgres log lines, and (for FT-N-08) reading `docker logs missions-sut`. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.7, 1.8, 1.9; AC-2 2.2, 2.6; AC-3 3.2; AC-4 4.1; AC-8 8.7; AC-10 10.1. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `VehicleService`, `MissionService`, `WaypointService`, the controllers, `ErrorHandlingMiddleware`, `AppDataConnection`, `DatabaseMigrator`, or `JwtExtensions`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-581_test_security_auth_claims.md b/_docs/tasks/todo/AZ-581_test_security_auth_claims.md new file mode 100644 index 0000000..484ac66 --- /dev/null +++ b/_docs/tasks/todo/AZ-581_test_security_auth_claims.md @@ -0,0 +1,125 @@ +# Security Tests — Auth & Claims + +**Task**: AZ-581_test_security_auth_claims +**Name**: Security tests — auth & claims (NFT-SEC-01..06 + 04b) +**Description**: Implement xUnit blackbox tests for the 7 JWT authn/authz scenarios — missing/invalid header, invalid signature (single-byte flip + foreign-keypair), expired-outside-skew vs inside-30s-skew, wrong `iss`, wrong `aud`, missing `permissions`, wrong/multi-value `permissions` claim (contains-match accepts `["FL","ADMIN"]`). +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-581 +**Epic**: AZ-575 + +## Problem + +JWT validation is the only thing standing between the open `e2e-net` and the protected `/vehicles` + `/missions` + `/missions/{id}/waypoints` surface. Six failure modes (no header / bad signature / expired / wrong iss / wrong aud / wrong perm) MUST all produce `401` or `403` deterministically — any drift means an attacker who learns the JWKS public bytes could shape a token that bypasses one rule and rides through. The drift re-verification of 2026-05-14 split AC-5.3 into two checks (`iss` AND `aud`) and tightened the clock skew from .NET's 5-min default to 30s; this task pins both. NFT-SEC-06 specifically asserts the `RequireClaim("permissions","FL")` is contains-match — a multi-permission token `["FL","ADMIN"]` must be accepted, while `"fl"` / `"FLight"` / `"ADMIN"` alone must be rejected. + +## Outcome + +- All seven NFT-SEC-01..06 + 04b scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=Sec`, `Traces=AC-5.x` or `AC-9.x`, `Result=pass`. +- NFT-SEC-02 covers BOTH the single-byte-flip case AND the foreign-keypair case (token signed by a separate ECDSA keypair never published in the JWKS). +- NFT-SEC-03 verifies the 30s skew BOTH ways — `exp_offset_seconds=-60` rejected, `exp_offset_seconds=-15` accepted. +- NFT-SEC-06 verifies multi-permission token acceptance — `permissions: ["FL","ADMIN"]` → `200`. +- NFT-SEC-01 asserts no DB side-effect on the `POST /vehicles` 401 path (side-channel count unchanged). + +## Scope + +### Included + +- NFT-SEC-01 Missing `Authorization` header on `/vehicles` GET/POST, `/missions` GET, `/missions/{any}/waypoints` GET — all `401`, no DB row written on the POST. +- NFT-SEC-02 Invalid signature — single-byte-flipped signature segment AND foreign-keypair tokens. +- NFT-SEC-03 Expired token — `exp_offset_seconds=-60` → `401`; `exp_offset_seconds=-15` → `200` (inside 30s skew). +- NFT-SEC-04 Wrong `iss` — `POST /sign { "iss": "https://attacker.example.com" }` → `401`; default `iss` → `200`. +- NFT-SEC-04b Wrong `aud` — `POST /sign { "aud": "wrong-audience" }` → `401`. +- NFT-SEC-05 Missing `permissions` claim — `403`. +- NFT-SEC-06 Wrong `permissions` value AND multi-permission acceptance — `"fl"`, `"FLight"`, `"ADMIN"` → `403`; `["FL","ADMIN"]` → `200`. + +### Excluded + +- NFT-SEC-07 health-exempt-from-auth lives in Task 15. +- NFT-SEC-08 stacktrace-not-leaked overlaps with FT-N-08 in Task 13 (and lives in Task 15 for the security-shaped variant). +- NFT-SEC-09 SQL injection guard lives in Task 15. +- NFT-SEC-10 alg-pin lives in Task 15. +- NFT-SEC-11 unknown-kid rotation lag lives in Task 15. +- NFT-SEC-12 missing-env startup throw lives in Task 15. +- NFT-SEC-13 CORS Production-gate lives in Task 15. + +## Acceptance Criteria + +**AC-1: NFT-SEC-01 missing header rejects every protected endpoint with 401, no side-effect** +Given the running test stack +When the consumer issues `GET /vehicles`, `GET /missions`, `GET /missions/{any}/waypoints`, and `POST /vehicles` with a valid body — all without an `Authorization` header +Then each response is `401`, AND side-channel `SELECT COUNT(*) FROM vehicles` before and after the `POST` are equal + +**AC-2: NFT-SEC-02 invalid signature rejects two attack shapes** +Given a valid signed token `T_good` from `jwks-mock POST /sign` +When the consumer flips a single byte in `T_good`'s signature segment producing `T_bad`, and separately mints `T_foreign` signed by an ECDSA keypair never published in the JWKS +Then `GET /vehicles` with `T_bad` returns `401` AND `GET /vehicles` with `T_foreign` returns `401` + +**AC-3: NFT-SEC-03 30s clock skew is enforced on both sides** +Given the mock with default issuer/audience +When the consumer mints two tokens via `POST /sign { exp_offset_seconds: -60 }` and `POST /sign { exp_offset_seconds: -15 }` +Then `GET /vehicles` with the −60s token returns `401` AND `GET /vehicles` with the −15s token returns `200` + +**AC-4: NFT-SEC-04 wrong `iss` rejected, matching `iss` accepted** +When the consumer mints a token via `POST /sign { iss: "https://attacker.example.com" }` and another via `POST /sign {}` (default iss) +Then `GET /vehicles` with the attacker-iss token returns `401` AND with the default-iss token returns `200` + +**AC-5: NFT-SEC-04b wrong `aud` rejected** +When the consumer mints a token via `POST /sign { aud: "wrong-audience" }` +Then `GET /vehicles` returns `401` + +**AC-6: NFT-SEC-05 missing `permissions` claim rejected with 403** +When the consumer mints a token with no `permissions` claim (mock body `{ permissions: "" }` or `{ permissions: null }` per the mock's contract) +Then `GET /vehicles` returns `403` (NOT 401 — signature is valid) + +**AC-7: NFT-SEC-06 contains-match policy on `permissions`** +When the consumer mints tokens with `permissions` values `"ADMIN"`, `"fl"` (lowercase), `"FLight"`, AND `["FL","ADMIN"]` (multi-value array) +Then `GET /vehicles` returns `403` for the first three AND `200` for the multi-value `["FL","ADMIN"]` array (contains-match accepts `"FL"` among the values) + +## Non-Functional Requirements + +**Performance** +- NFT-SEC-01..06: ≤ 5s each. The Authorization-header failure paths are cheap (no DB round-trip on the 401/403 short-circuit). + +**Reliability** +- NFT-SEC-02 requires an out-of-band ECDSA-keypair helper that lives inside the test project, NOT in `jwks-mock` (the mock must never publish a public key it does not control). The helper generates a P-256 keypair at test-start and signs a token directly using `System.Security.Cryptography.ECDsa` — the public key is never registered with `missions`. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | running stack | 4 endpoints w/o Authorization | all 401; POST no DB write | AC-5.4 | +| AC-2 | `T_good` from mock + foreign keypair | flipped signature; foreign-keypair token | both 401 | AC-5.5 | +| AC-3 | mock with default iss/aud | exp_offset −60s vs −15s | 401 / 200 | AC-5.2, AC-5.6 | +| AC-4 | mock | iss=attacker vs default | 401 / 200 | AC-5.3, AC-5.11 | +| AC-5 | mock | aud=wrong | 401 | AC-5.3, AC-5.12 | +| AC-6 | mock | permissions missing | 403 | AC-5.8, AC-9.1 | +| AC-7 | mock | permissions=ADMIN/fl/FLight/["FL","ADMIN"] | 403/403/403/200 | AC-9.1, AC-9.2 | + +## Constraints + +- HTTP only against `http://missions:8080`. Tokens minted via `https://jwks-mock:8443/sign` with parameterised overrides. +- NFT-SEC-02 foreign-keypair: a test-only helper inside `Azaion.Missions.E2E.Tests` MAY use `System.Security.Cryptography.ECDsa` directly for the attack-token construction; this is the ONLY in-test signing path allowed — every other test must use the mock. +- NFT-SEC-06 multi-permission token requires the mock's `POST /sign` body to accept `permissions` as either a string OR a JSON array; the test-infrastructure ticket (AZ-576) covers this in the mock's contract. +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: NFT-SEC-03 flaky due to wall-clock variability** +- *Risk*: A −15s offset could fail if Docker time skew between the mock and `missions` is large. +- *Mitigation*: Both containers run on the same host clock (no `--init` time isolation); test asserts only at offsets well clear of the 30s boundary (−60s and −15s — 30s and 15s away from the boundary respectively). + +**Risk 2: NFT-SEC-06 multi-permission shape varies between systems** +- *Risk*: If the spec for `permissions` claim later changes from "contains-match string" to "exact-array-membership", the multi-value assertion breaks. +- *Mitigation*: Test traces explicitly to AC-9.2 and references `Auth/JwtExtensions.cs` policy registration; any change there must update this test in the same commit. + +**Risk 3: Foreign-keypair token validation might pass if the SUT silently trusts any well-formed ECDSA token** +- *Risk*: A regression that disables `IssuerSigningKeyResolver` would let the foreign-keypair token through. +- *Mitigation*: Mitigated by the structure of AC-2 — both bad-signature shapes (flipped byte AND foreign keypair) must return 401. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080/{vehicles,missions}*`) and acquire signed tokens via `https://jwks-mock:8443/sign` (with the test-only foreign-keypair helper for NFT-SEC-02). Expected outputs are the documented HTTP status codes from `_docs/00_problem/input_data/expected_results/results_report.md` AC-5 rows and AC-9 rows. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs` (auth pipeline registration), the `[Authorize(Policy = "FL")]` filter, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-582_test_security_alg_rotation_cors.md b/_docs/tasks/todo/AZ-582_test_security_alg_rotation_cors.md new file mode 100644 index 0000000..56721c4 --- /dev/null +++ b/_docs/tasks/todo/AZ-582_test_security_alg_rotation_cors.md @@ -0,0 +1,140 @@ +# Security Tests — Alg-pin / Rotation / CORS / No-leak + +**Task**: AZ-582_test_security_alg_rotation_cors +**Name**: Security tests — alg-pin, rotation, CORS, no-leak (NFT-SEC-07..13) +**Description**: Implement xUnit blackbox tests for the 7 cross-cutting security scenarios — health endpoint anonymous-OK (NFT-SEC-07), 500 redacted body shape (NFT-SEC-08), SQL-injection guard via parameterised queries (NFT-SEC-09), algorithm-pin defends against HS256-confusion and unsigned tokens (NFT-SEC-10), unknown-`kid` rotation lag with old-key grace window (NFT-SEC-11), startup fail-fast on missing required env vars + HTTPS-only JWKS URL (NFT-SEC-12), and CORS Production-gate fail-fast + permissive-default-warning in non-Production (NFT-SEC-13). +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-582 +**Epic**: AZ-575 + +## Problem + +Six of these scenarios pin invariants that were broken in earlier code paths and structurally fixed during the 2026-05-14 drift cycle. NFT-SEC-10 (alg-pin) defends against the most common JWKS-public-key-as-HMAC-secret attack. NFT-SEC-11 (kid rotation) verifies that the test-infrastructure JWKS cache shortening (C01) actually shrinks rotation lag inside the 15-minute CI gate. NFT-SEC-12 verifies all four `Infrastructure/ConfigurationResolver.ResolveRequiredOrThrow` calls — `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL`. NFT-SEC-13 verifies `CorsConfigurationValidator.EnsureSafeForEnvironment` actually throws on `ASPNETCORE_ENVIRONMENT=Production` with empty allow-list, AND falls back to permissive with a warning log in `Test`/`Development`. Each is a separate failure mode; together they form the "static config and cryptographic posture" surface that nothing else in the suite covers. + +## Outcome + +- All seven NFT-SEC-07..13 scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=Sec`, `Traces=AC-5.x`/`AC-6.x`/`AC-7.x`/`AC-8.x`/`AC-9.x`/`AC-10.x`, `Result=pass`. +- NFT-SEC-10 covers BOTH HS256-confusion (mock signs with the public key as HMAC secret) AND `alg: none` (mock emits unsigned JWT) — both must return `401`. +- NFT-SEC-11 (rotation lag) completes inside 120s and exercises the three windows: cached-misses-new-kid → 401, cache-refreshed → 200, old-kid-still-valid-during-grace → 200, post-grace-old-kid → mock refuses to sign. +- NFT-SEC-12 runs five separate `docker run` invocations (four missing-env + one HTTP-not-HTTPS JWKS URL); each asserts non-zero exit / log line. +- NFT-SEC-13 runs five separate `docker run` invocations spanning Production-fail-fast, Production-AllowAny-warning, Production-with-origins, Production-cross-origin-rejection, Test-permissive-warning. + +## Scope + +### Included + +- NFT-SEC-07 Health endpoint anonymous + accepted with expired token (auth pipeline not evaluated). +- NFT-SEC-08 500 redacted body — no `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/type-name in body; log has the stack info. +- NFT-SEC-09 SQL-injection guard — `?name=' OR '1'='1` and `?name=; DROP TABLE vehicles; --` are treated as literal strings. +- NFT-SEC-10 Alg-pin — HS256-confusion AND unsigned token both rejected. +- NFT-SEC-11 Unknown-kid rotation lag with old-key grace window. +- NFT-SEC-12 Missing required env vars (4 vars) + HTTP-JWKS-URL warning path. +- NFT-SEC-13 CORS Production-gate fail-fast + AllowAnyOrigin warning + explicit-origin preflight + cross-origin preflight rejection + non-Production permissive-default warning. + +### Excluded + +- The 401/403 auth pipeline (NFT-SEC-01..06 + 04b) lives in Task 14. +- The destructive `DROP TABLE` mid-test for the 500 path (FT-N-08) lives in Task 13. NFT-SEC-08 here REUSES the same fixture but adds the response-body redaction assertions. + +## Acceptance Criteria + +**AC-1: NFT-SEC-07 health is anonymous and skips the auth pipeline** +When `GET /health` is issued (a) with no `Authorization` header AND (b) with `Authorization: Bearer ` +Then both responses are `200` with body `{ "status": "healthy" }` — proving the auth pipeline does not run for `/health` + +**AC-2: NFT-SEC-08 500 redacted body** +Given the same fixture as FT-N-08 (`DROP TABLE vehicles CASCADE`) +When `GET /vehicles/{any uuid}` is issued +Then response body is EXACTLY `{ "statusCode":500, "message":"Internal server error" }`, contains NO key matching `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/exception-type-name +And `docker logs missions-sut` contains an `Unhandled exception` line including the exception type or file path of the throw site + +**AC-3: NFT-SEC-09 SQL-injection guard** +Given a running stack with `seed_3_vehicles_2_default` +When `GET /vehicles?name=' OR '1'='1` (URL-encoded) is issued +Then response is `200` with `body.length == 0` (the literal string does not match any `Name`) +And when `GET /missions?name=; DROP TABLE vehicles; --` (URL-encoded) is issued +Then response is `200` with `body.TotalCount == 0` AND side-channel `SELECT to_regclass('vehicles')` returns a non-null oid (the table still exists) + +**AC-4: NFT-SEC-10 algorithm-pin rejects HS256-confusion and unsigned** +When the consumer mints a token via `POST /sign { alg_override: "HS256" }` (mock signs with the JWKS public key as HMAC secret) +Then `GET /vehicles` returns `401` +And when the consumer mints a token via `POST /sign { alg_override: "none" }` (unsigned JWT) +Then `GET /vehicles` returns `401` + +**AC-5: NFT-SEC-11 unknown-kid rotation completes within 120s with grace window honoured** +Given `missions` has a warm JWKS cache and `jwks-mock` is configured with `OLD_KEY_GRACE_SECONDS=5` +When the consumer issues `POST jwks-mock:8443/rotate-key {}`, immediately mints a token signed with the new kid, and calls `GET /vehicles` BEFORE missions has refreshed +Then the first call returns `401` (new kid not yet in cache) +And after waiting for the JWKS refresh window (≤ 90s; the mock sets `max-age=60` and missions has `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30` per C01), the same token returns `200` +And during the 5s grace window, a token still signed with the OLD kid is accepted (`200`) +And after the grace window expires, the mock refuses to sign with the old kid (`400`/`410` from `POST /sign`) + +**AC-6: NFT-SEC-12 startup fail-fast on required env vars + HTTPS-only JWKS** +When `missions` is launched via separate `docker run` invocations, each missing exactly one of `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL` (4 cases) +Then in each case the container exits non-zero within 5s AND its logs contain `InvalidOperationException` mentioning the corresponding variable (or its `Database:Url`/`Jwt:Issuer`/`Jwt:Audience`/`Jwt:JwksUrl` config alias) +And when `missions` is launched with `JWT_JWKS_URL=http://jwks-mock:8443/...` (HTTP not HTTPS) and the other three set +Then the container STARTS, AND the first protected request fails (`500` body or `401` with `RequireHttps` mention) AND the log contains a line mentioning `HTTPS` / `RequireHttps` + +**AC-7: NFT-SEC-13 CORS Production-gate fail-fast + non-Production warning** +When `missions` is launched with `ASPNETCORE_ENVIRONMENT=Production` and no `CorsConfig` env vars +Then the container exits non-zero within 5s AND its logs contain `InvalidOperationException` mentioning `CorsConfig`/`AllowedOrigins`/Production +And when launched with `ASPNETCORE_ENVIRONMENT=Production` + `CorsConfig__AllowAnyOrigin=true` +Then the container starts AND the logs contain a warning that CORS is permissive in Production +And when launched with `ASPNETCORE_ENVIRONMENT=Production` + `CorsConfig__AllowedOrigins__0=https://operator.example.com` +Then `OPTIONS /vehicles` preflight from `https://operator.example.com` returns `200` with `Access-Control-Allow-Origin: https://operator.example.com` +And the same preflight from `https://attacker.example.com` responds without the allow-origin echo +And when launched with `ASPNETCORE_ENVIRONMENT=Test` and no `CorsConfig`, the container starts AND the logs contain the documented `PermissiveDefaultWarning` + +## Non-Functional Requirements + +**Performance** +- NFT-SEC-07..10: ≤ 5s each. +- NFT-SEC-11: ≤ 120s (rotation + cache refresh). +- NFT-SEC-12: ≤ 60s (5 docker-run cycles). +- NFT-SEC-13: ≤ 90s (5 docker-run cycles + preflight requests). + +**Reliability** +- NFT-SEC-11 must run in its own xUnit `[Collection("JwksRotation")]` because rotating the mock affects every subsequent test that already has tokens in flight. After the test, the fixture restores the original key by calling `POST /rotate-key` once more and waits the grace window. +- NFT-SEC-12 and NFT-SEC-13 spawn `docker run` from inside the test runner — the runner container must have access to a Docker socket OR the suite-level test orchestrator must run these as separate compose profiles. AZ-576 covers the runner-side Docker access. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | running stack | `GET /health` no-auth and with expired token | both 200 | AC-7.1, AC-9.4 | +| AC-2 | dropped `vehicles` table | `GET /vehicles/{any}` | 500 + body has only `statusCode,message` + log has stacktrace | AC-8.6, AC-10.3 | +| AC-3 | `seed_3_vehicles_2_default` | `?name=' OR '1'='1` then `?name=; DROP TABLE…` | 200 + len 0 + table still exists | AC-1.6, AC-2.3 defensive | +| AC-4 | mock with alg overrides | HS256-confusion token then unsigned token | both 401 | AC-5.1, AC-5.10 | +| AC-5 | warm JWKS cache | `POST /rotate-key` + 3 timing checks | 401 → wait → 200; old-kid grace; post-grace mock refuses | AC-5.7 | +| AC-6 | 5 docker-run cases | missing DATABASE_URL/JWT_ISSUER/JWT_AUDIENCE/JWT_JWKS_URL + HTTP-not-HTTPS | 4 fail-fast + 1 start-then-500 | AC-6.1, AC-6.2, E1, E3 | +| AC-7 | 5 docker-run cases | Production fail-fast, AllowAnyOrigin warn, explicit-origin allow, cross-origin reject, Test permissive warn | per scenario | AC-6.11, E9 | + +## Constraints + +- HTTP only against `http://missions:8080` for the cases that run inside the standard compose stack. NFT-SEC-12 and NFT-SEC-13 use `docker run` directly against `azaion/missions:test`. +- NFT-SEC-09 second probe (`SELECT to_regclass('vehicles')`) requires side-channel Npgsql access AFTER the SUT response — if the table was dropped, the test was wrong. +- NFT-SEC-11 fixture must restore the original key before exit (otherwise every test in subsequent collections fails with `kid` mismatch). +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: NFT-SEC-10 false-pass if the mock cannot produce an HS256 token** +- *Risk*: If the mock implementation rejects `alg_override="HS256"`, the test never exercises the attack — it gets `400` from the mock and incorrectly thinks `missions` rejected. +- *Mitigation*: The test asserts a successful `200 OK` from `jwks-mock POST /sign` BEFORE issuing `GET /vehicles`; mock failure fails Arrange, not Assert. + +**Risk 2: NFT-SEC-11 flake on slow CI** +- *Risk*: The 60s `max-age` + 30s `AutoRefresh` + clock variance might push refresh past 120s on a heavily loaded runner. +- *Mitigation*: The test polls every 5s for ≤ 120s; if no transition by 120s, fails with a clear "rotation not observed inside the budget" message. The 120s budget already includes margin per `environment.md` § CI gate. + +**Risk 3: NFT-SEC-13 cross-origin preflight assertion misreads CORS header presence** +- *Risk*: ASP.NET Core's CORS middleware returns `200` for OPTIONS even when origin is disallowed, just without the allow-origin header. A loose assertion would miss the rejection. +- *Mitigation*: Test asserts `Access-Control-Allow-Origin` header EXACTLY: present and matching the allowed origin in the allow case; absent (header == null) in the reject case. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface and verify startup behaviour via `docker run` and `docker logs missions-sut` scrape. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-5 (NFT-SEC-10/11), AC-6 (NFT-SEC-12/13), AC-7 (NFT-SEC-07), AC-8 (NFT-SEC-08), AC-9 (NFT-SEC-07), AC-10 (NFT-SEC-08). +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs` (config resolution + CORS + auth pipeline), `Infrastructure/ConfigurationResolver`, `Infrastructure/CorsConfigurationValidator`, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-583_test_resilience_cascade_migrator.md b/_docs/tasks/todo/AZ-583_test_resilience_cascade_migrator.md new file mode 100644 index 0000000..290305e --- /dev/null +++ b/_docs/tasks/todo/AZ-583_test_resilience_cascade_migrator.md @@ -0,0 +1,115 @@ +# Resilience Tests — Cascade + Migrator + +**Task**: AZ-583_test_resilience_cascade_migrator +**Name**: Resilience tests — cascade + migrator (NFT-RES-01..04) +**Description**: Implement xUnit blackbox tests for the 4 cascade and migrator resilience scenarios — mission cascade NOT transaction-wrapped (partial deletes survive mid-walk failure; AC-3.3 / ADR-006 carry-forward), waypoint cascade same invariant (AC-4.6), migrator idempotent on container restart (AC-6.6), and the B9 one-shot legacy table drop is destructive on first run + idempotent on subsequent restarts (AC-6.5, AC-10.5). +**Complexity**: 3 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-583 +**Epic**: AZ-575 + +## Problem + +The cascade tests encode TWO documented carry-forwards — the F3 (mission) and F4 (waypoint) cascades are NOT transaction-wrapped, so when the walk fails mid-way (e.g., `media` table absent), the rows deleted BEFORE the failure stay deleted while the rows deleted AFTER do not. This is documented under ADR-006 and AC-3.3 / AC-3.4 / AC-4.6 / AC-10.2 as deferred work. The tests intentionally pin the current behaviour so a future transaction-wrap change is caught loudly. The migrator tests pin two operational invariants needed for blue-green / restart-during-deploy patterns: NFT-RES-03 verifies a vanilla restart is a no-op, and NFT-RES-04 verifies the post-B9 `DROP TABLE IF EXISTS orthophotos/gps_corrections` block runs once and is idempotent thereafter. + +## Outcome + +- All four NFT-RES-01..04 scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=Res`, `Traces=AC-3.3` / `AC-4.6` / `AC-6.6` / `AC-6.5`, `Result=pass`. +- NFT-RES-01 and NFT-RES-02 assert BOTH the partial-state observation (some rows deleted, some not) AND the 500 response shape (envelope keys, no leak) — fail loudly when a future transaction wrap rolls everything back. +- NFT-RES-03 asserts no NEW error log lines appear after the restart timestamp (not just "any error", which would conflate pre-existing startup-time warnings). +- NFT-RES-04 includes a build-time / source-inspection gate so it only meaningfully runs on a post-B9 build (B9 landed locally 2026-05-15 — verified via `_docs/_process_leftovers/2026-05-14_rename-flights-to-missions.md`). + +## Scope + +### Included + +- NFT-RES-01 Cascade NOT transaction-wrapped (mission, F3) — `DROP TABLE media CASCADE` before request; `500` response; `map_objects` count `0` (committed); `missions` count `1` (uncommitted). +- NFT-RES-02 Cascade NOT transaction-wrapped (waypoint, F4) — same shape against F4 fixture. +- NFT-RES-03 Idempotent migrator on restart — `docker compose restart missions`; no NEW error log lines; schema unchanged. +- NFT-RES-04 B9 one-shot legacy drop — `seed_legacy_gps_tables` precondition; on first start `orthophotos` + `gps_corrections` are dropped; subsequent restart is no-op. + +### Excluded + +- NFT-RES-05 Required config missing → fail-fast (4 docker-run cases + DB-unreachable) lives in Task 17. +- NFT-RES-06 DB does not exist (Npgsql 3D000) lives in Task 17. +- NFT-RES-07 JWKS rotation lives in Task 17 (NOTE: also touched by NFT-SEC-11 in Task 15 from a security angle; this resilience variant focuses on the no-restart operational property). +- NFT-RES-08 TOCTOU on default-vehicle exclusivity lives in Task 17. + +## Acceptance Criteria + +**AC-1: NFT-RES-01 mission cascade partial-state survives mid-walk failure** +Given `fixture_cascade_F3` applied to a running stack +When the side-channel executes `DROP TABLE media CASCADE` THEN the consumer issues `DELETE /missions/{mid}` with JWT `FL` +Then the response is `500` with envelope `{ statusCode:500, message:"Internal server error" }` +And side-channel `SELECT COUNT(*) FROM map_objects WHERE mission_id={mid}` returns `0` (committed before the failure) +And side-channel `SELECT COUNT(*) FROM missions WHERE id={mid}` returns `1` (uncommitted after the failure) +And `docker logs missions-sut` contains an `Unhandled exception` line mentioning `relation` and `media` within 2s of the request + +**AC-2: NFT-RES-02 waypoint cascade partial-state same invariant** +Given `fixture_cascade_F4` applied +When the side-channel executes `DROP TABLE media CASCADE` THEN the consumer issues `DELETE /missions/{mid}/waypoints/{wp1}` +Then the response is `500` +And side-channel `SELECT COUNT(*) FROM detection WHERE annotation_id IN (wp1 chain)` returns `0` +And side-channel `SELECT COUNT(*) FROM waypoints WHERE id={wp1}` returns `1` + +**AC-3: NFT-RES-03 migrator is idempotent on restart** +Given `missions` has been started once (schema migrated; `seed_empty` state) +When `docker compose -f docker-compose.test.yml restart missions` is invoked AND health returns 200 within 30s +Then `docker logs missions-sut` since the restart timestamp contains NO new lines matching `(error|Error|exception)` +And the side-channel `\d+ vehicles` table description is unchanged from the post-first-start state + +**AC-4: NFT-RES-04 B9 one-shot legacy drop is destructive then idempotent** +Given `seed_legacy_gps_tables` (legacy `orthophotos` + `gps_corrections` present), `missions` not yet started for this scenario, AND the build is post-B9 (verified via `to_regclass` or source inspection of `DatabaseMigrator.cs`) +When `docker compose up -d missions` is invoked and health returns 200 +Then side-channel `SELECT to_regclass('orthophotos'), to_regclass('gps_corrections')` returns both NULL (tables dropped) +And when `docker compose restart missions` is invoked and health returns 200 again +Then side-channel queries still return both NULL, AND `docker logs missions-sut` since the restart contains NO `does not exist` line (the `IF EXISTS` suppresses the no-op error) + +## Non-Functional Requirements + +**Performance** +- NFT-RES-01..02: ≤ 10s each (cascade walk + fault injection setup). +- NFT-RES-03..04: ≤ 60s each (container restart + health poll). + +**Reliability** +- NFT-RES-01 and NFT-RES-02 are destructive (drop `media` table); each runs in its own xUnit `[Collection("ResCascadeF3")]` / `[Collection("ResCascadeF4")]` with `ComposeRestartFixture` teardown (full `down -v && up -d`). +- NFT-RES-04 has a build-time gate: the test queries the migrator source (or checks if the legacy tables exist after start) and SKIPS with a recorded reason on pre-B9 builds. Skipped rows appear in the CSV report with `Result=skip` and a clear `ErrorMessage` field. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `fixture_cascade_F3` + DROP `media` | `DELETE /missions/{mid}` | 500 + map_objects=0 + missions=1 + log mentions `media` | AC-3.3, AC-10.2 | +| AC-2 | `fixture_cascade_F4` + DROP `media` | `DELETE /missions/{mid}/waypoints/{wp1}` | 500 + detection=0 + wp1=1 | AC-4.6, AC-3.3 | +| AC-3 | post-first-start `seed_empty` | `docker compose restart missions` | health back in 30s + no new error logs + schema unchanged | AC-6.6, AC-6.4 | +| AC-4 | `seed_legacy_gps_tables` + post-B9 build | first start + restart | first drops legacy tables; restart is no-op (no error log) | AC-6.5, AC-10.5 | + +## Constraints + +- HTTP only against `http://missions:8080` for the cascade requests; side-channel Npgsql for fixture state + post-state assertions. +- NFT-RES-01..02 use the same `fixture_cascade_F3.sql` / `fixture_cascade_F4.sql` from Tasks 11/12; do NOT re-author seed SQL. +- NFT-RES-03..04 use `docker compose` from inside the runner (Docker-socket-mounted) OR from the suite orchestrator — AZ-576 covers this. +- NFT-RES-04 must verify B9 has landed before running; otherwise SKIP with a clear reason (record in CSV). +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: NFT-RES-01/02 false-pass when transaction wrap lands** +- *Risk*: A future ADR-006 closure wraps the cascade in a transaction; `map_objects` count becomes `> 0` (rolled back) and `missions` count stays `1`. The test would interpret this as a failure of the partial-state invariant — but that failure means the system is BETTER. +- *Mitigation*: Both tests include a source-level comment `// CARRY-FORWARD: AC-3.3 / ADR-006 — flip assertions when transaction wrap lands` and `[Trait("carry_forward","ADR-006")]` so a future filter finds them. + +**Risk 2: NFT-RES-03 false-pass when restart-time errors are tolerated** +- *Risk*: A simple `docker logs | grep -i error` over the entire log returns the migrator's pre-existing warnings. +- *Mitigation*: The test captures `docker logs missions-sut --since=` and greps from THAT slice only. + +**Risk 3: NFT-RES-04 incorrectly runs on a pre-B9 build** +- *Risk*: If the build-time gate is silently bypassed, the test asserts dropping the legacy tables — which would never happen, and the test would fail with a misleading message. +- *Mitigation*: The gate checks BOTH the migrator source for the `DROP TABLE IF EXISTS orthophotos` line AND verifies the legacy tables are present in the seed BEFORE the SUT starts. If either check fails, the test SKIPS with `Result=skip` and a clear `ErrorMessage`. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface plus container orchestration (`docker compose restart`, `docker compose up -d`) and `docker logs missions-sut` scrape. Side-channel Npgsql for fixture state and post-state assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-3 3.3, AC-4 4.6, AC-6 6.4-6.6, AC-10 10.2/10.5. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `MissionService`, `WaypointService`, `MissionsController`, `Database/DatabaseMigrator`, `ErrorHandlingMiddleware`, or `AppDataConnection`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-584_test_resilience_config_db_rotation_race.md b/_docs/tasks/todo/AZ-584_test_resilience_config_db_rotation_race.md new file mode 100644 index 0000000..d98f29c --- /dev/null +++ b/_docs/tasks/todo/AZ-584_test_resilience_config_db_rotation_race.md @@ -0,0 +1,120 @@ +# Resilience Tests — Config / DB / JWKS Rotation / TOCTOU Race + +**Task**: AZ-584_test_resilience_config_db_rotation_race +**Name**: Resilience tests — config / DB / rotation / race (NFT-RES-05..08) +**Description**: Implement xUnit blackbox tests for the 4 resilience scenarios — startup fail-fast on missing required config (6 docker-run cases including the DB-unreachable differentiator), database missing → Npgsql 3D000 process exit, JWKS rotation propagates without `missions` restart, and TOCTOU race on default-vehicle exclusivity (probabilistic, expected to produce `default_count ≥ 2` in at least one iteration). +**Complexity**: 5 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-584 +**Epic**: AZ-575 + +## Problem + +These four scenarios pin the documented operational and concurrency posture of the service in places nothing else covers. NFT-RES-05 verifies BOTH the new fail-fast resolver path (rows 1–5: missing env vars throw `InvalidOperationException` BEFORE the HTTP server binds) AND the DB-down differentiator (row 6: config resolution succeeds, then Npgsql throws a recognisable connection error). NFT-RES-06 verifies the "database does not exist" case is observably different from "DB host unreachable" — Postgres returns SQLSTATE `3D000` and the container exits non-zero within 30s. NFT-RES-07 is the operational counterpart to NFT-SEC-11 — same JWKS rotation flow, but asserts the no-restart property (`docker inspect StartedAt` unchanged) instead of the kid-cache mechanics. NFT-RES-08 is intentionally probabilistic: it asserts the documented AC-1.4 race window EXISTS by running 100 parallel concurrent INSERTs and verifying that at least one iteration produces `is_default=true count ≥ 2`. + +## Outcome + +- All four NFT-RES-05..08 scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=Res`, `Traces=AC-6.1..2/AC-6.7..8/AC-5.7/AC-1.4`, `Result=pass`. +- NFT-RES-05 covers 6 cases — 4 missing-env (rows 1–4), 1 whitespace-only (`JWT_ISSUER=""`), and 1 DB-down-after-config-resolution (row 6 with `Connection refused`). +- NFT-RES-06 asserts the Postgres error code `3D000` appears in the container logs and the container exit code is non-zero within 30s. +- NFT-RES-07 asserts `docker inspect --format '{{.State.StartedAt}}' missions-sut` returns the SAME value before and after the rotation flow — the service did NOT restart. +- NFT-RES-08 records the observed `default_count ≥ 2` iteration count and includes `[Trait("Stability","probabilistic")]` so CI tolerates ≤ 1 failed run per 5. If 0 iterations produce the race, the test FAILS with a clear "race window closed — update AC-1.4 and rewrite this test" message. + +## Scope + +### Included + +- NFT-RES-05 6 docker-run cases (4 missing-env + 1 whitespace + 1 DB-down differentiator). +- NFT-RES-06 `DROP DATABASE azaion` → `docker compose up -d missions` → assert non-zero exit + `3D000` in logs. +- NFT-RES-07 JWKS rotation flow — `T1` works pre-rotation; `T2` rejected pre-cache-refresh; `T2` accepted post-refresh; `T1` eventually rejected post-grace; `missions` startup timestamp unchanged. +- NFT-RES-08 100 parallel `(POST /vehicles { IsDefault:true } || side-channel INSERT (..., is_default=true))` iterations; at least one produces `default_count ≥ 2`. + +### Excluded + +- NFT-SEC-11 (security-shaped variant of JWKS rotation) lives in Task 15. +- NFT-SEC-12 (security-shaped variant of startup fail-fast) lives in Task 15. NOTE: NFT-RES-05 and NFT-SEC-12 share 4 of 5 docker-run cases — the test infrastructure (AZ-576) provides a shared `MissionsContainerHelper` so both tasks can reuse the same docker-run primitive without duplicating logic. + +## Acceptance Criteria + +**AC-1: NFT-RES-05 startup fail-fast on missing required config + DB-down differentiator** +When `missions` is launched via 6 separate `docker run` invocations: +- (1) all 4 required env vars unset +- (2) `DATABASE_URL` unset, JWT vars set +- (3) `JWT_ISSUER=""` (whitespace-only), others set +- (4) `JWT_AUDIENCE` unset, others set +- (5) `JWT_JWKS_URL` unset, others set +- (6) all 4 vars set correctly, BUT `postgres-test` is stopped before `missions` starts +Then rows 1–5 → container exits non-zero within 5s, logs contain `InvalidOperationException`, logs mention the corresponding key (or its config alias) +And row 6 → container exits non-zero within 30s, logs contain a Npgsql `Connection refused` line (NOT an `InvalidOperationException` — proving config resolution succeeded BEFORE DB-connect failed) + +**AC-2: NFT-RES-06 database missing → process exits with Npgsql 3D000** +Given `postgres-test` running with the `azaion` database NOT yet created (or just dropped via side-channel) +When `docker compose -f docker-compose.test.yml up -d missions` is invoked +Then the container exits non-zero within 30s AND `docker logs missions-sut` contains at least one line matching `3D000` + +**AC-3: NFT-RES-07 JWKS rotation propagates without missions restart** +Given `missions` running with a warm JWKS cache, `jwks-mock` running with `OLD_KEY_GRACE_SECONDS=5` and `Cache-Control: max-age=60`, and Token `T1` minted with the current kid `kid_v1` +When `GET /vehicles` is issued with `T1` +Then response is `200` +And when `POST jwks-mock:8443/rotate-key {}` is invoked, `T2` is minted with `kid_v2`, and `GET /vehicles` is issued with `T2` BEFORE the JWKS cache refresh +Then response is `401` +And after waiting up to 90s for cache refresh (mock `max-age=60` + service `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30`), `GET /vehicles` with the same `T2` returns `200` +And `GET /vehicles` with `T1` (still has unexpired lifetime) returns `401` AFTER the grace window expires +And `docker inspect --format '{{.State.StartedAt}}' missions-sut` returns the SAME ISO-8601 timestamp before and after the entire rotation flow (the service did NOT restart) + +**AC-4: NFT-RES-08 TOCTOU race produces default_count ≥ 2 in at least one iteration** +Given `seed_one_default_vehicle` (default `P1`) +When the test runs 100 concurrent iterations, each issuing `POST /vehicles { IsDefault:true }` to the API in parallel with a side-channel `INSERT INTO vehicles (..., is_default=true)` +Then after all iterations complete, at least one iteration's post-state shows `SELECT COUNT(*) FROM vehicles WHERE is_default=true ≥ 2` +And if 0 iterations produce the race, the test FAILS with `"race window closed — update AC-1.4 carry-forward and rewrite this test"` (this is a structural test failure, not a flake) + +## Non-Functional Requirements + +**Performance** +- NFT-RES-05: ≤ 180s (6 docker-run cycles). +- NFT-RES-06: ≤ 60s (DROP DATABASE + docker-run + exit poll). +- NFT-RES-07: ≤ 180s (JWKS cache refresh window). +- NFT-RES-08: ≤ 30s (100 parallel iterations). + +**Reliability** +- NFT-RES-07 fixture MUST restore the original key by calling `POST /rotate-key` again at the end AND wait the grace window before yielding control — otherwise every subsequent test runs against an unfamiliar kid. +- NFT-RES-08 is probabilistic: `[Trait("Stability","probabilistic")]`. CI tolerates ≤ 1 failed run per 5 — but the structural failure mode ("race never observed in any iteration") still fails the suite. A deterministic-via-advisory-lock follow-up is recorded as a Refactor Backlog item. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `missions` not running | 6 docker-run cases | 5 fail-fast (InvalidOperationException) + 1 DB-down (Connection refused) | AC-6.1, AC-6.2, AC-6.7, E3, E4 | +| AC-2 | `DROP DATABASE azaion` | `docker compose up -d missions` | exit non-zero in 30s + log has `3D000` | AC-6.8 | +| AC-3 | warm JWKS cache + mock with grace=5/max-age=60 | rotate + 3 timing probes | T1→200; T2→401→wait→200; T1→401 post-grace; StartedAt unchanged | AC-5.7 | +| AC-4 | `seed_one_default_vehicle` | 100 parallel (POST + side-channel INSERT) | ≥ 1 iteration shows default_count ≥ 2 | AC-1.4 | + +## Constraints + +- HTTP only against `http://missions:8080` for the runtime cases; `docker run` and `docker compose` for the startup/DB cases. +- NFT-RES-05 row 6 (DB-down differentiator) is critical: the test must assert the log is `Connection refused`-shaped, NOT an `InvalidOperationException`. This rules out a regression where the resolver silently accepts an empty DB URL. +- NFT-RES-07 must clean up: rotate back to the original key in teardown AND wait `OLD_KEY_GRACE_SECONDS` so subsequent tests do not encounter a stale-kid edge case. +- NFT-RES-08 records the per-iteration timing and observed counts to the CSV report's `Traces` field for diagnosis. +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: NFT-RES-05 row 6 false-pass when config resolution silently accepts empty `DATABASE_URL`** +- *Risk*: A regression that returns an empty default for `DATABASE_URL` would make rows 2/6 indistinguishable — both would log a Npgsql error, but row 2 should log `InvalidOperationException` first. +- *Mitigation*: Test asserts row 2 logs the `InvalidOperationException` BEFORE any Npgsql output; row 6 logs Npgsql `Connection refused` directly without `InvalidOperationException`. Failure of either differentiator fails the test. + +**Risk 2: NFT-RES-07 flake on slow CI** +- *Risk*: Same as NFT-SEC-11 — slow refresh window. +- *Mitigation*: Same — poll every 5s for ≤ 90s; fail clearly if no transition observed in budget. + +**Risk 3: NFT-RES-08 deterministic-pass when race window closes** +- *Risk*: If a future TOCTOU fix lands (e.g., adding a `UNIQUE WHERE is_default=true` constraint), the test's "race observed" assertion fails — but the system is BETTER. +- *Mitigation*: Test failure message includes `"race window closed — update AC-1.4 carry-forward and rewrite this test"` so a future engineer knows the failure is expected and what to do. The test is gated by `[Trait("carry_forward","AC-1.4")]`. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface plus `docker run`, `docker compose`, `docker inspect`, and `docker logs missions-sut` scrape. Side-channel Npgsql for fixture state, post-state assertions, and concurrent INSERTs. JWKS rotation via `POST https://jwks-mock:8443/rotate-key`. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.4, AC-5 5.7, AC-6 6.1/6.2/6.7/6.8, E3/E4. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container). +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs`, `Infrastructure/ConfigurationResolver`, `Database/AppDataConnection`, `Database/DatabaseMigrator`, `Services/VehicleService` (for the TOCTOU race), or `Auth/JwtExtensions`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-585_test_resource_limits.md b/_docs/tasks/todo/AZ-585_test_resource_limits.md new file mode 100644 index 0000000..4ae5a4c --- /dev/null +++ b/_docs/tasks/todo/AZ-585_test_resource_limits.md @@ -0,0 +1,116 @@ +# Resource Limit Tests + +**Task**: AZ-585_test_resource_limits +**Name**: Resource limit tests (NFT-RES-LIM-01..04) +**Description**: Implement xUnit blackbox tests for the 4 resource-limit observation scenarios — steady-state RSS memory under 5-min sustained load (P95 ≤ 250 MiB; no monotonic climb), Npgsql connection pool ≤ 100 with no unbounded growth, file-descriptor count ≤ 1024 with no leak, and cold-start RSS ≤ 200 MiB at `t=30s` after health-ok. Provisional gates documented per `restrictions.md` H6 — locked in after first green run. +**Complexity**: 3 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-585 +**Epic**: AZ-575 + +## Problem + +Per H6, container-level resource limits are NOT enforced inside the container — they will be set at the suite level (`_infra/_compose/`) per device type once locked. These tests establish baseline observations so the suite can size the cgroup limits correctly AND provide an upper-bound regression gate so future changes do not silently 10× the memory or FD footprint. The 8 GB Jetson Orin must accommodate ~6 .NET edge services + Postgres + UI; `missions`'s budget is ~200 MiB cold + ~250 MiB hot. Without these observation tests, a leak or library bloat could ship to the device and force a re-sizing decision late in deployment. + +## Outcome + +- All four NFT-RES-LIM-01..04 scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=ResLim`, `Traces=H1|H3|H6|O10`, `Result=pass`, AND records the measured value (e.g., `P95_RSS_MiB=187`) in the `Traces` column so suite-level deployment planning can read it. +- NFT-RES-LIM-01 measures P95 RSS over 5 minutes of mixed sustained load AND asserts `final_RSS - P95_RSS ≤ 20% * P95_RSS` (no monotonic climb). +- NFT-RES-LIM-02 measures Npgsql connection count via `pg_stat_activity` every 5s AND asserts both `max ≤ 100` AND `final ≤ 1.3 * first_minute_steady_state`. +- NFT-RES-LIM-03 measures `/proc//fd | wc -l` inside the container every 5s AND asserts both `max ≤ 1024` AND `final ≤ 1.3 * minute_one_count`. +- NFT-RES-LIM-04 measures cold-start RSS exactly 30s after `GET /health` first returns 200 (no requests issued yet) AND asserts `RSS ≤ 200 MiB`. + +## Scope + +### Included + +- NFT-RES-LIM-01 Steady-state memory under 5-min sustained load. +- NFT-RES-LIM-02 Connection pool steady-state. +- NFT-RES-LIM-03 File-descriptor steady-state. +- NFT-RES-LIM-04 Cold-start RSS budget. +- Each test records the measured value to the CSV `Traces` field so deployment planning can pick it up. +- Provisional gates: 250 MiB hot, 200 MiB cold, 100 connections, 1024 FDs. On first green run, replace provisional gates with `measured + 50%` and open a Refactor Backlog ticket if the provisional gate was exceeded. + +### Excluded + +- Performance (latency / throughput) tests live in Task 19. +- GPU / temperature / disk-I/O monitoring (per `restrictions.md` H8 — no specialised hardware on a CRUD service). +- Long-soak / endurance tests (> 5 min) — explicitly deferred per `restrictions.md` H8. + +## Acceptance Criteria + +**AC-1: NFT-RES-LIM-01 steady-state RSS ≤ provisional 250 MiB with no monotonic climb** +Given `missions` running with `seed_25_missions` + `seed_3_vehicles_2_default` and no host-side memory limit +When the test orchestrator drives ~50 RPS of mixed `GET /vehicles`, `GET /missions`, `GET /missions/{id}/waypoints` for 5 minutes from a single concurrent client, while polling `docker stats --no-stream missions-sut` every 5s +Then the P95 of the 60 RSS samples is `≤ 250 MiB` (provisional gate) +And the final-sample RSS is within ± 20% of the P95 RSS (no sustained leak — RSS does not climb monotonically) +And the measured P95 is recorded to the CSV `Traces` column as `P95_RSS_MiB=` + +**AC-2: NFT-RES-LIM-02 connection pool ≤ 100 with no unbounded growth** +Given the same setup as NFT-RES-LIM-01 +When the test orchestrator polls side-channel `SELECT count(*) FROM pg_stat_activity WHERE application_name LIKE 'Npgsql%' OR (usename='postgres' AND backend_type='client backend')` every 5s for 5 minutes +Then the max sampled connection count is `≤ 100` +And the final-sample count is `≤ 1.3 × (mean of samples in the first minute)` +And the measured max is recorded as `MAX_NPGSQL_CONNS=` + +**AC-3: NFT-RES-LIM-03 file descriptors ≤ 1024 with no leak** +Given the same setup as NFT-RES-LIM-01 +When the test orchestrator executes `docker exec missions-sut sh -c 'ls /proc/$(pgrep -f Azaion.Missions.dll | head -1)/fd | wc -l'` every 5s for 5 minutes +Then the max sampled FD count is `≤ 1024` +And the final-sample count is `≤ 1.3 × (count at t=1min)` +And the measured max is recorded as `MAX_FD=` + +**AC-4: NFT-RES-LIM-04 cold-start RSS ≤ 200 MiB** +Given `missions` has been started fresh (via `docker compose up -d missions` after `down -v`), no requests issued yet +When `GET /health` first returns `200` AND 30s have elapsed +Then `docker stats --no-stream missions-sut` reports `MEM USAGE` ≤ 200 MiB +And the measured cold-start RSS is recorded as `COLD_RSS_MiB=` + +## Non-Functional Requirements + +**Performance** +- NFT-RES-LIM-01..03: each take exactly 5 minutes (sampling window). With Arrange/teardown, ≤ 6 minutes wall-clock. +- NFT-RES-LIM-04: ≤ 60s wall-clock (fresh start + health-poll + 30s wait + measurement). +- The total task runtime budget is ≤ 20 minutes, fitting inside the documented 15-min suite CI gate per `environment.md`. NFT-RES-LIM-01..03 share the same 5-minute window and run concurrently against a single dockerised `missions`; NFT-RES-LIM-04 runs separately because it requires a fresh start. + +**Reliability** +- The load generator is a single-thread `HttpClient` driving requests in a tight loop; this is documented at 50 RPS approximately for the in-suite test runner. If the runner is unable to sustain 50 RPS (CI infrastructure too slow), the test SKIPS NFT-RES-LIM-01..03 with `Result=skip` and a clear `ErrorMessage=runner cannot sustain target load`. CI then reruns these on a beefier worker. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | `seed_25_missions` + 50 RPS for 5 min | P95 RSS sampling | P95 ≤ 250 MiB + no monotonic climb | H1, H6, O10 | +| AC-2 | same | `pg_stat_activity` polling | max ≤ 100 + final ≤ 1.3×steady | O10 | +| AC-3 | same | `/proc//fd` polling | max ≤ 1024 + final ≤ 1.3×minute-one | H6, O10 | +| AC-4 | fresh `docker compose up -d` | cold-start RSS at t=30s | RSS ≤ 200 MiB | H1, H3 | + +## Constraints + +- `docker stats` and `docker exec` from inside the runner: requires Docker socket access; AZ-576 covers this. +- NFT-RES-LIM-03 requires `pgrep` inside the `missions` image; the test FAILS in Arrange (not Assert) if `pgrep` is unavailable. Alternative: parse `/proc/1/comm` if PID 1 is the .NET process (preferred for the small Dockerfile). +- All measurements are recorded to the CSV report's `Traces` field so deployment planning can pick them up; this is more important than the pass/fail gate. +- Provisional gates are documented per `restrictions.md` H6 — locked in based on first measured run. +- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test. + +## Risks & Mitigation + +**Risk 1: Measurement variance on shared CI runners** +- *Risk*: A runner under noisy-neighbour load reports inflated RSS, flaking the gate. +- *Mitigation*: Gates are provisional and generous (250 MiB vs. typical .NET service of ~150 MiB; 100 connections vs. typical idle pool of ~5–10). After the first green run, the gate is locked at `measured + 50%`. + +**Risk 2: NFT-RES-LIM-01..03 share a 5-minute window — flake correlation** +- *Risk*: A CI hiccup that kills the SUT mid-window flakes all three at once. +- *Mitigation*: Each test asserts its own metric; on `missions-sut` exit during the window, the test FAILS with a `"SUT exited during measurement window"` ErrorMessage rather than reporting a misleading metric value. + +**Risk 3: Provisional gates silently accepted as the locked gate** +- *Risk*: If the first green run measures 200 MiB and the test passes, a future engineer treats 250 MiB as the gate forever — but actual headroom is only 50 MiB. +- *Mitigation*: The test logs `(measured / gate) ratio`; CI dashboards flag ratios > 0.8 for re-tuning consideration. The lock-in workflow is documented in `restrictions.md` H6. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface for load generation; `docker stats`, `docker exec`, and side-channel `pg_stat_activity` for measurement. Expected outputs are the documented gates from `_docs/02_document/tests/resource-limit-tests.md` (provisional) and the corresponding entries in `_docs/00_problem/input_data/expected_results/results_report.md` (when locked). +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`. +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the Npgsql connection pool, the `AppDataConnection` lifetime, or the `Program.cs` startup path. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub. diff --git a/_docs/tasks/todo/AZ-586_test_performance.md b/_docs/tasks/todo/AZ-586_test_performance.md new file mode 100644 index 0000000..e332ffa --- /dev/null +++ b/_docs/tasks/todo/AZ-586_test_performance.md @@ -0,0 +1,117 @@ +# Performance Tests + +**Task**: AZ-586_test_performance +**Name**: Performance tests (NFT-PERF-01..04) +**Description**: Implement xUnit blackbox tests for the 4 performance scenarios — F3 cascade-delete P50 ≤ 50ms on a 1-waypoint mission, F3 cascade-delete P50 ≤ 200ms on the full chain (provisional baseline; lock after first green run), `GET /health` P50 ≤ 10ms, and `GET /missions?page=1&pageSize=20` P95 ≤ 100ms against a 1000-mission seed (provisional baseline). Every test runs 5 warm-up calls + the documented N measured calls; cold-start passes excluded. +**Complexity**: 3 points +**Dependencies**: AZ-576_test_infrastructure +**Component**: Blackbox Tests +**Tracker**: AZ-586 +**Epic**: AZ-575 + +## Problem + +Three latency thresholds are documented (AC-3.6 P50 ≤ 50ms for minimal cascade, AC-7.3 P50 ≤ 10ms for health, AC-2.3 implicit list latency) and one (NFT-PERF-02 full-chain cascade) is a baseline that subsequent runs must not regress by more than 50%. Without these tests, an unintentional N+1 query, missing index, or accidental serialization layer overhead could silently 10× the response time before the next manual perf benchmark catches it. The full-chain cascade test is especially load-bearing because the F3 cascade walks 5 dependency tables — a future indexing regression or transaction-wrap addition would show up here first. + +## Outcome + +- All four NFT-PERF-01..04 scenarios run and pass against the dockerised `missions` service. +- Each test produces a CSV row with `Category=Perf`, `Traces=AC-3.6` / `AC-3.1` / `AC-7.3` / `AC-2.3`, `Result=pass`, AND records P50 and P95 numeric values in the `Traces` column (e.g., `P50_MS=23.4, P95_MS=41.8`). +- 5 warm-up calls precede every measured set; cold-start passes are excluded from the percentile computation. +- All tests run sequentially against a single client (no concurrent connections) so HTTP/1.1 connection-reuse and JIT warm-up are deterministic. +- Tests run only when `[Trait("Category","Perf")]` filter is active (default test suite filter excludes performance to keep the standard CI gate ≤ 15 min); a separate `scripts/run-performance-tests.sh` invocation runs them. + +## Scope + +### Included + +- NFT-PERF-01 F3 minimal cascade — `DELETE /missions/{id}` on 1-waypoint missions; P50 ≤ 50ms over 100 sequential calls. +- NFT-PERF-02 F3 full cascade — `DELETE /missions/{id}` on `fixture_cascade_F3`-shaped missions; P50 ≤ 200ms over 50 sequential calls (provisional baseline). +- NFT-PERF-03 Health endpoint — `GET /health` P50 ≤ 10ms over 100 sequential calls. +- NFT-PERF-04 List pagination — `GET /missions?page=1&pageSize=20` P95 ≤ 100ms over 100 sequential calls against a 1000-mission seed (provisional baseline). +- Recording P50/P95 to CSV `Traces` column for trend tracking even when not gated. +- Performance suite is gated behind the `[Trait("Category","Perf")]` filter; standard CI gate excludes these. + +### Excluded + +- Concurrency / contention tests (race scenarios) live in Task 17 (NFT-RES-08). +- Resource consumption (RSS, FDs, connections) lives in Task 18 (NFT-RES-LIM). +- Production-hardware (Jetson Orin) latency baselines — documented as a follow-up in `restrictions.md` H8; test environment baselines stand in. +- Concurrent-client throughput / RPS — not in scope today; documented as Refactor Backlog. + +## Acceptance Criteria + +**AC-1: NFT-PERF-01 F3 minimal cascade P50 ≤ 50ms** +Given `missions` + `postgres-test` colocated on the same Docker network, `seed_one_default_vehicle` + 100 minimal missions (each with 1 waypoint, no media/annotations/detection/map_objects rows), AND 5 warm-up `DELETE` calls have completed on missions outside the measured set +When the consumer issues 100 sequential `DELETE /missions/{id_i}` calls (one per seeded mission, 1 ≤ i ≤ 100) and records per-call wall-clock latency +Then the P50 (median) of the 100 latencies is `≤ 50ms` +And P50 + P95 are recorded to the CSV `Traces` column as `P50_MS=, P95_MS=` + +**AC-2: NFT-PERF-02 F3 full-chain cascade P50 ≤ 200ms** +Given 50 missions each with the `fixture_cascade_F3` chain (3 map_objects, 2 waypoints, 2 media, 2 annotations, 2 detection rows) AND 5 warm-up calls on additional fixtures outside the measured set +When the consumer issues 50 sequential `DELETE /missions/{id_i}` calls and records per-call wall-clock latency +Then P50 ≤ 200ms (provisional baseline — to be locked at `measured + 50%` on first green run) +And P50 + P95 recorded to CSV + +**AC-3: NFT-PERF-03 health endpoint P50 ≤ 10ms** +Given `missions` running, no special seed, AND 5 warm-up `GET /health` calls +When the consumer issues 100 sequential `GET /health` calls (no `Authorization` header) and records per-call wall-clock latency +Then P50 ≤ 10ms +And P50 + P95 recorded to CSV + +**AC-4: NFT-PERF-04 list pagination P95 ≤ 100ms (provisional)** +Given `seed_one_default_vehicle` + 1000 missions referencing it, AND 5 warm-up `GET /missions?page=1&pageSize=20` calls +When the consumer issues 100 sequential `GET /missions?page=1&pageSize=20` calls and records per-call wall-clock latency +Then P95 ≤ 100ms (provisional baseline — to be locked at `measured + 50%` on first green run) +And P50 + P95 recorded to CSV + +## Non-Functional Requirements + +**Performance** +- NFT-PERF-01: ≤ 30s wall-clock (100 calls × ≤ 50ms each + measurement overhead). Per `[Trait("max_ms","30000")]` xUnit timeout. +- NFT-PERF-02: ≤ 60s wall-clock. +- NFT-PERF-03: ≤ 5s wall-clock. +- NFT-PERF-04: ≤ 30s wall-clock. + +**Reliability** +- All tests SKIP if the runner cannot allocate ≥ 2 CPU cores and ≥ 2 GB free RAM (per `performance-tests.md` Notes). SKIP records `Result=skip` and `ErrorMessage=insufficient CPU/RAM`. Default CI runner spec must meet this — but degraded runners must not produce false-fail noise. +- All tests assume `missions` and `postgres-test` are colocated on the same Docker network (no inter-host link). The fixture verifies this via `docker inspect missions-sut --format '{{.NetworkSettings.Networks.testnet.IPAddress}}'` returns non-empty. + +## Blackbox Tests + +| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References | +|--------|------------------------|-------------|-------------------|----------------| +| AC-1 | 100 minimal missions + 5 warm-ups | 100 sequential `DELETE /missions/{id}` | P50 ≤ 50ms; record P50/P95 | AC-3.6 | +| AC-2 | 50 F3-fixture missions + 5 warm-ups | 50 sequential `DELETE /missions/{id}` | P50 ≤ 200ms (provisional); record P50/P95 | AC-3.1, AC-3.6 | +| AC-3 | warm runtime + 5 warm-ups | 100 sequential `GET /health` | P50 ≤ 10ms; record P50/P95 | AC-7.3 | +| AC-4 | 1000 missions + 5 warm-ups | 100 sequential `GET /missions?page=1&pageSize=20` | P95 ≤ 100ms (provisional); record P50/P95 | AC-2.3 | + +## Constraints + +- Tests live in `Tests/Performance/` and are tagged `[Trait("Category","Perf")]` so the default CI gate excludes them. +- A separate `scripts/run-performance-tests.sh` (created by AZ-576) invokes only this category. The standard `scripts/run-tests.sh` skips them. +- Sequential single-client execution — no `Parallel.For` or `Task.WhenAll`; each call awaits the previous response. +- Warm-up calls are NOT included in the percentile computation. Per `// Warmup` comment block in the test, the first 5 calls go to fixtures created specifically for warm-up (not the measured set). +- The `Stopwatch`-based timing measures `HttpClient.SendAsync` wall-clock; serialization/deserialization overhead is INCLUDED (this is what end-users observe). +- Provisional gates (NFT-PERF-02, NFT-PERF-04) are documented in source as `// PROVISIONAL — lock at measured + 50% on first green run` and `[Trait("provisional","yes")]`. +- AAA pattern with `// Arrange` (seed + warm-up), `// Act` (measured calls + percentile compute), `// Assert` (gate + CSV record). + +## Risks & Mitigation + +**Risk 1: CI variance breaks tight P50 ≤ 10ms gate (NFT-PERF-03)** +- *Risk*: On a noisy-neighbour CI runner, even a static `/health` route can hiccup once per 100 calls; if the hiccup lands in the P50 region, the median exceeds 10ms. +- *Mitigation*: P50 is robust to single outliers (median position 50 of 100). If the test still flakes, lock the gate at `measured P50 + 50%` after the first green run. + +**Risk 2: NFT-PERF-04 1000-mission seed overlaps with other tests' DB state** +- *Risk*: Seeding 1000 missions affects pagination tests, list-shape tests, and date-filter tests — if NFT-PERF-04 runs before them in the same SUT lifetime, results drift. +- *Mitigation*: NFT-PERF-04 lives in `[Collection("Perf1k")]` and uses `IClassFixture` to TRUNCATE all rows before its seed AND restore `seed_empty` after. Functional tests' fixtures handle their own seed; no cross-pollination. + +**Risk 3: Provisional gates accepted as locked gates** +- *Risk*: Same as NFT-RES-LIM Risk 3 — if first run measures 80ms and the test passes, future engineers see the 100ms gate as the standard. +- *Mitigation*: CI dashboards flag `measured / gate ratio > 0.8` for re-tuning. Lock-in workflow documented in `performance-tests.md`. + +## System Under Test Boundary + +- Tests drive the product through the public HTTP surface (`http://missions:8080`) plus Npgsql side-channel for seed setup. Bearer tokens (NFT-PERF-01, 02, 04) minted via `https://jwks-mock:8443/sign`; NFT-PERF-03 sends no Authorization header. Expected outputs are the documented latency thresholds from `_docs/02_document/tests/performance-tests.md`. +- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`. +- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the controllers, service classes, `AppDataConnection`, or any layer affecting response time. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.