mirror of
https://github.com/azaion/missions.git
synced 2026-06-21 08:21:08 +00:00
[AZ-575] Add 11 blackbox test task specs from decompose Step 5
Decompose Step 5 (tests-only mode) produced the test-task ladder for the Blackbox Tests epic. Test infrastructure (AZ-576) blocks the rest; all 10 blackbox child tasks fan out from it. Tasks (epic AZ-575): - AZ-576 test_infrastructure (5 SP) - AZ-577 test_vehicles_positive (5 SP) - AZ-578 test_missions_positive (5 SP) - AZ-579 test_waypoints_health_positive (5 SP) - AZ-580 test_validation_authz_negative (3 SP) - AZ-581 test_security_auth_claims (5 SP) - AZ-582 test_security_alg_rotation_cors (5 SP) - AZ-583 test_resilience_cascade_migrator (3 SP) - AZ-584 test_resilience_config_db_rotation_race (5 SP) - AZ-585 test_resource_limits (3 SP) - AZ-586 test_performance (3 SP) Total: 45 SP across 11 tasks. Coverage verified against blackbox/security/resilience/resource-limit/performance test specs (56 scenarios). _docs/_autodev_state.md advanced to Step 6 (Implement Tests). Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -2,8 +2,8 @@
|
|||||||
|
|
||||||
## Current Step
|
## Current Step
|
||||||
flow: existing-code
|
flow: existing-code
|
||||||
step: 5
|
step: 6
|
||||||
name: Decompose Tests
|
name: Implement Tests
|
||||||
status: not_started
|
status: not_started
|
||||||
sub_step:
|
sub_step:
|
||||||
phase: 0
|
phase: 0
|
||||||
@@ -13,5 +13,5 @@ retry_count: 0
|
|||||||
cycle: 1
|
cycle: 1
|
||||||
tracker: jira
|
tracker: jira
|
||||||
|
|
||||||
## Rename tracking (Jira AZ-EPIC + child stories B1-B12)
|
## Last Updated
|
||||||
See `_docs/_process_leftovers/2026-05-14_rename-flights-to-missions.md`. Local code work for B5, B6, B7, B8, B9, B12 landed 2026-05-15; .woodpecker tag rename done. Cross-repo work pending: B4 (suite), B10-suite, B11 (autopilot + ui), B12 spec catch-up in suite. Leftover stays until those land.
|
2026-05-15
|
||||||
|
|||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# Dependencies Table
|
||||||
|
|
||||||
|
**Date**: 2026-05-15
|
||||||
|
**Mode**: tests-only decomposition (Step 5 of `existing-code` autodev flow)
|
||||||
|
**Epic**: AZ-575 — Blackbox Tests — Missions
|
||||||
|
**Total Tasks**: 11
|
||||||
|
**Total Complexity Points**: 45 (5 + 5 + 5 + 5 + 3 + 5 + 5 + 3 + 5 + 3 + 3)
|
||||||
|
|
||||||
|
| Task | Name | Complexity | Dependencies | Epic |
|
||||||
|
|------|------|-----------|-------------|------|
|
||||||
|
| AZ-576 | test_infrastructure | 5 | None | AZ-575 |
|
||||||
|
| AZ-577 | test_vehicles_positive | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-578 | test_missions_positive | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-579 | test_waypoints_health_positive | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-580 | test_validation_authz_negative | 3 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-581 | test_security_auth_claims | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-582 | test_security_alg_rotation_cors | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-583 | test_resilience_cascade_migrator | 3 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-584 | test_resilience_config_db_rotation_race | 5 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-585 | test_resource_limits | 3 | AZ-576 | AZ-575 |
|
||||||
|
| AZ-586 | test_performance | 3 | AZ-576 | AZ-575 |
|
||||||
|
|
||||||
|
## Coverage Verification
|
||||||
|
|
||||||
|
| Spec file | Scenarios | Covered by |
|
||||||
|
|-----------|-----------|------------|
|
||||||
|
| `tests/blackbox-tests.md` § Positive | FT-P-01..06 (Vehicles) | AZ-577 |
|
||||||
|
| `tests/blackbox-tests.md` § Positive | FT-P-07..12 (Missions) | AZ-578 |
|
||||||
|
| `tests/blackbox-tests.md` § Positive | FT-P-13..18 (Waypoints + Health) | AZ-579 |
|
||||||
|
| `tests/blackbox-tests.md` § Negative | FT-N-01..08 | AZ-580 |
|
||||||
|
| `tests/security-tests.md` | NFT-SEC-01..06 + 04b | AZ-581 |
|
||||||
|
| `tests/security-tests.md` | NFT-SEC-07..13 | AZ-582 |
|
||||||
|
| `tests/resilience-tests.md` | NFT-RES-01..04 | AZ-583 |
|
||||||
|
| `tests/resilience-tests.md` | NFT-RES-05..08 | AZ-584 |
|
||||||
|
| `tests/resource-limit-tests.md` | NFT-RES-LIM-01..04 | AZ-585 |
|
||||||
|
| `tests/performance-tests.md` | NFT-PERF-01..04 | AZ-586 |
|
||||||
|
|
||||||
|
**Total scenarios covered**: 56 (18 FT-P + 8 FT-N + 14 NFT-SEC + 8 NFT-RES + 4 NFT-RES-LIM + 4 NFT-PERF).
|
||||||
|
|
||||||
|
## Cross-Task Consistency Checks
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|-------|--------|
|
||||||
|
| Every scenario from `blackbox-tests.md` § Positive (FT-P-01..18) is covered | PASS |
|
||||||
|
| Every scenario from `blackbox-tests.md` § Negative (FT-N-01..08) is covered | PASS |
|
||||||
|
| Every scenario from `security-tests.md` (NFT-SEC-01..13 + 04b) is covered | PASS |
|
||||||
|
| Every scenario from `resilience-tests.md` (NFT-RES-01..08) is covered | PASS |
|
||||||
|
| Every scenario from `resource-limit-tests.md` (NFT-RES-LIM-01..04) is covered | PASS |
|
||||||
|
| Every scenario from `performance-tests.md` (NFT-PERF-01..04) is covered | PASS |
|
||||||
|
| No task exceeds 5 complexity points | PASS |
|
||||||
|
| Every blackbox test task depends on the test-infrastructure task (AZ-576) | PASS |
|
||||||
|
| Test-infrastructure task (AZ-576) has no upstream test dependencies | PASS |
|
||||||
|
| No circular dependencies in the task graph | PASS — graph is a fan-out: AZ-576 → {AZ-577..AZ-586} |
|
||||||
|
| Every e2e/blackbox task has a System Under Test Boundary section | PASS — all 10 child tasks include the section |
|
||||||
|
| System Under Test Boundary forbids stubbing internal product modules | PASS — verified in each task spec |
|
||||||
|
| System Under Test Boundary requires comparison to expected-results artifacts | PASS — every task references `_docs/00_problem/input_data/expected_results/results_report.md` and/or the relevant machine-readable expected-result JSON |
|
||||||
|
|
||||||
|
## Overlap & Shared-Concern Notes
|
||||||
|
|
||||||
|
- **NFT-SEC-08 (Task 15) ↔ FT-N-08 (Task 13)** both exercise the 500 error envelope. FT-N-08 owns the destructive `DROP TABLE vehicles` fault injection and asserts redaction + log line presence; NFT-SEC-08 additionally asserts the body has NO key matching `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/type-name. No work duplication — the two tests share the fixture but assert distinct invariants.
|
||||||
|
- **NFT-SEC-11 (Task 15) ↔ NFT-RES-07 (Task 17)** both exercise JWKS rotation. NFT-SEC-11 focuses on the `kid`-cache mechanics + grace-window timing; NFT-RES-07 additionally asserts the `docker inspect StartedAt` invariant (no restart). Sharing the same primitive via the `JwksRotateFixture` from AZ-576.
|
||||||
|
- **NFT-SEC-12 (Task 15) ↔ NFT-RES-05 (Task 17)** both exercise startup fail-fast on missing required env vars. NFT-SEC-12 covers 4 missing-env cases + HTTP-JWKS-URL path. NFT-RES-05 covers the same 4 missing-env cases + an additional whitespace-only case + the DB-down-after-config-resolution differentiator (proves config resolution succeeded before Npgsql failed). Tasks share the `MissionsContainerHelper` docker-run primitive from AZ-576.
|
||||||
|
|
||||||
|
## Execution Order Hint
|
||||||
|
|
||||||
|
Recommended dependency-aware batches for `/implement`:
|
||||||
|
|
||||||
|
1. **Batch 1 (sequential, blocking the rest)**: AZ-576 — test_infrastructure
|
||||||
|
2. **Batch 2 (parallel, fan-out from AZ-576)**: AZ-577..AZ-586 in any order. Independent test classes within a single xUnit assembly; no inter-task ordering needed.
|
||||||
|
|
||||||
|
CSV report sorting at suite end: by `Category` (Blackbox / Sec / Res / ResLim / Perf), then by test ID within category.
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
# Test Infrastructure
|
||||||
|
|
||||||
|
**Task**: AZ-576_test_infrastructure
|
||||||
|
**Name**: Test Infrastructure (Missions e2e)
|
||||||
|
**Description**: Scaffold the Blackbox test project — xUnit runner, JWKS mock service, Docker test environment wiring, test data fixtures, reporting. Compose file already exists at repo root and references not-yet-built build contexts; this task fills in those contexts.
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: None (C01 + C02 testability refactor already landed; see `_docs/04_refactoring/01-testability-refactoring/testability_changes_summary.md`)
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-576
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
Two artifacts that the existing `docker-compose.test.yml` references but does not yet build, plus the run script the suite expects:
|
||||||
|
|
||||||
|
1. `tests/Azaion.Missions.JwksMock/` — minimal HTTPS service holding an ECDSA P-256 keypair in memory, serving JWKS + `POST /sign` + `POST /rotate-key`. Image tag `azaion/jwks-mock:test`.
|
||||||
|
2. `tests/Azaion.Missions.E2E.Tests/` — xUnit 2.x test project that drives the running `missions` service over HTTP, mints tokens via `https://jwks-mock:8443/sign`, asserts DB side-effects through a side-channel Npgsql connection, and produces `report.csv`.
|
||||||
|
3. `tests/jwks-mock-ca.crt` — the self-signed CA cert that both `missions` and `e2e-consumer` mount and `update-ca-certificates --fresh` adds to the OS trust bundle (per `docker-entrypoint.sh` from C02).
|
||||||
|
4. `scripts/run-tests.sh` — wraps `docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer`, collects `report.csv`, then `down -v`.
|
||||||
|
5. `scripts/run-performance-tests.sh` — same compose stack with the `[Trait("Category","Perf")]` filter and the perf seed.
|
||||||
|
|
||||||
|
The `missions` and `postgres-test` services already exist in `docker-compose.test.yml`; the `jwks-mock` and `e2e-consumer` services exist but point at build contexts that this task creates.
|
||||||
|
|
||||||
|
## Test Project Folder Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
tests/
|
||||||
|
├── jwks-mock-ca.crt # self-signed CA (mounted into missions + e2e-consumer)
|
||||||
|
├── Azaion.Missions.JwksMock/
|
||||||
|
│ ├── Azaion.Missions.JwksMock.csproj
|
||||||
|
│ ├── Dockerfile # builds azaion/jwks-mock:test, exposes 8443/tcp
|
||||||
|
│ ├── Program.cs # ASP.NET Core minimal API
|
||||||
|
│ ├── Endpoints/
|
||||||
|
│ │ ├── JwksEndpoint.cs # GET /.well-known/jwks.json
|
||||||
|
│ │ ├── SignEndpoint.cs # POST /sign
|
||||||
|
│ │ └── RotateKeyEndpoint.cs # POST /rotate-key
|
||||||
|
│ ├── Services/
|
||||||
|
│ │ ├── KeyStore.cs # in-memory ECDSA P-256 keypair + old-key grace window
|
||||||
|
│ │ └── TokenSigner.cs # ECDSA signing with alg_override/kid_override support
|
||||||
|
│ └── appsettings.json # JWT_ISSUER, JWT_AUDIENCE, OLD_KEY_GRACE_SECONDS
|
||||||
|
└── Azaion.Missions.E2E.Tests/
|
||||||
|
├── Azaion.Missions.E2E.Tests.csproj # xUnit 2.x + Bogus 35.x + Npgsql 10.x
|
||||||
|
├── Dockerfile # runs `dotnet test --logger trx` + trx→csv post-step
|
||||||
|
├── TestBase.cs # HttpClient factory, default JWT, shared MissionsBaseUrl
|
||||||
|
├── TokenMinter.cs # POST jwks-mock:8443/sign with overrides
|
||||||
|
├── Fixtures/
|
||||||
|
│ ├── DbResetFixture.cs # IClassFixture<>: TRUNCATE between classes
|
||||||
|
│ ├── DbSeedFixture.cs # base for the named seed sets in test-data.md
|
||||||
|
│ ├── ComposeRestartFixture.cs # docker compose down -v && up -d for bootstrap-sensitive tests
|
||||||
|
│ └── JwksRotateFixture.cs # POST /rotate-key + wait for missions to refresh JWKS cache
|
||||||
|
├── Helpers/
|
||||||
|
│ ├── DbAssertions.cs # Npgsql side-channel asserts (row counts, default-vehicle invariants)
|
||||||
|
│ ├── HttpAssertions.cs # PascalCase shape, error-envelope shape, ordering, pagination
|
||||||
|
│ └── FixtureSql.cs # loads fixture_cascade_F3.sql / fixture_cascade_F4.sql
|
||||||
|
├── Tests/
|
||||||
|
│ ├── Vehicles/ # FT-P-01..06, FT-N-01..03
|
||||||
|
│ ├── Missions/ # FT-P-07..12, FT-N-04..06
|
||||||
|
│ ├── Waypoints/ # FT-P-13..15, FT-P-18, FT-N-07
|
||||||
|
│ ├── Health/ # FT-P-16..17, FT-N-08
|
||||||
|
│ ├── Security/ # NFT-SEC-01..13, 04b
|
||||||
|
│ ├── Resilience/ # NFT-RES-01..08
|
||||||
|
│ ├── ResourceLimits/ # NFT-RES-LIM-01..04
|
||||||
|
│ └── Performance/ # NFT-PERF-01..04
|
||||||
|
└── Reporting/
|
||||||
|
├── TrxToCsvPostProcessor.cs # produces /app/results/report.csv per environment.md § Reporting
|
||||||
|
└── ResultRow.cs # TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage
|
||||||
|
```
|
||||||
|
|
||||||
|
### Layout Rationale
|
||||||
|
|
||||||
|
- **Per-feature test folders** (`Vehicles/`, `Missions/`, etc.) match the natural decomposition surface in `blackbox-tests.md` and let `dotnet test --filter` slice the suite per Jira child ticket.
|
||||||
|
- **`Fixtures/` separate from `Tests/`** so xUnit `IClassFixture<>` lifetime is explicit (class-scoped DB reset) and not entangled with test cases.
|
||||||
|
- **`Helpers/` named for the assertion family** (DB, HTTP, FixtureSql) so each test reads as a single `// Arrange` + `// Act` + `// Assert` block per `coderule.mdc`.
|
||||||
|
- **JwksMock is a SEPARATE csproj**, not nested inside the e2e tests, because the build context is mounted as a service in `docker-compose.test.yml` (`tests/Azaion.Missions.JwksMock/`). Sharing a project would force the e2e runner to ship JWKS code into its container.
|
||||||
|
- **CA cert lives at `tests/jwks-mock-ca.crt`** rather than inside a project so both consumers (missions + e2e-consumer) mount the same file. The cert is regenerated only when the keypair changes — committed to the repo for deterministic test runs.
|
||||||
|
|
||||||
|
## Mock Services
|
||||||
|
|
||||||
|
| Mock Service | Replaces | Endpoints | Behavior |
|
||||||
|
|-------------|----------|-----------|----------|
|
||||||
|
| `jwks-mock` | `admin` JWT issuer + JWKS endpoint | `GET https://jwks-mock:8443/.well-known/jwks.json`; `POST https://jwks-mock:8443/sign`; `POST https://jwks-mock:8443/rotate-key` | Holds one ECDSA P-256 keypair in memory; serves the public half as JWKS with `Cache-Control: public, max-age=60`; signs ECDSA-SHA256 JWTs on `/sign` honoring optional `iss`/`aud`/`exp_offset_seconds`/`permissions`/`alg_override`/`kid_override`; rotates keypair on `/rotate-key` while retaining the old public key for `OLD_KEY_GRACE_SECONDS` (5s in tests). Private key never leaves the container. |
|
||||||
|
|
||||||
|
DB-only stubs (no service running, side-channel SQL inserts only): `annotations`, `detection`, `media`, `map_objects` — see `_docs/02_document/tests/test-data.md` § External Dependency Mocks.
|
||||||
|
|
||||||
|
### Mock Control API
|
||||||
|
|
||||||
|
`jwks-mock` exposes `POST /sign` and `POST /rotate-key` as its full control surface. The `/sign` body shape is documented in `test-data.md` § "JWKS mock token-minting contract":
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST https://jwks-mock:8443/sign
|
||||||
|
{
|
||||||
|
"iss": "https://admin-test.azaion.local", # optional
|
||||||
|
"aud": "azaion-edge", # optional
|
||||||
|
"exp_offset_seconds": 3600, # optional; negative for expired
|
||||||
|
"permissions": "FL", # optional; "" / "ADMIN" / "fl" / "FLight" for claim-mismatch
|
||||||
|
"alg_override": null, # "HS256" to test alg-confusion (NFT-SEC-10)
|
||||||
|
"kid_override": null # non-existent kid for unknown-key tests (NFT-SEC-11)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response: `{ "token": "<encoded JWT>", "kid": "<key id>" }`.
|
||||||
|
|
||||||
|
## Docker Test Environment
|
||||||
|
|
||||||
|
### docker-compose.test.yml Structure
|
||||||
|
|
||||||
|
| Service | Image / Build | Purpose | Depends On |
|
||||||
|
|---------|--------------|---------|------------|
|
||||||
|
| `postgres-test` | `postgres:16-alpine` | Owned test PostgreSQL; `tmpfs:/var/lib/postgresql/data` for `down -v` isolation | — |
|
||||||
|
| `jwks-mock` | build `tests/Azaion.Missions.JwksMock/` → `azaion/jwks-mock:test` | Mock JWKS issuer | — |
|
||||||
|
| `missions` | build `.` (repo root `Dockerfile`) → `azaion/missions:test` | System under test | `postgres-test` (healthy), `jwks-mock` (healthy) |
|
||||||
|
| `e2e-consumer` | build `tests/Azaion.Missions.E2E.Tests/` | xUnit runner; emits `report.csv` to host-mounted `./test-results/` | `missions` (healthy), `jwks-mock` (healthy) |
|
||||||
|
|
||||||
|
The compose file is already authored at the repo root. This task does NOT modify it — the file IS the contract; the task fills in the two missing build contexts so the references resolve.
|
||||||
|
|
||||||
|
### Networks and Volumes
|
||||||
|
|
||||||
|
| Resource | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| `e2e-net` (bridge) | Isolated test network; no host network access. All four services attach. |
|
||||||
|
| `tmpfs:/var/lib/postgresql/data` | Ephemeral PG data; recreated per `docker compose down -v`. |
|
||||||
|
| `./test-results:/app/results` | `e2e-consumer` mounts this for `report.csv` output to the host. |
|
||||||
|
| `./tests/jwks-mock-ca.crt:/usr/local/share/ca-certificates/jwks-mock-ca.crt:ro` | Mounted into `missions` AND `e2e-consumer` so both trust the mock's HTTPS cert after `update-ca-certificates --fresh` runs in `docker-entrypoint.sh`. |
|
||||||
|
|
||||||
|
## Test Runner Configuration
|
||||||
|
|
||||||
|
**Framework**: xUnit 2.x
|
||||||
|
**Plugins**: `Microsoft.NET.Test.Sdk`, `xunit.runner.visualstudio`, `Bogus 35.x` (synthetic data), `Npgsql 10.x` (side-channel only — NO `Azaion.Missions.*` project reference)
|
||||||
|
**Entry point**: `dotnet test tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj --logger "trx;LogFileName=results.trx"` followed by `TrxToCsvPostProcessor` converting `results.trx` → `report.csv`
|
||||||
|
**AAA convention**: every test method has `// Arrange` / `// Act` / `// Assert` comments per `.cursor/rules/coderule.mdc`.
|
||||||
|
|
||||||
|
### Fixture Strategy
|
||||||
|
|
||||||
|
| Fixture | Scope | Purpose |
|
||||||
|
|---------|-------|---------|
|
||||||
|
| `DbResetFixture` | Class (`IClassFixture<>`) | `TRUNCATE TABLE` for all schema tables between classes; cheap reset for read-path tests (AC-1, AC-2, AC-4) |
|
||||||
|
| `DbSeedFixture<TSeed>` | Class | Applies the named seed sets from `test-data.md` (`seed_empty`, `seed_one_default_vehicle`, `seed_3_vehicles_2_default`, `seed_25_missions`, `fixture_cascade_F3`, `fixture_cascade_F4`, `seed_5_waypoints_unordered`, `seed_legacy_gps_tables`) via Npgsql side-channel |
|
||||||
|
| `ComposeRestartFixture` | Collection | `docker compose -f docker-compose.test.yml down -v && up -d` between scenarios that assert startup-time behavior (AC-6.3..6.7, AC-5.7) |
|
||||||
|
| `JwksRotateFixture` | Scenario | `POST jwks-mock:8443/rotate-key` then waits for missions to refresh its JWKS cache (≤ 30s in tests, capped by `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS`) |
|
||||||
|
| `JwksMockReverseFixture` | Scenario | Boots `missions` outside compose via `docker run` with `ASPNETCORE_ENVIRONMENT=Production` + empty `CorsConfig:AllowedOrigins` to test E9 lock (NFT-SEC-13) |
|
||||||
|
|
||||||
|
### xUnit traits
|
||||||
|
|
||||||
|
Every test method MUST set `[Trait("Category", "Blackbox" | "Sec" | "Res" | "ResLim" | "Perf")]`. The CSV `Category` column reads from this trait. Traceability IDs go into a second `[Trait("Traces", "AC-1.2,AC-1.4")]` trait, comma-separated.
|
||||||
|
|
||||||
|
## Test Data Fixtures
|
||||||
|
|
||||||
|
Loaded entirely from `_docs/02_document/tests/test-data.md` § Seed Data Sets. The fixtures bind the named seeds to the AC IDs that consume them:
|
||||||
|
|
||||||
|
| Data Set | Source | Format | Used By |
|
||||||
|
|----------|--------|--------|---------|
|
||||||
|
| `seed_empty` | `down -v` + `missions` startup migrator | Schema only, no rows | bootstrap, unauth, 404 scenarios |
|
||||||
|
| `seed_one_default_vehicle` | Side-channel `INSERT INTO vehicles ...` | Inline SQL string | AC-1.2 default-clear, AC-1.3 TOCTOU, AC-1.4 setDefault, AC-2.1 mission-create |
|
||||||
|
| `seed_3_vehicles_2_default` | Side-channel SQL | Inline | AC-1.5 list, AC-1.6 filter |
|
||||||
|
| `seed_25_missions` | Side-channel SQL with deterministic UUIDs | Inline | AC-2.3..2.5 pagination + date filter |
|
||||||
|
| `fixture_cascade_F3` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql` | SQL file | AC-3.1, 3.3, 3.4, 10.2 |
|
||||||
|
| `fixture_cascade_F4` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql` | SQL file | AC-4.5, 4.6 |
|
||||||
|
| `seed_5_waypoints_unordered` | Side-channel SQL with `order_num [3,1,2,5,4]` | Inline | AC-4.3 unpaginated ordering |
|
||||||
|
| `seed_legacy_gps_tables` | `CREATE TABLE orthophotos / gps_corrections` + `INSERT` | Inline | AC-3.5 absence, AC-6.5 one-shot drop, AC-10.5 legacy migration |
|
||||||
|
|
||||||
|
### Data Isolation
|
||||||
|
|
||||||
|
Three tiers, by scenario type (per `test-data.md` § Data Isolation Strategy):
|
||||||
|
|
||||||
|
- **Class-scoped DB reset** (`IClassFixture<DbResetFixture>`): for scenarios that share a seed within a class but must not leak across classes. Used for AC-1, AC-2, AC-4 read paths.
|
||||||
|
- **Scenario-scoped container restart** (`docker compose down -v && up -d`): for scenarios that assert startup-time behavior or migrator side-effects (AC-6.3..6.7, AC-6.11, AC-5.7).
|
||||||
|
- **No per-test transaction rollback** — the system under test is a separate process; its `DataConnection` is not in the test transaction.
|
||||||
|
|
||||||
|
## Test Reporting
|
||||||
|
|
||||||
|
**Format**: CSV
|
||||||
|
**Columns**: `TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage`
|
||||||
|
**Output path**: `/app/results/report.csv` inside `e2e-consumer`, mounted to `./test-results/report.csv` on the host
|
||||||
|
**Source**: post-processor reads `results.trx` (xUnit logger output), joins each test's `[Trait("Category",...)]` and `[Trait("Traces",...)]` into the CSV columns. `Result` is `pass` / `fail` / `skip`. `ErrorMessage` is the first line of the failure message (CRs stripped).
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: Test environment starts**
|
||||||
|
Given the `docker-compose.test.yml` at repo root
|
||||||
|
When `docker compose -f docker-compose.test.yml up --build` runs
|
||||||
|
Then `postgres-test`, `jwks-mock`, and `missions` all reach `healthy`, and `e2e-consumer` starts after them
|
||||||
|
|
||||||
|
**AC-2: Mock JWKS service responds**
|
||||||
|
Given the test environment is running
|
||||||
|
When `GET https://jwks-mock:8443/.well-known/jwks.json` is issued from inside `e2e-net`
|
||||||
|
Then the response is `200 OK` with a JWKS body containing exactly one ECDSA P-256 public key
|
||||||
|
And `POST https://jwks-mock:8443/sign` with body `{}` returns a valid ECDSA-SHA256 JWT whose `iss` / `aud` match the mock's env vars
|
||||||
|
|
||||||
|
**AC-3: Test runner executes**
|
||||||
|
Given the test environment is running
|
||||||
|
When `e2e-consumer` starts and `dotnet test` runs
|
||||||
|
Then the runner discovers ≥ 1 test in each of the eight test folders (`Vehicles/`, `Missions/`, `Waypoints/`, `Health/`, `Security/`, `Resilience/`, `ResourceLimits/`, `Performance/`)
|
||||||
|
|
||||||
|
**AC-4: Test report generated**
|
||||||
|
Given tests have been executed
|
||||||
|
When `e2e-consumer` exits
|
||||||
|
Then `./test-results/report.csv` exists on the host
|
||||||
|
And the first line is the documented column header `TestId,TestName,Category,Traces,ExecutionTimeMs,Result,ErrorMessage`
|
||||||
|
And every executed test has exactly one CSV row
|
||||||
|
|
||||||
|
**AC-5: CA trust works end-to-end**
|
||||||
|
Given `tests/jwks-mock-ca.crt` is mounted into both `missions` and `e2e-consumer`
|
||||||
|
When `docker-entrypoint.sh` runs `update-ca-certificates --fresh` and `missions` issues `GET https://jwks-mock:8443/.well-known/jwks.json` to populate its JWKS cache
|
||||||
|
Then the TLS handshake succeeds (no `RemoteCertificateNotAvailable` / `RemoteCertificateNameMismatch`)
|
||||||
|
And the cached JWKS contains the public key the consumer-issued tokens are signed with
|
||||||
|
|
||||||
|
**AC-6: JWKS rotation observable inside the 15-minute CI gate**
|
||||||
|
Given the test compose sets `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30` and `JWT_JWKS_REFRESH_INTERVAL_SECONDS=10` (per C01)
|
||||||
|
When `POST https://jwks-mock:8443/rotate-key` is called
|
||||||
|
Then within 30s `missions` refreshes its JWKS cache and accepts tokens signed with the new `kid`
|
||||||
|
And during the 5s `OLD_KEY_GRACE_SECONDS` window tokens signed with the old `kid` are still accepted
|
||||||
|
|
||||||
|
**AC-7: AAA pattern enforced**
|
||||||
|
Given the xUnit test project compiles
|
||||||
|
When `dotnet build` runs
|
||||||
|
Then every `[Fact]` / `[Theory]` method in `tests/Azaion.Missions.E2E.Tests/Tests/` contains the literal comment lines `// Arrange` (when setup exists), `// Act`, and `// Assert` in that order — verified by a Roslyn analyzer test or a single integration assertion that greps the source files
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- `restrictions.md` SW-01: target framework .NET 10 (matches `Azaion.Missions.csproj`)
|
||||||
|
- `restrictions.md` HW-01: ARM64 + AMD64 (multi-arch base images on both projects)
|
||||||
|
- `restrictions.md` ENV-01: HTTPS-only for the JWKS endpoint (HTTP would short-circuit AC-6.12)
|
||||||
|
- `coderule.mdc`: AAA pattern with `// Arrange` / `// Act` / `// Assert` comments, no narrative comments otherwise
|
||||||
|
- No project reference from `Azaion.Missions.E2E.Tests` → `Azaion.Missions.csproj` (consumer must remain blackbox; assertions only via HTTP and Npgsql side-channel)
|
||||||
|
- Side-channel DB access limited to fixture seeding + post-call assertions; marked with `[Trait("db_access","seed-or-assert-only")]` where used
|
||||||
|
- Token signing happens ONLY inside `jwks-mock`; the consumer never imports a JWT signing library
|
||||||
|
- `report.csv` lives in `./test-results/` (host-mounted); this directory MUST be in `.gitignore`
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
# Vehicles Positive Flow Tests
|
||||||
|
|
||||||
|
**Task**: AZ-577_test_vehicles_positive
|
||||||
|
**Name**: Vehicles positive tests (FT-P-01..06)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 6 happy-path Vehicle CRUD scenarios — create non-default, create default (demotes prior), setDefault, list (no-pagination + Name ASC), filter (case-INSENSITIVE name + exact isDefault), delete with no references.
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-577
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The `/vehicles` surface implements two non-obvious invariants that documentation alone cannot guarantee: (1) creating a default vehicle clears any prior default in the same logical step, and (2) the list filter is case-INSENSITIVE on `name` (the docs said case-sensitive until 2026-05-14 — drift now corrected, but only an executable test can pin the actual code path). Without these tests, a future refactor of `VehicleService` could silently re-introduce two default rows or a case-sensitive filter and break consumers (`autopilot` reads the default vehicle on boot).
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All six FT-P-01..06 scenarios run against the dockerised `missions` service via HTTP + Npgsql side-channel and pass.
|
||||||
|
- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-1.x`, `Result=pass`, and an `ExecutionTimeMs` under the documented `Max execution time` (5s for create paths, 2s for read/delete).
|
||||||
|
- The list test asserts both shape (`array` not `PaginatedResponse`) and ordering (`Name ASC`).
|
||||||
|
- The filter test asserts case-INSENSITIVE matching for two casings (`BR` and `br`).
|
||||||
|
- The default-clear invariant is verified via DB count (`is_default=true` count == 1 after every default-creating action).
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- FT-P-01 Create non-default — `POST /vehicles` body shape + PascalCase response + DB row count.
|
||||||
|
- FT-P-02 Create default demotes prior default — `seed_one_default_vehicle` precondition; assert exactly one default after.
|
||||||
|
- FT-P-03 setDefault promotes existing vehicle — `POST /vehicles/{id}/setDefault`; assert clear-then-set via side-channel.
|
||||||
|
- FT-P-04 List unpaginated + Name ASC — assert body is JSON array (not `{Items,Page,…}`), assert length and ordering.
|
||||||
|
- FT-P-05 Filter `name=BR&isDefault=true` then `name=br&…` — assert case-INSENSITIVE substring match against `seed_3_vehicles_2_default`.
|
||||||
|
- FT-P-06 Delete with no references — `204` + DB count 0.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- FT-N-03 "delete vehicle in use returns 409" lives in Task 13 (negative tests).
|
||||||
|
- Validation-of-input scenarios (empty `Name`, negative `BatteryCapacity`, unknown `Type` int) are carry-forwards documented in `test-data.md` § Data Validation Rules; they are NOT tested here because the spec marks them as "accepted today" — they belong to the Refactor Backlog, not this task.
|
||||||
|
- TOCTOU race on default-vehicle exclusivity (NFT-RES-08) lives in Task 17.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: FT-P-01 returns 201 with PascalCase body**
|
||||||
|
Given `seed_empty` and a JWT with `permissions=FL`
|
||||||
|
When `POST /vehicles` is issued with the documented body
|
||||||
|
Then response is `201 Created`, body parses as `Vehicle` with PascalCase keys, `Id` parses as UUID, side-channel `SELECT COUNT(*) FROM vehicles WHERE id=<returned>` returns 1
|
||||||
|
|
||||||
|
**AC-2: FT-P-02 demotes prior default**
|
||||||
|
Given `seed_one_default_vehicle` (prior row `P1.is_default=true`)
|
||||||
|
When `POST /vehicles { …, IsDefault:true }` is issued
|
||||||
|
Then response is `201`, side-channel shows new row `is_default=true`, row `P1.is_default=false`, and `SELECT COUNT(*) WHERE is_default=true` == 1
|
||||||
|
|
||||||
|
**AC-3: FT-P-03 setDefault clears prior**
|
||||||
|
Given `seed_one_default_vehicle` plus a non-default row `P2`
|
||||||
|
When `POST /vehicles/{P2}/setDefault { IsDefault:true }` is issued
|
||||||
|
Then response is `200` with `Id==P2, IsDefault==true`, and side-channel shows `P2.is_default=true`, `P1.is_default=false`, count==1
|
||||||
|
|
||||||
|
**AC-4: FT-P-04 list is unpaginated and ordered**
|
||||||
|
Given `seed_3_vehicles_2_default` containing `BR-01, BR-02, MQ-9` in any insert order
|
||||||
|
When `GET /vehicles` is issued
|
||||||
|
Then response is `200`, body parses as a JSON array (NOT an object with `Items`), `body.length == 3`, and `[v.Name for v in body] == ["BR-01","BR-02","MQ-9"]`
|
||||||
|
|
||||||
|
**AC-5: FT-P-05 filter is case-INSENSITIVE**
|
||||||
|
Given `seed_3_vehicles_2_default`
|
||||||
|
When `GET /vehicles?name=BR&isDefault=true` AND `GET /vehicles?name=br&isDefault=true` are issued
|
||||||
|
Then both responses are `200` with `body.length == 1` and `body[0].Name == "BR-01"`
|
||||||
|
|
||||||
|
**AC-6: FT-P-06 delete is 204 + row gone**
|
||||||
|
Given one vehicle row with no missions referencing it
|
||||||
|
When `DELETE /vehicles/{id}` is issued
|
||||||
|
Then response is `204 No Content` with empty body, and side-channel shows `count == 0` for that id
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- Each test must complete inside the documented `Max execution time` from `blackbox-tests.md` (5s for FT-P-01..03, 5s for FT-P-07-style writes, 2s for FT-P-04..06). The xUnit `[Trait("max_ms", "5000")]` or per-test `Timeout` must reflect this.
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- Tests share a `[Collection("Vehicles")]` xUnit collection and use `IClassFixture<DbResetFixture>` to TRUNCATE between scenarios. No state must leak between FT-P-01 and FT-P-04.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `seed_empty`, JWT permissions=FL | `POST /vehicles` non-default body | `201` + PascalCase `Vehicle` + DB count 1 | — |
|
||||||
|
| AC-2 | `seed_one_default_vehicle` (P1) | `POST /vehicles { IsDefault:true }` | `201` + DB shows count==1 default after | AC-1.2 invariant |
|
||||||
|
| AC-3 | `seed_one_default_vehicle` + extra P2 | `POST /vehicles/{P2}/setDefault` | `200` + DB count==1 default; P1 cleared | AC-1.2 / AC-1.4 |
|
||||||
|
| AC-4 | `seed_3_vehicles_2_default` (`BR-01,BR-02,MQ-9`) | `GET /vehicles` shape + order | `200` + array + Name ASC | AC-1.5 |
|
||||||
|
| AC-5 | `seed_3_vehicles_2_default` | `GET /vehicles?name=BR…` + `?name=br…` | `200` + len 1 + `BR-01` for both casings | AC-1.6 |
|
||||||
|
| AC-6 | One row, zero missions | `DELETE /vehicles/{id}` | `204` + DB count 0 | AC-1.10 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080` (no project reference to `Azaion.Missions.csproj`).
|
||||||
|
- Bearer token minted via `https://jwks-mock:8443/sign` with `permissions=FL`.
|
||||||
|
- DB assertions through the Npgsql side-channel only; marked `[Trait("db_access","seed-or-assert-only")]`.
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` comments per `coderule.mdc`.
|
||||||
|
- PascalCase JSON contract (`PropertyNamingPolicy = null`) is part of the SUT contract; the test must NOT silently accept camelCase.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: Tests depend on side-channel SQL that drifts from the SUT migrator**
|
||||||
|
- *Risk*: If the migrator changes the `vehicles` column set, hand-rolled `INSERT` in the seed fixture breaks.
|
||||||
|
- *Mitigation*: Seed fixtures use the schema produced by the SUT's own startup migrator — `docker compose up` runs first, then the fixture inserts into the already-migrated tables.
|
||||||
|
|
||||||
|
**Risk 2: Ordering test (AC-4) is flaky if insert order accidentally matches alphabetic order**
|
||||||
|
- *Risk*: A non-deterministic seed insert could mask a missing `OrderBy`.
|
||||||
|
- *Mitigation*: Seed fixture inserts rows in `[MQ-9, BR-02, BR-01]` order (reverse alphabetic) so the test fails if the SUT omits the `OrderBy(a => a.Name)`.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080/vehicles*`) plus the documented DB side-channel for fixture seeding and post-call assertions; expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.1, 1.2, 1.4, 1.5, 1.6, 1.10.
|
||||||
|
- Stubs are allowed ONLY for the external `admin` JWT issuer (the `jwks-mock` container per `tests/Azaion.Missions.JwksMock/`).
|
||||||
|
- Stubs, fakes, monkeypatches, deterministic fallbacks, or direct imports are NOT allowed for any internal product module — including `VehicleService`, `VehiclesController`, `AppDataConnection`, `DatabaseMigrator`, `JwtExtensions`, or `ErrorHandlingMiddleware`. If any of these is not implemented (e.g., the SUT image hasn't been built), the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
# Missions Positive Flow Tests
|
||||||
|
|
||||||
|
**Task**: AZ-578_test_missions_positive
|
||||||
|
**Name**: Missions positive tests (FT-P-07..12)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 6 happy-path Mission scenarios — create with default CreatedDate, paginated list (PageSize=20, CreatedDate DESC, case-INSENSITIVE name filter), page 2, date-range filter, partial update preserving null fields, and full cascade delete across map_objects/detection/annotations/media/waypoints/missions.
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-578
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The `/missions` surface is the project's most consequential read+write path. Three behaviours are easy to silently break: (1) the default `CreatedDate = UtcNow` when the body omits it (AC-2.1), (2) `PaginatedResponse<Mission>` envelope with `Page,PageSize,TotalCount,Items` PascalCase keys + `CreatedDate DESC` ordering (AC-2.3), and (3) the cascade delete walking every dependency table including DB-only stub tables `map_objects`, `detection`, `annotations`, `media` (AC-3.1). The cascade is **not** transaction-wrapped (NFT-RES-01 in Task 16 pins that invariant); the positive scenario here verifies the happy-path walk completes.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All six FT-P-07..12 scenarios run against the dockerised `missions` service and pass.
|
||||||
|
- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-2.x` or `AC-3.1`, `Result=pass`, within the documented `Max execution time` (5s for create, 2s for list/update, 10s for cascade delete).
|
||||||
|
- The pagination test asserts both the envelope shape (`Items, TotalCount, Page, PageSize` PascalCase) AND `CreatedDate` DESC ordering across all 20 items.
|
||||||
|
- The cascade test compares per-table delete counts against `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` via `json_diff`.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- FT-P-07 Mission create with default CreatedDate — assert `|body.CreatedDate - t0| ≤ 5s`.
|
||||||
|
- FT-P-08 Mission list default page — envelope shape, `Page==1`, `PageSize==20`, `TotalCount==25`, `Items.length==20`, `CreatedDate` DESC ordering, plus case-INSENSITIVE `?name=re` filter.
|
||||||
|
- FT-P-09 Mission list page 2 — `Page==2`, `Items.length==5`, UUID-set disjoint from page 1.
|
||||||
|
- FT-P-10 Mission list date range — `?fromDate=&toDate=` inclusivity (January 2026 returns 5 of 25).
|
||||||
|
- FT-P-11 Mission partial update — `PUT /missions/{id}` with `VehicleId:null` preserves prior `VehicleId`.
|
||||||
|
- FT-P-12 Mission cascade delete (F3) — `DELETE /missions/{id}` walks every dependency table; per-table counts compared against `cascade_F3_walk.json`.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- FT-N-04 "create mission with non-existent VehicleId returns 400" lives in Task 13.
|
||||||
|
- FT-N-05 "GET mission 404" lives in Task 13.
|
||||||
|
- FT-N-06 "cascade delete short-circuits on missing mission (no DELETE issued against dependency tables)" lives in Task 13.
|
||||||
|
- Cascade NOT-transaction-wrapped invariant (NFT-RES-01) lives in Task 16.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: FT-P-07 mission create defaults CreatedDate to UtcNow**
|
||||||
|
Given `seed_one_default_vehicle` and a JWT with `permissions=FL`
|
||||||
|
When the consumer captures `t0 = UtcNow` then issues `POST /missions { Name:"Recon-01", VehicleId:<id>, CreatedDate:null }`
|
||||||
|
Then response is `201`, `body.CreatedDate` parses as UTC, and `abs(body.CreatedDate - t0) ≤ 5s`
|
||||||
|
|
||||||
|
**AC-2: FT-P-08 list returns PaginatedResponse with DESC ordering and case-INSENSITIVE name filter**
|
||||||
|
Given `seed_25_missions` (5 January, 20 February 2026, mix of `Recon-*` names)
|
||||||
|
When `GET /missions` is issued
|
||||||
|
Then response is `200` with `Page==1, PageSize==20, TotalCount==25, Items.length==20`, all PascalCase keys, AND for every `i ∈ [0..18]` `Items[i].CreatedDate >= Items[i+1].CreatedDate` (strictly DESC ordering)
|
||||||
|
And when `GET /missions?name=re` (lowercase) is issued, `body.TotalCount > 0` (case-INSENSITIVE substring match against `Recon-*`)
|
||||||
|
|
||||||
|
**AC-3: FT-P-09 page 2 returns the remaining 5 items, disjoint from page 1**
|
||||||
|
Given `seed_25_missions`
|
||||||
|
When `GET /missions?page=2&pageSize=20` is issued
|
||||||
|
Then response is `200`, `Page==2`, `Items.length==5`, AND the set of `Items[*].Id` is disjoint from the page-1 response
|
||||||
|
|
||||||
|
**AC-4: FT-P-10 date range filter is inclusive of bounds**
|
||||||
|
Given `seed_25_missions` (5 in January 2026, 20 in February 2026)
|
||||||
|
When `GET /missions?fromDate=2026-01-01T00:00:00Z&toDate=2026-01-31T23:59:59Z` is issued
|
||||||
|
Then response is `200`, `TotalCount==5`, and every `Items[i].CreatedDate` is within January 2026 UTC
|
||||||
|
|
||||||
|
**AC-5: FT-P-11 partial update preserves null fields**
|
||||||
|
Given one mission row with known `Name="Original"` and `VehicleId=V1`
|
||||||
|
When `PUT /missions/{id} { Name:"Renamed", VehicleId:null }` is issued
|
||||||
|
Then response is `200`, `body.Name == "Renamed"`, AND `body.VehicleId == V1` (preserved)
|
||||||
|
|
||||||
|
**AC-6: FT-P-12 cascade delete walks every dependency table**
|
||||||
|
Given `fixture_cascade_F3` applied (one mission with 2 waypoints → 2 media → 2 annotations → 2 detection rows + 3 map_objects)
|
||||||
|
When `DELETE /missions/{mid}` is issued
|
||||||
|
Then response is `204`, AND side-channel `SELECT COUNT(*)` returns 0 for `map_objects`, `detection`, `annotations`, `media`, `waypoints`, `missions` rows in the seeded chain
|
||||||
|
And the per-table counts after deletion match `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` via deep JSON diff
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- FT-P-07: ≤ 5s. FT-P-08..11: ≤ 2s each. FT-P-12: ≤ 10s (cascade through 5 tables).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- FT-P-12 must use `IClassFixture<DbResetFixture>` that recreates `fixture_cascade_F3` fresh per scenario (the fixture is destructive). FT-P-08..10 share `seed_25_missions` across the same class.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `seed_one_default_vehicle` | `POST /missions { CreatedDate:null }` | `201` + `\|body.CreatedDate - t0\| ≤ 5s` | AC-2.1 |
|
||||||
|
| AC-2 | `seed_25_missions` | `GET /missions` then `GET /missions?name=re` | `200` + envelope + DESC + case-INSENSITIVE match | AC-2.3, AC-8.7 |
|
||||||
|
| AC-3 | `seed_25_missions` | `GET /missions?page=2&pageSize=20` | `200` + `Page=2` + len 5 + disjoint UUIDs | AC-2.3 |
|
||||||
|
| AC-4 | `seed_25_missions` | `GET /missions?fromDate=…&toDate=…` (January window) | `200` + `TotalCount=5` + all in window | AC-2.3 |
|
||||||
|
| AC-5 | One row with `Name=Original, VehicleId=V1` | `PUT /missions/{id} { Name:"Renamed", VehicleId:null }` | `200` + Name updated + VehicleId preserved | AC-2.5 |
|
||||||
|
| AC-6 | `fixture_cascade_F3` | `DELETE /missions/{mid}` | `204` + DB counts 0 across 6 tables + `cascade_F3_walk.json` match | AC-3.1 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080/missions*` (no project reference to `Azaion.Missions.csproj`).
|
||||||
|
- Bearer token minted via `https://jwks-mock:8443/sign` with `permissions=FL`.
|
||||||
|
- FT-P-12 fixture uses the SQL file at `_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql` (NOT a hand-rolled INSERT — the SQL file is the contract).
|
||||||
|
- Per-table count comparison in FT-P-12 uses `json_diff` against `cascade_F3_walk.json`; if the file is missing, the test must fail (not silently pass).
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
- `seed_25_missions` MUST use deterministic UUIDs and deterministic `CreatedDate` values so the disjoint-set assertion in AC-3 and the date-range assertion in AC-4 are reproducible.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: cascade_F3_walk.json drifts from fixture_cascade_F3.sql**
|
||||||
|
- *Risk*: Updating the seed SQL without updating the walk JSON makes AC-6 silently pass with wrong counts.
|
||||||
|
- *Mitigation*: Both files live under the same `expected_results/` directory; the test loads the walk JSON at runtime and verifies BOTH that pre-delete counts match the walk's `before` values AND post-delete counts match the walk's `after` values. A drift fails the "before" assertion first.
|
||||||
|
|
||||||
|
**Risk 2: AC-2 ordering assertion is flaky if seed CreatedDate values collide**
|
||||||
|
- *Risk*: Two missions with identical `CreatedDate` produce a tie-breaker-dependent order; the DESC assertion would be deterministic only if the comparator is stable.
|
||||||
|
- *Mitigation*: `seed_25_missions` SQL assigns distinct `CreatedDate` values spaced ≥ 1 second apart; any future seed change must preserve this invariant.
|
||||||
|
|
||||||
|
**Risk 3: cascade test pollutes neighbour scenarios**
|
||||||
|
- *Risk*: F3 fixture deletes rows across 6 tables; if FT-P-12 runs in the same xUnit class as a read-path test, that test sees an empty DB.
|
||||||
|
- *Mitigation*: FT-P-12 lives in its own xUnit `[Collection("CascadeF3")]` and uses `IClassFixture<DbResetFixture>` to reset between every scenario in the class.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080/missions*`) plus the documented DB side-channel for fixture seeding and post-call assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-2 2.1, 2.3, 2.4, 2.5, 2.7 and AC-3 row 3.1, and against the machine-readable file `_docs/00_problem/input_data/expected_results/cascade_F3_walk.json` for the cascade walk.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL because the owning services are out of scope per `environment.md`).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `MissionService`, `MissionsController`, `WaypointService`, `AppDataConnection`, `DatabaseMigrator`, `JwtExtensions`, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# Waypoints + Health Positive Flow Tests
|
||||||
|
|
||||||
|
**Task**: AZ-579_test_waypoints_health_positive
|
||||||
|
**Name**: Waypoints + Health positive tests (FT-P-13..18)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 6 happy-path Waypoint + Health scenarios — waypoint list ordered by OrderNum ASC, waypoint create echoes geo fields (no auto-conversion), waypoint update is full overwrite, health 200 anonymous, health 200 with Postgres stopped (no DB ping), and waypoint cascade delete scoped to one waypoint (sibling chain intact).
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-579
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Waypoints carry two non-obvious behaviors: (1) the list endpoint orders by `OrderNum` ASC regardless of insert order (AC-4.3), and (2) `PUT /missions/{id}/waypoints/{wpId}` is a FULL overwrite even though the DTO looks "partial" (non-nullable enums + numerics) — passing `Height:0` overwrites the previous `Height:120` (AC-4.4). The waypoint cascade delete (AC-4.5) is the tighter sibling of the mission cascade — it must remove the target waypoint's chain (`media → annotations → detection`) without touching a sibling waypoint's chain. The health endpoint (AC-7.1, AC-7.2) is the suite's probe contract: it MUST return 200 anonymously AND MUST NOT ping the database, because the suite reverse proxy uses `/health` to decide whether to route traffic — a DB outage must not depool a healthy process.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All six FT-P-13..18 scenarios run against the dockerised `missions` service and pass.
|
||||||
|
- Each test produces a CSV row with `Category=Blackbox`, `Traces=AC-4.x` or `AC-7.x`, `Result=pass`, within the documented `Max execution time` (2s for FT-P-13..16, 5s for FT-P-17 to allow PG stop, 10s for FT-P-18 cascade).
|
||||||
|
- The list test asserts both shape (JSON array) and ordering (`[1,2,3,4,5]` ASC from a `[3,1,2,5,4]` insert order).
|
||||||
|
- The update test asserts the FULL overwrite by passing `Height:0` and checking the new value is 0 (not the preserved 120).
|
||||||
|
- The "PG stopped" health test asserts the process answers `200` even with `postgres-test` stopped — proving the probe does not ping the DB.
|
||||||
|
- The cascade test (F4) asserts target-waypoint chain deleted AND sibling-waypoint chain preserved, with per-table counts compared against `cascade_F4_walk.json`.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- FT-P-13 Waypoint list ordered by `OrderNum` ASC — `seed_5_waypoints_unordered` inserts in `[3,1,2,5,4]` order.
|
||||||
|
- FT-P-14 Waypoint create echoes `GeoPoint` fields (no auto lat/lon ↔ MGRS conversion today — preserves the documented divergence from spec).
|
||||||
|
- FT-P-15 Waypoint update is full overwrite — `Height:0` overwrites `Height:120`, `OrderNum` changes, `GeoPoint:null` clears.
|
||||||
|
- FT-P-16 Health 200 anonymous — no `Authorization` header, exact JSON `{ "status": "healthy" }`.
|
||||||
|
- FT-P-17 Health 200 with PG stopped — proves process-liveness only, no DB ping.
|
||||||
|
- FT-P-18 Waypoint cascade delete (F4) — `DELETE /missions/{mid}/waypoints/{wp1}`; per-table counts on `wp1` chain go to 0; sibling `wp2` chain intact.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- FT-N-07 "waypoint operation against missing mission returns 404" lives in Task 13.
|
||||||
|
- Waypoint nested existence check (single composite-FK predicate per `state.json` drift entry) is implementation detail; the blackbox test only asserts the observable 404 in FT-N-07.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: FT-P-13 waypoint list is ordered by OrderNum ASC**
|
||||||
|
Given `seed_5_waypoints_unordered` under one mission, with `order_num` values `[3,1,2,5,4]` inserted in that order
|
||||||
|
When `GET /missions/{id}/waypoints` is issued with a valid JWT
|
||||||
|
Then response is `200`, body parses as JSON array, `body.length == 5`, AND `[w.OrderNum for w in body] == [1,2,3,4,5]`
|
||||||
|
|
||||||
|
**AC-2: FT-P-14 waypoint create echoes geo fields, no MGRS conversion**
|
||||||
|
Given one mission row
|
||||||
|
When `POST /missions/{id}/waypoints { GeoPoint:{Lat:50.45, Lon:30.52, Mgrs:null}, WaypointSource:0, WaypointObjective:0, OrderNum:1, Height:120 }` is issued
|
||||||
|
Then response is `201`, `body.GeoPoint.Lat == 50.45`, `body.GeoPoint.Lon == 30.52`, AND `body.GeoPoint.Mgrs == null` (NO auto-conversion)
|
||||||
|
|
||||||
|
**AC-3: FT-P-15 waypoint update is full overwrite**
|
||||||
|
Given one waypoint with `Height=120, OrderNum=1, GeoPoint=(Lat:50.45, …)`
|
||||||
|
When `PUT /missions/{id}/waypoints/{wpId} { GeoPoint:null, WaypointSource:1, WaypointObjective:1, OrderNum:2, Height:0 }` is issued
|
||||||
|
Then response is `200`, `body.Height == 0` (overwritten from 120), `body.OrderNum == 2`, AND `body.GeoPoint == null`
|
||||||
|
|
||||||
|
**AC-4: FT-P-16 health is 200 anonymous**
|
||||||
|
Given a running `missions` container
|
||||||
|
When `GET /health` is issued with NO `Authorization` header
|
||||||
|
Then response is `200`, body is exactly `{ "status": "healthy" }` with case-sensitive key
|
||||||
|
|
||||||
|
**AC-5: FT-P-17 health is 200 with PG stopped**
|
||||||
|
Given `missions` is running AND `docker compose stop postgres-test` has succeeded
|
||||||
|
When `GET /health` is issued
|
||||||
|
Then response is `200`, body is exactly `{ "status": "healthy" }` — proving the probe does NOT ping the DB
|
||||||
|
|
||||||
|
**AC-6: FT-P-18 waypoint cascade scope is one waypoint**
|
||||||
|
Given `fixture_cascade_F4` (target waypoint `wp1` with chain `media → annotations → detection`; sibling waypoint `wp2` with its own chain)
|
||||||
|
When `DELETE /missions/{mid}/waypoints/{wp1}` is issued
|
||||||
|
Then response is `204`, AND side-channel `SELECT COUNT(*)` returns 0 for the `wp1` chain rows in `detection`, `annotations`, `media`, AND for `wp1` itself in `waypoints`
|
||||||
|
And side-channel returns `1` for `wp2` in `waypoints` AND `> 0` for the `wp2` chain rows in `media, annotations, detection`
|
||||||
|
And the per-table counts after deletion match `_docs/00_problem/input_data/expected_results/cascade_F4_walk.json` via deep JSON diff
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- FT-P-13..16: ≤ 2s each. FT-P-17: ≤ 5s (allow PG stop time). FT-P-18: ≤ 10s (cascade through 4 tables).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- FT-P-17 must restore `postgres-test` to `Up` before exiting (try/finally with `docker compose start postgres-test` in the fixture teardown) — otherwise subsequent tests fail with `ConnectionRefused`.
|
||||||
|
- FT-P-18 uses `IClassFixture<DbResetFixture>` with the F4 fixture recreated per scenario.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `seed_5_waypoints_unordered` ([3,1,2,5,4]) | `GET /missions/{id}/waypoints` | `200` + array + OrderNum ASC | AC-4.3 |
|
||||||
|
| AC-2 | One mission row | `POST /missions/{id}/waypoints { GeoPoint:{Lat,Lon,Mgrs:null} }` | `201` + GeoPoint echoed + Mgrs null (no conversion) | AC-4 (data_parameters § 2.3) |
|
||||||
|
| AC-3 | One waypoint Height=120 | `PUT … { Height:0, GeoPoint:null }` | `200` + Height=0 + GeoPoint=null (full overwrite) | AC-4.4 |
|
||||||
|
| AC-4 | Running container | `GET /health` no auth | `200` + exact `{"status":"healthy"}` | AC-7.1 |
|
||||||
|
| AC-5 | PG stopped | `GET /health` | `200` + exact `{"status":"healthy"}` | AC-7.2, AC-7.3 |
|
||||||
|
| AC-6 | `fixture_cascade_F4` | `DELETE /missions/{mid}/waypoints/{wp1}` | `204` + wp1 chain 0 + wp2 chain intact + `cascade_F4_walk.json` match | AC-4.5 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080`; bearer token via `https://jwks-mock:8443/sign` with `permissions=FL` (for waypoint endpoints); FT-P-16 and FT-P-17 explicitly send no `Authorization` header.
|
||||||
|
- FT-P-17 uses `ComposeRestartFixture`-style helper that runs `docker compose -f docker-compose.test.yml stop postgres-test` then `docker compose -f docker-compose.test.yml start postgres-test` in teardown.
|
||||||
|
- FT-P-18 fixture uses `_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql` (NOT a hand-rolled INSERT).
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: FT-P-15 silently passes if SUT exposes a "partial" update path**
|
||||||
|
- *Risk*: If a future refactor adds a JSON-merge update mode, sending `Height:0` might be interpreted as "leave Height unchanged" rather than overwrite.
|
||||||
|
- *Mitigation*: The test ALSO sets `GeoPoint:null` and asserts the value is null after — proving the path is full-overwrite, not patch.
|
||||||
|
|
||||||
|
**Risk 2: FT-P-17 PG-stop leaks to other tests**
|
||||||
|
- *Risk*: If the test fails before teardown, subsequent tests run against a dead DB.
|
||||||
|
- *Mitigation*: The fixture uses `try/finally`; the teardown waits for `postgres-test` to reach `healthy` (poll `pg_isready`) before yielding control back to xUnit.
|
||||||
|
|
||||||
|
**Risk 3: FT-P-18 sibling-intact assertion gives false-pass if F4 fixture is empty**
|
||||||
|
- *Risk*: If `fixture_cascade_F4.sql` failed to insert `wp2`'s chain, the post-delete assertion `wp2 chain > 0` fails trivially — but with a misleading message.
|
||||||
|
- *Mitigation*: The test asserts pre-delete counts FIRST (`wp1` chain > 0 AND `wp2` chain > 0); fixture failure is caught in the Arrange phase, not the Assert phase.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080/missions/{id}/waypoints*` and `http://missions:8080/health`) plus the documented DB side-channel for fixture seeding and post-call assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-4 4.2, 4.3, 4.4, 4.5 and AC-7 rows 7.1, 7.2, and against the machine-readable file `_docs/00_problem/input_data/expected_results/cascade_F4_walk.json`.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection` (seeded via side-channel SQL).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `WaypointService`, `MissionsController` (health route), `AppDataConnection`, or `Program.cs`'s health middleware. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
# Validation + 404 + Authz Negative Tests
|
||||||
|
|
||||||
|
**Task**: AZ-580_test_validation_authz_negative
|
||||||
|
**Name**: Functional negative tests (FT-N-01..08)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 8 negative scenarios — case-insensitive filter no-match, 404 for missing GET vehicle/mission/waypoint-parent, 409 for delete-vehicle-in-use, 400 for create-mission-with-bogus-VehicleId (carry-forward divergence), cascade short-circuit on missing mission (no dependency DELETEs issued), and the generic 500 redacted-body + stacktrace-in-log contract.
|
||||||
|
**Complexity**: 3 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-580
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The negative-path contract is what protects clients from undefined behaviour: every documented failure must produce a predictable status code + `{ statusCode, message }` envelope, and no failure mode may silently mutate state. Three behaviors are especially load-bearing: (1) `DELETE /missions/{missing}` must 404 *before* any dependency-table DELETE issues — otherwise a typo'd UUID could remove rows from `map_objects` belonging to a different mission (AC-3.2); (2) `DELETE /vehicles/{used}` must 409 and leave the row in place (AC-1.8); (3) the generic 500 must redact internals — `Internal server error` body, full stack only in container logs (AC-8.6, AC-10.3).
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All eight FT-N-01..08 scenarios run against the dockerised `missions` service and pass.
|
||||||
|
- Each test produces a CSV row with `Category=Blackbox` (negative subset; `Traces=AC-1.6, AC-1.7, AC-1.8, AC-2.2, AC-2.4, AC-3.2, AC-4.2, AC-8.6, AC-10.3`), `Result=pass`.
|
||||||
|
- The 500 test asserts BOTH that the body is exactly `{ "statusCode":500, "message":"Internal server error" }` AND that the container log emitted an `"Unhandled exception"` line within 2s.
|
||||||
|
- FT-N-06 asserts via `pg_stat_statements` (or post-request log scrape) that NO `DELETE FROM map_objects/waypoints/media/annotations/detection` SQL ran during the 404 request — the existence check short-circuits before the cascade.
|
||||||
|
- FT-N-04 explicitly pins the documented spec-divergence (returns 400 today, spec wants 404); test must include a comment marking it as a carry-forward to revisit when the divergence is closed.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- FT-N-01 Vehicle name filter no-match — `?name=ZZ` and `?name=zz` against `seed_3_vehicles_2_default` both return `body.length == 0`.
|
||||||
|
- FT-N-02 GET vehicle 404 — random UUID returns `{ statusCode:404, message:… }`.
|
||||||
|
- FT-N-03 Delete vehicle in use 409 — row not deleted afterwards.
|
||||||
|
- FT-N-04 Create mission with bogus VehicleId returns 400 today (CARRY-FORWARD comment).
|
||||||
|
- FT-N-05 GET mission 404 — envelope shape.
|
||||||
|
- FT-N-06 Cascade short-circuit — 404 + zero DELETE SQL issued.
|
||||||
|
- FT-N-07 Waypoint operation against missing mission — 404.
|
||||||
|
- FT-N-08 Generic 500 — redacted body + stacktrace in log.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- 401 / 403 auth-failure paths (NFT-SEC-01..06) live in Task 14.
|
||||||
|
- 400/422 spec-divergence carry-forwards that are NOT executable today (input validation for empty `Name`, negative `BatteryCapacity`, unknown `Type` int) are documented as Refactor Backlog items in `tests/blackbox-tests.md` and are NOT in scope here.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: FT-N-01 vehicle filter no-match returns empty array for both casings**
|
||||||
|
Given `seed_3_vehicles_2_default` (`BR-01, BR-02, MQ-9`)
|
||||||
|
When `GET /vehicles?name=ZZ` then `GET /vehicles?name=zz` are issued
|
||||||
|
Then both responses are `200` with `body.length == 0`
|
||||||
|
|
||||||
|
**AC-2: FT-N-02 GET vehicle 404 returns the standard envelope**
|
||||||
|
Given any DB state and a valid JWT
|
||||||
|
When `GET /vehicles/{random uuid}` is issued
|
||||||
|
Then response is `404` with body parsing to JSON object having EXACTLY the keys `statusCode` and `message`, and `statusCode == 404`
|
||||||
|
|
||||||
|
**AC-3: FT-N-03 delete in-use vehicle returns 409 and leaves row**
|
||||||
|
Given one vehicle and ≥ 1 mission referencing it
|
||||||
|
When `DELETE /vehicles/{id}` is issued
|
||||||
|
Then response is `409` with envelope `{ statusCode:409, message:<non-empty> }`, and side-channel `SELECT COUNT(*) FROM vehicles WHERE id={id}` returns `1`
|
||||||
|
|
||||||
|
**AC-4: FT-N-04 create mission with bogus VehicleId returns 400 today (carry-forward)**
|
||||||
|
Given `seed_empty`
|
||||||
|
When `POST /missions { Name:"x", VehicleId:<random uuid>, CreatedDate:null }` is issued
|
||||||
|
Then response is `400` with envelope (carry-forward: spec wants 404; the test must include a `// CARRY-FORWARD: expected to flip to 404 when AC-2.2 divergence is closed` comment)
|
||||||
|
And side-channel `SELECT COUNT(*) FROM missions` returns `0`
|
||||||
|
|
||||||
|
**AC-5: FT-N-05 GET mission 404 returns the standard envelope**
|
||||||
|
Given any DB state and a valid JWT
|
||||||
|
When `GET /missions/{random uuid}` is issued
|
||||||
|
Then response is `404` with envelope `{ statusCode:404, message:<non-empty> }`
|
||||||
|
|
||||||
|
**AC-6: FT-N-06 cascade short-circuit issues zero dependency-table DELETEs**
|
||||||
|
Given `fixture_cascade_F3` (seeded chain rooted at `mid`) and a `postgres-test` started with `log_statement=all`
|
||||||
|
When `DELETE /missions/{mid'}` (random UUID, not `mid`) is issued
|
||||||
|
Then response is `404`, side-channel `SELECT COUNT(*) FROM map_objects` is unchanged, AND the `postgres-test` log (or `pg_stat_statements`) shows NO `DELETE FROM map_objects/waypoints/media/annotations/detection` SQL emitted by the request connection
|
||||||
|
|
||||||
|
**AC-7: FT-N-07 waypoint operation against missing mission returns 404**
|
||||||
|
Given any DB state and a valid JWT
|
||||||
|
When `GET /missions/{random uuid}/waypoints` is issued
|
||||||
|
Then response is `404` with envelope `{ statusCode:404, message:<non-empty> }`
|
||||||
|
|
||||||
|
**AC-8: FT-N-08 generic 500 redacts body, stacktrace lands in log**
|
||||||
|
Given side-channel has executed `DROP TABLE vehicles CASCADE`
|
||||||
|
When `GET /vehicles/{any uuid}` is issued with JWT `FL`
|
||||||
|
Then response is `500` with body EXACTLY `{ "statusCode":500, "message":"Internal server error" }`
|
||||||
|
And `docker logs missions-sut` contains an `"Unhandled exception"` line emitted ≤ 2s after the request timestamp, containing the exception type name (`PostgresException` or similar)
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- FT-N-01..05, FT-N-07: ≤ 2s each. FT-N-06: ≤ 5s. FT-N-08: ≤ 5s (allow log scrape).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- FT-N-06 requires `postgres-test` to be started with `log_statement=all` (`command: ["postgres", "-c", "log_statement=all"]` overlay in `docker-compose.test.yml`, OR `ALTER SYSTEM SET` via side-channel in the fixture). The test must FAIL if logging is not enabled — not silently pass.
|
||||||
|
- FT-N-08 is destructive (drops the `vehicles` table). It MUST run in its own xUnit `[Collection("ErrorEnvelope500")]` with `ComposeRestartFixture` teardown (full `down -v && up -d`).
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `seed_3_vehicles_2_default` | `?name=ZZ` then `?name=zz` | `200` + `body.length == 0` for both | AC-1.6 |
|
||||||
|
| AC-2 | any | `GET /vehicles/{random}` | `404` + envelope | AC-1.7, AC-8.2 |
|
||||||
|
| AC-3 | Vehicle + mission referencing it | `DELETE /vehicles/{id}` | `409` + row preserved | AC-1.8, AC-8.5 |
|
||||||
|
| AC-4 | `seed_empty` | `POST /missions { VehicleId:<random> }` | `400` (today) + no row written + carry-forward comment | AC-2.2 |
|
||||||
|
| AC-5 | any | `GET /missions/{random}` | `404` + envelope | AC-2.4, AC-8.2 |
|
||||||
|
| AC-6 | `fixture_cascade_F3` + PG logging on | `DELETE /missions/{random}` | `404` + zero dependency-table DELETE SQL | AC-3.2 |
|
||||||
|
| AC-7 | any | `GET /missions/{random}/waypoints` | `404` + envelope | AC-4.2 |
|
||||||
|
| AC-8 | side-channel DROPped vehicles | `GET /vehicles/{any}` | `500` + redacted body + stacktrace logged within 2s | AC-8.6, AC-10.3 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080`; bearer token via `https://jwks-mock:8443/sign` with `permissions=FL`.
|
||||||
|
- FT-N-06 requires Postgres logging mode `log_statement=all`; the fixture must verify (via `SHOW log_statement`) that logging is on BEFORE running the test — fail in Arrange if not.
|
||||||
|
- FT-N-08 fixture teardown must restart the compose stack (`down -v && up -d`); subsequent tests would otherwise hit a missing table.
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
- Carry-forward comments (FT-N-04) are required so future spec-vs-code work knows where to update.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: FT-N-06 false-pass when PG logging is off**
|
||||||
|
- *Risk*: If `postgres-test` runs without `log_statement=all`, the "no DELETE issued" assertion trivially passes — the log is empty.
|
||||||
|
- *Mitigation*: Arrange phase runs `SHOW log_statement` via side-channel and fails fast if the result is not `"all"`. The compose overlay setting this MUST be loaded.
|
||||||
|
|
||||||
|
**Risk 2: FT-N-08 leaves the SUT in a broken state**
|
||||||
|
- *Risk*: After `DROP TABLE vehicles CASCADE`, every subsequent test against `/vehicles` returns 500 until the migrator re-creates the table on next startup.
|
||||||
|
- *Mitigation*: Fixture runs `docker compose -f docker-compose.test.yml down -v && up -d` in teardown; subsequent tests wait for `missions` to reach `healthy`.
|
||||||
|
|
||||||
|
**Risk 3: FT-N-04 expectation flips silently when spec divergence closes**
|
||||||
|
- *Risk*: When the spec-aligned 404 lands, this test will fail with a status mismatch — and the test author needs context to know it's intentional.
|
||||||
|
- *Mitigation*: The test includes a `// CARRY-FORWARD: AC-2.2 — expected to flip to 404 when bogus-VehicleId divergence is closed` source-level comment AND `[Trait("carry_forward", "AC-2.2")]` so a future filter can find it.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080/{vehicles,missions}*`) plus the documented DB side-channel for fixture seeding, post-call assertions, and (for FT-N-06) reading `pg_stat_statements` / Postgres log lines, and (for FT-N-08) reading `docker logs missions-sut`. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.7, 1.8, 1.9; AC-2 2.2, 2.6; AC-3 3.2; AC-4 4.1; AC-8 8.7; AC-10 10.1.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `VehicleService`, `MissionService`, `WaypointService`, the controllers, `ErrorHandlingMiddleware`, `AppDataConnection`, `DatabaseMigrator`, or `JwtExtensions`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
# Security Tests — Auth & Claims
|
||||||
|
|
||||||
|
**Task**: AZ-581_test_security_auth_claims
|
||||||
|
**Name**: Security tests — auth & claims (NFT-SEC-01..06 + 04b)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 7 JWT authn/authz scenarios — missing/invalid header, invalid signature (single-byte flip + foreign-keypair), expired-outside-skew vs inside-30s-skew, wrong `iss`, wrong `aud`, missing `permissions`, wrong/multi-value `permissions` claim (contains-match accepts `["FL","ADMIN"]`).
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-581
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
JWT validation is the only thing standing between the open `e2e-net` and the protected `/vehicles` + `/missions` + `/missions/{id}/waypoints` surface. Six failure modes (no header / bad signature / expired / wrong iss / wrong aud / wrong perm) MUST all produce `401` or `403` deterministically — any drift means an attacker who learns the JWKS public bytes could shape a token that bypasses one rule and rides through. The drift re-verification of 2026-05-14 split AC-5.3 into two checks (`iss` AND `aud`) and tightened the clock skew from .NET's 5-min default to 30s; this task pins both. NFT-SEC-06 specifically asserts the `RequireClaim("permissions","FL")` is contains-match — a multi-permission token `["FL","ADMIN"]` must be accepted, while `"fl"` / `"FLight"` / `"ADMIN"` alone must be rejected.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All seven NFT-SEC-01..06 + 04b scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=Sec`, `Traces=AC-5.x` or `AC-9.x`, `Result=pass`.
|
||||||
|
- NFT-SEC-02 covers BOTH the single-byte-flip case AND the foreign-keypair case (token signed by a separate ECDSA keypair never published in the JWKS).
|
||||||
|
- NFT-SEC-03 verifies the 30s skew BOTH ways — `exp_offset_seconds=-60` rejected, `exp_offset_seconds=-15` accepted.
|
||||||
|
- NFT-SEC-06 verifies multi-permission token acceptance — `permissions: ["FL","ADMIN"]` → `200`.
|
||||||
|
- NFT-SEC-01 asserts no DB side-effect on the `POST /vehicles` 401 path (side-channel count unchanged).
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-SEC-01 Missing `Authorization` header on `/vehicles` GET/POST, `/missions` GET, `/missions/{any}/waypoints` GET — all `401`, no DB row written on the POST.
|
||||||
|
- NFT-SEC-02 Invalid signature — single-byte-flipped signature segment AND foreign-keypair tokens.
|
||||||
|
- NFT-SEC-03 Expired token — `exp_offset_seconds=-60` → `401`; `exp_offset_seconds=-15` → `200` (inside 30s skew).
|
||||||
|
- NFT-SEC-04 Wrong `iss` — `POST /sign { "iss": "https://attacker.example.com" }` → `401`; default `iss` → `200`.
|
||||||
|
- NFT-SEC-04b Wrong `aud` — `POST /sign { "aud": "wrong-audience" }` → `401`.
|
||||||
|
- NFT-SEC-05 Missing `permissions` claim — `403`.
|
||||||
|
- NFT-SEC-06 Wrong `permissions` value AND multi-permission acceptance — `"fl"`, `"FLight"`, `"ADMIN"` → `403`; `["FL","ADMIN"]` → `200`.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- NFT-SEC-07 health-exempt-from-auth lives in Task 15.
|
||||||
|
- NFT-SEC-08 stacktrace-not-leaked overlaps with FT-N-08 in Task 13 (and lives in Task 15 for the security-shaped variant).
|
||||||
|
- NFT-SEC-09 SQL injection guard lives in Task 15.
|
||||||
|
- NFT-SEC-10 alg-pin lives in Task 15.
|
||||||
|
- NFT-SEC-11 unknown-kid rotation lag lives in Task 15.
|
||||||
|
- NFT-SEC-12 missing-env startup throw lives in Task 15.
|
||||||
|
- NFT-SEC-13 CORS Production-gate lives in Task 15.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-SEC-01 missing header rejects every protected endpoint with 401, no side-effect**
|
||||||
|
Given the running test stack
|
||||||
|
When the consumer issues `GET /vehicles`, `GET /missions`, `GET /missions/{any}/waypoints`, and `POST /vehicles` with a valid body — all without an `Authorization` header
|
||||||
|
Then each response is `401`, AND side-channel `SELECT COUNT(*) FROM vehicles` before and after the `POST` are equal
|
||||||
|
|
||||||
|
**AC-2: NFT-SEC-02 invalid signature rejects two attack shapes**
|
||||||
|
Given a valid signed token `T_good` from `jwks-mock POST /sign`
|
||||||
|
When the consumer flips a single byte in `T_good`'s signature segment producing `T_bad`, and separately mints `T_foreign` signed by an ECDSA keypair never published in the JWKS
|
||||||
|
Then `GET /vehicles` with `T_bad` returns `401` AND `GET /vehicles` with `T_foreign` returns `401`
|
||||||
|
|
||||||
|
**AC-3: NFT-SEC-03 30s clock skew is enforced on both sides**
|
||||||
|
Given the mock with default issuer/audience
|
||||||
|
When the consumer mints two tokens via `POST /sign { exp_offset_seconds: -60 }` and `POST /sign { exp_offset_seconds: -15 }`
|
||||||
|
Then `GET /vehicles` with the −60s token returns `401` AND `GET /vehicles` with the −15s token returns `200`
|
||||||
|
|
||||||
|
**AC-4: NFT-SEC-04 wrong `iss` rejected, matching `iss` accepted**
|
||||||
|
When the consumer mints a token via `POST /sign { iss: "https://attacker.example.com" }` and another via `POST /sign {}` (default iss)
|
||||||
|
Then `GET /vehicles` with the attacker-iss token returns `401` AND with the default-iss token returns `200`
|
||||||
|
|
||||||
|
**AC-5: NFT-SEC-04b wrong `aud` rejected**
|
||||||
|
When the consumer mints a token via `POST /sign { aud: "wrong-audience" }`
|
||||||
|
Then `GET /vehicles` returns `401`
|
||||||
|
|
||||||
|
**AC-6: NFT-SEC-05 missing `permissions` claim rejected with 403**
|
||||||
|
When the consumer mints a token with no `permissions` claim (mock body `{ permissions: "" }` or `{ permissions: null }` per the mock's contract)
|
||||||
|
Then `GET /vehicles` returns `403` (NOT 401 — signature is valid)
|
||||||
|
|
||||||
|
**AC-7: NFT-SEC-06 contains-match policy on `permissions`**
|
||||||
|
When the consumer mints tokens with `permissions` values `"ADMIN"`, `"fl"` (lowercase), `"FLight"`, AND `["FL","ADMIN"]` (multi-value array)
|
||||||
|
Then `GET /vehicles` returns `403` for the first three AND `200` for the multi-value `["FL","ADMIN"]` array (contains-match accepts `"FL"` among the values)
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-SEC-01..06: ≤ 5s each. The Authorization-header failure paths are cheap (no DB round-trip on the 401/403 short-circuit).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- NFT-SEC-02 requires an out-of-band ECDSA-keypair helper that lives inside the test project, NOT in `jwks-mock` (the mock must never publish a public key it does not control). The helper generates a P-256 keypair at test-start and signs a token directly using `System.Security.Cryptography.ECDsa` — the public key is never registered with `missions`.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | running stack | 4 endpoints w/o Authorization | all 401; POST no DB write | AC-5.4 |
|
||||||
|
| AC-2 | `T_good` from mock + foreign keypair | flipped signature; foreign-keypair token | both 401 | AC-5.5 |
|
||||||
|
| AC-3 | mock with default iss/aud | exp_offset −60s vs −15s | 401 / 200 | AC-5.2, AC-5.6 |
|
||||||
|
| AC-4 | mock | iss=attacker vs default | 401 / 200 | AC-5.3, AC-5.11 |
|
||||||
|
| AC-5 | mock | aud=wrong | 401 | AC-5.3, AC-5.12 |
|
||||||
|
| AC-6 | mock | permissions missing | 403 | AC-5.8, AC-9.1 |
|
||||||
|
| AC-7 | mock | permissions=ADMIN/fl/FLight/["FL","ADMIN"] | 403/403/403/200 | AC-9.1, AC-9.2 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080`. Tokens minted via `https://jwks-mock:8443/sign` with parameterised overrides.
|
||||||
|
- NFT-SEC-02 foreign-keypair: a test-only helper inside `Azaion.Missions.E2E.Tests` MAY use `System.Security.Cryptography.ECDsa` directly for the attack-token construction; this is the ONLY in-test signing path allowed — every other test must use the mock.
|
||||||
|
- NFT-SEC-06 multi-permission token requires the mock's `POST /sign` body to accept `permissions` as either a string OR a JSON array; the test-infrastructure ticket (AZ-576) covers this in the mock's contract.
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: NFT-SEC-03 flaky due to wall-clock variability**
|
||||||
|
- *Risk*: A −15s offset could fail if Docker time skew between the mock and `missions` is large.
|
||||||
|
- *Mitigation*: Both containers run on the same host clock (no `--init` time isolation); test asserts only at offsets well clear of the 30s boundary (−60s and −15s — 30s and 15s away from the boundary respectively).
|
||||||
|
|
||||||
|
**Risk 2: NFT-SEC-06 multi-permission shape varies between systems**
|
||||||
|
- *Risk*: If the spec for `permissions` claim later changes from "contains-match string" to "exact-array-membership", the multi-value assertion breaks.
|
||||||
|
- *Mitigation*: Test traces explicitly to AC-9.2 and references `Auth/JwtExtensions.cs` policy registration; any change there must update this test in the same commit.
|
||||||
|
|
||||||
|
**Risk 3: Foreign-keypair token validation might pass if the SUT silently trusts any well-formed ECDSA token**
|
||||||
|
- *Risk*: A regression that disables `IssuerSigningKeyResolver` would let the foreign-keypair token through.
|
||||||
|
- *Mitigation*: Mitigated by the structure of AC-2 — both bad-signature shapes (flipped byte AND foreign keypair) must return 401.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080/{vehicles,missions}*`) and acquire signed tokens via `https://jwks-mock:8443/sign` (with the test-only foreign-keypair helper for NFT-SEC-02). Expected outputs are the documented HTTP status codes from `_docs/00_problem/input_data/expected_results/results_report.md` AC-5 rows and AC-9 rows.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs` (auth pipeline registration), the `[Authorize(Policy = "FL")]` filter, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# Security Tests — Alg-pin / Rotation / CORS / No-leak
|
||||||
|
|
||||||
|
**Task**: AZ-582_test_security_alg_rotation_cors
|
||||||
|
**Name**: Security tests — alg-pin, rotation, CORS, no-leak (NFT-SEC-07..13)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 7 cross-cutting security scenarios — health endpoint anonymous-OK (NFT-SEC-07), 500 redacted body shape (NFT-SEC-08), SQL-injection guard via parameterised queries (NFT-SEC-09), algorithm-pin defends against HS256-confusion and unsigned tokens (NFT-SEC-10), unknown-`kid` rotation lag with old-key grace window (NFT-SEC-11), startup fail-fast on missing required env vars + HTTPS-only JWKS URL (NFT-SEC-12), and CORS Production-gate fail-fast + permissive-default-warning in non-Production (NFT-SEC-13).
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-582
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Six of these scenarios pin invariants that were broken in earlier code paths and structurally fixed during the 2026-05-14 drift cycle. NFT-SEC-10 (alg-pin) defends against the most common JWKS-public-key-as-HMAC-secret attack. NFT-SEC-11 (kid rotation) verifies that the test-infrastructure JWKS cache shortening (C01) actually shrinks rotation lag inside the 15-minute CI gate. NFT-SEC-12 verifies all four `Infrastructure/ConfigurationResolver.ResolveRequiredOrThrow` calls — `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL`. NFT-SEC-13 verifies `CorsConfigurationValidator.EnsureSafeForEnvironment` actually throws on `ASPNETCORE_ENVIRONMENT=Production` with empty allow-list, AND falls back to permissive with a warning log in `Test`/`Development`. Each is a separate failure mode; together they form the "static config and cryptographic posture" surface that nothing else in the suite covers.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All seven NFT-SEC-07..13 scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=Sec`, `Traces=AC-5.x`/`AC-6.x`/`AC-7.x`/`AC-8.x`/`AC-9.x`/`AC-10.x`, `Result=pass`.
|
||||||
|
- NFT-SEC-10 covers BOTH HS256-confusion (mock signs with the public key as HMAC secret) AND `alg: none` (mock emits unsigned JWT) — both must return `401`.
|
||||||
|
- NFT-SEC-11 (rotation lag) completes inside 120s and exercises the three windows: cached-misses-new-kid → 401, cache-refreshed → 200, old-kid-still-valid-during-grace → 200, post-grace-old-kid → mock refuses to sign.
|
||||||
|
- NFT-SEC-12 runs five separate `docker run` invocations (four missing-env + one HTTP-not-HTTPS JWKS URL); each asserts non-zero exit / log line.
|
||||||
|
- NFT-SEC-13 runs five separate `docker run` invocations spanning Production-fail-fast, Production-AllowAny-warning, Production-with-origins, Production-cross-origin-rejection, Test-permissive-warning.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-SEC-07 Health endpoint anonymous + accepted with expired token (auth pipeline not evaluated).
|
||||||
|
- NFT-SEC-08 500 redacted body — no `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/type-name in body; log has the stack info.
|
||||||
|
- NFT-SEC-09 SQL-injection guard — `?name=' OR '1'='1` and `?name=; DROP TABLE vehicles; --` are treated as literal strings.
|
||||||
|
- NFT-SEC-10 Alg-pin — HS256-confusion AND unsigned token both rejected.
|
||||||
|
- NFT-SEC-11 Unknown-kid rotation lag with old-key grace window.
|
||||||
|
- NFT-SEC-12 Missing required env vars (4 vars) + HTTP-JWKS-URL warning path.
|
||||||
|
- NFT-SEC-13 CORS Production-gate fail-fast + AllowAnyOrigin warning + explicit-origin preflight + cross-origin preflight rejection + non-Production permissive-default warning.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- The 401/403 auth pipeline (NFT-SEC-01..06 + 04b) lives in Task 14.
|
||||||
|
- The destructive `DROP TABLE` mid-test for the 500 path (FT-N-08) lives in Task 13. NFT-SEC-08 here REUSES the same fixture but adds the response-body redaction assertions.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-SEC-07 health is anonymous and skips the auth pipeline**
|
||||||
|
When `GET /health` is issued (a) with no `Authorization` header AND (b) with `Authorization: Bearer <expired token>`
|
||||||
|
Then both responses are `200` with body `{ "status": "healthy" }` — proving the auth pipeline does not run for `/health`
|
||||||
|
|
||||||
|
**AC-2: NFT-SEC-08 500 redacted body**
|
||||||
|
Given the same fixture as FT-N-08 (`DROP TABLE vehicles CASCADE`)
|
||||||
|
When `GET /vehicles/{any uuid}` is issued
|
||||||
|
Then response body is EXACTLY `{ "statusCode":500, "message":"Internal server error" }`, contains NO key matching `stack`/`stackTrace`/`exception`/`inner`/`trace`/file-path/exception-type-name
|
||||||
|
And `docker logs missions-sut` contains an `Unhandled exception` line including the exception type or file path of the throw site
|
||||||
|
|
||||||
|
**AC-3: NFT-SEC-09 SQL-injection guard**
|
||||||
|
Given a running stack with `seed_3_vehicles_2_default`
|
||||||
|
When `GET /vehicles?name=' OR '1'='1` (URL-encoded) is issued
|
||||||
|
Then response is `200` with `body.length == 0` (the literal string does not match any `Name`)
|
||||||
|
And when `GET /missions?name=; DROP TABLE vehicles; --` (URL-encoded) is issued
|
||||||
|
Then response is `200` with `body.TotalCount == 0` AND side-channel `SELECT to_regclass('vehicles')` returns a non-null oid (the table still exists)
|
||||||
|
|
||||||
|
**AC-4: NFT-SEC-10 algorithm-pin rejects HS256-confusion and unsigned**
|
||||||
|
When the consumer mints a token via `POST /sign { alg_override: "HS256" }` (mock signs with the JWKS public key as HMAC secret)
|
||||||
|
Then `GET /vehicles` returns `401`
|
||||||
|
And when the consumer mints a token via `POST /sign { alg_override: "none" }` (unsigned JWT)
|
||||||
|
Then `GET /vehicles` returns `401`
|
||||||
|
|
||||||
|
**AC-5: NFT-SEC-11 unknown-kid rotation completes within 120s with grace window honoured**
|
||||||
|
Given `missions` has a warm JWKS cache and `jwks-mock` is configured with `OLD_KEY_GRACE_SECONDS=5`
|
||||||
|
When the consumer issues `POST jwks-mock:8443/rotate-key {}`, immediately mints a token signed with the new kid, and calls `GET /vehicles` BEFORE missions has refreshed
|
||||||
|
Then the first call returns `401` (new kid not yet in cache)
|
||||||
|
And after waiting for the JWKS refresh window (≤ 90s; the mock sets `max-age=60` and missions has `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30` per C01), the same token returns `200`
|
||||||
|
And during the 5s grace window, a token still signed with the OLD kid is accepted (`200`)
|
||||||
|
And after the grace window expires, the mock refuses to sign with the old kid (`400`/`410` from `POST /sign`)
|
||||||
|
|
||||||
|
**AC-6: NFT-SEC-12 startup fail-fast on required env vars + HTTPS-only JWKS**
|
||||||
|
When `missions` is launched via separate `docker run` invocations, each missing exactly one of `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL` (4 cases)
|
||||||
|
Then in each case the container exits non-zero within 5s AND its logs contain `InvalidOperationException` mentioning the corresponding variable (or its `Database:Url`/`Jwt:Issuer`/`Jwt:Audience`/`Jwt:JwksUrl` config alias)
|
||||||
|
And when `missions` is launched with `JWT_JWKS_URL=http://jwks-mock:8443/...` (HTTP not HTTPS) and the other three set
|
||||||
|
Then the container STARTS, AND the first protected request fails (`500` body or `401` with `RequireHttps` mention) AND the log contains a line mentioning `HTTPS` / `RequireHttps`
|
||||||
|
|
||||||
|
**AC-7: NFT-SEC-13 CORS Production-gate fail-fast + non-Production warning**
|
||||||
|
When `missions` is launched with `ASPNETCORE_ENVIRONMENT=Production` and no `CorsConfig` env vars
|
||||||
|
Then the container exits non-zero within 5s AND its logs contain `InvalidOperationException` mentioning `CorsConfig`/`AllowedOrigins`/Production
|
||||||
|
And when launched with `ASPNETCORE_ENVIRONMENT=Production` + `CorsConfig__AllowAnyOrigin=true`
|
||||||
|
Then the container starts AND the logs contain a warning that CORS is permissive in Production
|
||||||
|
And when launched with `ASPNETCORE_ENVIRONMENT=Production` + `CorsConfig__AllowedOrigins__0=https://operator.example.com`
|
||||||
|
Then `OPTIONS /vehicles` preflight from `https://operator.example.com` returns `200` with `Access-Control-Allow-Origin: https://operator.example.com`
|
||||||
|
And the same preflight from `https://attacker.example.com` responds without the allow-origin echo
|
||||||
|
And when launched with `ASPNETCORE_ENVIRONMENT=Test` and no `CorsConfig`, the container starts AND the logs contain the documented `PermissiveDefaultWarning`
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-SEC-07..10: ≤ 5s each.
|
||||||
|
- NFT-SEC-11: ≤ 120s (rotation + cache refresh).
|
||||||
|
- NFT-SEC-12: ≤ 60s (5 docker-run cycles).
|
||||||
|
- NFT-SEC-13: ≤ 90s (5 docker-run cycles + preflight requests).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- NFT-SEC-11 must run in its own xUnit `[Collection("JwksRotation")]` because rotating the mock affects every subsequent test that already has tokens in flight. After the test, the fixture restores the original key by calling `POST /rotate-key` once more and waits the grace window.
|
||||||
|
- NFT-SEC-12 and NFT-SEC-13 spawn `docker run` from inside the test runner — the runner container must have access to a Docker socket OR the suite-level test orchestrator must run these as separate compose profiles. AZ-576 covers the runner-side Docker access.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | running stack | `GET /health` no-auth and with expired token | both 200 | AC-7.1, AC-9.4 |
|
||||||
|
| AC-2 | dropped `vehicles` table | `GET /vehicles/{any}` | 500 + body has only `statusCode,message` + log has stacktrace | AC-8.6, AC-10.3 |
|
||||||
|
| AC-3 | `seed_3_vehicles_2_default` | `?name=' OR '1'='1` then `?name=; DROP TABLE…` | 200 + len 0 + table still exists | AC-1.6, AC-2.3 defensive |
|
||||||
|
| AC-4 | mock with alg overrides | HS256-confusion token then unsigned token | both 401 | AC-5.1, AC-5.10 |
|
||||||
|
| AC-5 | warm JWKS cache | `POST /rotate-key` + 3 timing checks | 401 → wait → 200; old-kid grace; post-grace mock refuses | AC-5.7 |
|
||||||
|
| AC-6 | 5 docker-run cases | missing DATABASE_URL/JWT_ISSUER/JWT_AUDIENCE/JWT_JWKS_URL + HTTP-not-HTTPS | 4 fail-fast + 1 start-then-500 | AC-6.1, AC-6.2, E1, E3 |
|
||||||
|
| AC-7 | 5 docker-run cases | Production fail-fast, AllowAnyOrigin warn, explicit-origin allow, cross-origin reject, Test permissive warn | per scenario | AC-6.11, E9 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080` for the cases that run inside the standard compose stack. NFT-SEC-12 and NFT-SEC-13 use `docker run` directly against `azaion/missions:test`.
|
||||||
|
- NFT-SEC-09 second probe (`SELECT to_regclass('vehicles')`) requires side-channel Npgsql access AFTER the SUT response — if the table was dropped, the test was wrong.
|
||||||
|
- NFT-SEC-11 fixture must restore the original key before exit (otherwise every test in subsequent collections fails with `kid` mismatch).
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: NFT-SEC-10 false-pass if the mock cannot produce an HS256 token**
|
||||||
|
- *Risk*: If the mock implementation rejects `alg_override="HS256"`, the test never exercises the attack — it gets `400` from the mock and incorrectly thinks `missions` rejected.
|
||||||
|
- *Mitigation*: The test asserts a successful `200 OK` from `jwks-mock POST /sign` BEFORE issuing `GET /vehicles`; mock failure fails Arrange, not Assert.
|
||||||
|
|
||||||
|
**Risk 2: NFT-SEC-11 flake on slow CI**
|
||||||
|
- *Risk*: The 60s `max-age` + 30s `AutoRefresh` + clock variance might push refresh past 120s on a heavily loaded runner.
|
||||||
|
- *Mitigation*: The test polls every 5s for ≤ 120s; if no transition by 120s, fails with a clear "rotation not observed inside the budget" message. The 120s budget already includes margin per `environment.md` § CI gate.
|
||||||
|
|
||||||
|
**Risk 3: NFT-SEC-13 cross-origin preflight assertion misreads CORS header presence**
|
||||||
|
- *Risk*: ASP.NET Core's CORS middleware returns `200` for OPTIONS even when origin is disallowed, just without the allow-origin header. A loose assertion would miss the rejection.
|
||||||
|
- *Mitigation*: Test asserts `Access-Control-Allow-Origin` header EXACTLY: present and matching the allowed origin in the allow case; absent (header == null) in the reject case.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface and verify startup behaviour via `docker run` and `docker logs missions-sut` scrape. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-5 (NFT-SEC-10/11), AC-6 (NFT-SEC-12/13), AC-7 (NFT-SEC-07), AC-8 (NFT-SEC-08), AC-9 (NFT-SEC-07), AC-10 (NFT-SEC-08).
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs` (config resolution + CORS + auth pipeline), `Infrastructure/ConfigurationResolver`, `Infrastructure/CorsConfigurationValidator`, or `ErrorHandlingMiddleware`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# Resilience Tests — Cascade + Migrator
|
||||||
|
|
||||||
|
**Task**: AZ-583_test_resilience_cascade_migrator
|
||||||
|
**Name**: Resilience tests — cascade + migrator (NFT-RES-01..04)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 4 cascade and migrator resilience scenarios — mission cascade NOT transaction-wrapped (partial deletes survive mid-walk failure; AC-3.3 / ADR-006 carry-forward), waypoint cascade same invariant (AC-4.6), migrator idempotent on container restart (AC-6.6), and the B9 one-shot legacy table drop is destructive on first run + idempotent on subsequent restarts (AC-6.5, AC-10.5).
|
||||||
|
**Complexity**: 3 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-583
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The cascade tests encode TWO documented carry-forwards — the F3 (mission) and F4 (waypoint) cascades are NOT transaction-wrapped, so when the walk fails mid-way (e.g., `media` table absent), the rows deleted BEFORE the failure stay deleted while the rows deleted AFTER do not. This is documented under ADR-006 and AC-3.3 / AC-3.4 / AC-4.6 / AC-10.2 as deferred work. The tests intentionally pin the current behaviour so a future transaction-wrap change is caught loudly. The migrator tests pin two operational invariants needed for blue-green / restart-during-deploy patterns: NFT-RES-03 verifies a vanilla restart is a no-op, and NFT-RES-04 verifies the post-B9 `DROP TABLE IF EXISTS orthophotos/gps_corrections` block runs once and is idempotent thereafter.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All four NFT-RES-01..04 scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=Res`, `Traces=AC-3.3` / `AC-4.6` / `AC-6.6` / `AC-6.5`, `Result=pass`.
|
||||||
|
- NFT-RES-01 and NFT-RES-02 assert BOTH the partial-state observation (some rows deleted, some not) AND the 500 response shape (envelope keys, no leak) — fail loudly when a future transaction wrap rolls everything back.
|
||||||
|
- NFT-RES-03 asserts no NEW error log lines appear after the restart timestamp (not just "any error", which would conflate pre-existing startup-time warnings).
|
||||||
|
- NFT-RES-04 includes a build-time / source-inspection gate so it only meaningfully runs on a post-B9 build (B9 landed locally 2026-05-15 — verified via `_docs/_process_leftovers/2026-05-14_rename-flights-to-missions.md`).
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-RES-01 Cascade NOT transaction-wrapped (mission, F3) — `DROP TABLE media CASCADE` before request; `500` response; `map_objects` count `0` (committed); `missions` count `1` (uncommitted).
|
||||||
|
- NFT-RES-02 Cascade NOT transaction-wrapped (waypoint, F4) — same shape against F4 fixture.
|
||||||
|
- NFT-RES-03 Idempotent migrator on restart — `docker compose restart missions`; no NEW error log lines; schema unchanged.
|
||||||
|
- NFT-RES-04 B9 one-shot legacy drop — `seed_legacy_gps_tables` precondition; on first start `orthophotos` + `gps_corrections` are dropped; subsequent restart is no-op.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- NFT-RES-05 Required config missing → fail-fast (4 docker-run cases + DB-unreachable) lives in Task 17.
|
||||||
|
- NFT-RES-06 DB does not exist (Npgsql 3D000) lives in Task 17.
|
||||||
|
- NFT-RES-07 JWKS rotation lives in Task 17 (NOTE: also touched by NFT-SEC-11 in Task 15 from a security angle; this resilience variant focuses on the no-restart operational property).
|
||||||
|
- NFT-RES-08 TOCTOU on default-vehicle exclusivity lives in Task 17.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-RES-01 mission cascade partial-state survives mid-walk failure**
|
||||||
|
Given `fixture_cascade_F3` applied to a running stack
|
||||||
|
When the side-channel executes `DROP TABLE media CASCADE` THEN the consumer issues `DELETE /missions/{mid}` with JWT `FL`
|
||||||
|
Then the response is `500` with envelope `{ statusCode:500, message:"Internal server error" }`
|
||||||
|
And side-channel `SELECT COUNT(*) FROM map_objects WHERE mission_id={mid}` returns `0` (committed before the failure)
|
||||||
|
And side-channel `SELECT COUNT(*) FROM missions WHERE id={mid}` returns `1` (uncommitted after the failure)
|
||||||
|
And `docker logs missions-sut` contains an `Unhandled exception` line mentioning `relation` and `media` within 2s of the request
|
||||||
|
|
||||||
|
**AC-2: NFT-RES-02 waypoint cascade partial-state same invariant**
|
||||||
|
Given `fixture_cascade_F4` applied
|
||||||
|
When the side-channel executes `DROP TABLE media CASCADE` THEN the consumer issues `DELETE /missions/{mid}/waypoints/{wp1}`
|
||||||
|
Then the response is `500`
|
||||||
|
And side-channel `SELECT COUNT(*) FROM detection WHERE annotation_id IN (wp1 chain)` returns `0`
|
||||||
|
And side-channel `SELECT COUNT(*) FROM waypoints WHERE id={wp1}` returns `1`
|
||||||
|
|
||||||
|
**AC-3: NFT-RES-03 migrator is idempotent on restart**
|
||||||
|
Given `missions` has been started once (schema migrated; `seed_empty` state)
|
||||||
|
When `docker compose -f docker-compose.test.yml restart missions` is invoked AND health returns 200 within 30s
|
||||||
|
Then `docker logs missions-sut` since the restart timestamp contains NO new lines matching `(error|Error|exception)`
|
||||||
|
And the side-channel `\d+ vehicles` table description is unchanged from the post-first-start state
|
||||||
|
|
||||||
|
**AC-4: NFT-RES-04 B9 one-shot legacy drop is destructive then idempotent**
|
||||||
|
Given `seed_legacy_gps_tables` (legacy `orthophotos` + `gps_corrections` present), `missions` not yet started for this scenario, AND the build is post-B9 (verified via `to_regclass` or source inspection of `DatabaseMigrator.cs`)
|
||||||
|
When `docker compose up -d missions` is invoked and health returns 200
|
||||||
|
Then side-channel `SELECT to_regclass('orthophotos'), to_regclass('gps_corrections')` returns both NULL (tables dropped)
|
||||||
|
And when `docker compose restart missions` is invoked and health returns 200 again
|
||||||
|
Then side-channel queries still return both NULL, AND `docker logs missions-sut` since the restart contains NO `does not exist` line (the `IF EXISTS` suppresses the no-op error)
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-RES-01..02: ≤ 10s each (cascade walk + fault injection setup).
|
||||||
|
- NFT-RES-03..04: ≤ 60s each (container restart + health poll).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- NFT-RES-01 and NFT-RES-02 are destructive (drop `media` table); each runs in its own xUnit `[Collection("ResCascadeF3")]` / `[Collection("ResCascadeF4")]` with `ComposeRestartFixture` teardown (full `down -v && up -d`).
|
||||||
|
- NFT-RES-04 has a build-time gate: the test queries the migrator source (or checks if the legacy tables exist after start) and SKIPS with a recorded reason on pre-B9 builds. Skipped rows appear in the CSV report with `Result=skip` and a clear `ErrorMessage` field.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `fixture_cascade_F3` + DROP `media` | `DELETE /missions/{mid}` | 500 + map_objects=0 + missions=1 + log mentions `media` | AC-3.3, AC-10.2 |
|
||||||
|
| AC-2 | `fixture_cascade_F4` + DROP `media` | `DELETE /missions/{mid}/waypoints/{wp1}` | 500 + detection=0 + wp1=1 | AC-4.6, AC-3.3 |
|
||||||
|
| AC-3 | post-first-start `seed_empty` | `docker compose restart missions` | health back in 30s + no new error logs + schema unchanged | AC-6.6, AC-6.4 |
|
||||||
|
| AC-4 | `seed_legacy_gps_tables` + post-B9 build | first start + restart | first drops legacy tables; restart is no-op (no error log) | AC-6.5, AC-10.5 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080` for the cascade requests; side-channel Npgsql for fixture state + post-state assertions.
|
||||||
|
- NFT-RES-01..02 use the same `fixture_cascade_F3.sql` / `fixture_cascade_F4.sql` from Tasks 11/12; do NOT re-author seed SQL.
|
||||||
|
- NFT-RES-03..04 use `docker compose` from inside the runner (Docker-socket-mounted) OR from the suite orchestrator — AZ-576 covers this.
|
||||||
|
- NFT-RES-04 must verify B9 has landed before running; otherwise SKIP with a clear reason (record in CSV).
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: NFT-RES-01/02 false-pass when transaction wrap lands**
|
||||||
|
- *Risk*: A future ADR-006 closure wraps the cascade in a transaction; `map_objects` count becomes `> 0` (rolled back) and `missions` count stays `1`. The test would interpret this as a failure of the partial-state invariant — but that failure means the system is BETTER.
|
||||||
|
- *Mitigation*: Both tests include a source-level comment `// CARRY-FORWARD: AC-3.3 / ADR-006 — flip assertions when transaction wrap lands` and `[Trait("carry_forward","ADR-006")]` so a future filter finds them.
|
||||||
|
|
||||||
|
**Risk 2: NFT-RES-03 false-pass when restart-time errors are tolerated**
|
||||||
|
- *Risk*: A simple `docker logs | grep -i error` over the entire log returns the migrator's pre-existing warnings.
|
||||||
|
- *Mitigation*: The test captures `docker logs missions-sut --since=<restart_timestamp>` and greps from THAT slice only.
|
||||||
|
|
||||||
|
**Risk 3: NFT-RES-04 incorrectly runs on a pre-B9 build**
|
||||||
|
- *Risk*: If the build-time gate is silently bypassed, the test asserts dropping the legacy tables — which would never happen, and the test would fail with a misleading message.
|
||||||
|
- *Mitigation*: The gate checks BOTH the migrator source for the `DROP TABLE IF EXISTS orthophotos` line AND verifies the legacy tables are present in the seed BEFORE the SUT starts. If either check fails, the test SKIPS with `Result=skip` and a clear `ErrorMessage`.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface plus container orchestration (`docker compose restart`, `docker compose up -d`) and `docker logs missions-sut` scrape. Side-channel Npgsql for fixture state and post-state assertions. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-3 3.3, AC-4 4.6, AC-6 6.4-6.6, AC-10 10.2/10.5.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects` (seeded via side-channel SQL).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `MissionService`, `WaypointService`, `MissionsController`, `Database/DatabaseMigrator`, `ErrorHandlingMiddleware`, or `AppDataConnection`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# Resilience Tests — Config / DB / JWKS Rotation / TOCTOU Race
|
||||||
|
|
||||||
|
**Task**: AZ-584_test_resilience_config_db_rotation_race
|
||||||
|
**Name**: Resilience tests — config / DB / rotation / race (NFT-RES-05..08)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 4 resilience scenarios — startup fail-fast on missing required config (6 docker-run cases including the DB-unreachable differentiator), database missing → Npgsql 3D000 process exit, JWKS rotation propagates without `missions` restart, and TOCTOU race on default-vehicle exclusivity (probabilistic, expected to produce `default_count ≥ 2` in at least one iteration).
|
||||||
|
**Complexity**: 5 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-584
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
These four scenarios pin the documented operational and concurrency posture of the service in places nothing else covers. NFT-RES-05 verifies BOTH the new fail-fast resolver path (rows 1–5: missing env vars throw `InvalidOperationException` BEFORE the HTTP server binds) AND the DB-down differentiator (row 6: config resolution succeeds, then Npgsql throws a recognisable connection error). NFT-RES-06 verifies the "database does not exist" case is observably different from "DB host unreachable" — Postgres returns SQLSTATE `3D000` and the container exits non-zero within 30s. NFT-RES-07 is the operational counterpart to NFT-SEC-11 — same JWKS rotation flow, but asserts the no-restart property (`docker inspect StartedAt` unchanged) instead of the kid-cache mechanics. NFT-RES-08 is intentionally probabilistic: it asserts the documented AC-1.4 race window EXISTS by running 100 parallel concurrent INSERTs and verifying that at least one iteration produces `is_default=true count ≥ 2`.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All four NFT-RES-05..08 scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=Res`, `Traces=AC-6.1..2/AC-6.7..8/AC-5.7/AC-1.4`, `Result=pass`.
|
||||||
|
- NFT-RES-05 covers 6 cases — 4 missing-env (rows 1–4), 1 whitespace-only (`JWT_ISSUER=""`), and 1 DB-down-after-config-resolution (row 6 with `Connection refused`).
|
||||||
|
- NFT-RES-06 asserts the Postgres error code `3D000` appears in the container logs and the container exit code is non-zero within 30s.
|
||||||
|
- NFT-RES-07 asserts `docker inspect --format '{{.State.StartedAt}}' missions-sut` returns the SAME value before and after the rotation flow — the service did NOT restart.
|
||||||
|
- NFT-RES-08 records the observed `default_count ≥ 2` iteration count and includes `[Trait("Stability","probabilistic")]` so CI tolerates ≤ 1 failed run per 5. If 0 iterations produce the race, the test FAILS with a clear "race window closed — update AC-1.4 and rewrite this test" message.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-RES-05 6 docker-run cases (4 missing-env + 1 whitespace + 1 DB-down differentiator).
|
||||||
|
- NFT-RES-06 `DROP DATABASE azaion` → `docker compose up -d missions` → assert non-zero exit + `3D000` in logs.
|
||||||
|
- NFT-RES-07 JWKS rotation flow — `T1` works pre-rotation; `T2` rejected pre-cache-refresh; `T2` accepted post-refresh; `T1` eventually rejected post-grace; `missions` startup timestamp unchanged.
|
||||||
|
- NFT-RES-08 100 parallel `(POST /vehicles { IsDefault:true } || side-channel INSERT (..., is_default=true))` iterations; at least one produces `default_count ≥ 2`.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- NFT-SEC-11 (security-shaped variant of JWKS rotation) lives in Task 15.
|
||||||
|
- NFT-SEC-12 (security-shaped variant of startup fail-fast) lives in Task 15. NOTE: NFT-RES-05 and NFT-SEC-12 share 4 of 5 docker-run cases — the test infrastructure (AZ-576) provides a shared `MissionsContainerHelper` so both tasks can reuse the same docker-run primitive without duplicating logic.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-RES-05 startup fail-fast on missing required config + DB-down differentiator**
|
||||||
|
When `missions` is launched via 6 separate `docker run` invocations:
|
||||||
|
- (1) all 4 required env vars unset
|
||||||
|
- (2) `DATABASE_URL` unset, JWT vars set
|
||||||
|
- (3) `JWT_ISSUER=""` (whitespace-only), others set
|
||||||
|
- (4) `JWT_AUDIENCE` unset, others set
|
||||||
|
- (5) `JWT_JWKS_URL` unset, others set
|
||||||
|
- (6) all 4 vars set correctly, BUT `postgres-test` is stopped before `missions` starts
|
||||||
|
Then rows 1–5 → container exits non-zero within 5s, logs contain `InvalidOperationException`, logs mention the corresponding key (or its config alias)
|
||||||
|
And row 6 → container exits non-zero within 30s, logs contain a Npgsql `Connection refused` line (NOT an `InvalidOperationException` — proving config resolution succeeded BEFORE DB-connect failed)
|
||||||
|
|
||||||
|
**AC-2: NFT-RES-06 database missing → process exits with Npgsql 3D000**
|
||||||
|
Given `postgres-test` running with the `azaion` database NOT yet created (or just dropped via side-channel)
|
||||||
|
When `docker compose -f docker-compose.test.yml up -d missions` is invoked
|
||||||
|
Then the container exits non-zero within 30s AND `docker logs missions-sut` contains at least one line matching `3D000`
|
||||||
|
|
||||||
|
**AC-3: NFT-RES-07 JWKS rotation propagates without missions restart**
|
||||||
|
Given `missions` running with a warm JWKS cache, `jwks-mock` running with `OLD_KEY_GRACE_SECONDS=5` and `Cache-Control: max-age=60`, and Token `T1` minted with the current kid `kid_v1`
|
||||||
|
When `GET /vehicles` is issued with `T1`
|
||||||
|
Then response is `200`
|
||||||
|
And when `POST jwks-mock:8443/rotate-key {}` is invoked, `T2` is minted with `kid_v2`, and `GET /vehicles` is issued with `T2` BEFORE the JWKS cache refresh
|
||||||
|
Then response is `401`
|
||||||
|
And after waiting up to 90s for cache refresh (mock `max-age=60` + service `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30`), `GET /vehicles` with the same `T2` returns `200`
|
||||||
|
And `GET /vehicles` with `T1` (still has unexpired lifetime) returns `401` AFTER the grace window expires
|
||||||
|
And `docker inspect --format '{{.State.StartedAt}}' missions-sut` returns the SAME ISO-8601 timestamp before and after the entire rotation flow (the service did NOT restart)
|
||||||
|
|
||||||
|
**AC-4: NFT-RES-08 TOCTOU race produces default_count ≥ 2 in at least one iteration**
|
||||||
|
Given `seed_one_default_vehicle` (default `P1`)
|
||||||
|
When the test runs 100 concurrent iterations, each issuing `POST /vehicles { IsDefault:true }` to the API in parallel with a side-channel `INSERT INTO vehicles (..., is_default=true)`
|
||||||
|
Then after all iterations complete, at least one iteration's post-state shows `SELECT COUNT(*) FROM vehicles WHERE is_default=true ≥ 2`
|
||||||
|
And if 0 iterations produce the race, the test FAILS with `"race window closed — update AC-1.4 carry-forward and rewrite this test"` (this is a structural test failure, not a flake)
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-RES-05: ≤ 180s (6 docker-run cycles).
|
||||||
|
- NFT-RES-06: ≤ 60s (DROP DATABASE + docker-run + exit poll).
|
||||||
|
- NFT-RES-07: ≤ 180s (JWKS cache refresh window).
|
||||||
|
- NFT-RES-08: ≤ 30s (100 parallel iterations).
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- NFT-RES-07 fixture MUST restore the original key by calling `POST /rotate-key` again at the end AND wait the grace window before yielding control — otherwise every subsequent test runs against an unfamiliar kid.
|
||||||
|
- NFT-RES-08 is probabilistic: `[Trait("Stability","probabilistic")]`. CI tolerates ≤ 1 failed run per 5 — but the structural failure mode ("race never observed in any iteration") still fails the suite. A deterministic-via-advisory-lock follow-up is recorded as a Refactor Backlog item.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `missions` not running | 6 docker-run cases | 5 fail-fast (InvalidOperationException) + 1 DB-down (Connection refused) | AC-6.1, AC-6.2, AC-6.7, E3, E4 |
|
||||||
|
| AC-2 | `DROP DATABASE azaion` | `docker compose up -d missions` | exit non-zero in 30s + log has `3D000` | AC-6.8 |
|
||||||
|
| AC-3 | warm JWKS cache + mock with grace=5/max-age=60 | rotate + 3 timing probes | T1→200; T2→401→wait→200; T1→401 post-grace; StartedAt unchanged | AC-5.7 |
|
||||||
|
| AC-4 | `seed_one_default_vehicle` | 100 parallel (POST + side-channel INSERT) | ≥ 1 iteration shows default_count ≥ 2 | AC-1.4 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- HTTP only against `http://missions:8080` for the runtime cases; `docker run` and `docker compose` for the startup/DB cases.
|
||||||
|
- NFT-RES-05 row 6 (DB-down differentiator) is critical: the test must assert the log is `Connection refused`-shaped, NOT an `InvalidOperationException`. This rules out a regression where the resolver silently accepts an empty DB URL.
|
||||||
|
- NFT-RES-07 must clean up: rotate back to the original key in teardown AND wait `OLD_KEY_GRACE_SECONDS` so subsequent tests do not encounter a stale-kid edge case.
|
||||||
|
- NFT-RES-08 records the per-iteration timing and observed counts to the CSV report's `Traces` field for diagnosis.
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: NFT-RES-05 row 6 false-pass when config resolution silently accepts empty `DATABASE_URL`**
|
||||||
|
- *Risk*: A regression that returns an empty default for `DATABASE_URL` would make rows 2/6 indistinguishable — both would log a Npgsql error, but row 2 should log `InvalidOperationException` first.
|
||||||
|
- *Mitigation*: Test asserts row 2 logs the `InvalidOperationException` BEFORE any Npgsql output; row 6 logs Npgsql `Connection refused` directly without `InvalidOperationException`. Failure of either differentiator fails the test.
|
||||||
|
|
||||||
|
**Risk 2: NFT-RES-07 flake on slow CI**
|
||||||
|
- *Risk*: Same as NFT-SEC-11 — slow refresh window.
|
||||||
|
- *Mitigation*: Same — poll every 5s for ≤ 90s; fail clearly if no transition observed in budget.
|
||||||
|
|
||||||
|
**Risk 3: NFT-RES-08 deterministic-pass when race window closes**
|
||||||
|
- *Risk*: If a future TOCTOU fix lands (e.g., adding a `UNIQUE WHERE is_default=true` constraint), the test's "race observed" assertion fails — but the system is BETTER.
|
||||||
|
- *Mitigation*: Test failure message includes `"race window closed — update AC-1.4 carry-forward and rewrite this test"` so a future engineer knows the failure is expected and what to do. The test is gated by `[Trait("carry_forward","AC-1.4")]`.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface plus `docker run`, `docker compose`, `docker inspect`, and `docker logs missions-sut` scrape. Side-channel Npgsql for fixture state, post-state assertions, and concurrent INSERTs. JWKS rotation via `POST https://jwks-mock:8443/rotate-key`. Expected outputs are compared against `_docs/00_problem/input_data/expected_results/results_report.md` rows AC-1 1.4, AC-5 5.7, AC-6 6.1/6.2/6.7/6.8, E3/E4.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container).
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including `JwtExtensions`, `Program.cs`, `Infrastructure/ConfigurationResolver`, `Database/AppDataConnection`, `Database/DatabaseMigrator`, `Services/VehicleService` (for the TOCTOU race), or `Auth/JwtExtensions`. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# Resource Limit Tests
|
||||||
|
|
||||||
|
**Task**: AZ-585_test_resource_limits
|
||||||
|
**Name**: Resource limit tests (NFT-RES-LIM-01..04)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 4 resource-limit observation scenarios — steady-state RSS memory under 5-min sustained load (P95 ≤ 250 MiB; no monotonic climb), Npgsql connection pool ≤ 100 with no unbounded growth, file-descriptor count ≤ 1024 with no leak, and cold-start RSS ≤ 200 MiB at `t=30s` after health-ok. Provisional gates documented per `restrictions.md` H6 — locked in after first green run.
|
||||||
|
**Complexity**: 3 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-585
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Per H6, container-level resource limits are NOT enforced inside the container — they will be set at the suite level (`_infra/_compose/`) per device type once locked. These tests establish baseline observations so the suite can size the cgroup limits correctly AND provide an upper-bound regression gate so future changes do not silently 10× the memory or FD footprint. The 8 GB Jetson Orin must accommodate ~6 .NET edge services + Postgres + UI; `missions`'s budget is ~200 MiB cold + ~250 MiB hot. Without these observation tests, a leak or library bloat could ship to the device and force a re-sizing decision late in deployment.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All four NFT-RES-LIM-01..04 scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=ResLim`, `Traces=H1|H3|H6|O10`, `Result=pass`, AND records the measured value (e.g., `P95_RSS_MiB=187`) in the `Traces` column so suite-level deployment planning can read it.
|
||||||
|
- NFT-RES-LIM-01 measures P95 RSS over 5 minutes of mixed sustained load AND asserts `final_RSS - P95_RSS ≤ 20% * P95_RSS` (no monotonic climb).
|
||||||
|
- NFT-RES-LIM-02 measures Npgsql connection count via `pg_stat_activity` every 5s AND asserts both `max ≤ 100` AND `final ≤ 1.3 * first_minute_steady_state`.
|
||||||
|
- NFT-RES-LIM-03 measures `/proc/<pid>/fd | wc -l` inside the container every 5s AND asserts both `max ≤ 1024` AND `final ≤ 1.3 * minute_one_count`.
|
||||||
|
- NFT-RES-LIM-04 measures cold-start RSS exactly 30s after `GET /health` first returns 200 (no requests issued yet) AND asserts `RSS ≤ 200 MiB`.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-RES-LIM-01 Steady-state memory under 5-min sustained load.
|
||||||
|
- NFT-RES-LIM-02 Connection pool steady-state.
|
||||||
|
- NFT-RES-LIM-03 File-descriptor steady-state.
|
||||||
|
- NFT-RES-LIM-04 Cold-start RSS budget.
|
||||||
|
- Each test records the measured value to the CSV `Traces` field so deployment planning can pick it up.
|
||||||
|
- Provisional gates: 250 MiB hot, 200 MiB cold, 100 connections, 1024 FDs. On first green run, replace provisional gates with `measured + 50%` and open a Refactor Backlog ticket if the provisional gate was exceeded.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- Performance (latency / throughput) tests live in Task 19.
|
||||||
|
- GPU / temperature / disk-I/O monitoring (per `restrictions.md` H8 — no specialised hardware on a CRUD service).
|
||||||
|
- Long-soak / endurance tests (> 5 min) — explicitly deferred per `restrictions.md` H8.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-RES-LIM-01 steady-state RSS ≤ provisional 250 MiB with no monotonic climb**
|
||||||
|
Given `missions` running with `seed_25_missions` + `seed_3_vehicles_2_default` and no host-side memory limit
|
||||||
|
When the test orchestrator drives ~50 RPS of mixed `GET /vehicles`, `GET /missions`, `GET /missions/{id}/waypoints` for 5 minutes from a single concurrent client, while polling `docker stats --no-stream missions-sut` every 5s
|
||||||
|
Then the P95 of the 60 RSS samples is `≤ 250 MiB` (provisional gate)
|
||||||
|
And the final-sample RSS is within ± 20% of the P95 RSS (no sustained leak — RSS does not climb monotonically)
|
||||||
|
And the measured P95 is recorded to the CSV `Traces` column as `P95_RSS_MiB=<n>`
|
||||||
|
|
||||||
|
**AC-2: NFT-RES-LIM-02 connection pool ≤ 100 with no unbounded growth**
|
||||||
|
Given the same setup as NFT-RES-LIM-01
|
||||||
|
When the test orchestrator polls side-channel `SELECT count(*) FROM pg_stat_activity WHERE application_name LIKE 'Npgsql%' OR (usename='postgres' AND backend_type='client backend')` every 5s for 5 minutes
|
||||||
|
Then the max sampled connection count is `≤ 100`
|
||||||
|
And the final-sample count is `≤ 1.3 × (mean of samples in the first minute)`
|
||||||
|
And the measured max is recorded as `MAX_NPGSQL_CONNS=<n>`
|
||||||
|
|
||||||
|
**AC-3: NFT-RES-LIM-03 file descriptors ≤ 1024 with no leak**
|
||||||
|
Given the same setup as NFT-RES-LIM-01
|
||||||
|
When the test orchestrator executes `docker exec missions-sut sh -c 'ls /proc/$(pgrep -f Azaion.Missions.dll | head -1)/fd | wc -l'` every 5s for 5 minutes
|
||||||
|
Then the max sampled FD count is `≤ 1024`
|
||||||
|
And the final-sample count is `≤ 1.3 × (count at t=1min)`
|
||||||
|
And the measured max is recorded as `MAX_FD=<n>`
|
||||||
|
|
||||||
|
**AC-4: NFT-RES-LIM-04 cold-start RSS ≤ 200 MiB**
|
||||||
|
Given `missions` has been started fresh (via `docker compose up -d missions` after `down -v`), no requests issued yet
|
||||||
|
When `GET /health` first returns `200` AND 30s have elapsed
|
||||||
|
Then `docker stats --no-stream missions-sut` reports `MEM USAGE` ≤ 200 MiB
|
||||||
|
And the measured cold-start RSS is recorded as `COLD_RSS_MiB=<n>`
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-RES-LIM-01..03: each take exactly 5 minutes (sampling window). With Arrange/teardown, ≤ 6 minutes wall-clock.
|
||||||
|
- NFT-RES-LIM-04: ≤ 60s wall-clock (fresh start + health-poll + 30s wait + measurement).
|
||||||
|
- The total task runtime budget is ≤ 20 minutes, fitting inside the documented 15-min suite CI gate per `environment.md`. NFT-RES-LIM-01..03 share the same 5-minute window and run concurrently against a single dockerised `missions`; NFT-RES-LIM-04 runs separately because it requires a fresh start.
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- The load generator is a single-thread `HttpClient` driving requests in a tight loop; this is documented at 50 RPS approximately for the in-suite test runner. If the runner is unable to sustain 50 RPS (CI infrastructure too slow), the test SKIPS NFT-RES-LIM-01..03 with `Result=skip` and a clear `ErrorMessage=runner cannot sustain target load`. CI then reruns these on a beefier worker.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | `seed_25_missions` + 50 RPS for 5 min | P95 RSS sampling | P95 ≤ 250 MiB + no monotonic climb | H1, H6, O10 |
|
||||||
|
| AC-2 | same | `pg_stat_activity` polling | max ≤ 100 + final ≤ 1.3×steady | O10 |
|
||||||
|
| AC-3 | same | `/proc/<pid>/fd` polling | max ≤ 1024 + final ≤ 1.3×minute-one | H6, O10 |
|
||||||
|
| AC-4 | fresh `docker compose up -d` | cold-start RSS at t=30s | RSS ≤ 200 MiB | H1, H3 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- `docker stats` and `docker exec` from inside the runner: requires Docker socket access; AZ-576 covers this.
|
||||||
|
- NFT-RES-LIM-03 requires `pgrep` inside the `missions` image; the test FAILS in Arrange (not Assert) if `pgrep` is unavailable. Alternative: parse `/proc/1/comm` if PID 1 is the .NET process (preferred for the small Dockerfile).
|
||||||
|
- All measurements are recorded to the CSV report's `Traces` field so deployment planning can pick them up; this is more important than the pass/fail gate.
|
||||||
|
- Provisional gates are documented per `restrictions.md` H6 — locked in based on first measured run.
|
||||||
|
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: Measurement variance on shared CI runners**
|
||||||
|
- *Risk*: A runner under noisy-neighbour load reports inflated RSS, flaking the gate.
|
||||||
|
- *Mitigation*: Gates are provisional and generous (250 MiB vs. typical .NET service of ~150 MiB; 100 connections vs. typical idle pool of ~5–10). After the first green run, the gate is locked at `measured + 50%`.
|
||||||
|
|
||||||
|
**Risk 2: NFT-RES-LIM-01..03 share a 5-minute window — flake correlation**
|
||||||
|
- *Risk*: A CI hiccup that kills the SUT mid-window flakes all three at once.
|
||||||
|
- *Mitigation*: Each test asserts its own metric; on `missions-sut` exit during the window, the test FAILS with a `"SUT exited during measurement window"` ErrorMessage rather than reporting a misleading metric value.
|
||||||
|
|
||||||
|
**Risk 3: Provisional gates silently accepted as the locked gate**
|
||||||
|
- *Risk*: If the first green run measures 200 MiB and the test passes, a future engineer treats 250 MiB as the gate forever — but actual headroom is only 50 MiB.
|
||||||
|
- *Mitigation*: The test logs `(measured / gate) ratio`; CI dashboards flag ratios > 0.8 for re-tuning consideration. The lock-in workflow is documented in `restrictions.md` H6.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface for load generation; `docker stats`, `docker exec`, and side-channel `pg_stat_activity` for measurement. Expected outputs are the documented gates from `_docs/02_document/tests/resource-limit-tests.md` (provisional) and the corresponding entries in `_docs/00_problem/input_data/expected_results/results_report.md` (when locked).
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`.
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the Npgsql connection pool, the `AppDataConnection` lifetime, or the `Program.cs` startup path. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
# Performance Tests
|
||||||
|
|
||||||
|
**Task**: AZ-586_test_performance
|
||||||
|
**Name**: Performance tests (NFT-PERF-01..04)
|
||||||
|
**Description**: Implement xUnit blackbox tests for the 4 performance scenarios — F3 cascade-delete P50 ≤ 50ms on a 1-waypoint mission, F3 cascade-delete P50 ≤ 200ms on the full chain (provisional baseline; lock after first green run), `GET /health` P50 ≤ 10ms, and `GET /missions?page=1&pageSize=20` P95 ≤ 100ms against a 1000-mission seed (provisional baseline). Every test runs 5 warm-up calls + the documented N measured calls; cold-start passes excluded.
|
||||||
|
**Complexity**: 3 points
|
||||||
|
**Dependencies**: AZ-576_test_infrastructure
|
||||||
|
**Component**: Blackbox Tests
|
||||||
|
**Tracker**: AZ-586
|
||||||
|
**Epic**: AZ-575
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Three latency thresholds are documented (AC-3.6 P50 ≤ 50ms for minimal cascade, AC-7.3 P50 ≤ 10ms for health, AC-2.3 implicit list latency) and one (NFT-PERF-02 full-chain cascade) is a baseline that subsequent runs must not regress by more than 50%. Without these tests, an unintentional N+1 query, missing index, or accidental serialization layer overhead could silently 10× the response time before the next manual perf benchmark catches it. The full-chain cascade test is especially load-bearing because the F3 cascade walks 5 dependency tables — a future indexing regression or transaction-wrap addition would show up here first.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
- All four NFT-PERF-01..04 scenarios run and pass against the dockerised `missions` service.
|
||||||
|
- Each test produces a CSV row with `Category=Perf`, `Traces=AC-3.6` / `AC-3.1` / `AC-7.3` / `AC-2.3`, `Result=pass`, AND records P50 and P95 numeric values in the `Traces` column (e.g., `P50_MS=23.4, P95_MS=41.8`).
|
||||||
|
- 5 warm-up calls precede every measured set; cold-start passes are excluded from the percentile computation.
|
||||||
|
- All tests run sequentially against a single client (no concurrent connections) so HTTP/1.1 connection-reuse and JIT warm-up are deterministic.
|
||||||
|
- Tests run only when `[Trait("Category","Perf")]` filter is active (default test suite filter excludes performance to keep the standard CI gate ≤ 15 min); a separate `scripts/run-performance-tests.sh` invocation runs them.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### Included
|
||||||
|
|
||||||
|
- NFT-PERF-01 F3 minimal cascade — `DELETE /missions/{id}` on 1-waypoint missions; P50 ≤ 50ms over 100 sequential calls.
|
||||||
|
- NFT-PERF-02 F3 full cascade — `DELETE /missions/{id}` on `fixture_cascade_F3`-shaped missions; P50 ≤ 200ms over 50 sequential calls (provisional baseline).
|
||||||
|
- NFT-PERF-03 Health endpoint — `GET /health` P50 ≤ 10ms over 100 sequential calls.
|
||||||
|
- NFT-PERF-04 List pagination — `GET /missions?page=1&pageSize=20` P95 ≤ 100ms over 100 sequential calls against a 1000-mission seed (provisional baseline).
|
||||||
|
- Recording P50/P95 to CSV `Traces` column for trend tracking even when not gated.
|
||||||
|
- Performance suite is gated behind the `[Trait("Category","Perf")]` filter; standard CI gate excludes these.
|
||||||
|
|
||||||
|
### Excluded
|
||||||
|
|
||||||
|
- Concurrency / contention tests (race scenarios) live in Task 17 (NFT-RES-08).
|
||||||
|
- Resource consumption (RSS, FDs, connections) lives in Task 18 (NFT-RES-LIM).
|
||||||
|
- Production-hardware (Jetson Orin) latency baselines — documented as a follow-up in `restrictions.md` H8; test environment baselines stand in.
|
||||||
|
- Concurrent-client throughput / RPS — not in scope today; documented as Refactor Backlog.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
**AC-1: NFT-PERF-01 F3 minimal cascade P50 ≤ 50ms**
|
||||||
|
Given `missions` + `postgres-test` colocated on the same Docker network, `seed_one_default_vehicle` + 100 minimal missions (each with 1 waypoint, no media/annotations/detection/map_objects rows), AND 5 warm-up `DELETE` calls have completed on missions outside the measured set
|
||||||
|
When the consumer issues 100 sequential `DELETE /missions/{id_i}` calls (one per seeded mission, 1 ≤ i ≤ 100) and records per-call wall-clock latency
|
||||||
|
Then the P50 (median) of the 100 latencies is `≤ 50ms`
|
||||||
|
And P50 + P95 are recorded to the CSV `Traces` column as `P50_MS=<v1>, P95_MS=<v2>`
|
||||||
|
|
||||||
|
**AC-2: NFT-PERF-02 F3 full-chain cascade P50 ≤ 200ms**
|
||||||
|
Given 50 missions each with the `fixture_cascade_F3` chain (3 map_objects, 2 waypoints, 2 media, 2 annotations, 2 detection rows) AND 5 warm-up calls on additional fixtures outside the measured set
|
||||||
|
When the consumer issues 50 sequential `DELETE /missions/{id_i}` calls and records per-call wall-clock latency
|
||||||
|
Then P50 ≤ 200ms (provisional baseline — to be locked at `measured + 50%` on first green run)
|
||||||
|
And P50 + P95 recorded to CSV
|
||||||
|
|
||||||
|
**AC-3: NFT-PERF-03 health endpoint P50 ≤ 10ms**
|
||||||
|
Given `missions` running, no special seed, AND 5 warm-up `GET /health` calls
|
||||||
|
When the consumer issues 100 sequential `GET /health` calls (no `Authorization` header) and records per-call wall-clock latency
|
||||||
|
Then P50 ≤ 10ms
|
||||||
|
And P50 + P95 recorded to CSV
|
||||||
|
|
||||||
|
**AC-4: NFT-PERF-04 list pagination P95 ≤ 100ms (provisional)**
|
||||||
|
Given `seed_one_default_vehicle` + 1000 missions referencing it, AND 5 warm-up `GET /missions?page=1&pageSize=20` calls
|
||||||
|
When the consumer issues 100 sequential `GET /missions?page=1&pageSize=20` calls and records per-call wall-clock latency
|
||||||
|
Then P95 ≤ 100ms (provisional baseline — to be locked at `measured + 50%` on first green run)
|
||||||
|
And P50 + P95 recorded to CSV
|
||||||
|
|
||||||
|
## Non-Functional Requirements
|
||||||
|
|
||||||
|
**Performance**
|
||||||
|
- NFT-PERF-01: ≤ 30s wall-clock (100 calls × ≤ 50ms each + measurement overhead). Per `[Trait("max_ms","30000")]` xUnit timeout.
|
||||||
|
- NFT-PERF-02: ≤ 60s wall-clock.
|
||||||
|
- NFT-PERF-03: ≤ 5s wall-clock.
|
||||||
|
- NFT-PERF-04: ≤ 30s wall-clock.
|
||||||
|
|
||||||
|
**Reliability**
|
||||||
|
- All tests SKIP if the runner cannot allocate ≥ 2 CPU cores and ≥ 2 GB free RAM (per `performance-tests.md` Notes). SKIP records `Result=skip` and `ErrorMessage=insufficient CPU/RAM`. Default CI runner spec must meet this — but degraded runners must not produce false-fail noise.
|
||||||
|
- All tests assume `missions` and `postgres-test` are colocated on the same Docker network (no inter-host link). The fixture verifies this via `docker inspect missions-sut --format '{{.NetworkSettings.Networks.testnet.IPAddress}}'` returns non-empty.
|
||||||
|
|
||||||
|
## Blackbox Tests
|
||||||
|
|
||||||
|
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||||
|
|--------|------------------------|-------------|-------------------|----------------|
|
||||||
|
| AC-1 | 100 minimal missions + 5 warm-ups | 100 sequential `DELETE /missions/{id}` | P50 ≤ 50ms; record P50/P95 | AC-3.6 |
|
||||||
|
| AC-2 | 50 F3-fixture missions + 5 warm-ups | 50 sequential `DELETE /missions/{id}` | P50 ≤ 200ms (provisional); record P50/P95 | AC-3.1, AC-3.6 |
|
||||||
|
| AC-3 | warm runtime + 5 warm-ups | 100 sequential `GET /health` | P50 ≤ 10ms; record P50/P95 | AC-7.3 |
|
||||||
|
| AC-4 | 1000 missions + 5 warm-ups | 100 sequential `GET /missions?page=1&pageSize=20` | P95 ≤ 100ms (provisional); record P50/P95 | AC-2.3 |
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- Tests live in `Tests/Performance/` and are tagged `[Trait("Category","Perf")]` so the default CI gate excludes them.
|
||||||
|
- A separate `scripts/run-performance-tests.sh` (created by AZ-576) invokes only this category. The standard `scripts/run-tests.sh` skips them.
|
||||||
|
- Sequential single-client execution — no `Parallel.For` or `Task.WhenAll`; each call awaits the previous response.
|
||||||
|
- Warm-up calls are NOT included in the percentile computation. Per `// Warmup` comment block in the test, the first 5 calls go to fixtures created specifically for warm-up (not the measured set).
|
||||||
|
- The `Stopwatch`-based timing measures `HttpClient.SendAsync` wall-clock; serialization/deserialization overhead is INCLUDED (this is what end-users observe).
|
||||||
|
- Provisional gates (NFT-PERF-02, NFT-PERF-04) are documented in source as `// PROVISIONAL — lock at measured + 50% on first green run` and `[Trait("provisional","yes")]`.
|
||||||
|
- AAA pattern with `// Arrange` (seed + warm-up), `// Act` (measured calls + percentile compute), `// Assert` (gate + CSV record).
|
||||||
|
|
||||||
|
## Risks & Mitigation
|
||||||
|
|
||||||
|
**Risk 1: CI variance breaks tight P50 ≤ 10ms gate (NFT-PERF-03)**
|
||||||
|
- *Risk*: On a noisy-neighbour CI runner, even a static `/health` route can hiccup once per 100 calls; if the hiccup lands in the P50 region, the median exceeds 10ms.
|
||||||
|
- *Mitigation*: P50 is robust to single outliers (median position 50 of 100). If the test still flakes, lock the gate at `measured P50 + 50%` after the first green run.
|
||||||
|
|
||||||
|
**Risk 2: NFT-PERF-04 1000-mission seed overlaps with other tests' DB state**
|
||||||
|
- *Risk*: Seeding 1000 missions affects pagination tests, list-shape tests, and date-filter tests — if NFT-PERF-04 runs before them in the same SUT lifetime, results drift.
|
||||||
|
- *Mitigation*: NFT-PERF-04 lives in `[Collection("Perf1k")]` and uses `IClassFixture<DbResetFixture>` to TRUNCATE all rows before its seed AND restore `seed_empty` after. Functional tests' fixtures handle their own seed; no cross-pollination.
|
||||||
|
|
||||||
|
**Risk 3: Provisional gates accepted as locked gates**
|
||||||
|
- *Risk*: Same as NFT-RES-LIM Risk 3 — if first run measures 80ms and the test passes, future engineers see the 100ms gate as the standard.
|
||||||
|
- *Mitigation*: CI dashboards flag `measured / gate ratio > 0.8` for re-tuning. Lock-in workflow documented in `performance-tests.md`.
|
||||||
|
|
||||||
|
## System Under Test Boundary
|
||||||
|
|
||||||
|
- Tests drive the product through the public HTTP surface (`http://missions:8080`) plus Npgsql side-channel for seed setup. Bearer tokens (NFT-PERF-01, 02, 04) minted via `https://jwks-mock:8443/sign`; NFT-PERF-03 sends no Authorization header. Expected outputs are the documented latency thresholds from `_docs/02_document/tests/performance-tests.md`.
|
||||||
|
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`.
|
||||||
|
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the controllers, service classes, `AppDataConnection`, or any layer affecting response time. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
|
||||||
Reference in New Issue
Block a user