missions/_docs/tasks/done/AZ-576_test_infrastructure.md

# Test Infrastructure

**Status**: Done (2026-05-15)
**Task**: AZ-576_test_infrastructure
**Name**: Test Infrastructure (Missions e2e)
**Description**: Scaffold the Blackbox test project — xUnit runner, JWKS mock service, Docker test environment wiring, test data fixtures, reporting. Compose file already exists at repo root and references not-yet-built build contexts; this task fills in those contexts.
**Complexity**: 5 points
**Dependencies**: None (C01 + C02 testability refactor already landed; see `_docs/04_refactoring/01-testability-refactoring/testability_changes_summary.md`)
**Component**: Blackbox Tests
**Tracker**: AZ-576
**Epic**: AZ-575

## Scope

Two artifacts that the existing `docker-compose.test.yml` references but does not yet build, plus the run script the suite expects:

1. `tests/Azaion.Missions.JwksMock/` — minimal HTTPS service holding an ECDSA P-256 keypair in memory, serving JWKS + `POST /sign` + `POST /rotate-key`. Image tag `azaion/jwks-mock:test`.
2. `tests/Azaion.Missions.E2E.Tests/` — xUnit 2.x test project that drives the running `missions` service over HTTP, mints tokens via `https://jwks-mock:8443/sign`, asserts DB side-effects through a side-channel Npgsql connection, and produces `report.csv`.
3. `tests/jwks-mock-ca.crt` — the self-signed CA cert that both `missions` and `e2e-consumer` mount and `update-ca-certificates --fresh` adds to the OS trust bundle (per `docker-entrypoint.sh` from C02).
4. `scripts/run-tests.sh` — wraps `docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer`, collects `report.csv`, then `down -v`.
5. `scripts/run-performance-tests.sh` — same compose stack with the `[Trait("Category","Perf")]` filter and the perf seed.

The `missions` and `postgres-test` services already exist in `docker-compose.test.yml`; the `jwks-mock` and `e2e-consumer` services exist but point at build contexts that this task creates.

## Test Project Folder Layout

```
tests/
├── jwks-mock-ca.crt                        # self-signed CA (mounted into missions + e2e-consumer)
├── Azaion.Missions.JwksMock/
│   ├── Azaion.Missions.JwksMock.csproj
│   ├── Dockerfile                          # builds azaion/jwks-mock:test, exposes 8443/tcp
│   ├── Program.cs                          # ASP.NET Core minimal API
│   ├── Endpoints/
│   │   ├── JwksEndpoint.cs                 # GET /.well-known/jwks.json
│   │   ├── SignEndpoint.cs                 # POST /sign
│   │   └── RotateKeyEndpoint.cs            # POST /rotate-key
│   ├── Services/
│   │   ├── KeyStore.cs                     # in-memory ECDSA P-256 keypair + old-key grace window
│   │   └── TokenSigner.cs                  # ECDSA signing with alg_override/kid_override support
│   └── appsettings.json                    # JWT_ISSUER, JWT_AUDIENCE, OLD_KEY_GRACE_SECONDS
└── Azaion.Missions.E2E.Tests/
    ├── Azaion.Missions.E2E.Tests.csproj    # xUnit 2.x + Bogus 35.x + Npgsql 10.x
    ├── Dockerfile                          # runs `dotnet test --logger trx` + trx→csv post-step
    ├── TestBase.cs                         # HttpClient factory, default JWT, shared MissionsBaseUrl
    ├── TokenMinter.cs                      # POST jwks-mock:8443/sign with overrides
    ├── Fixtures/
    │   ├── DbResetFixture.cs               # IClassFixture<>: TRUNCATE between classes
    │   ├── DbSeedFixture.cs                # base for the named seed sets in test-data.md
    │   ├── ComposeRestartFixture.cs        # docker compose down -v && up -d for bootstrap-sensitive tests
    │   └── JwksRotateFixture.cs            # POST /rotate-key + wait for missions to refresh JWKS cache
    ├── Helpers/
    │   ├── DbAssertions.cs                 # Npgsql side-channel asserts (row counts, default-vehicle invariants)
    │   ├── HttpAssertions.cs               # PascalCase shape, error-envelope shape, ordering, pagination
    │   └── FixtureSql.cs                   # loads fixture_cascade_F3.sql / fixture_cascade_F4.sql
    ├── Tests/
    │   ├── Vehicles/                       # FT-P-01..06, FT-N-01..03
    │   ├── Missions/                       # FT-P-07..12, FT-N-04..06
    │   ├── Waypoints/                      # FT-P-13..15, FT-P-18, FT-N-07
    │   ├── Health/                         # FT-P-16..17, FT-N-08
    │   ├── Security/                       # NFT-SEC-01..13, 04b
    │   ├── Resilience/                     # NFT-RES-01..08
    │   ├── ResourceLimits/                 # NFT-RES-LIM-01..04
    │   └── Performance/                    # NFT-PERF-01..04
    └── Reporting/
        ├── TrxToCsvPostProcessor.cs        # produces /app/results/report.csv per environment.md § Reporting
        └── ResultRow.cs                    # TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage
```

### Layout Rationale

- **Per-feature test folders** (`Vehicles/`, `Missions/`, etc.) match the natural decomposition surface in `blackbox-tests.md` and let `dotnet test --filter` slice the suite per Jira child ticket.
- **`Fixtures/` separate from `Tests/`** so xUnit `IClassFixture<>` lifetime is explicit (class-scoped DB reset) and not entangled with test cases.
- **`Helpers/` named for the assertion family** (DB, HTTP, FixtureSql) so each test reads as a single `// Arrange` + `// Act` + `// Assert` block per `coderule.mdc`.
- **JwksMock is a SEPARATE csproj**, not nested inside the e2e tests, because the build context is mounted as a service in `docker-compose.test.yml` (`tests/Azaion.Missions.JwksMock/`). Sharing a project would force the e2e runner to ship JWKS code into its container.
- **CA cert lives at `tests/jwks-mock-ca.crt`** rather than inside a project so both consumers (missions + e2e-consumer) mount the same file. The cert is regenerated only when the keypair changes — committed to the repo for deterministic test runs.

## Mock Services

| Mock Service | Replaces | Endpoints | Behavior |
|-------------|----------|-----------|----------|
| `jwks-mock` | `admin` JWT issuer + JWKS endpoint | `GET https://jwks-mock:8443/.well-known/jwks.json`; `POST https://jwks-mock:8443/sign`; `POST https://jwks-mock:8443/rotate-key` | Holds one ECDSA P-256 keypair in memory; serves the public half as JWKS with `Cache-Control: public, max-age=60`; signs ECDSA-SHA256 JWTs on `/sign` honoring optional `iss`/`aud`/`exp_offset_seconds`/`permissions`/`alg_override`/`kid_override`; rotates keypair on `/rotate-key` while retaining the old public key for `OLD_KEY_GRACE_SECONDS` (5s in tests). Private key never leaves the container. |

DB-only stubs (no service running, side-channel SQL inserts only): `annotations`, `detection`, `media`, `map_objects` — see `_docs/02_document/tests/test-data.md` § External Dependency Mocks.

### Mock Control API

`jwks-mock` exposes `POST /sign` and `POST /rotate-key` as its full control surface. The `/sign` body shape is documented in `test-data.md` § "JWKS mock token-minting contract":

```http
POST https://jwks-mock:8443/sign
{
  "iss": "https://admin-test.azaion.local",   # optional
  "aud": "azaion-edge",                        # optional
  "exp_offset_seconds": 3600,                   # optional; negative for expired
  "permissions": "FL",                          # optional; "" / "ADMIN" / "fl" / "FLight" for claim-mismatch
  "alg_override": null,                         # "HS256" to test alg-confusion (NFT-SEC-10)
  "kid_override": null                          # non-existent kid for unknown-key tests (NFT-SEC-11)
}
```

Response: `{ "token": "<encoded JWT>", "kid": "<key id>" }`.

## Docker Test Environment

### docker-compose.test.yml Structure

| Service | Image / Build | Purpose | Depends On |
|---------|--------------|---------|------------|
| `postgres-test` | `postgres:16-alpine` | Owned test PostgreSQL; `tmpfs:/var/lib/postgresql/data` for `down -v` isolation | — |
| `jwks-mock` | build `tests/Azaion.Missions.JwksMock/` → `azaion/jwks-mock:test` | Mock JWKS issuer | — |
| `missions` | build `.` (repo root `Dockerfile`) → `azaion/missions:test` | System under test | `postgres-test` (healthy), `jwks-mock` (healthy) |
| `e2e-consumer` | build `tests/Azaion.Missions.E2E.Tests/` | xUnit runner; emits `report.csv` to host-mounted `./test-results/` | `missions` (healthy), `jwks-mock` (healthy) |

The compose file is already authored at the repo root. This task does NOT modify it — the file IS the contract; the task fills in the two missing build contexts so the references resolve.

### Networks and Volumes

| Resource | Purpose |
|----------|---------|
| `e2e-net` (bridge) | Isolated test network; no host network access. All four services attach. |
| `tmpfs:/var/lib/postgresql/data` | Ephemeral PG data; recreated per `docker compose down -v`. |
| `./test-results:/app/results` | `e2e-consumer` mounts this for `report.csv` output to the host. |
| `./tests/jwks-mock-ca.crt:/usr/local/share/ca-certificates/jwks-mock-ca.crt:ro` | Mounted into `missions` AND `e2e-consumer` so both trust the mock's HTTPS cert after `update-ca-certificates --fresh` runs in `docker-entrypoint.sh`. |

## Test Runner Configuration

**Framework**: xUnit 2.x
**Plugins**: `Microsoft.NET.Test.Sdk`, `xunit.runner.visualstudio`, `Bogus 35.x` (synthetic data), `Npgsql 10.x` (side-channel only — NO `Azaion.Missions.*` project reference)
**Entry point**: `dotnet test tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj --logger "trx;LogFileName=results.trx"` followed by `TrxToCsvPostProcessor` converting `results.trx` → `report.csv`
**AAA convention**: every test method has `// Arrange` / `// Act` / `// Assert` comments per `.cursor/rules/coderule.mdc`.

### Fixture Strategy

| Fixture | Scope | Purpose |
|---------|-------|---------|
| `DbResetFixture` | Class (`IClassFixture<>`) | `TRUNCATE TABLE` for all schema tables between classes; cheap reset for read-path tests (AC-1, AC-2, AC-4) |
| `DbSeedFixture<TSeed>` | Class | Applies the named seed sets from `test-data.md` (`seed_empty`, `seed_one_default_vehicle`, `seed_3_vehicles_2_default`, `seed_25_missions`, `fixture_cascade_F3`, `fixture_cascade_F4`, `seed_5_waypoints_unordered`, `seed_legacy_gps_tables`) via Npgsql side-channel |
| `ComposeRestartFixture` | Collection | `docker compose -f docker-compose.test.yml down -v && up -d` between scenarios that assert startup-time behavior (AC-6.3..6.7, AC-5.7) |
| `JwksRotateFixture` | Scenario | `POST jwks-mock:8443/rotate-key` then waits for missions to refresh its JWKS cache (≤ 30s in tests, capped by `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS`) |
| `JwksMockReverseFixture` | Scenario | Boots `missions` outside compose via `docker run` with `ASPNETCORE_ENVIRONMENT=Production` + empty `CorsConfig:AllowedOrigins` to test E9 lock (NFT-SEC-13) |

### xUnit traits

Every test method MUST set `[Trait("Category", "Blackbox" | "Sec" | "Res" | "ResLim" | "Perf")]`. The CSV `Category` column reads from this trait. Traceability IDs go into a second `[Trait("Traces", "AC-1.2,AC-1.4")]` trait, comma-separated.

## Test Data Fixtures

Loaded entirely from `_docs/02_document/tests/test-data.md` § Seed Data Sets. The fixtures bind the named seeds to the AC IDs that consume them:

| Data Set | Source | Format | Used By |
|----------|--------|--------|---------|
| `seed_empty` | `down -v` + `missions` startup migrator | Schema only, no rows | bootstrap, unauth, 404 scenarios |
| `seed_one_default_vehicle` | Side-channel `INSERT INTO vehicles ...` | Inline SQL string | AC-1.2 default-clear, AC-1.3 TOCTOU, AC-1.4 setDefault, AC-2.1 mission-create |
| `seed_3_vehicles_2_default` | Side-channel SQL | Inline | AC-1.5 list, AC-1.6 filter |
| `seed_25_missions` | Side-channel SQL with deterministic UUIDs | Inline | AC-2.3..2.5 pagination + date filter |
| `fixture_cascade_F3` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql` | SQL file | AC-3.1, 3.3, 3.4, 10.2 |
| `fixture_cascade_F4` | `_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql` | SQL file | AC-4.5, 4.6 |
| `seed_5_waypoints_unordered` | Side-channel SQL with `order_num [3,1,2,5,4]` | Inline | AC-4.3 unpaginated ordering |
| `seed_legacy_gps_tables` | `CREATE TABLE orthophotos / gps_corrections` + `INSERT` | Inline | AC-3.5 absence, AC-6.5 one-shot drop, AC-10.5 legacy migration |

### Data Isolation

Three tiers, by scenario type (per `test-data.md` § Data Isolation Strategy):

- **Class-scoped DB reset** (`IClassFixture<DbResetFixture>`): for scenarios that share a seed within a class but must not leak across classes. Used for AC-1, AC-2, AC-4 read paths.
- **Scenario-scoped container restart** (`docker compose down -v && up -d`): for scenarios that assert startup-time behavior or migrator side-effects (AC-6.3..6.7, AC-6.11, AC-5.7).
- **No per-test transaction rollback** — the system under test is a separate process; its `DataConnection` is not in the test transaction.

## Test Reporting

**Format**: CSV
**Columns**: `TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage`
**Output path**: `/app/results/report.csv` inside `e2e-consumer`, mounted to `./test-results/report.csv` on the host
**Source**: post-processor reads `results.trx` (xUnit logger output), joins each test's `[Trait("Category",...)]` and `[Trait("Traces",...)]` into the CSV columns. `Result` is `pass` / `fail` / `skip`. `ErrorMessage` is the first line of the failure message (CRs stripped).

## Acceptance Criteria

**AC-1: Test environment starts**
Given the `docker-compose.test.yml` at repo root
When `docker compose -f docker-compose.test.yml up --build` runs
Then `postgres-test`, `jwks-mock`, and `missions` all reach `healthy`, and `e2e-consumer` starts after them

**AC-2: Mock JWKS service responds**
Given the test environment is running
When `GET https://jwks-mock:8443/.well-known/jwks.json` is issued from inside `e2e-net`
Then the response is `200 OK` with a JWKS body containing exactly one ECDSA P-256 public key
And `POST https://jwks-mock:8443/sign` with body `{}` returns a valid ECDSA-SHA256 JWT whose `iss` / `aud` match the mock's env vars

**AC-3: Test runner executes**
Given the test environment is running
When `e2e-consumer` starts and `dotnet test` runs
Then the runner discovers ≥ 1 test in each of the eight test folders (`Vehicles/`, `Missions/`, `Waypoints/`, `Health/`, `Security/`, `Resilience/`, `ResourceLimits/`, `Performance/`)

**AC-4: Test report generated**
Given tests have been executed
When `e2e-consumer` exits
Then `./test-results/report.csv` exists on the host
And the first line is the documented column header `TestId,TestName,Category,Traces,ExecutionTimeMs,Result,ErrorMessage`
And every executed test has exactly one CSV row

**AC-5: CA trust works end-to-end**
Given `tests/jwks-mock-ca.crt` is mounted into both `missions` and `e2e-consumer`
When `docker-entrypoint.sh` runs `update-ca-certificates --fresh` and `missions` issues `GET https://jwks-mock:8443/.well-known/jwks.json` to populate its JWKS cache
Then the TLS handshake succeeds (no `RemoteCertificateNotAvailable` / `RemoteCertificateNameMismatch`)
And the cached JWKS contains the public key the consumer-issued tokens are signed with

**AC-6: JWKS rotation observable inside the 15-minute CI gate**
Given the test compose sets `JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30` and `JWT_JWKS_REFRESH_INTERVAL_SECONDS=10` (per C01)
When `POST https://jwks-mock:8443/rotate-key` is called
Then within 30s `missions` refreshes its JWKS cache and accepts tokens signed with the new `kid`
And during the 5s `OLD_KEY_GRACE_SECONDS` window tokens signed with the old `kid` are still accepted

**AC-7: AAA pattern enforced**
Given the xUnit test project compiles
When `dotnet build` runs
Then every `[Fact]` / `[Theory]` method in `tests/Azaion.Missions.E2E.Tests/Tests/` contains the literal comment lines `// Arrange` (when setup exists), `// Act`, and `// Assert` in that order — verified by a Roslyn analyzer test or a single integration assertion that greps the source files

## Constraints

- `restrictions.md` SW-01: target framework .NET 10 (matches `Azaion.Missions.csproj`)
- `restrictions.md` HW-01: ARM64 + AMD64 (multi-arch base images on both projects)
- `restrictions.md` ENV-01: HTTPS-only for the JWKS endpoint (HTTP would short-circuit AC-6.12)
- `coderule.mdc`: AAA pattern with `// Arrange` / `// Act` / `// Assert` comments, no narrative comments otherwise
- No project reference from `Azaion.Missions.E2E.Tests` → `Azaion.Missions.csproj` (consumer must remain blackbox; assertions only via HTTP and Npgsql side-channel)
- Side-channel DB access limited to fixture seeding + post-call assertions; marked with `[Trait("db_access","seed-or-assert-only")]` where used
- Token signing happens ONLY inside `jwks-mock`; the consumer never imports a JWT signing library
- `report.csv` lives in `./test-results/` (host-mounted); this directory MUST be in `.gitignore`