missions/_docs/tasks/todo/AZ-576_test_infrastructure.md

Test Infrastructure

Task: AZ-576_test_infrastructure Name: Test Infrastructure (Missions e2e) Description: Scaffold the Blackbox test project — xUnit runner, JWKS mock service, Docker test environment wiring, test data fixtures, reporting. Compose file already exists at repo root and references not-yet-built build contexts; this task fills in those contexts. Complexity: 5 points Dependencies: None (C01 + C02 testability refactor already landed; see _docs/04_refactoring/01-testability-refactoring/testability_changes_summary.md) Component: Blackbox Tests Tracker: AZ-576 Epic: AZ-575

Scope

Two artifacts that the existing docker-compose.test.yml references but does not yet build, plus the run script the suite expects:

1. tests/Azaion.Missions.JwksMock/ — minimal HTTPS service holding an ECDSA P-256 keypair in memory, serving JWKS + POST /sign + POST /rotate-key. Image tag azaion/jwks-mock:test.
2. tests/Azaion.Missions.E2E.Tests/ — xUnit 2.x test project that drives the running missions service over HTTP, mints tokens via https://jwks-mock:8443/sign, asserts DB side-effects through a side-channel Npgsql connection, and produces report.csv.
3. tests/jwks-mock-ca.crt — the self-signed CA cert that both missions and e2e-consumer mount and update-ca-certificates --fresh adds to the OS trust bundle (per docker-entrypoint.sh from C02).
4. scripts/run-tests.sh — wraps docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer, collects report.csv, then down -v.
5. scripts/run-performance-tests.sh — same compose stack with the [Trait("Category","Perf")] filter and the perf seed.

The missions and postgres-test services already exist in docker-compose.test.yml; the jwks-mock and e2e-consumer services exist but point at build contexts that this task creates.

Test Project Folder Layout

tests/
├── jwks-mock-ca.crt                        # self-signed CA (mounted into missions + e2e-consumer)
├── Azaion.Missions.JwksMock/
│   ├── Azaion.Missions.JwksMock.csproj
│   ├── Dockerfile                          # builds azaion/jwks-mock:test, exposes 8443/tcp
│   ├── Program.cs                          # ASP.NET Core minimal API
│   ├── Endpoints/
│   │   ├── JwksEndpoint.cs                 # GET /.well-known/jwks.json
│   │   ├── SignEndpoint.cs                 # POST /sign
│   │   └── RotateKeyEndpoint.cs            # POST /rotate-key
│   ├── Services/
│   │   ├── KeyStore.cs                     # in-memory ECDSA P-256 keypair + old-key grace window
│   │   └── TokenSigner.cs                  # ECDSA signing with alg_override/kid_override support
│   └── appsettings.json                    # JWT_ISSUER, JWT_AUDIENCE, OLD_KEY_GRACE_SECONDS
└── Azaion.Missions.E2E.Tests/
    ├── Azaion.Missions.E2E.Tests.csproj    # xUnit 2.x + Bogus 35.x + Npgsql 10.x
    ├── Dockerfile                          # runs `dotnet test --logger trx` + trx→csv post-step
    ├── TestBase.cs                         # HttpClient factory, default JWT, shared MissionsBaseUrl
    ├── TokenMinter.cs                      # POST jwks-mock:8443/sign with overrides
    ├── Fixtures/
    │   ├── DbResetFixture.cs               # IClassFixture<>: TRUNCATE between classes
    │   ├── DbSeedFixture.cs                # base for the named seed sets in test-data.md
    │   ├── ComposeRestartFixture.cs        # docker compose down -v && up -d for bootstrap-sensitive tests
    │   └── JwksRotateFixture.cs            # POST /rotate-key + wait for missions to refresh JWKS cache
    ├── Helpers/
    │   ├── DbAssertions.cs                 # Npgsql side-channel asserts (row counts, default-vehicle invariants)
    │   ├── HttpAssertions.cs               # PascalCase shape, error-envelope shape, ordering, pagination
    │   └── FixtureSql.cs                   # loads fixture_cascade_F3.sql / fixture_cascade_F4.sql
    ├── Tests/
    │   ├── Vehicles/                       # FT-P-01..06, FT-N-01..03
    │   ├── Missions/                       # FT-P-07..12, FT-N-04..06
    │   ├── Waypoints/                      # FT-P-13..15, FT-P-18, FT-N-07
    │   ├── Health/                         # FT-P-16..17, FT-N-08
    │   ├── Security/                       # NFT-SEC-01..13, 04b
    │   ├── Resilience/                     # NFT-RES-01..08
    │   ├── ResourceLimits/                 # NFT-RES-LIM-01..04
    │   └── Performance/                    # NFT-PERF-01..04
    └── Reporting/
        ├── TrxToCsvPostProcessor.cs        # produces /app/results/report.csv per environment.md § Reporting
        └── ResultRow.cs                    # TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage

Layout Rationale

• Per-feature test folders (Vehicles/, Missions/, etc.) match the natural decomposition surface in blackbox-tests.md and let dotnet test --filter slice the suite per Jira child ticket.
• Fixtures/ separate from Tests/ so xUnit IClassFixture<> lifetime is explicit (class-scoped DB reset) and not entangled with test cases.
• Helpers/ named for the assertion family (DB, HTTP, FixtureSql) so each test reads as a single // Arrange + // Act + // Assert block per coderule.mdc.
• JwksMock is a SEPARATE csproj, not nested inside the e2e tests, because the build context is mounted as a service in docker-compose.test.yml (tests/Azaion.Missions.JwksMock/). Sharing a project would force the e2e runner to ship JWKS code into its container.
• CA cert lives at tests/jwks-mock-ca.crt rather than inside a project so both consumers (missions + e2e-consumer) mount the same file. The cert is regenerated only when the keypair changes — committed to the repo for deterministic test runs.

Mock Services

Mock Service	Replaces	Endpoints	Behavior
jwks-mock	admin JWT issuer + JWKS endpoint	GET https://jwks-mock:8443/.well-known/jwks.json; POST https://jwks-mock:8443/sign; POST https://jwks-mock:8443/rotate-key	Holds one ECDSA P-256 keypair in memory; serves the public half as JWKS with Cache-Control: public, max-age=60; signs ECDSA-SHA256 JWTs on /sign honoring optional iss/aud/exp_offset_seconds/permissions/alg_override/kid_override; rotates keypair on /rotate-key while retaining the old public key for OLD_KEY_GRACE_SECONDS (5s in tests). Private key never leaves the container.

DB-only stubs (no service running, side-channel SQL inserts only): annotations, detection, media, map_objects — see _docs/02_document/tests/test-data.md § External Dependency Mocks.

Mock Control API

jwks-mock exposes POST /sign and POST /rotate-key as its full control surface. The /sign body shape is documented in test-data.md § "JWKS mock token-minting contract":

POST https://jwks-mock:8443/sign
{
  "iss": "https://admin-test.azaion.local",   # optional
  "aud": "azaion-edge",                        # optional
  "exp_offset_seconds": 3600,                   # optional; negative for expired
  "permissions": "FL",                          # optional; "" / "ADMIN" / "fl" / "FLight" for claim-mismatch
  "alg_override": null,                         # "HS256" to test alg-confusion (NFT-SEC-10)
  "kid_override": null                          # non-existent kid for unknown-key tests (NFT-SEC-11)
}

Response: { "token": "<encoded JWT>", "kid": "<key id>" }.

Docker Test Environment

docker-compose.test.yml Structure

Service	Image / Build	Purpose	Depends On
postgres-test	postgres:16-alpine	Owned test PostgreSQL; tmpfs:/var/lib/postgresql/data for down -v isolation	—
jwks-mock	build tests/Azaion.Missions.JwksMock/ → azaion/jwks-mock:test	Mock JWKS issuer	—
missions	build . (repo root Dockerfile) → azaion/missions:test	System under test	postgres-test (healthy), jwks-mock (healthy)
e2e-consumer	build tests/Azaion.Missions.E2E.Tests/	xUnit runner; emits report.csv to host-mounted ./test-results/	missions (healthy), jwks-mock (healthy)

The compose file is already authored at the repo root. This task does NOT modify it — the file IS the contract; the task fills in the two missing build contexts so the references resolve.

Networks and Volumes

Resource	Purpose
e2e-net (bridge)	Isolated test network; no host network access. All four services attach.
tmpfs:/var/lib/postgresql/data	Ephemeral PG data; recreated per docker compose down -v.
./test-results:/app/results	e2e-consumer mounts this for report.csv output to the host.
./tests/jwks-mock-ca.crt:/usr/local/share/ca-certificates/jwks-mock-ca.crt:ro	Mounted into missions AND e2e-consumer so both trust the mock's HTTPS cert after update-ca-certificates --fresh runs in docker-entrypoint.sh.

Test Runner Configuration

Framework: xUnit 2.x Plugins: Microsoft.NET.Test.Sdk, xunit.runner.visualstudio, Bogus 35.x (synthetic data), Npgsql 10.x (side-channel only — NO Azaion.Missions.* project reference) Entry point: dotnet test tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj --logger "trx;LogFileName=results.trx" followed by TrxToCsvPostProcessor converting results.trx → report.csv AAA convention: every test method has // Arrange / // Act / // Assert comments per .cursor/rules/coderule.mdc.

Fixture Strategy

Fixture	Scope	Purpose
DbResetFixture	Class (IClassFixture<>)	TRUNCATE TABLE for all schema tables between classes; cheap reset for read-path tests (AC-1, AC-2, AC-4)
DbSeedFixture<TSeed>	Class	Applies the named seed sets from test-data.md (seed_empty, seed_one_default_vehicle, seed_3_vehicles_2_default, seed_25_missions, fixture_cascade_F3, fixture_cascade_F4, seed_5_waypoints_unordered, seed_legacy_gps_tables) via Npgsql side-channel
ComposeRestartFixture	Collection	docker compose -f docker-compose.test.yml down -v && up -d between scenarios that assert startup-time behavior (AC-6.3..6.7, AC-5.7)
JwksRotateFixture	Scenario	POST jwks-mock:8443/rotate-key then waits for missions to refresh its JWKS cache (≤ 30s in tests, capped by JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS)
JwksMockReverseFixture	Scenario	Boots missions outside compose via docker run with ASPNETCORE_ENVIRONMENT=Production + empty CorsConfig:AllowedOrigins to test E9 lock (NFT-SEC-13)

xUnit traits

Every test method MUST set [Trait("Category", "Blackbox" | "Sec" | "Res" | "ResLim" | "Perf")]. The CSV Category column reads from this trait. Traceability IDs go into a second [Trait("Traces", "AC-1.2,AC-1.4")] trait, comma-separated.

Test Data Fixtures

Loaded entirely from _docs/02_document/tests/test-data.md § Seed Data Sets. The fixtures bind the named seeds to the AC IDs that consume them:

Data Set	Source	Format	Used By
seed_empty	down -v + missions startup migrator	Schema only, no rows	bootstrap, unauth, 404 scenarios
seed_one_default_vehicle	Side-channel INSERT INTO vehicles ...	Inline SQL string	AC-1.2 default-clear, AC-1.3 TOCTOU, AC-1.4 setDefault, AC-2.1 mission-create
seed_3_vehicles_2_default	Side-channel SQL	Inline	AC-1.5 list, AC-1.6 filter
seed_25_missions	Side-channel SQL with deterministic UUIDs	Inline	AC-2.3..2.5 pagination + date filter
fixture_cascade_F3	_docs/00_problem/input_data/expected_results/fixture_cascade_F3.sql	SQL file	AC-3.1, 3.3, 3.4, 10.2
fixture_cascade_F4	_docs/00_problem/input_data/expected_results/fixture_cascade_F4.sql	SQL file	AC-4.5, 4.6
seed_5_waypoints_unordered	Side-channel SQL with order_num [3,1,2,5,4]	Inline	AC-4.3 unpaginated ordering
seed_legacy_gps_tables	CREATE TABLE orthophotos / gps_corrections + INSERT	Inline	AC-3.5 absence, AC-6.5 one-shot drop, AC-10.5 legacy migration

Data Isolation

Three tiers, by scenario type (per test-data.md § Data Isolation Strategy):

• Class-scoped DB reset (IClassFixture<DbResetFixture>): for scenarios that share a seed within a class but must not leak across classes. Used for AC-1, AC-2, AC-4 read paths.
• Scenario-scoped container restart (docker compose down -v && up -d): for scenarios that assert startup-time behavior or migrator side-effects (AC-6.3..6.7, AC-6.11, AC-5.7).
• No per-test transaction rollback — the system under test is a separate process; its DataConnection is not in the test transaction.

Test Reporting

Format: CSV Columns: TestId, TestName, Category, Traces, ExecutionTimeMs, Result, ErrorMessage Output path: /app/results/report.csv inside e2e-consumer, mounted to ./test-results/report.csv on the host Source: post-processor reads results.trx (xUnit logger output), joins each test's [Trait("Category",...)] and [Trait("Traces",...)] into the CSV columns. Result is pass / fail / skip. ErrorMessage is the first line of the failure message (CRs stripped).

Acceptance Criteria

AC-1: Test environment starts Given the docker-compose.test.yml at repo root When docker compose -f docker-compose.test.yml up --build runs Then postgres-test, jwks-mock, and missions all reach healthy, and e2e-consumer starts after them

AC-2: Mock JWKS service responds Given the test environment is running When GET https://jwks-mock:8443/.well-known/jwks.json is issued from inside e2e-net Then the response is 200 OK with a JWKS body containing exactly one ECDSA P-256 public key And POST https://jwks-mock:8443/sign with body {} returns a valid ECDSA-SHA256 JWT whose iss / aud match the mock's env vars

AC-3: Test runner executes Given the test environment is running When e2e-consumer starts and dotnet test runs Then the runner discovers ≥ 1 test in each of the eight test folders (Vehicles/, Missions/, Waypoints/, Health/, Security/, Resilience/, ResourceLimits/, Performance/)

AC-4: Test report generated Given tests have been executed When e2e-consumer exits Then ./test-results/report.csv exists on the host And the first line is the documented column header TestId,TestName,Category,Traces,ExecutionTimeMs,Result,ErrorMessage And every executed test has exactly one CSV row

AC-5: CA trust works end-to-end Given tests/jwks-mock-ca.crt is mounted into both missions and e2e-consumer When docker-entrypoint.sh runs update-ca-certificates --fresh and missions issues GET https://jwks-mock:8443/.well-known/jwks.json to populate its JWKS cache Then the TLS handshake succeeds (no RemoteCertificateNotAvailable / RemoteCertificateNameMismatch) And the cached JWKS contains the public key the consumer-issued tokens are signed with

AC-6: JWKS rotation observable inside the 15-minute CI gate Given the test compose sets JWT_JWKS_AUTO_REFRESH_INTERVAL_SECONDS=30 and JWT_JWKS_REFRESH_INTERVAL_SECONDS=10 (per C01) When POST https://jwks-mock:8443/rotate-key is called Then within 30s missions refreshes its JWKS cache and accepts tokens signed with the new kid And during the 5s OLD_KEY_GRACE_SECONDS window tokens signed with the old kid are still accepted

AC-7: AAA pattern enforced Given the xUnit test project compiles When dotnet build runs Then every [Fact] / [Theory] method in tests/Azaion.Missions.E2E.Tests/Tests/ contains the literal comment lines // Arrange (when setup exists), // Act, and // Assert in that order — verified by a Roslyn analyzer test or a single integration assertion that greps the source files

Constraints

• restrictions.md SW-01: target framework .NET 10 (matches Azaion.Missions.csproj)
• restrictions.md HW-01: ARM64 + AMD64 (multi-arch base images on both projects)
• restrictions.md ENV-01: HTTPS-only for the JWKS endpoint (HTTP would short-circuit AC-6.12)
• coderule.mdc: AAA pattern with // Arrange / // Act / // Assert comments, no narrative comments otherwise
• No project reference from Azaion.Missions.E2E.Tests → Azaion.Missions.csproj (consumer must remain blackbox; assertions only via HTTP and Npgsql side-channel)
• Side-channel DB access limited to fixture seeding + post-call assertions; marked with [Trait("db_access","seed-or-assert-only")] where used
• Token signing happens ONLY inside jwks-mock; the consumer never imports a JWT signing library
• report.csv lives in ./test-results/ (host-mounted); this directory MUST be in .gitignore