[AZ-585] [AZ-586] ResLim+Perf NFT tests; close test cycle 1

Batch 4 of test implementation cycle 1 (existing-code Step 6, final batch).

- AZ-585 SteadyStateLoadTests + ColdStartRssTests: NFT-RES-LIM-01..04.
  SteadyStateLoadFixture runs one 5-min sustained-load window and samples
  RSS (docker stats), Npgsql conns (pg_stat_activity), and FDs
  (/proc/1/fd) every 5s; three test methods assert independently. All
  SkippableFact-gated on docker primitives.
- AZ-586 PerformanceTests: NFT-PERF-01..04. Sequential single-client,
  5 warm-ups + N measured calls, P50+P95 via LatencyPercentiles, recorded
  to PERF_RESULTS_FILE. Tagged Category=Perf so default gate excludes them.

Infrastructure:
- entrypoint.sh now applies --filter "${TEST_FILTER:-Category!=Perf}"
  per AZ-586 (default CI gate excludes performance).
- MetricCsvRecorder: idempotent CSV appender keyed on env var, used by
  both Perf and ResLim categories.

Step 6 (Implement Tests) is complete. Final report at
_docs/03_implementation/implementation_report_tests.md handoffs the
full-suite gate to test-run/SKILL.md (Step 7).

Co-authored-by: Cursor <cursoragent@cursor.com>

_docs/03_implementation/batch_04_report.md

@@ -0,0 +1,80 @@
+# Batch Report
+
+**Batch**: 4
+**Tasks**: AZ-585, AZ-586
+**Date**: 2026-05-15
+**Run mode**: Test implementation (existing-code Step 6)
+**Total complexity**: 6 SP (3 + 3)
+
+## Task Results
+
+| Task | Status | Files Modified | Tests | AC Coverage | Issues |
+|------|--------|----------------|-------|-------------|--------|
+| AZ-585_test_resource_limits | Done | 3 added, 1 deleted | 4 / 4 discovery | 4/4 NFT-RES-LIM covered | 0 |
+| AZ-586_test_performance     | Done | 1 added, 1 deleted, 2 helpers added, entrypoint.sh modified | 4 / 4 discovery | 4/4 NFT-PERF covered | 0 |
+
+## AC Test Coverage: All 8 NFT scenarios covered
+
+- **AZ-585 (4/4)**: NFT-RES-LIM-01 → `SteadyStateLoadTests.NFT_RES_LIM_01_*` (P95 RSS + no-leak ratio), NFT-RES-LIM-02 → `SteadyStateLoadTests.NFT_RES_LIM_02_*` (Npgsql conn cap + minute-1 mean), NFT-RES-LIM-03 → `SteadyStateLoadTests.NFT_RES_LIM_03_*` (FD cap + minute-1 anchor), NFT-RES-LIM-04 → `ColdStartRssTests.NFT_RES_LIM_04_*` (30s settle + cold-RSS cap).
+- **AZ-586 (4/4)**: NFT-PERF-01 → `PerformanceTests.NFT_PERF_01_*` (100 minimal-cascade DELETEs, P50 ≤ 50ms), NFT-PERF-02 → `*.NFT_PERF_02_*` (50 F3-shape cascade DELETEs, provisional P50 ≤ 200ms), NFT-PERF-03 → `*.NFT_PERF_03_*` (100 `/health`, P50 ≤ 10ms), NFT-PERF-04 → `*.NFT_PERF_04_*` (100 paginated lists vs 1000-mission seed, provisional P95 ≤ 100ms).
+
+## Code Review Verdict: PASS_WITH_WARNINGS (self-review)
+
+- 0 Critical, 0 High, 0 Medium.
+- **Low — coverage**: 4 of 4 ResLim tests are `SkippableFact` gated on `COMPOSE_RESTART_ENABLED=1` + docker CLI in the e2e-consumer image — same Docker-socket follow-up already flagged in batch 3 report. NFT-RES-LIM-04 additionally requires `docker compose stop|rm|up` access; same gate.
+- **Low — maintainability**: `SteadyStateLoadFixture.ParseHumanBytes` and `ColdStartRssTests.ParseHumanBytes` are duplicated. Both files parse the LHS of `docker stats --no-stream --format '{{.MemUsage}}'`; the duplication is intentional today because the two files have different gating predicates (fixture uses `Enabled` property + `CommandAvailable` probe, ColdStart uses `MissionsContainerHelper.Enabled`), and lifting the helper to `Helpers/HumanBytes.cs` would be a shared-helper change worth a separate refactor. Captured as a follow-up note; not auto-fixed because it touches both files. **Recommend folding into the docker-CLI follow-up task.**
+- **Low — observability**: `PerformanceTests` swallows non-2xx-non-404 with `InvalidOperationException` (warmup + measured), so a misbehaving SUT mid-run yields a clear stack trace; no silent pass. This is intended.
+
+## Auto-Fix Attempts: 1
+
+`SteadyStateLoadFixture.cs:59` initially called `new TokenMinter()` (parameter-less ctor); `TokenMinter` requires `signUrl`. Fixed to `new TokenMinter(TestEnvironment.JwksMockBaseUrl + "/sign")` — same pattern as `TestBase`. Style/Low under the Auto-Fix Gate matrix. Rebuild: 0 warnings, 0 errors.
+
+## Stuck Agents: None
+
+## Files Created (5) + 2 deletions + 1 modified script
+
+### Helpers (2)
+
+- `tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs` — nearest-rank P50/P95/Percentile/Mean over `IReadOnlyList<double>`. Sorts a defensive copy.
+- `tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs` — appends one row per scenario (Timestamp, Category, Scenario, Result, Traces, ErrorMessage) to a CSV referenced by `PERF_RESULTS_FILE` (perf) or `RESLIM_RESULTS_FILE` (reslim). No-op when the env var is unset.
+
+### Fixtures (1)
+
+- `tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs` — class-scoped 5-minute sustained-load fixture. Generates ~50 RPS via a single-threaded `HttpClient` loop, samples RSS / Npgsql conn count / FD count every 5s. Exposes the time series + `LoadGeneratorMetTargetRps` + `SutExitedDuringWindow` + `SkipReason`. Tests inspect `SkipReason` to surface explicit skips when docker primitives are unavailable.
+
+### Test classes (3)
+
+- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs` — NFT-RES-LIM-01..03 share the fixture window. Each test asserts one metric independently. `[Collection("ResLimSteadyState")]`.
+- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs` — NFT-RES-LIM-04. Runs `docker compose stop|rm|up missions` for a fresh start, waits 30s after `/health` returns 200, reads RSS, asserts ≤ 200 MiB. Lives in the `MigratorRestart` collection to serialise with the other compose-restarting tests.
+- `tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs` — NFT-PERF-01..04, all `[Trait("Category","Perf")]`. Sequential single-client, 5 warm-ups + N measured, records P50 + P95 to `PERF_RESULTS_FILE`.
+
+### Deleted (2 Sanity placeholders)
+
+- `tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs` — dead placeholder from AZ-576; replaced by `PerformanceTests`.
+- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs` — same.
+
+### Modified (entrypoint filter, per AZ-586 Spec)
+
+- `tests/Azaion.Missions.E2E.Tests/entrypoint.sh` — added `--filter "${TEST_FILTER:-Category!=Perf}"`. The default CI gate now excludes the Performance category (AZ-586 Spec § Outcome: "default test suite filter excludes performance to keep the standard CI gate ≤ 15 min"); `scripts/run-performance-tests.sh` bypasses the entrypoint anyway and invokes `dotnet test --filter Category=Perf` directly. The shell variable `TEST_FILTER` is overridable for ad-hoc invocations (e.g., to include Perf during a local profiling session).
+
+## Local Verification
+
+- `dotnet build tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj` — 0 warnings, 0 errors.
+- 8 new NFT methods discoverable via `[Trait("Category","Perf")]` (4) and `[Trait("Category","ResLim")]` (4).
+
+## Pre-existing issues NOT in scope
+
+- `scripts/run-performance-tests.sh` line 104 references `/app/Azaion.Missions.E2E.Tests.csproj`, but the Dockerfile copies the test project to `/src/`. Pre-existing script bug — flag for the docker-CLI follow-up task that re-validates the run-perf script end-to-end. Not introduced by this batch.
+- Root `Azaion.Missions.csproj` Sdk.Web globs still pull `tests/**/*.cs` into the main project compilation — same flag as batch 3 cumulative review report; pre-existing.
+
+## Docker Stack Validation
+
+Not run as part of this batch — same hand-off as batches 1-3. Step 7 (`test-run/SKILL.md`) owns the `docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer` gate. The 5 SkippableFact tests in this batch activate when the consumer image has `docker` CLI + socket bind; otherwise they emit explicit skip reasons (no silent pass).
+
+## Tracker Updates
+
+AZ-585, AZ-586 transitioned to `In Testing` via the Atlassian MCP after this commit (Step 12).
+
+## Next Batch
+
+All 11 test tasks (AZ-576 + AZ-577..AZ-586) are now done. Step 6 (Implement Tests) is **complete**. Autodev advances to Step 7 (Run Tests) — `test-run/SKILL.md` owns the full-suite gate.

_docs/03_implementation/implementation_report_tests.md

@@ -0,0 +1,119 @@
+# Test Implementation Final Report
+
+**Run**: existing-code Step 6 (Implement Tests)
+**Date**: 2026-05-15
+**Cycle**: 1
+**Verdict**: HANDOFF — full-suite gate owned by `.cursor/skills/test-run/SKILL.md` (Step 7)
+
+## Scope
+
+11 test tasks decomposed by `/decompose-tests` and tracked under epic **AZ-575**:
+
+| Task    | Description                                              | SP | Batch |
+|---------|----------------------------------------------------------|----|-------|
+| AZ-576  | Test infrastructure (compose, csproj, mocks, helpers)    | 5  | 1     |
+| AZ-577  | Vehicles positive (FT-P-01..06)                          | 5  | 2     |
+| AZ-578  | Missions positive (FT-P-07..12)                          | 5  | 2     |
+| AZ-579  | Waypoints + health positive (FT-P-13..18)                | 5  | 2     |
+| AZ-580  | Validation + authz negative (FT-N-01..08)                | 3  | 2     |
+| AZ-581  | Security auth/claims (NFT-SEC-01..06+04b)                | 5  | 3     |
+| AZ-582  | Security alg/rotation/CORS (NFT-SEC-07..13)              | 5  | 3     |
+| AZ-583  | Resilience cascade + migrator (NFT-RES-01..04)           | 3  | 3     |
+| AZ-584  | Resilience config/DB/rotation/race (NFT-RES-05..08)      | 5  | 3     |
+| AZ-585  | Resource limits (NFT-RES-LIM-01..04)                     | 3  | 4     |
+| AZ-586  | Performance (NFT-PERF-01..04)                            | 3  | 4     |
+| **Total** |                                                        | **47** |     |
+
+## Results
+
+| Batch | Tasks                            | SP | Verdict              | Carry-forwards |
+|-------|----------------------------------|----|----------------------|----------------|
+| 1     | AZ-576                           | 5  | PASS_WITH_WARNINGS   | 0              |
+| 2     | AZ-577..AZ-580                   | 18 | PASS_WITH_WARNINGS   | 3              |
+| 3     | AZ-581..AZ-584                   | 18 | PASS_WITH_WARNINGS   | 3              |
+| 4     | AZ-585, AZ-586                   | 6  | PASS_WITH_WARNINGS   | 0              |
+
+**Cumulative reviews**: 1 (`cumulative_review_batches_01-03_cycle1_report.md`, PASS_WITH_WARNINGS, 4 Low findings).
+
+## AC Test Coverage
+
+| Source | ACs | Tests | Coverage |
+|--------|-----|-------|----------|
+| FT-P (functional positive) | 18 | 18 | 18/18 |
+| FT-N (negative)             | 8  | 8  | 8/8   |
+| NFT-SEC (security)          | 14 | 22 | 14/14 (some scenarios → multiple `Theory` rows) |
+| NFT-RES (resilience)        | 8  | 12 | 8/8   |
+| NFT-RES-LIM (resource lim)  | 4  | 4  | 4/4   |
+| NFT-PERF (performance)      | 4  | 4  | 4/4   |
+| **Total**                   | **56** | **68** | **56/56** |
+
+Every AC has at least one trace via `[Trait("Traces", "AC-X.Y")]`; structural carry-forwards (6 total) are pinned with `[Trait("carry_forward", "...")]` so `dotnet test --filter "carry_forward~..."` surfaces them as a set when the underlying spec/code reconciliation lands.
+
+## Spec-vs-Code Carry-forwards (6 total)
+
+| Site                                    | Spec says                                    | Code says                                                  | Carry-forward tag             |
+|-----------------------------------------|----------------------------------------------|------------------------------------------------------------|-------------------------------|
+| FT-P-03 `Vehicles/PositiveTests.cs`     | `POST /vehicles/{id}/setDefault` → 200 + body| `[HttpPatch("{id:guid}/default")]` → 204 NoContent          | `AC-1.4/route-shape`          |
+| FT-P-14/15 `Waypoints/PositiveTests.cs` | Nested `GeoPoint:{Lat,Lon,Mgrs}`             | LinqToDB entity flat `Lat`/`Lon`/`Mgrs`                    | `flat-waypoint-shape`         |
+| FT-N-07 `Waypoints/NegativeTests.cs`    | Missing parent → 404 + problem envelope      | `GetWaypoints` returns `[]`                                | `AC-4.2/missing-parent-soft`  |
+| NFT-RES-01 `Resilience/CascadeF3Tests.cs` | Mid-walk cascade is transactional          | `MissionService.DeleteMission` is non-transactional        | `ADR-006`                     |
+| NFT-RES-02 `Resilience/CascadeF4Tests.cs` | Waypoint cascade leaves detection=0/waypoint=1 partial state | `WaypointService.DeleteWaypoint` queries `media` BEFORE any deletion — aborts at step 1 with nothing deleted | `AC-4.6/walk-order`           |
+| NFT-RES-08 `Resilience/DefaultVehicleRaceTests.cs` | TOCTOU race observable                   | `ux_vehicles_one_default` partial unique index closes the race | `AC-1.4/index-closes-race`    |
+
+These carry-forwards flip the moment the spec or the code is reconciled; the tests fail loudly at that point — intentional.
+
+## Code Review Summary
+
+- **0 Critical / 0 High / 0 Medium** across all four batches.
+- **4 Low findings** captured in cumulative review (3 follow-up + 1 baseline-carried) — see `_docs/03_implementation/cumulative_review_batches_01-03_cycle1_report.md`.
+- Auto-fix rounds across the cycle: batch 2 (89× xUnit1030 warnings), batch 3 (3× missing-using errors), batch 4 (1× TokenMinter parameter-less ctor). All auto-fix-eligible per the Auto-Fix Gate matrix; no escalations.
+
+## Files Added (high level)
+
+- **Helpers** (10): `ApiDtos`, `DbAssertions`, `DockerLogs`, `FixtureSql`, `ForeignKeypair`, `HttpAssertions`, `LatencyPercentiles`, `MetricCsvRecorder`, `MissionsContainerHelper` — plus the existing `TestEnvironment`.
+- **Fixtures** (9): `CascadeF3Fixture`, `CascadeF4Fixture`, `ComposeRestartFixture`, `DbResetFixture`, `JwksMockReverseFixture` (spec-only stub), `JwksRotateFixture`, `PostgresStopStartFixture`, `Seeds`, `StubSchema`, `SteadyStateLoadFixture`.
+- **Test classes** (24): grouped under `Tests/{Vehicles,Missions,Waypoints,Health,Errors,Security,Resilience,Performance,ResourceLimits,Reporting}/` per the AZ-576 layout.
+- **Infrastructure**: `docker-compose.test.yml` extensions (fixtures volume), `entrypoint.sh` Category-filter, `Reporting/TrxToCsvPostProcessor.cs` (from batch 1).
+- **JWKS mock**: extended `SignBody` (permissions_array) + `TokenSigner` (kid_override validation) — required by NFT-SEC-06 and NFT-SEC-11.
+
+## SkippableFact / SkippableTheory inventory
+
+| Test                                                       | Skip predicate                                                   | Reason when skipped |
+|------------------------------------------------------------|------------------------------------------------------------------|----------------------|
+| `Tests/Health/HealthTests.NFT_P_17` (FT-P-17)              | `COMPOSE_RESTART_ENABLED=1`                                       | postgres-test stop/start |
+| `Tests/Errors/Error500Tests.NFT_N_08`                       | `COMPOSE_RESTART_ENABLED=1`                                       | drops vehicles table |
+| `Tests/Security/ErrorRedactionTests.NFT_SEC_08`             | `COMPOSE_RESTART_ENABLED=1`                                       | drops vehicles table |
+| `Tests/Security/StartupConfigTests.NFT_SEC_12` (theory + HTTP-JWKS) | `MissionsContainerHelper.Enabled`                          | docker run primitives |
+| `Tests/Security/CorsConfigTests.NFT_SEC_13` (4 scenarios)   | `MissionsContainerHelper.Enabled`                                | docker run primitives |
+| `Tests/Resilience/CascadeF3Tests.NFT_RES_01`               | `COMPOSE_RESTART_ENABLED=1`                                       | drops media table |
+| `Tests/Resilience/CascadeF4Tests.NFT_RES_02`               | `COMPOSE_RESTART_ENABLED=1`                                       | drops media table |
+| `Tests/Resilience/MigratorRestartTests.NFT_RES_03/04`       | `ComposeRestartFixture.Enabled`                                  | docker compose restart |
+| `Tests/Resilience/ConfigDbStartupTests.*` (8 methods)       | `MissionsContainerHelper.Enabled`                                | docker run primitives |
+| `Tests/Resilience/JwksRotationNoRestartTests.NFT_RES_07`    | `MissionsContainerHelper.Enabled` (for StartedAt read)           | docker inspect |
+| `Tests/ResourceLimits/SteadyStateLoadTests.*` (3 methods)   | `SteadyStateLoadFixture.SkipReason` (set on missing docker)      | docker stats / docker exec |
+| `Tests/ResourceLimits/ColdStartRssTests.NFT_RES_LIM_04`     | `COMPOSE_RESTART_ENABLED=1` + `MissionsContainerHelper.Enabled`  | docker compose stop/start |
+
+Every Skippable test surfaces an explicit reason; none silent-pass.
+
+## Handoff to Step 7 (Run Tests)
+
+This report is a **HANDOFF** — the full-suite gate is owned by `.cursor/skills/test-run/SKILL.md`. That skill is responsible for:
+
+1. Building the docker compose stack (`docker compose -f docker-compose.test.yml --profile test build`).
+2. Running the e2e-consumer (`docker compose ... up --abort-on-container-exit --exit-code-from e2e-consumer e2e-consumer postgres-test missions jwks-mock`).
+3. Inspecting `test-results/report.csv` + the Skippable test reasons.
+4. Surfacing any blocking failure to the user via the test-run-skill's BLOCKING-gate protocol.
+5. Optionally enabling the Docker-CLI Skippable subset via a one-time consumer-image upgrade (`docker-cli` install + socket bind) before the next cycle.
+
+The performance suite is intentionally NOT part of the default gate — it runs via `scripts/run-performance-tests.sh` only.
+
+## Outstanding follow-ups (NOT blocking Step 7)
+
+1. **Docker-CLI inside e2e-consumer image** — needed to activate the 12 Skippable methods. Recommend a separate ticket sized 3 SP (Dockerfile add of `docker-cli` package + `docker-compose.test.yml` `/var/run/docker.sock` mount). Validates run-perf script's `/app/` → `/src/` path bug at the same time.
+2. **Test/source compilation separation** — `Azaion.Missions.csproj` Sdk.Web globs pull `tests/**/*.cs`. Recommend `<Compile Remove="tests/**" />` or moving to a `.sln`. Pre-existing project layout drift.
+3. **AC-1.4 carry-forward decision** — see NFT-RES-08 carry-forward. The product team should decide whether the partial unique index OR an application-level guard is the canonical solution; today the test pins the index behaviour.
+4. **AC-4.6 walk-order decision** — see NFT-RES-02 carry-forward. The waypoint cascade walks dependency tables in a different order than the spec implied; the team should reconcile spec and code.
+
+## Sign-off
+
+Cycle 1 test implementation complete. 4 batches, 11 tasks, 47 SP. All ACs traced; no blocking findings; tracker tickets transitioned to **In Testing**. Autodev advances to Step 7 (Run Tests).