From 001e80fe965e932308871ca96f069ffa82c0ad63 Mon Sep 17 00:00:00 2001 From: Oleksandr Bezdieniezhnykh Date: Fri, 15 May 2026 09:11:53 +0300 Subject: [PATCH] [AZ-585] [AZ-586] ResLim+Perf NFT tests; close test cycle 1 Batch 4 of test implementation cycle 1 (existing-code Step 6, final batch). - AZ-585 SteadyStateLoadTests + ColdStartRssTests: NFT-RES-LIM-01..04. SteadyStateLoadFixture runs one 5-min sustained-load window and samples RSS (docker stats), Npgsql conns (pg_stat_activity), and FDs (/proc/1/fd) every 5s; three test methods assert independently. All SkippableFact-gated on docker primitives. - AZ-586 PerformanceTests: NFT-PERF-01..04. Sequential single-client, 5 warm-ups + N measured calls, P50+P95 via LatencyPercentiles, recorded to PERF_RESULTS_FILE. Tagged Category=Perf so default gate excludes them. Infrastructure: - entrypoint.sh now applies --filter "${TEST_FILTER:-Category!=Perf}" per AZ-586 (default CI gate excludes performance). - MetricCsvRecorder: idempotent CSV appender keyed on env var, used by both Perf and ResLim categories. Step 6 (Implement Tests) is complete. Final report at _docs/03_implementation/implementation_report_tests.md handoffs the full-suite gate to test-run/SKILL.md (Step 7). Co-authored-by: Cursor --- _docs/03_implementation/batch_04_report.md | 80 ++++ .../implementation_report_tests.md | 119 ++++++ _docs/_autodev_state.md | 12 +- .../AZ-585_test_resource_limits.md | 0 .../{todo => done}/AZ-586_test_performance.md | 0 .../Fixtures/SteadyStateLoadFixture.cs | 206 ++++++++++ .../Helpers/LatencyPercentiles.cs | 38 ++ .../Helpers/MetricCsvRecorder.cs | 58 +++ .../Tests/Performance/PerformanceTests.cs | 380 ++++++++++++++++++ .../Tests/Performance/Sanity.cs | 23 -- .../Tests/ResourceLimits/ColdStartRssTests.cs | 131 ++++++ .../Tests/ResourceLimits/Sanity.cs | 23 -- .../ResourceLimits/SteadyStateLoadTests.cs | 156 +++++++ tests/Azaion.Missions.E2E.Tests/entrypoint.sh | 7 + 14 files changed, 1181 insertions(+), 52 deletions(-) create mode 100644 _docs/03_implementation/batch_04_report.md create mode 100644 _docs/03_implementation/implementation_report_tests.md rename _docs/tasks/{todo => done}/AZ-585_test_resource_limits.md (100%) rename _docs/tasks/{todo => done}/AZ-586_test_performance.md (100%) create mode 100644 tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs create mode 100644 tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs create mode 100644 tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs create mode 100644 tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs delete mode 100644 tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs create mode 100644 tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs delete mode 100644 tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs create mode 100644 tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs diff --git a/_docs/03_implementation/batch_04_report.md b/_docs/03_implementation/batch_04_report.md new file mode 100644 index 0000000..6155887 --- /dev/null +++ b/_docs/03_implementation/batch_04_report.md @@ -0,0 +1,80 @@ +# Batch Report + +**Batch**: 4 +**Tasks**: AZ-585, AZ-586 +**Date**: 2026-05-15 +**Run mode**: Test implementation (existing-code Step 6) +**Total complexity**: 6 SP (3 + 3) + +## Task Results + +| Task | Status | Files Modified | Tests | AC Coverage | Issues | +|------|--------|----------------|-------|-------------|--------| +| AZ-585_test_resource_limits | Done | 3 added, 1 deleted | 4 / 4 discovery | 4/4 NFT-RES-LIM covered | 0 | +| AZ-586_test_performance | Done | 1 added, 1 deleted, 2 helpers added, entrypoint.sh modified | 4 / 4 discovery | 4/4 NFT-PERF covered | 0 | + +## AC Test Coverage: All 8 NFT scenarios covered + +- **AZ-585 (4/4)**: NFT-RES-LIM-01 → `SteadyStateLoadTests.NFT_RES_LIM_01_*` (P95 RSS + no-leak ratio), NFT-RES-LIM-02 → `SteadyStateLoadTests.NFT_RES_LIM_02_*` (Npgsql conn cap + minute-1 mean), NFT-RES-LIM-03 → `SteadyStateLoadTests.NFT_RES_LIM_03_*` (FD cap + minute-1 anchor), NFT-RES-LIM-04 → `ColdStartRssTests.NFT_RES_LIM_04_*` (30s settle + cold-RSS cap). +- **AZ-586 (4/4)**: NFT-PERF-01 → `PerformanceTests.NFT_PERF_01_*` (100 minimal-cascade DELETEs, P50 ≤ 50ms), NFT-PERF-02 → `*.NFT_PERF_02_*` (50 F3-shape cascade DELETEs, provisional P50 ≤ 200ms), NFT-PERF-03 → `*.NFT_PERF_03_*` (100 `/health`, P50 ≤ 10ms), NFT-PERF-04 → `*.NFT_PERF_04_*` (100 paginated lists vs 1000-mission seed, provisional P95 ≤ 100ms). + +## Code Review Verdict: PASS_WITH_WARNINGS (self-review) + +- 0 Critical, 0 High, 0 Medium. +- **Low — coverage**: 4 of 4 ResLim tests are `SkippableFact` gated on `COMPOSE_RESTART_ENABLED=1` + docker CLI in the e2e-consumer image — same Docker-socket follow-up already flagged in batch 3 report. NFT-RES-LIM-04 additionally requires `docker compose stop|rm|up` access; same gate. +- **Low — maintainability**: `SteadyStateLoadFixture.ParseHumanBytes` and `ColdStartRssTests.ParseHumanBytes` are duplicated. Both files parse the LHS of `docker stats --no-stream --format '{{.MemUsage}}'`; the duplication is intentional today because the two files have different gating predicates (fixture uses `Enabled` property + `CommandAvailable` probe, ColdStart uses `MissionsContainerHelper.Enabled`), and lifting the helper to `Helpers/HumanBytes.cs` would be a shared-helper change worth a separate refactor. Captured as a follow-up note; not auto-fixed because it touches both files. **Recommend folding into the docker-CLI follow-up task.** +- **Low — observability**: `PerformanceTests` swallows non-2xx-non-404 with `InvalidOperationException` (warmup + measured), so a misbehaving SUT mid-run yields a clear stack trace; no silent pass. This is intended. + +## Auto-Fix Attempts: 1 + +`SteadyStateLoadFixture.cs:59` initially called `new TokenMinter()` (parameter-less ctor); `TokenMinter` requires `signUrl`. Fixed to `new TokenMinter(TestEnvironment.JwksMockBaseUrl + "/sign")` — same pattern as `TestBase`. Style/Low under the Auto-Fix Gate matrix. Rebuild: 0 warnings, 0 errors. + +## Stuck Agents: None + +## Files Created (5) + 2 deletions + 1 modified script + +### Helpers (2) + +- `tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs` — nearest-rank P50/P95/Percentile/Mean over `IReadOnlyList`. Sorts a defensive copy. +- `tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs` — appends one row per scenario (Timestamp, Category, Scenario, Result, Traces, ErrorMessage) to a CSV referenced by `PERF_RESULTS_FILE` (perf) or `RESLIM_RESULTS_FILE` (reslim). No-op when the env var is unset. + +### Fixtures (1) + +- `tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs` — class-scoped 5-minute sustained-load fixture. Generates ~50 RPS via a single-threaded `HttpClient` loop, samples RSS / Npgsql conn count / FD count every 5s. Exposes the time series + `LoadGeneratorMetTargetRps` + `SutExitedDuringWindow` + `SkipReason`. Tests inspect `SkipReason` to surface explicit skips when docker primitives are unavailable. + +### Test classes (3) + +- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs` — NFT-RES-LIM-01..03 share the fixture window. Each test asserts one metric independently. `[Collection("ResLimSteadyState")]`. +- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs` — NFT-RES-LIM-04. Runs `docker compose stop|rm|up missions` for a fresh start, waits 30s after `/health` returns 200, reads RSS, asserts ≤ 200 MiB. Lives in the `MigratorRestart` collection to serialise with the other compose-restarting tests. +- `tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs` — NFT-PERF-01..04, all `[Trait("Category","Perf")]`. Sequential single-client, 5 warm-ups + N measured, records P50 + P95 to `PERF_RESULTS_FILE`. + +### Deleted (2 Sanity placeholders) + +- `tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs` — dead placeholder from AZ-576; replaced by `PerformanceTests`. +- `tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs` — same. + +### Modified (entrypoint filter, per AZ-586 Spec) + +- `tests/Azaion.Missions.E2E.Tests/entrypoint.sh` — added `--filter "${TEST_FILTER:-Category!=Perf}"`. The default CI gate now excludes the Performance category (AZ-586 Spec § Outcome: "default test suite filter excludes performance to keep the standard CI gate ≤ 15 min"); `scripts/run-performance-tests.sh` bypasses the entrypoint anyway and invokes `dotnet test --filter Category=Perf` directly. The shell variable `TEST_FILTER` is overridable for ad-hoc invocations (e.g., to include Perf during a local profiling session). + +## Local Verification + +- `dotnet build tests/Azaion.Missions.E2E.Tests/Azaion.Missions.E2E.Tests.csproj` — 0 warnings, 0 errors. +- 8 new NFT methods discoverable via `[Trait("Category","Perf")]` (4) and `[Trait("Category","ResLim")]` (4). + +## Pre-existing issues NOT in scope + +- `scripts/run-performance-tests.sh` line 104 references `/app/Azaion.Missions.E2E.Tests.csproj`, but the Dockerfile copies the test project to `/src/`. Pre-existing script bug — flag for the docker-CLI follow-up task that re-validates the run-perf script end-to-end. Not introduced by this batch. +- Root `Azaion.Missions.csproj` Sdk.Web globs still pull `tests/**/*.cs` into the main project compilation — same flag as batch 3 cumulative review report; pre-existing. + +## Docker Stack Validation + +Not run as part of this batch — same hand-off as batches 1-3. Step 7 (`test-run/SKILL.md`) owns the `docker compose -f docker-compose.test.yml up --build --abort-on-container-exit e2e-consumer` gate. The 5 SkippableFact tests in this batch activate when the consumer image has `docker` CLI + socket bind; otherwise they emit explicit skip reasons (no silent pass). + +## Tracker Updates + +AZ-585, AZ-586 transitioned to `In Testing` via the Atlassian MCP after this commit (Step 12). + +## Next Batch + +All 11 test tasks (AZ-576 + AZ-577..AZ-586) are now done. Step 6 (Implement Tests) is **complete**. Autodev advances to Step 7 (Run Tests) — `test-run/SKILL.md` owns the full-suite gate. diff --git a/_docs/03_implementation/implementation_report_tests.md b/_docs/03_implementation/implementation_report_tests.md new file mode 100644 index 0000000..81c8081 --- /dev/null +++ b/_docs/03_implementation/implementation_report_tests.md @@ -0,0 +1,119 @@ +# Test Implementation Final Report + +**Run**: existing-code Step 6 (Implement Tests) +**Date**: 2026-05-15 +**Cycle**: 1 +**Verdict**: HANDOFF — full-suite gate owned by `.cursor/skills/test-run/SKILL.md` (Step 7) + +## Scope + +11 test tasks decomposed by `/decompose-tests` and tracked under epic **AZ-575**: + +| Task | Description | SP | Batch | +|---------|----------------------------------------------------------|----|-------| +| AZ-576 | Test infrastructure (compose, csproj, mocks, helpers) | 5 | 1 | +| AZ-577 | Vehicles positive (FT-P-01..06) | 5 | 2 | +| AZ-578 | Missions positive (FT-P-07..12) | 5 | 2 | +| AZ-579 | Waypoints + health positive (FT-P-13..18) | 5 | 2 | +| AZ-580 | Validation + authz negative (FT-N-01..08) | 3 | 2 | +| AZ-581 | Security auth/claims (NFT-SEC-01..06+04b) | 5 | 3 | +| AZ-582 | Security alg/rotation/CORS (NFT-SEC-07..13) | 5 | 3 | +| AZ-583 | Resilience cascade + migrator (NFT-RES-01..04) | 3 | 3 | +| AZ-584 | Resilience config/DB/rotation/race (NFT-RES-05..08) | 5 | 3 | +| AZ-585 | Resource limits (NFT-RES-LIM-01..04) | 3 | 4 | +| AZ-586 | Performance (NFT-PERF-01..04) | 3 | 4 | +| **Total** | | **47** | | + +## Results + +| Batch | Tasks | SP | Verdict | Carry-forwards | +|-------|----------------------------------|----|----------------------|----------------| +| 1 | AZ-576 | 5 | PASS_WITH_WARNINGS | 0 | +| 2 | AZ-577..AZ-580 | 18 | PASS_WITH_WARNINGS | 3 | +| 3 | AZ-581..AZ-584 | 18 | PASS_WITH_WARNINGS | 3 | +| 4 | AZ-585, AZ-586 | 6 | PASS_WITH_WARNINGS | 0 | + +**Cumulative reviews**: 1 (`cumulative_review_batches_01-03_cycle1_report.md`, PASS_WITH_WARNINGS, 4 Low findings). + +## AC Test Coverage + +| Source | ACs | Tests | Coverage | +|--------|-----|-------|----------| +| FT-P (functional positive) | 18 | 18 | 18/18 | +| FT-N (negative) | 8 | 8 | 8/8 | +| NFT-SEC (security) | 14 | 22 | 14/14 (some scenarios → multiple `Theory` rows) | +| NFT-RES (resilience) | 8 | 12 | 8/8 | +| NFT-RES-LIM (resource lim) | 4 | 4 | 4/4 | +| NFT-PERF (performance) | 4 | 4 | 4/4 | +| **Total** | **56** | **68** | **56/56** | + +Every AC has at least one trace via `[Trait("Traces", "AC-X.Y")]`; structural carry-forwards (6 total) are pinned with `[Trait("carry_forward", "...")]` so `dotnet test --filter "carry_forward~..."` surfaces them as a set when the underlying spec/code reconciliation lands. + +## Spec-vs-Code Carry-forwards (6 total) + +| Site | Spec says | Code says | Carry-forward tag | +|-----------------------------------------|----------------------------------------------|------------------------------------------------------------|-------------------------------| +| FT-P-03 `Vehicles/PositiveTests.cs` | `POST /vehicles/{id}/setDefault` → 200 + body| `[HttpPatch("{id:guid}/default")]` → 204 NoContent | `AC-1.4/route-shape` | +| FT-P-14/15 `Waypoints/PositiveTests.cs` | Nested `GeoPoint:{Lat,Lon,Mgrs}` | LinqToDB entity flat `Lat`/`Lon`/`Mgrs` | `flat-waypoint-shape` | +| FT-N-07 `Waypoints/NegativeTests.cs` | Missing parent → 404 + problem envelope | `GetWaypoints` returns `[]` | `AC-4.2/missing-parent-soft` | +| NFT-RES-01 `Resilience/CascadeF3Tests.cs` | Mid-walk cascade is transactional | `MissionService.DeleteMission` is non-transactional | `ADR-006` | +| NFT-RES-02 `Resilience/CascadeF4Tests.cs` | Waypoint cascade leaves detection=0/waypoint=1 partial state | `WaypointService.DeleteWaypoint` queries `media` BEFORE any deletion — aborts at step 1 with nothing deleted | `AC-4.6/walk-order` | +| NFT-RES-08 `Resilience/DefaultVehicleRaceTests.cs` | TOCTOU race observable | `ux_vehicles_one_default` partial unique index closes the race | `AC-1.4/index-closes-race` | + +These carry-forwards flip the moment the spec or the code is reconciled; the tests fail loudly at that point — intentional. + +## Code Review Summary + +- **0 Critical / 0 High / 0 Medium** across all four batches. +- **4 Low findings** captured in cumulative review (3 follow-up + 1 baseline-carried) — see `_docs/03_implementation/cumulative_review_batches_01-03_cycle1_report.md`. +- Auto-fix rounds across the cycle: batch 2 (89× xUnit1030 warnings), batch 3 (3× missing-using errors), batch 4 (1× TokenMinter parameter-less ctor). All auto-fix-eligible per the Auto-Fix Gate matrix; no escalations. + +## Files Added (high level) + +- **Helpers** (10): `ApiDtos`, `DbAssertions`, `DockerLogs`, `FixtureSql`, `ForeignKeypair`, `HttpAssertions`, `LatencyPercentiles`, `MetricCsvRecorder`, `MissionsContainerHelper` — plus the existing `TestEnvironment`. +- **Fixtures** (9): `CascadeF3Fixture`, `CascadeF4Fixture`, `ComposeRestartFixture`, `DbResetFixture`, `JwksMockReverseFixture` (spec-only stub), `JwksRotateFixture`, `PostgresStopStartFixture`, `Seeds`, `StubSchema`, `SteadyStateLoadFixture`. +- **Test classes** (24): grouped under `Tests/{Vehicles,Missions,Waypoints,Health,Errors,Security,Resilience,Performance,ResourceLimits,Reporting}/` per the AZ-576 layout. +- **Infrastructure**: `docker-compose.test.yml` extensions (fixtures volume), `entrypoint.sh` Category-filter, `Reporting/TrxToCsvPostProcessor.cs` (from batch 1). +- **JWKS mock**: extended `SignBody` (permissions_array) + `TokenSigner` (kid_override validation) — required by NFT-SEC-06 and NFT-SEC-11. + +## SkippableFact / SkippableTheory inventory + +| Test | Skip predicate | Reason when skipped | +|------------------------------------------------------------|------------------------------------------------------------------|----------------------| +| `Tests/Health/HealthTests.NFT_P_17` (FT-P-17) | `COMPOSE_RESTART_ENABLED=1` | postgres-test stop/start | +| `Tests/Errors/Error500Tests.NFT_N_08` | `COMPOSE_RESTART_ENABLED=1` | drops vehicles table | +| `Tests/Security/ErrorRedactionTests.NFT_SEC_08` | `COMPOSE_RESTART_ENABLED=1` | drops vehicles table | +| `Tests/Security/StartupConfigTests.NFT_SEC_12` (theory + HTTP-JWKS) | `MissionsContainerHelper.Enabled` | docker run primitives | +| `Tests/Security/CorsConfigTests.NFT_SEC_13` (4 scenarios) | `MissionsContainerHelper.Enabled` | docker run primitives | +| `Tests/Resilience/CascadeF3Tests.NFT_RES_01` | `COMPOSE_RESTART_ENABLED=1` | drops media table | +| `Tests/Resilience/CascadeF4Tests.NFT_RES_02` | `COMPOSE_RESTART_ENABLED=1` | drops media table | +| `Tests/Resilience/MigratorRestartTests.NFT_RES_03/04` | `ComposeRestartFixture.Enabled` | docker compose restart | +| `Tests/Resilience/ConfigDbStartupTests.*` (8 methods) | `MissionsContainerHelper.Enabled` | docker run primitives | +| `Tests/Resilience/JwksRotationNoRestartTests.NFT_RES_07` | `MissionsContainerHelper.Enabled` (for StartedAt read) | docker inspect | +| `Tests/ResourceLimits/SteadyStateLoadTests.*` (3 methods) | `SteadyStateLoadFixture.SkipReason` (set on missing docker) | docker stats / docker exec | +| `Tests/ResourceLimits/ColdStartRssTests.NFT_RES_LIM_04` | `COMPOSE_RESTART_ENABLED=1` + `MissionsContainerHelper.Enabled` | docker compose stop/start | + +Every Skippable test surfaces an explicit reason; none silent-pass. + +## Handoff to Step 7 (Run Tests) + +This report is a **HANDOFF** — the full-suite gate is owned by `.cursor/skills/test-run/SKILL.md`. That skill is responsible for: + +1. Building the docker compose stack (`docker compose -f docker-compose.test.yml --profile test build`). +2. Running the e2e-consumer (`docker compose ... up --abort-on-container-exit --exit-code-from e2e-consumer e2e-consumer postgres-test missions jwks-mock`). +3. Inspecting `test-results/report.csv` + the Skippable test reasons. +4. Surfacing any blocking failure to the user via the test-run-skill's BLOCKING-gate protocol. +5. Optionally enabling the Docker-CLI Skippable subset via a one-time consumer-image upgrade (`docker-cli` install + socket bind) before the next cycle. + +The performance suite is intentionally NOT part of the default gate — it runs via `scripts/run-performance-tests.sh` only. + +## Outstanding follow-ups (NOT blocking Step 7) + +1. **Docker-CLI inside e2e-consumer image** — needed to activate the 12 Skippable methods. Recommend a separate ticket sized 3 SP (Dockerfile add of `docker-cli` package + `docker-compose.test.yml` `/var/run/docker.sock` mount). Validates run-perf script's `/app/` → `/src/` path bug at the same time. +2. **Test/source compilation separation** — `Azaion.Missions.csproj` Sdk.Web globs pull `tests/**/*.cs`. Recommend `` or moving to a `.sln`. Pre-existing project layout drift. +3. **AC-1.4 carry-forward decision** — see NFT-RES-08 carry-forward. The product team should decide whether the partial unique index OR an application-level guard is the canonical solution; today the test pins the index behaviour. +4. **AC-4.6 walk-order decision** — see NFT-RES-02 carry-forward. The waypoint cascade walks dependency tables in a different order than the spec implied; the team should reconcile spec and code. + +## Sign-off + +Cycle 1 test implementation complete. 4 batches, 11 tasks, 47 SP. All ACs traced; no blocking findings; tracker tickets transitioned to **In Testing**. Autodev advances to Step 7 (Run Tests). diff --git a/_docs/_autodev_state.md b/_docs/_autodev_state.md index 4986c73..2f9f675 100644 --- a/_docs/_autodev_state.md +++ b/_docs/_autodev_state.md @@ -2,13 +2,13 @@ ## Current Step flow: existing-code -step: 6 -name: Implement Tests -status: in_progress +step: 7 +name: Run Tests +status: not_started sub_step: - phase: 16 - name: cumulative-review - detail: "batches 1-3 covered; batch 4 (AZ-585..AZ-586) pending" + phase: 0 + name: awaiting-invocation + detail: "" retry_count: 0 cycle: 1 tracker: jira diff --git a/_docs/tasks/todo/AZ-585_test_resource_limits.md b/_docs/tasks/done/AZ-585_test_resource_limits.md similarity index 100% rename from _docs/tasks/todo/AZ-585_test_resource_limits.md rename to _docs/tasks/done/AZ-585_test_resource_limits.md diff --git a/_docs/tasks/todo/AZ-586_test_performance.md b/_docs/tasks/done/AZ-586_test_performance.md similarity index 100% rename from _docs/tasks/todo/AZ-586_test_performance.md rename to _docs/tasks/done/AZ-586_test_performance.md diff --git a/tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs b/tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs new file mode 100644 index 0000000..b27d50d --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Fixtures/SteadyStateLoadFixture.cs @@ -0,0 +1,206 @@ +using System.Diagnostics; +using System.Globalization; +using System.Net.Http.Headers; +using Azaion.Missions.E2E.Helpers; +using Npgsql; +using Xunit; + +namespace Azaion.Missions.E2E.Fixtures; + +/// +/// Shared 5-minute steady-state load fixture for NFT-RES-LIM-01 / -02 / -03. +/// Runs the load generator once, samples RSS / connection count / FD count +/// every 5s, and exposes the time series + sentinel "did the SUT exit" flag. +/// +/// +/// The fixture is class-scoped (xUnit ) +/// and shared across all three NFT-RES-LIM-01/02/03 tests so the 5-minute +/// window runs once per CI invocation. +/// Disabled when COMPOSE_RESTART_ENABLED != 1 or docker CLI +/// is missing. Disabled state is observable via ; tests +/// must call at the top of the +/// method body — initialising the fixture without docker would throw inside +/// and surface as a hard failure instead of +/// the explicit skip the spec requires. +/// +public sealed class SteadyStateLoadFixture : IAsyncLifetime +{ + public const int SampleIntervalSeconds = 5; + public const int LoadDurationSeconds = 300; + public const int TargetRps = 50; + public const string ContainerName = "missions-sut"; + + public bool Enabled => + Environment.GetEnvironmentVariable("COMPOSE_RESTART_ENABLED") == "1"; + + public bool LoadGeneratorMetTargetRps { get; private set; } + public bool SutExitedDuringWindow { get; private set; } + public string? SkipReason { get; private set; } + + public List RssBytesSamples { get; } = new(); + public List NpgsqlConnectionSamples { get; } = new(); + public List FileDescriptorSamples { get; } = new(); + public List SampleTimestamps { get; } = new(); + + public async Task InitializeAsync() + { + if (!Enabled) + { + SkipReason = "COMPOSE_RESTART_ENABLED!=1 — docker CLI primitives unavailable"; + return; + } + if (!CommandAvailable("docker")) + { + SkipReason = "docker CLI not on PATH in this consumer image"; + return; + } + + using var http = new HttpClient { BaseAddress = new Uri(TestEnvironment.MissionsBaseUrl) }; + var token = await new TokenMinter(TestEnvironment.JwksMockBaseUrl + "/sign").MintDefaultAsync(); + http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.Jwt); + + var cancel = new CancellationTokenSource(); + var endpoints = new[] { "/vehicles", "/missions", "/missions?page=1&pageSize=20" }; + long requestsSent = 0; + + var loadTask = Task.Run(async () => + { + var ix = 0; + while (!cancel.IsCancellationRequested) + { + try + { + var url = endpoints[ix++ % endpoints.Length]; + using var resp = await http.GetAsync(url, cancel.Token); + Interlocked.Increment(ref requestsSent); + } + catch (HttpRequestException) { /* surfaces via SutExitedDuringWindow below */ } + catch (OperationCanceledException) { return; } + } + }, cancel.Token); + + var samplingDeadline = DateTime.UtcNow.AddSeconds(LoadDurationSeconds); + while (DateTime.UtcNow < samplingDeadline) + { + await Task.Delay(TimeSpan.FromSeconds(SampleIntervalSeconds), cancel.Token); + if (!ContainerIsRunning(ContainerName)) + { + SutExitedDuringWindow = true; + break; + } + SampleTimestamps.Add(DateTime.UtcNow); + RssBytesSamples.Add(ReadRssBytes(ContainerName)); + NpgsqlConnectionSamples.Add(ReadNpgsqlConnectionCount()); + FileDescriptorSamples.Add(ReadFileDescriptorCount(ContainerName)); + } + + cancel.Cancel(); + try { await loadTask; } catch (OperationCanceledException) { } + + // Sustained 50 RPS over 300s = 15000 requests; allow 10% slack for + // CI variance / connection-refused retries. + LoadGeneratorMetTargetRps = + requestsSent >= (long)(TargetRps * LoadDurationSeconds * 0.9); + } + + public Task DisposeAsync() => Task.CompletedTask; + + private static long ReadRssBytes(string containerName) + { + // `docker stats --no-stream --format '{{.MemUsage}}'` prints e.g. + // "187.4MiB / 7.7GiB". We need the LHS in bytes. + var raw = Run("docker", + $"stats --no-stream --format '{{{{.MemUsage}}}}' {containerName}"); + var lhs = raw.Split('/')[0].Trim().Trim('\''); + return ParseHumanBytes(lhs); + } + + private static int ReadNpgsqlConnectionCount() + { + using var conn = new NpgsqlConnection(TestEnvironment.DbSideChannel); + conn.Open(); + using var cmd = conn.CreateCommand(); + cmd.CommandText = """ + SELECT count(*)::INTEGER FROM pg_stat_activity + WHERE application_name LIKE 'Npgsql%' + OR (usename = 'postgres' AND backend_type = 'client backend'); + """; + return Convert.ToInt32(cmd.ExecuteScalar()); + } + + private static int ReadFileDescriptorCount(string containerName) + { + // `pgrep` is not guaranteed in the runtime image; we walk /proc + // directly. `/proc/1/comm` is the entrypoint process name; for the + // ASP.NET Core SDK image this is `dotnet`. + var stdout = Run("docker", + $"exec {containerName} sh -c 'ls /proc/1/fd | wc -l'"); + return int.Parse(stdout.Trim(), CultureInfo.InvariantCulture); + } + + private static long ParseHumanBytes(string text) + { + // "187.4MiB" / "1.2GiB" / "234KiB" / "987B" + var unitIx = text.IndexOfAny(new[] { 'K', 'M', 'G', 'T', 'B' }); + if (unitIx < 0) return long.Parse(text, CultureInfo.InvariantCulture); + var num = double.Parse(text.Substring(0, unitIx), CultureInfo.InvariantCulture); + var unit = text.Substring(unitIx); + return unit switch + { + "B" => (long)num, + "KiB" or "KB" or "K" => (long)(num * 1024), + "MiB" or "MB" or "M" => (long)(num * 1024 * 1024), + "GiB" or "GB" or "G" => (long)(num * 1024 * 1024 * 1024), + "TiB" or "TB" or "T" => (long)(num * 1024L * 1024 * 1024 * 1024), + _ => throw new FormatException($"unknown human-bytes unit in '{text}'") + }; + } + + private static bool ContainerIsRunning(string containerName) + { + try + { + var stdout = Run("docker", + $"inspect --format '{{{{.State.Running}}}}' {containerName}"); + return stdout.Trim().Trim('\'').Equals("true", StringComparison.Ordinal); + } + catch (InvalidOperationException) { return false; } + } + + private static bool CommandAvailable(string command) + { + try + { + var psi = new ProcessStartInfo(command, "--version") + { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + using var p = Process.Start(psi); + if (p is null) return false; + p.WaitForExit(); + return p.ExitCode == 0; + } + catch (System.ComponentModel.Win32Exception) { return false; } + } + + private static string Run(string file, string args) + { + var psi = new ProcessStartInfo(file, args) + { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + using var p = Process.Start(psi) + ?? throw new InvalidOperationException($"failed to launch `{file} {args}`"); + var stdout = p.StandardOutput.ReadToEnd(); + var stderr = p.StandardError.ReadToEnd(); + p.WaitForExit(); + if (p.ExitCode != 0) + throw new InvalidOperationException( + $"`{file} {args}` exited {p.ExitCode}: {stderr}"); + return stdout; + } +} diff --git a/tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs b/tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs new file mode 100644 index 0000000..3fe1fba --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Helpers/LatencyPercentiles.cs @@ -0,0 +1,38 @@ +namespace Azaion.Missions.E2E.Helpers; + +/// +/// Median + percentile helper for the NFT-PERF-* and NFT-RES-LIM-01 +/// scenarios. Inputs are wall-clock latency samples (or RSS samples) +/// in any orderable numeric type; the helper sorts a defensive copy +/// and uses the "nearest-rank" definition of percentile (matching the +/// percentile defaults used in `docker stats` and most CI dashboards). +/// +public static class LatencyPercentiles +{ + public static double P50(IReadOnlyList samples) => Percentile(samples, 50); + public static double P95(IReadOnlyList samples) => Percentile(samples, 95); + + public static double Percentile(IReadOnlyList samples, int percentile) + { + if (samples.Count == 0) + throw new ArgumentException("samples must contain at least one value", nameof(samples)); + if (percentile < 0 || percentile > 100) + throw new ArgumentOutOfRangeException(nameof(percentile), "percentile must be in [0, 100]"); + + var sorted = samples.ToArray(); + Array.Sort(sorted); + + // Nearest-rank: rank = ceil(p/100 * N); index = rank - 1. + var rank = (int)Math.Ceiling(percentile / 100.0 * sorted.Length); + if (rank < 1) rank = 1; + if (rank > sorted.Length) rank = sorted.Length; + return sorted[rank - 1]; + } + + public static double Mean(IReadOnlyList samples) + { + if (samples.Count == 0) + throw new ArgumentException("samples must contain at least one value", nameof(samples)); + return samples.Average(); + } +} diff --git a/tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs b/tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs new file mode 100644 index 0000000..2545d40 --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Helpers/MetricCsvRecorder.cs @@ -0,0 +1,58 @@ +using System.Globalization; + +namespace Azaion.Missions.E2E.Helpers; + +/// +/// Appends one row per NFT-PERF / NFT-RES-LIM scenario to a side-channel +/// CSV referenced by an environment variable. The Reporting.Cli converter +/// only knows about compile-time [Trait] data — runtime measurements +/// (P50/P95, MAX_FD, P95_RSS_MiB, etc.) need this separate file so +/// deployment planning + trend dashboards can read them. +/// +/// +/// File schema (idempotent header written on first append): +/// Timestamp,Category,Scenario,Result,Traces,ErrorMessage +/// The Traces column carries the dynamic key=value pairs the spec requires +/// (e.g., "AC-3.6; P50_MS=23.4; P95_MS=41.8"); the recorder just +/// joins them with semicolons — callers compose the right shape. +/// +public sealed class MetricCsvRecorder +{ + private readonly string? _path; + private static readonly object Lock = new(); + + /// name of the env var that carries the target CSV path + /// (e.g., PERF_RESULTS_FILE for NFT-PERF, RESLIM_RESULTS_FILE + /// for NFT-RES-LIM). When the env var is missing or whitespace, every + /// call is a no-op — the recorder is intentionally + /// silent inside the standard CI run. + public MetricCsvRecorder(string envVar) + { + var v = Environment.GetEnvironmentVariable(envVar); + _path = string.IsNullOrWhiteSpace(v) ? null : v; + } + + public bool IsEnabled => _path is not null; + + public void Record(string category, string scenario, string result, string traces, string? errorMessage = null) + { + if (_path is null) return; + lock (Lock) + { + var dir = Path.GetDirectoryName(_path); + if (!string.IsNullOrEmpty(dir)) Directory.CreateDirectory(dir); + var newFile = !File.Exists(_path); + using var sw = new StreamWriter(_path, append: true); + if (newFile) + sw.WriteLine("Timestamp,Category,Scenario,Result,Traces,ErrorMessage"); + sw.WriteLine( + $"{DateTime.UtcNow.ToString("O", CultureInfo.InvariantCulture)}," + + $"{Csv(category)},{Csv(scenario)},{Csv(result)},{Csv(traces)},{Csv(errorMessage ?? "")}"); + } + } + + private static string Csv(string value) => + value.Contains(',') || value.Contains('"') || value.Contains('\n') + ? "\"" + value.Replace("\"", "\"\"", StringComparison.Ordinal) + "\"" + : value; +} diff --git a/tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs b/tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs new file mode 100644 index 0000000..21ae029 --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Tests/Performance/PerformanceTests.cs @@ -0,0 +1,380 @@ +using System.Diagnostics; +using System.Globalization; +using System.Net.Http.Headers; +using Azaion.Missions.E2E.Fixtures; +using Azaion.Missions.E2E.Helpers; +using Npgsql; +using Xunit; + +namespace Azaion.Missions.E2E.Tests.Performance; + +/// +/// NFT-PERF-01..04 — wall-clock latency observations against the dockerised +/// missions service. Excluded from the default CI gate via +/// --filter "Category!=Perf" in entrypoint.sh; run via +/// scripts/run-performance-tests.sh. +/// +/// +/// Each scenario follows the same shape: seed deterministic data, warm-up +/// 5 calls (excluded from the percentile), run N measured sequential calls +/// recording wall-clock, compute P50 + P95, record +/// them to the runtime CSV referenced by PERF_RESULTS_FILE, then +/// assert against the documented gate. Sequential single-client execution +/// keeps HTTP/1.1 connection-reuse and JIT warm-up deterministic. +/// +[Collection("Perf")] +[Trait("Category", "Perf")] +public sealed class PerformanceTests : TestBase, IClassFixture +{ + private static readonly MetricCsvRecorder Csv = new("PERF_RESULTS_FILE"); + private const int WarmupCalls = 5; + + [Fact(Timeout = 60_000)] + [Trait("Traces", "AC-3.6")] + [Trait("max_ms", "30000")] + public async Task NFT_PERF_01_minimal_cascade_delete_p50_within_50ms() + { + // Arrange — 105 missions (100 measured + 5 warmup), each with one waypoint. + DbResetFixture.ResetDatabase(TestEnvironment.DbSideChannel); + Seeds.Apply(Seeds.OneDefaultVehicle.Sql); + var (measured, warmup) = SeedSequentialMissions(105, waypointsPerMission: 1); + await AttachAuthAsync(); + + await WarmupDeletesAsync(warmup); + + // Act + var latenciesMs = await MeasureSequentialDeletesAsync(measured); + var p50 = LatencyPercentiles.P50(latenciesMs); + var p95 = LatencyPercentiles.P95(latenciesMs); + + Csv.Record( + category: "Perf", + scenario: "NFT-PERF-01", + result: p50 <= 50.0 ? "pass" : "fail", + traces: $"AC-3.6; P50_MS={p50.ToString("F2", CultureInfo.InvariantCulture)}; " + + $"P95_MS={p95.ToString("F2", CultureInfo.InvariantCulture)}"); + + // Assert + Assert.True(p50 <= 50.0, + $"NFT-PERF-01 P50 budget exceeded: P50={p50:F2}ms (gate=50ms), P95={p95:F2}ms"); + } + + [Fact(Timeout = 120_000)] + [Trait("Traces", "AC-3.1,AC-3.6")] + [Trait("max_ms", "60000")] + [Trait("provisional", "yes")] + public async Task NFT_PERF_02_full_chain_cascade_delete_p50_within_200ms_provisional() + { + // PROVISIONAL — lock at measured + 50% on first green run. + // Arrange — 55 F3-shaped missions (50 measured + 5 warmup). + DbResetFixture.ResetDatabase(TestEnvironment.DbSideChannel); + Seeds.Apply(Seeds.OneDefaultVehicle.Sql); + StubSchema.EnsureCreated(); + var (measured, warmup) = SeedF3MissionsCascadeChains(55); + await AttachAuthAsync(); + + await WarmupDeletesAsync(warmup); + + // Act + var latenciesMs = await MeasureSequentialDeletesAsync(measured); + var p50 = LatencyPercentiles.P50(latenciesMs); + var p95 = LatencyPercentiles.P95(latenciesMs); + + Csv.Record( + category: "Perf", + scenario: "NFT-PERF-02", + result: p50 <= 200.0 ? "pass" : "fail", + traces: $"AC-3.1; P50_MS={p50.ToString("F2", CultureInfo.InvariantCulture)}; " + + $"P95_MS={p95.ToString("F2", CultureInfo.InvariantCulture)}"); + + // Assert + Assert.True(p50 <= 200.0, + $"NFT-PERF-02 P50 (provisional 200ms) exceeded: P50={p50:F2}ms, P95={p95:F2}ms"); + } + + [Fact(Timeout = 30_000)] + [Trait("Traces", "AC-7.3")] + [Trait("max_ms", "5000")] + public async Task NFT_PERF_03_health_p50_within_10ms() + { + // Arrange — no seed needed; /health is anonymous. + for (int i = 0; i < WarmupCalls; i++) + { + using var resp = await Missions.GetAsync("/health"); + resp.EnsureSuccessStatusCode(); + } + + // Act + var latenciesMs = new List(100); + for (int i = 0; i < 100; i++) + { + var sw = Stopwatch.StartNew(); + using var resp = await Missions.GetAsync("/health"); + sw.Stop(); + resp.EnsureSuccessStatusCode(); + latenciesMs.Add(sw.Elapsed.TotalMilliseconds); + } + var p50 = LatencyPercentiles.P50(latenciesMs); + var p95 = LatencyPercentiles.P95(latenciesMs); + + Csv.Record( + category: "Perf", + scenario: "NFT-PERF-03", + result: p50 <= 10.0 ? "pass" : "fail", + traces: $"AC-7.3; P50_MS={p50.ToString("F2", CultureInfo.InvariantCulture)}; " + + $"P95_MS={p95.ToString("F2", CultureInfo.InvariantCulture)}"); + + // Assert + Assert.True(p50 <= 10.0, + $"NFT-PERF-03 P50 budget exceeded: P50={p50:F2}ms (gate=10ms), P95={p95:F2}ms"); + } + + [Fact(Timeout = 90_000)] + [Trait("Traces", "AC-2.3")] + [Trait("max_ms", "30000")] + [Trait("provisional", "yes")] + public async Task NFT_PERF_04_missions_list_pagination_p95_within_100ms_provisional() + { + // PROVISIONAL — lock at measured + 50% on first green run. + // Arrange — 1000 missions referencing seed_one_default_vehicle. + DbResetFixture.ResetDatabase(TestEnvironment.DbSideChannel); + Seeds.Apply(Seeds.OneDefaultVehicle.Sql); + SeedSequentialMissionsNoWaypoints(1000); + await AttachAuthAsync(); + + for (int i = 0; i < WarmupCalls; i++) + { + using var resp = await Missions.GetAsync("/missions?page=1&pageSize=20"); + resp.EnsureSuccessStatusCode(); + } + + // Act + var latenciesMs = new List(100); + for (int i = 0; i < 100; i++) + { + var sw = Stopwatch.StartNew(); + using var resp = await Missions.GetAsync("/missions?page=1&pageSize=20"); + sw.Stop(); + resp.EnsureSuccessStatusCode(); + latenciesMs.Add(sw.Elapsed.TotalMilliseconds); + } + var p50 = LatencyPercentiles.P50(latenciesMs); + var p95 = LatencyPercentiles.P95(latenciesMs); + + Csv.Record( + category: "Perf", + scenario: "NFT-PERF-04", + result: p95 <= 100.0 ? "pass" : "fail", + traces: $"AC-2.3; P50_MS={p50.ToString("F2", CultureInfo.InvariantCulture)}; " + + $"P95_MS={p95.ToString("F2", CultureInfo.InvariantCulture)}"); + + // Assert + Assert.True(p95 <= 100.0, + $"NFT-PERF-04 P95 (provisional 100ms) exceeded: P50={p50:F2}ms, P95={p95:F2}ms"); + } + + private async Task AttachAuthAsync() + { + var t = await Tokens.MintDefaultAsync(); + Missions.DefaultRequestHeaders.Authorization = + new AuthenticationHeaderValue("Bearer", t.Jwt); + } + + private async Task WarmupDeletesAsync(IReadOnlyList warmupMissionIds) + { + foreach (var id in warmupMissionIds) + { + using var resp = await Missions.DeleteAsync($"/missions/{id}"); + // 200 or 204 are both acceptable; the cascade walks regardless. + // 4xx would indicate a seed problem — fail loudly. + if (!resp.IsSuccessStatusCode && (int)resp.StatusCode != 404) + throw new InvalidOperationException( + $"warmup DELETE /missions/{id} returned {(int)resp.StatusCode}"); + } + } + + private async Task> MeasureSequentialDeletesAsync(IReadOnlyList missionIds) + { + var latencies = new List(missionIds.Count); + foreach (var id in missionIds) + { + var sw = Stopwatch.StartNew(); + using var resp = await Missions.DeleteAsync($"/missions/{id}"); + sw.Stop(); + if (!resp.IsSuccessStatusCode && (int)resp.StatusCode != 404) + throw new InvalidOperationException( + $"measured DELETE /missions/{id} returned {(int)resp.StatusCode}"); + latencies.Add(sw.Elapsed.TotalMilliseconds); + } + return latencies; + } + + /// + /// Returns (measured, warmup) where the FIRST 5 IDs are the warmup set + /// and the remaining (count-5) IDs are the measured set. Each mission + /// gets the requested number of waypoints with deterministic IDs. + /// + private static (List Measured, List Warmup) SeedSequentialMissions( + int count, int waypointsPerMission) + { + using var conn = new NpgsqlConnection(TestEnvironment.DbSideChannel); + conn.Open(); + using var tx = conn.BeginTransaction(); + + var ids = new List(count); + var seed = new Random(98765); + + using (var insertMission = conn.CreateCommand()) + { + insertMission.Transaction = tx; + insertMission.CommandText = """ + INSERT INTO missions (id, name, vehicle_id) + VALUES (@id, @name, @vehicle_id); + """; + insertMission.Parameters.Add(new NpgsqlParameter("id", NpgsqlTypes.NpgsqlDbType.Uuid)); + insertMission.Parameters.Add(new NpgsqlParameter("name", NpgsqlTypes.NpgsqlDbType.Text)); + insertMission.Parameters.Add(new NpgsqlParameter("vehicle_id", NpgsqlTypes.NpgsqlDbType.Uuid)); + + using var insertWaypoint = conn.CreateCommand(); + insertWaypoint.Transaction = tx; + insertWaypoint.CommandText = """ + INSERT INTO waypoints (id, mission_id, lat, lon, mgrs, order_num) + VALUES (@id, @mission_id, 50.45, 30.52, '36UYA1234567', @order_num); + """; + insertWaypoint.Parameters.Add(new NpgsqlParameter("id", NpgsqlTypes.NpgsqlDbType.Uuid)); + insertWaypoint.Parameters.Add(new NpgsqlParameter("mission_id", NpgsqlTypes.NpgsqlDbType.Uuid)); + insertWaypoint.Parameters.Add(new NpgsqlParameter("order_num", NpgsqlTypes.NpgsqlDbType.Integer)); + + for (int i = 0; i < count; i++) + { + var id = NewDeterministicGuid(seed); + ids.Add(id); + insertMission.Parameters["id"].Value = id; + insertMission.Parameters["name"].Value = $"perf-mission-{i:D4}"; + insertMission.Parameters["vehicle_id"].Value = Seeds.OneDefaultVehicle.Id; + insertMission.ExecuteNonQuery(); + + for (int w = 0; w < waypointsPerMission; w++) + { + insertWaypoint.Parameters["id"].Value = NewDeterministicGuid(seed); + insertWaypoint.Parameters["mission_id"].Value = id; + insertWaypoint.Parameters["order_num"].Value = w; + insertWaypoint.ExecuteNonQuery(); + } + } + } + + tx.Commit(); + var warmup = ids.Take(WarmupCalls).ToList(); + var measured = ids.Skip(WarmupCalls).ToList(); + return (measured, warmup); + } + + private static void SeedSequentialMissionsNoWaypoints(int count) + { + using var conn = new NpgsqlConnection(TestEnvironment.DbSideChannel); + conn.Open(); + using var tx = conn.BeginTransaction(); + using var cmd = conn.CreateCommand(); + cmd.Transaction = tx; + cmd.CommandText = """ + INSERT INTO missions (id, name, vehicle_id) + VALUES (@id, @name, @vehicle_id); + """; + cmd.Parameters.Add(new NpgsqlParameter("id", NpgsqlTypes.NpgsqlDbType.Uuid)); + cmd.Parameters.Add(new NpgsqlParameter("name", NpgsqlTypes.NpgsqlDbType.Text)); + cmd.Parameters.Add(new NpgsqlParameter("vehicle_id", NpgsqlTypes.NpgsqlDbType.Uuid)); + + var seed = new Random(13579); + for (int i = 0; i < count; i++) + { + cmd.Parameters["id"].Value = NewDeterministicGuid(seed); + cmd.Parameters["name"].Value = $"list-perf-{i:D4}"; + cmd.Parameters["vehicle_id"].Value = Seeds.OneDefaultVehicle.Id; + cmd.ExecuteNonQuery(); + } + tx.Commit(); + } + + /// + /// Seeds missions, each with the F3 cascade shape: + /// 3 map_objects + 2 waypoints + (per waypoint: 2 media → 2 annotations → 2 detection). + /// + private static (List Measured, List Warmup) SeedF3MissionsCascadeChains(int count) + { + using var conn = new NpgsqlConnection(TestEnvironment.DbSideChannel); + conn.Open(); + var ids = new List(count); + var seed = new Random(24680); + + for (int i = 0; i < count; i++) + { + using var tx = conn.BeginTransaction(); + var missionId = NewDeterministicGuid(seed); + ids.Add(missionId); + + ExecScalar(conn, tx, """ + INSERT INTO missions (id, name, vehicle_id) VALUES (@id, @name, @vid); + """, ("id", missionId), ("name", $"f3-perf-{i:D4}"), + ("vid", Seeds.OneDefaultVehicle.Id)); + + for (int m = 0; m < 3; m++) + ExecScalar(conn, tx, """ + INSERT INTO map_objects (id, mission_id, h3_index, mgrs) + VALUES (@id, @mid, '8a2a1072b59ffff', '36UYA1234567'); + """, ("id", NewDeterministicGuid(seed)), ("mid", missionId)); + + for (int w = 0; w < 2; w++) + { + var wpId = NewDeterministicGuid(seed); + ExecScalar(conn, tx, """ + INSERT INTO waypoints (id, mission_id, lat, lon, mgrs, order_num) + VALUES (@id, @mid, 50.45, 30.52, '36UYA1234567', @ord); + """, ("id", wpId), ("mid", missionId), ("ord", w)); + + for (int md = 0; md < 2; md++) + { + var mediaId = $"media-{Guid.NewGuid():N}"; + ExecScalar(conn, tx, """ + INSERT INTO media (id, waypoint_id) VALUES (@id, @wid); + """, ("id", mediaId), ("wid", wpId)); + + var annId = $"ann-{Guid.NewGuid():N}"; + ExecScalar(conn, tx, """ + INSERT INTO annotations (id, media_id) VALUES (@id, @mid); + """, ("id", annId), ("mid", mediaId)); + + ExecScalar(conn, tx, """ + INSERT INTO detection (id, annotation_id) VALUES (@id, @aid); + """, ("id", NewDeterministicGuid(seed)), ("aid", annId)); + } + } + tx.Commit(); + } + + var warmup = ids.Take(WarmupCalls).ToList(); + var measured = ids.Skip(WarmupCalls).ToList(); + return (measured, warmup); + } + + private static void ExecScalar(NpgsqlConnection conn, NpgsqlTransaction tx, string sql, + params (string Name, object Value)[] args) + { + using var cmd = conn.CreateCommand(); + cmd.Transaction = tx; + cmd.CommandText = sql; + foreach (var (name, value) in args) + cmd.Parameters.AddWithValue(name, value); + cmd.ExecuteNonQuery(); + } + + private static Guid NewDeterministicGuid(Random rng) + { + var bytes = new byte[16]; + rng.NextBytes(bytes); + // Force version 4 + variant 1 so the value is a valid UUID Postgres accepts. + bytes[7] = (byte)((bytes[7] & 0x0F) | 0x40); + bytes[8] = (byte)((bytes[8] & 0x3F) | 0x80); + return new Guid(bytes); + } +} diff --git a/tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs b/tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs deleted file mode 100644 index fdd9980..0000000 --- a/tests/Azaion.Missions.E2E.Tests/Tests/Performance/Sanity.cs +++ /dev/null @@ -1,23 +0,0 @@ -using Xunit; - -namespace Azaion.Missions.E2E.Tests.Performance; - -/// -/// Discovery-only smoke test for the Performance category. Real Performance -/// scenarios (NFT-PERF-01..04) land in AZ-586. -/// -public sealed class Sanity -{ - [Fact] - [Trait("Category", "Perf")] - [Trait("Traces", "AC-3")] - public void Discovery_smoke_test_runs() - { - // Arrange - const int sentinel = 1; - // Act - var result = sentinel + 0; - // Assert - Assert.Equal(1, result); - } -} diff --git a/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs b/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs new file mode 100644 index 0000000..2395dd4 --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/ColdStartRssTests.cs @@ -0,0 +1,131 @@ +using System.Diagnostics; +using System.Globalization; +using System.Net; +using Azaion.Missions.E2E.Helpers; +using Xunit; + +namespace Azaion.Missions.E2E.Tests.ResourceLimits; + +/// +/// NFT-RES-LIM-04 — cold-start RSS. Driven independently from the +/// steady-state window because it requires a fresh container start; lives +/// in the MigratorRestart collection so it serialises with the +/// other docker-compose-restarting tests rather than racing them. +/// +/// +/// The 30-second wait between health-OK and the measurement is the spec's +/// way of letting the JIT and the JWKS prefetch settle without doing any +/// real work — measuring at health-OK alone would conflate the genuine cold +/// baseline with bootstrap noise. +/// +[Collection("MigratorRestart")] +[Trait("Category", "ResLim")] +public sealed class ColdStartRssTests +{ + private static readonly MetricCsvRecorder Csv = new("RESLIM_RESULTS_FILE"); + private const long ProvisionalColdRssCapMiB = 200; + private const string ComposeFile = "/workspace/docker-compose.test.yml"; + + [SkippableFact] + [Trait("Traces", "H1|H3")] + [Trait("max_ms", "120000")] + public async Task NFT_RES_LIM_04_cold_start_rss_within_provisional_200_MiB() + { + Skip.IfNot(Environment.GetEnvironmentVariable("COMPOSE_RESTART_ENABLED") == "1", + "COMPOSE_RESTART_ENABLED!=1 — docker compose restart unavailable in this consumer image"); + Skip.IfNot(MissionsContainerHelper.Enabled, + "MissionsContainerHelper disabled — docker CLI unavailable"); + + // Arrange — bring missions down hard and start it fresh. The + // surrounding "MigratorRestart" collection serialises us against + // any other test that touches the SUT. + DockerCompose("stop missions"); + DockerCompose("rm -f missions"); + DockerCompose("up -d missions"); + + await WaitForHealthOkAsync(TimeSpan.FromSeconds(60)); + + // Act — wait 30s after health-OK so JIT/JWKS settle, then measure. + await Task.Delay(TimeSpan.FromSeconds(30)); + var rssBytes = ReadRssBytes("missions-sut"); + var rssMiB = rssBytes / (double)(1024 * 1024); + + var pass = rssMiB <= ProvisionalColdRssCapMiB; + Csv.Record( + category: "ResLim", + scenario: "NFT-RES-LIM-04", + result: pass ? "pass" : "fail", + traces: $"H1|H3; COLD_RSS_MiB={rssMiB.ToString("F1", CultureInfo.InvariantCulture)}"); + + // Assert — provisional gate. + Assert.True(pass, + $"cold-start RSS {rssMiB:F1} MiB exceeds provisional {ProvisionalColdRssCapMiB} MiB gate"); + } + + private static async Task WaitForHealthOkAsync(TimeSpan timeout) + { + using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(2) }; + var deadline = DateTime.UtcNow + timeout; + var healthUrl = new Uri(TestEnvironment.MissionsBaseUrl + "/health"); + while (DateTime.UtcNow < deadline) + { + try + { + using var resp = await http.GetAsync(healthUrl); + if (resp.StatusCode == HttpStatusCode.OK) return; + } + catch (HttpRequestException) { /* not yet listening */ } + catch (TaskCanceledException) { /* slow first response */ } + await Task.Delay(500); + } + throw new TimeoutException( + $"missions did not become healthy within {timeout.TotalSeconds:F0}s of cold start"); + } + + private static long ReadRssBytes(string containerName) + { + var raw = Run("docker", + $"stats --no-stream --format '{{{{.MemUsage}}}}' {containerName}"); + var lhs = raw.Split('/')[0].Trim().Trim('\''); + return ParseHumanBytes(lhs); + } + + private static long ParseHumanBytes(string text) + { + var unitIx = text.IndexOfAny(new[] { 'K', 'M', 'G', 'T', 'B' }); + if (unitIx < 0) return long.Parse(text, CultureInfo.InvariantCulture); + var num = double.Parse(text.Substring(0, unitIx), CultureInfo.InvariantCulture); + var unit = text.Substring(unitIx); + return unit switch + { + "B" => (long)num, + "KiB" or "KB" or "K" => (long)(num * 1024), + "MiB" or "MB" or "M" => (long)(num * 1024 * 1024), + "GiB" or "GB" or "G" => (long)(num * 1024 * 1024 * 1024), + "TiB" or "TB" or "T" => (long)(num * 1024L * 1024 * 1024 * 1024), + _ => throw new FormatException($"unknown human-bytes unit in '{text}'") + }; + } + + private static void DockerCompose(string subcommand) => + Run("docker", $"compose -f {ComposeFile} {subcommand}"); + + private static string Run(string file, string args) + { + var psi = new ProcessStartInfo(file, args) + { + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + using var p = Process.Start(psi) + ?? throw new InvalidOperationException($"failed to launch `{file} {args}`"); + var stdout = p.StandardOutput.ReadToEnd(); + var stderr = p.StandardError.ReadToEnd(); + p.WaitForExit(); + if (p.ExitCode != 0) + throw new InvalidOperationException( + $"`{file} {args}` exited {p.ExitCode}: {stderr}"); + return stdout; + } +} diff --git a/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs b/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs deleted file mode 100644 index a4186fd..0000000 --- a/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/Sanity.cs +++ /dev/null @@ -1,23 +0,0 @@ -using Xunit; - -namespace Azaion.Missions.E2E.Tests.ResourceLimits; - -/// -/// Discovery-only smoke test for the ResourceLimits category. Real -/// ResourceLimits scenarios (NFT-RES-LIM-01..04) land in AZ-585. -/// -public sealed class Sanity -{ - [Fact] - [Trait("Category", "ResLim")] - [Trait("Traces", "AC-3")] - public void Discovery_smoke_test_runs() - { - // Arrange - const int sentinel = 1; - // Act - var result = sentinel + 0; - // Assert - Assert.Equal(1, result); - } -} diff --git a/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs b/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs new file mode 100644 index 0000000..fe4ba4c --- /dev/null +++ b/tests/Azaion.Missions.E2E.Tests/Tests/ResourceLimits/SteadyStateLoadTests.cs @@ -0,0 +1,156 @@ +using System.Globalization; +using Azaion.Missions.E2E.Fixtures; +using Azaion.Missions.E2E.Helpers; +using Xunit; + +namespace Azaion.Missions.E2E.Tests.ResourceLimits; + +/// +/// NFT-RES-LIM-01..03 — three observations on a SINGLE 5-minute sustained +/// load window. The window itself lives in +/// (class-scoped, runs once); each +/// test asserts one metric against its provisional gate. +/// +/// +/// The fixture skips itself when docker primitives are unavailable; the +/// tests detect that via +/// and surface the same reason through Skip.IfNot. The fixture +/// also flips +/// if the SUT crashes mid-window — every test fails fast with a clear +/// message rather than reporting a misleading metric. +/// +[Collection("ResLimSteadyState")] +[Trait("Category", "ResLim")] +public sealed class SteadyStateLoadTests : TestBase, IClassFixture +{ + private static readonly MetricCsvRecorder Csv = new("RESLIM_RESULTS_FILE"); + private const long ProvisionalRssCapMiB = 250; + private const int ProvisionalConnectionCap = 100; + private const int ProvisionalFdCap = 1024; + + private readonly SteadyStateLoadFixture _load; + + public SteadyStateLoadTests(SteadyStateLoadFixture load) => _load = load; + + [SkippableFact] + [Trait("Traces", "H1|H6|O10")] + [Trait("max_ms", "360000")] + public void NFT_RES_LIM_01_steady_state_rss_within_provisional_gate_and_no_leak() + { + Skip.If(_load.SkipReason is not null, _load.SkipReason); + Skip.IfNot(_load.LoadGeneratorMetTargetRps, + "runner cannot sustain target load (NFR Reliability — not a SUT defect)"); + Assert.False(_load.SutExitedDuringWindow, "SUT exited during measurement window"); + + // Arrange + var samplesMiB = _load.RssBytesSamples.Select(b => b / (double)(1024 * 1024)).ToList(); + Assert.True(samplesMiB.Count >= 30, + $"expected ≥ 30 RSS samples over 5-min window, got {samplesMiB.Count}"); + + // Act + var p95 = LatencyPercentiles.P95(samplesMiB); + var finalMiB = samplesMiB[^1]; + + var leakRatio = Math.Abs(finalMiB - p95) / Math.Max(p95, 1.0); + var withinCap = p95 <= ProvisionalRssCapMiB; + var noLeak = leakRatio <= 0.20; + var pass = withinCap && noLeak; + + Csv.Record( + category: "ResLim", + scenario: "NFT-RES-LIM-01", + result: pass ? "pass" : "fail", + traces: $"H1|H6|O10; " + + $"P95_RSS_MiB={p95.ToString("F1", CultureInfo.InvariantCulture)}; " + + $"FINAL_RSS_MiB={finalMiB.ToString("F1", CultureInfo.InvariantCulture)}; " + + $"LEAK_RATIO={leakRatio.ToString("F2", CultureInfo.InvariantCulture)}"); + + // Assert — provisional gate; lock at measured + 50% after first green run. + Assert.True(withinCap, + $"P95 RSS {p95:F1} MiB exceeds provisional {ProvisionalRssCapMiB} MiB gate"); + Assert.True(noLeak, + $"final RSS {finalMiB:F1} MiB diverges {leakRatio:P0} from P95 {p95:F1} MiB (gate 20%)"); + } + + [SkippableFact] + [Trait("Traces", "O10")] + [Trait("max_ms", "360000")] + public void NFT_RES_LIM_02_npgsql_connection_pool_within_100_no_unbounded_growth() + { + Skip.If(_load.SkipReason is not null, _load.SkipReason); + Skip.IfNot(_load.LoadGeneratorMetTargetRps, + "runner cannot sustain target load (NFR Reliability — not a SUT defect)"); + Assert.False(_load.SutExitedDuringWindow, "SUT exited during measurement window"); + + var samples = _load.NpgsqlConnectionSamples; + Assert.True(samples.Count >= 30, + $"expected ≥ 30 connection samples over 5-min window, got {samples.Count}"); + + // Act + var max = samples.Max(); + var firstMinuteSampleCount = 60 / SteadyStateLoadFixture.SampleIntervalSeconds; + var firstMinute = samples.Take(firstMinuteSampleCount).ToList(); + var firstMinuteMean = firstMinute.Average(); + var finalCount = samples[^1]; + + var withinCap = max <= ProvisionalConnectionCap; + var noUnboundedGrowth = finalCount <= 1.3 * Math.Max(firstMinuteMean, 1.0); + var pass = withinCap && noUnboundedGrowth; + + Csv.Record( + category: "ResLim", + scenario: "NFT-RES-LIM-02", + result: pass ? "pass" : "fail", + traces: $"O10; MAX_NPGSQL_CONNS={max}; " + + $"FINAL_CONNS={finalCount}; " + + $"MINUTE1_MEAN={firstMinuteMean.ToString("F1", CultureInfo.InvariantCulture)}"); + + // Assert + Assert.True(withinCap, + $"max Npgsql connections {max} exceeds provisional cap {ProvisionalConnectionCap}"); + Assert.True(noUnboundedGrowth, + $"final connection count {finalCount} > 1.3 × first-minute mean {firstMinuteMean:F1}"); + } + + [SkippableFact] + [Trait("Traces", "H6|O10")] + [Trait("max_ms", "360000")] + public void NFT_RES_LIM_03_file_descriptors_within_1024_no_leak() + { + Skip.If(_load.SkipReason is not null, _load.SkipReason); + Skip.IfNot(_load.LoadGeneratorMetTargetRps, + "runner cannot sustain target load (NFR Reliability — not a SUT defect)"); + Assert.False(_load.SutExitedDuringWindow, "SUT exited during measurement window"); + + var samples = _load.FileDescriptorSamples; + Assert.True(samples.Count >= 30, + $"expected ≥ 30 FD samples over 5-min window, got {samples.Count}"); + + // Act + var max = samples.Max(); + var minuteOneSampleCount = 60 / SteadyStateLoadFixture.SampleIntervalSeconds; + // The spec calls out "count at t=1min" — anchor on the sample whose + // timestamp is closest to (start + 60s). + var minuteOneIx = Math.Min(minuteOneSampleCount - 1, samples.Count - 1); + var minuteOneCount = samples[minuteOneIx]; + var finalCount = samples[^1]; + + var withinCap = max <= ProvisionalFdCap; + var noLeak = finalCount <= 1.3 * Math.Max(minuteOneCount, 1); + var pass = withinCap && noLeak; + + Csv.Record( + category: "ResLim", + scenario: "NFT-RES-LIM-03", + result: pass ? "pass" : "fail", + traces: $"H6|O10; MAX_FD={max}; " + + $"FINAL_FD={finalCount}; MINUTE1_FD={minuteOneCount}"); + + // Assert + Assert.True(withinCap, + $"max FD count {max} exceeds provisional cap {ProvisionalFdCap}"); + Assert.True(noLeak, + $"final FD count {finalCount} > 1.3 × minute-1 count {minuteOneCount}"); + } + +} diff --git a/tests/Azaion.Missions.E2E.Tests/entrypoint.sh b/tests/Azaion.Missions.E2E.Tests/entrypoint.sh index 9fd2750..d3aeca6 100755 --- a/tests/Azaion.Missions.E2E.Tests/entrypoint.sh +++ b/tests/Azaion.Missions.E2E.Tests/entrypoint.sh @@ -14,9 +14,16 @@ set -eu mkdir -p "$RESULTS_DIR" set +e +## Performance scenarios (Category=Perf) are excluded from the default gate +## per AZ-586. They are invoked from scripts/run-performance-tests.sh which +## passes its own --filter Category=Perf. ResLim tests (Category=ResLim) stay +## in the default gate because their docker-CLI gate causes them to skip +## with an explicit reason when COMPOSE_RESTART_ENABLED is not set. +TEST_FILTER="${TEST_FILTER:-Category!=Perf}" dotnet test /src/Azaion.Missions.E2E.Tests.csproj \ --no-build \ --configuration Release \ + --filter "$TEST_FILTER" \ --logger "trx;LogFileName=results.trx" \ --logger "console;verbosity=normal" \ --results-directory "$RESULTS_DIR"