azaion/missions
1
0
Fork 0
mirror of https://github.com/azaion/missions.git synced 2026-06-22 03:21:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-585] [AZ-586] ResLim+Perf NFT tests; close test cycle 1

Browse Source 
Batch 4 of test implementation cycle 1 (existing-code Step 6, final batch).

- AZ-585 SteadyStateLoadTests + ColdStartRssTests: NFT-RES-LIM-01..04.
  SteadyStateLoadFixture runs one 5-min sustained-load window and samples
  RSS (docker stats), Npgsql conns (pg_stat_activity), and FDs
  (/proc/1/fd) every 5s; three test methods assert independently. All
  SkippableFact-gated on docker primitives.
- AZ-586 PerformanceTests: NFT-PERF-01..04. Sequential single-client,
  5 warm-ups + N measured calls, P50+P95 via LatencyPercentiles, recorded
  to PERF_RESULTS_FILE. Tagged Category=Perf so default gate excludes them.

Infrastructure:
- entrypoint.sh now applies --filter "${TEST_FILTER:-Category!=Perf}"
  per AZ-586 (default CI gate excludes performance).
- MetricCsvRecorder: idempotent CSV appender keyed on env var, used by
  both Perf and ResLim categories.

Step 6 (Implement Tests) is complete. Final report at
_docs/03_implementation/implementation_report_tests.md handoffs the
full-suite gate to test-run/SKILL.md (Step 7).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-15 09:11:53 +03:00
parent 26126e6216
commit 001e80fe96
14 changed files with 1181 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/tasks/todo/AZ-585_test_resource_limits.md
-116
View File
@@ -1,116 +0,0 @@
# Resource Limit Tests
**Task**: AZ-585_test_resource_limits
**Name**: Resource limit tests (NFT-RES-LIM-01..04)
**Description**: Implement xUnit blackbox tests for the 4 resource-limit observation scenarios — steady-state RSS memory under 5-min sustained load (P95 ≤ 250 MiB; no monotonic climb), Npgsql connection pool ≤ 100 with no unbounded growth, file-descriptor count ≤ 1024 with no leak, and cold-start RSS ≤ 200 MiB at `t=30s` after health-ok. Provisional gates documented per `restrictions.md` H6 — locked in after first green run.
**Complexity**: 3 points
**Dependencies**: AZ-576_test_infrastructure
**Component**: Blackbox Tests
**Tracker**: AZ-585
**Epic**: AZ-575
## Problem
Per H6, container-level resource limits are NOT enforced inside the container — they will be set at the suite level (`_infra/_compose/`) per device type once locked. These tests establish baseline observations so the suite can size the cgroup limits correctly AND provide an upper-bound regression gate so future changes do not silently 10× the memory or FD footprint. The 8 GB Jetson Orin must accommodate ~6 .NET edge services + Postgres + UI; `missions`'s budget is ~200 MiB cold + ~250 MiB hot. Without these observation tests, a leak or library bloat could ship to the device and force a re-sizing decision late in deployment.
## Outcome
- All four NFT-RES-LIM-01..04 scenarios run and pass against the dockerised `missions` service.
- Each test produces a CSV row with `Category=ResLim`, `Traces=H1|H3|H6|O10`, `Result=pass`, AND records the measured value (e.g., `P95_RSS_MiB=187`) in the `Traces` column so suite-level deployment planning can read it.
- NFT-RES-LIM-01 measures P95 RSS over 5 minutes of mixed sustained load AND asserts `final_RSS - P95_RSS ≤ 20% * P95_RSS` (no monotonic climb).
- NFT-RES-LIM-02 measures Npgsql connection count via `pg_stat_activity` every 5s AND asserts both `max ≤ 100` AND `final ≤ 1.3 * first_minute_steady_state`.
- NFT-RES-LIM-03 measures `/proc/<pid>/fd | wc -l` inside the container every 5s AND asserts both `max ≤ 1024` AND `final ≤ 1.3 * minute_one_count`.
- NFT-RES-LIM-04 measures cold-start RSS exactly 30s after `GET /health` first returns 200 (no requests issued yet) AND asserts `RSS ≤ 200 MiB`.
## Scope
### Included
- NFT-RES-LIM-01 Steady-state memory under 5-min sustained load.
- NFT-RES-LIM-02 Connection pool steady-state.
- NFT-RES-LIM-03 File-descriptor steady-state.
- NFT-RES-LIM-04 Cold-start RSS budget.
- Each test records the measured value to the CSV `Traces` field so deployment planning can pick it up.
- Provisional gates: 250 MiB hot, 200 MiB cold, 100 connections, 1024 FDs. On first green run, replace provisional gates with `measured + 50%` and open a Refactor Backlog ticket if the provisional gate was exceeded.
### Excluded
- Performance (latency / throughput) tests live in Task 19.
- GPU / temperature / disk-I/O monitoring (per `restrictions.md` H8 — no specialised hardware on a CRUD service).
- Long-soak / endurance tests (> 5 min) — explicitly deferred per `restrictions.md` H8.
## Acceptance Criteria
**AC-1: NFT-RES-LIM-01 steady-state RSS ≤ provisional 250 MiB with no monotonic climb**
Given `missions` running with `seed_25_missions` + `seed_3_vehicles_2_default` and no host-side memory limit
When the test orchestrator drives ~50 RPS of mixed `GET /vehicles`, `GET /missions`, `GET /missions/{id}/waypoints` for 5 minutes from a single concurrent client, while polling `docker stats --no-stream missions-sut` every 5s
Then the P95 of the 60 RSS samples is `≤ 250 MiB` (provisional gate)
And the final-sample RSS is within ± 20% of the P95 RSS (no sustained leak — RSS does not climb monotonically)
And the measured P95 is recorded to the CSV `Traces` column as `P95_RSS_MiB=<n>`
**AC-2: NFT-RES-LIM-02 connection pool ≤ 100 with no unbounded growth**
Given the same setup as NFT-RES-LIM-01
When the test orchestrator polls side-channel `SELECT count(*) FROM pg_stat_activity WHERE application_name LIKE 'Npgsql%' OR (usename='postgres' AND backend_type='client backend')` every 5s for 5 minutes
Then the max sampled connection count is `≤ 100`
And the final-sample count is `≤ 1.3 × (mean of samples in the first minute)`
And the measured max is recorded as `MAX_NPGSQL_CONNS=<n>`
**AC-3: NFT-RES-LIM-03 file descriptors ≤ 1024 with no leak**
Given the same setup as NFT-RES-LIM-01
When the test orchestrator executes `docker exec missions-sut sh -c 'ls /proc/$(pgrep -f Azaion.Missions.dll | head -1)/fd | wc -l'` every 5s for 5 minutes
Then the max sampled FD count is `≤ 1024`
And the final-sample count is `≤ 1.3 × (count at t=1min)`
And the measured max is recorded as `MAX_FD=<n>`
**AC-4: NFT-RES-LIM-04 cold-start RSS ≤ 200 MiB**
Given `missions` has been started fresh (via `docker compose up -d missions` after `down -v`), no requests issued yet
When `GET /health` first returns `200` AND 30s have elapsed
Then `docker stats --no-stream missions-sut` reports `MEM USAGE` ≤ 200 MiB
And the measured cold-start RSS is recorded as `COLD_RSS_MiB=<n>`
## Non-Functional Requirements
**Performance**
- NFT-RES-LIM-01..03: each take exactly 5 minutes (sampling window). With Arrange/teardown, ≤ 6 minutes wall-clock.
- NFT-RES-LIM-04: ≤ 60s wall-clock (fresh start + health-poll + 30s wait + measurement).
- The total task runtime budget is ≤ 20 minutes, fitting inside the documented 15-min suite CI gate per `environment.md`. NFT-RES-LIM-01..03 share the same 5-minute window and run concurrently against a single dockerised `missions`; NFT-RES-LIM-04 runs separately because it requires a fresh start.
**Reliability**
- The load generator is a single-thread `HttpClient` driving requests in a tight loop; this is documented at 50 RPS approximately for the in-suite test runner. If the runner is unable to sustain 50 RPS (CI infrastructure too slow), the test SKIPS NFT-RES-LIM-01..03 with `Result=skip` and a clear `ErrorMessage=runner cannot sustain target load`. CI then reruns these on a beefier worker.
## Blackbox Tests
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|--------|------------------------|-------------|-------------------|----------------|
| AC-1 | `seed_25_missions` + 50 RPS for 5 min | P95 RSS sampling | P95 ≤ 250 MiB + no monotonic climb | H1, H6, O10 |
| AC-2 | same | `pg_stat_activity` polling | max ≤ 100 + final ≤ 1.3×steady | O10 |
| AC-3 | same | `/proc/<pid>/fd` polling | max ≤ 1024 + final ≤ 1.3×minute-one | H6, O10 |
| AC-4 | fresh `docker compose up -d` | cold-start RSS at t=30s | RSS ≤ 200 MiB | H1, H3 |
## Constraints
- `docker stats` and `docker exec` from inside the runner: requires Docker socket access; AZ-576 covers this.
- NFT-RES-LIM-03 requires `pgrep` inside the `missions` image; the test FAILS in Arrange (not Assert) if `pgrep` is unavailable. Alternative: parse `/proc/1/comm` if PID 1 is the .NET process (preferred for the small Dockerfile).
- All measurements are recorded to the CSV report's `Traces` field so deployment planning can pick them up; this is more important than the pass/fail gate.
- Provisional gates are documented per `restrictions.md` H6 — locked in based on first measured run.
- AAA pattern with `// Arrange` / `// Act` / `// Assert` per test.
## Risks & Mitigation
**Risk 1: Measurement variance on shared CI runners**
- *Risk*: A runner under noisy-neighbour load reports inflated RSS, flaking the gate.
- *Mitigation*: Gates are provisional and generous (250 MiB vs. typical .NET service of ~150 MiB; 100 connections vs. typical idle pool of ~5–10). After the first green run, the gate is locked at `measured + 50%`.
**Risk 2: NFT-RES-LIM-01..03 share a 5-minute window — flake correlation**
- *Risk*: A CI hiccup that kills the SUT mid-window flakes all three at once.
- *Mitigation*: Each test asserts its own metric; on `missions-sut` exit during the window, the test FAILS with a `"SUT exited during measurement window"` ErrorMessage rather than reporting a misleading metric value.
**Risk 3: Provisional gates silently accepted as the locked gate**
- *Risk*: If the first green run measures 200 MiB and the test passes, a future engineer treats 250 MiB as the gate forever — but actual headroom is only 50 MiB.
- *Mitigation*: The test logs `(measured / gate) ratio`; CI dashboards flag ratios > 0.8 for re-tuning consideration. The lock-in workflow is documented in `restrictions.md` H6.
## System Under Test Boundary
- Tests drive the product through the public HTTP surface for load generation; `docker stats`, `docker exec`, and side-channel `pg_stat_activity` for measurement. Expected outputs are the documented gates from `_docs/02_document/tests/resource-limit-tests.md` (provisional) and the corresponding entries in `_docs/00_problem/input_data/expected_results/results_report.md` (when locked).
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`.
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the Npgsql connection pool, the `AppDataConnection` lifetime, or the `Program.cs` startup path. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.
_docs/tasks/todo/AZ-586_test_performance.md
-117
View File
@@ -1,117 +0,0 @@
# Performance Tests
**Task**: AZ-586_test_performance
**Name**: Performance tests (NFT-PERF-01..04)
**Description**: Implement xUnit blackbox tests for the 4 performance scenarios — F3 cascade-delete P50 ≤ 50ms on a 1-waypoint mission, F3 cascade-delete P50 ≤ 200ms on the full chain (provisional baseline; lock after first green run), `GET /health` P50 ≤ 10ms, and `GET /missions?page=1&pageSize=20` P95 ≤ 100ms against a 1000-mission seed (provisional baseline). Every test runs 5 warm-up calls + the documented N measured calls; cold-start passes excluded.
**Complexity**: 3 points
**Dependencies**: AZ-576_test_infrastructure
**Component**: Blackbox Tests
**Tracker**: AZ-586
**Epic**: AZ-575
## Problem
Three latency thresholds are documented (AC-3.6 P50 ≤ 50ms for minimal cascade, AC-7.3 P50 ≤ 10ms for health, AC-2.3 implicit list latency) and one (NFT-PERF-02 full-chain cascade) is a baseline that subsequent runs must not regress by more than 50%. Without these tests, an unintentional N+1 query, missing index, or accidental serialization layer overhead could silently 10× the response time before the next manual perf benchmark catches it. The full-chain cascade test is especially load-bearing because the F3 cascade walks 5 dependency tables — a future indexing regression or transaction-wrap addition would show up here first.
## Outcome
- All four NFT-PERF-01..04 scenarios run and pass against the dockerised `missions` service.
- Each test produces a CSV row with `Category=Perf`, `Traces=AC-3.6` / `AC-3.1` / `AC-7.3` / `AC-2.3`, `Result=pass`, AND records P50 and P95 numeric values in the `Traces` column (e.g., `P50_MS=23.4, P95_MS=41.8`).
- 5 warm-up calls precede every measured set; cold-start passes are excluded from the percentile computation.
- All tests run sequentially against a single client (no concurrent connections) so HTTP/1.1 connection-reuse and JIT warm-up are deterministic.
- Tests run only when `[Trait("Category","Perf")]` filter is active (default test suite filter excludes performance to keep the standard CI gate ≤ 15 min); a separate `scripts/run-performance-tests.sh` invocation runs them.
## Scope
### Included
- NFT-PERF-01 F3 minimal cascade — `DELETE /missions/{id}` on 1-waypoint missions; P50 ≤ 50ms over 100 sequential calls.
- NFT-PERF-02 F3 full cascade — `DELETE /missions/{id}` on `fixture_cascade_F3`-shaped missions; P50 ≤ 200ms over 50 sequential calls (provisional baseline).
- NFT-PERF-03 Health endpoint — `GET /health` P50 ≤ 10ms over 100 sequential calls.
- NFT-PERF-04 List pagination — `GET /missions?page=1&pageSize=20` P95 ≤ 100ms over 100 sequential calls against a 1000-mission seed (provisional baseline).
- Recording P50/P95 to CSV `Traces` column for trend tracking even when not gated.
- Performance suite is gated behind the `[Trait("Category","Perf")]` filter; standard CI gate excludes these.
### Excluded
- Concurrency / contention tests (race scenarios) live in Task 17 (NFT-RES-08).
- Resource consumption (RSS, FDs, connections) lives in Task 18 (NFT-RES-LIM).
- Production-hardware (Jetson Orin) latency baselines — documented as a follow-up in `restrictions.md` H8; test environment baselines stand in.
- Concurrent-client throughput / RPS — not in scope today; documented as Refactor Backlog.
## Acceptance Criteria
**AC-1: NFT-PERF-01 F3 minimal cascade P50 ≤ 50ms**
Given `missions` + `postgres-test` colocated on the same Docker network, `seed_one_default_vehicle` + 100 minimal missions (each with 1 waypoint, no media/annotations/detection/map_objects rows), AND 5 warm-up `DELETE` calls have completed on missions outside the measured set
When the consumer issues 100 sequential `DELETE /missions/{id_i}` calls (one per seeded mission, 1 ≤ i ≤ 100) and records per-call wall-clock latency
Then the P50 (median) of the 100 latencies is `≤ 50ms`
And P50 + P95 are recorded to the CSV `Traces` column as `P50_MS=<v1>, P95_MS=<v2>`
**AC-2: NFT-PERF-02 F3 full-chain cascade P50 ≤ 200ms**
Given 50 missions each with the `fixture_cascade_F3` chain (3 map_objects, 2 waypoints, 2 media, 2 annotations, 2 detection rows) AND 5 warm-up calls on additional fixtures outside the measured set
When the consumer issues 50 sequential `DELETE /missions/{id_i}` calls and records per-call wall-clock latency
Then P50 ≤ 200ms (provisional baseline — to be locked at `measured + 50%` on first green run)
And P50 + P95 recorded to CSV
**AC-3: NFT-PERF-03 health endpoint P50 ≤ 10ms**
Given `missions` running, no special seed, AND 5 warm-up `GET /health` calls
When the consumer issues 100 sequential `GET /health` calls (no `Authorization` header) and records per-call wall-clock latency
Then P50 ≤ 10ms
And P50 + P95 recorded to CSV
**AC-4: NFT-PERF-04 list pagination P95 ≤ 100ms (provisional)**
Given `seed_one_default_vehicle` + 1000 missions referencing it, AND 5 warm-up `GET /missions?page=1&pageSize=20` calls
When the consumer issues 100 sequential `GET /missions?page=1&pageSize=20` calls and records per-call wall-clock latency
Then P95 ≤ 100ms (provisional baseline — to be locked at `measured + 50%` on first green run)
And P50 + P95 recorded to CSV
## Non-Functional Requirements
**Performance**
- NFT-PERF-01: ≤ 30s wall-clock (100 calls × ≤ 50ms each + measurement overhead). Per `[Trait("max_ms","30000")]` xUnit timeout.
- NFT-PERF-02: ≤ 60s wall-clock.
- NFT-PERF-03: ≤ 5s wall-clock.
- NFT-PERF-04: ≤ 30s wall-clock.
**Reliability**
- All tests SKIP if the runner cannot allocate ≥ 2 CPU cores and ≥ 2 GB free RAM (per `performance-tests.md` Notes). SKIP records `Result=skip` and `ErrorMessage=insufficient CPU/RAM`. Default CI runner spec must meet this — but degraded runners must not produce false-fail noise.
- All tests assume `missions` and `postgres-test` are colocated on the same Docker network (no inter-host link). The fixture verifies this via `docker inspect missions-sut --format '{{.NetworkSettings.Networks.testnet.IPAddress}}'` returns non-empty.
## Blackbox Tests
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|--------|------------------------|-------------|-------------------|----------------|
| AC-1 | 100 minimal missions + 5 warm-ups | 100 sequential `DELETE /missions/{id}` | P50 ≤ 50ms; record P50/P95 | AC-3.6 |
| AC-2 | 50 F3-fixture missions + 5 warm-ups | 50 sequential `DELETE /missions/{id}` | P50 ≤ 200ms (provisional); record P50/P95 | AC-3.1, AC-3.6 |
| AC-3 | warm runtime + 5 warm-ups | 100 sequential `GET /health` | P50 ≤ 10ms; record P50/P95 | AC-7.3 |
| AC-4 | 1000 missions + 5 warm-ups | 100 sequential `GET /missions?page=1&pageSize=20` | P95 ≤ 100ms (provisional); record P50/P95 | AC-2.3 |
## Constraints
- Tests live in `Tests/Performance/` and are tagged `[Trait("Category","Perf")]` so the default CI gate excludes them.
- A separate `scripts/run-performance-tests.sh` (created by AZ-576) invokes only this category. The standard `scripts/run-tests.sh` skips them.
- Sequential single-client execution — no `Parallel.For` or `Task.WhenAll`; each call awaits the previous response.
- Warm-up calls are NOT included in the percentile computation. Per `// Warmup` comment block in the test, the first 5 calls go to fixtures created specifically for warm-up (not the measured set).
- The `Stopwatch`-based timing measures `HttpClient.SendAsync` wall-clock; serialization/deserialization overhead is INCLUDED (this is what end-users observe).
- Provisional gates (NFT-PERF-02, NFT-PERF-04) are documented in source as `// PROVISIONAL — lock at measured + 50% on first green run` and `[Trait("provisional","yes")]`.
- AAA pattern with `// Arrange` (seed + warm-up), `// Act` (measured calls + percentile compute), `// Assert` (gate + CSV record).
## Risks & Mitigation
**Risk 1: CI variance breaks tight P50 ≤ 10ms gate (NFT-PERF-03)**
- *Risk*: On a noisy-neighbour CI runner, even a static `/health` route can hiccup once per 100 calls; if the hiccup lands in the P50 region, the median exceeds 10ms.
- *Mitigation*: P50 is robust to single outliers (median position 50 of 100). If the test still flakes, lock the gate at `measured P50 + 50%` after the first green run.
**Risk 2: NFT-PERF-04 1000-mission seed overlaps with other tests' DB state**
- *Risk*: Seeding 1000 missions affects pagination tests, list-shape tests, and date-filter tests — if NFT-PERF-04 runs before them in the same SUT lifetime, results drift.
- *Mitigation*: NFT-PERF-04 lives in `[Collection("Perf1k")]` and uses `IClassFixture<DbResetFixture>` to TRUNCATE all rows before its seed AND restore `seed_empty` after. Functional tests' fixtures handle their own seed; no cross-pollination.
**Risk 3: Provisional gates accepted as locked gates**
- *Risk*: Same as NFT-RES-LIM Risk 3 — if first run measures 80ms and the test passes, future engineers see the 100ms gate as the standard.
- *Mitigation*: CI dashboards flag `measured / gate ratio > 0.8` for re-tuning. Lock-in workflow documented in `performance-tests.md`.
## System Under Test Boundary
- Tests drive the product through the public HTTP surface (`http://missions:8080`) plus Npgsql side-channel for seed setup. Bearer tokens (NFT-PERF-01, 02, 04) minted via `https://jwks-mock:8443/sign`; NFT-PERF-03 sends no Authorization header. Expected outputs are the documented latency thresholds from `_docs/02_document/tests/performance-tests.md`.
- Stubs are allowed ONLY for: the external `admin` JWT issuer (`jwks-mock` container) and the DB-only stub tables for `media`, `annotations`, `detection`, `map_objects`.
- Stubs, fakes, deterministic fallbacks, monkeypatches, or direct imports are NOT allowed for any internal product module — including the controllers, service classes, `AppDataConnection`, or any layer affecting response time. If any of these is not implemented, the test MUST fail/block as missing product implementation — it must not pass by replacing the module with a test stub.