satellite-provider/_docs/_process_leftovers/2026-05-12_perf-cycle3-harness-execution.md

Leftover — Cycle 3 perf harness execution

Timestamp: 2026-05-12T02:25:00Z (replay #2 — post AZ-500 .NET 10 migration; original deferral 2026-05-12T00:00:00Z) Reason for deferral: User skipped the Step 15 (Performance Test) gate of cycle 3. Per meta-rule.mdc, performance tests require explicit approval; a skipped question is not approval. Defaulted to skip + record-as-leftover to avoid blocking cycle-3 progress through Steps 16-17.

Replay attempt #1 — 2026-05-12T01:11:00Z (cycle 4 /autodev start, pre-migration)

User picked A (run perf harness now). Stack came up cleanly via docker-compose up -d --build. Perf script scripts/run-performance-tests.sh failed at the bootstrap step (dotnet build SatelliteProvider.IntegrationTests for the --mint-only JWT subcommand) because the host had only .NET 10.0.103 SDK installed and global.json pinned sdk.version=8.0.0 with rollForward=latestMinor (only rolls within 8.0.x). Exit code 3.

Sibling script scripts/run-tests.sh does NOT have this problem because it shells out to docker run --rm ... mcr.microsoft.com/dotnet/sdk:8.0 for every dotnet invocation. The perf script was written without that pattern.

Per cycle-3 lesson "scenarios accumulate as Unverified across cycles" — this is a real script bug, not just a host quirk.

Replay attempt #2 — 2026-05-12T02:21:00Z (cycle 4, AC-5 of AZ-500 short bootstrap-smoke)

After AZ-500 landed (.NET 10 migration: TFM, global.json sdk.version=10.0.0, all Docker images, all Microsoft.AspNetCore.* / Microsoft.Extensions.* packages, scripts/run-performance-tests.sh:49 bin/Release/net8.0/ → bin/Release/net10.0/), re-ran the AC-5 short variant:

PERF_REPEAT_COUNT=2 PERF_UAV_BATCH_SIZE=2 ./scripts/run-performance-tests.sh

against docker-compose up -d --build (api healthy on :18980, swagger 200, anonymous request 401). Trace summary:

Step	Result
Build SatelliteProvider.IntegrationTests (Release)	OK (build succeeded, 11 NU1902/CA2227 warnings, 0 errors, 41.5s)
--mint-only JWT subcommand	OK (341-byte token, 4h lifetime)
PT-01 cold tile download	PASS (2538ms / 30000ms threshold)
PT-02 cached tile retrieval	PASS (195ms / 500ms)
PT-03 region 200m / z18	PASS (384ms / 60000ms)
PT-04 region 500m / z18 + stitch	PASS (2202ms / 120000ms)
PT-05 5 concurrent regions	PASS (3258ms / 300000ms)
PT-06 route creation (2 points)	PASS (178ms / 5000ms)
PT-07 cold/warm region request	PASS (warm p95 2340ms < cold p95 3241ms)
PT-08 UAV batch upload	CRASHED at first batch summarisation — see below

Bootstrap step DID NOT exit with code 3 — host SDK / global.json mismatch is gone. AC-5 met.

Replay attempt #2 — root cause of PT-08 crash (NOT an SDK / .NET 10 issue)

bash -x trace shows the script silently exits right after rejected=0 and the cleanup trap fires. The script bug is at scripts/run-performance-tests.sh:417:

rejected=$(grep -o '"status":"rejected"' "$PERF_TMP_DIR/pt08_resp.json" | wc -l | tr -d ' ')

When the upload response has zero rejected items (the happy-path case), grep -o exits 1 (no matches). With set -o pipefail (line 16) the pipeline returns 1; with set -e the assignment kills the script. The sibling line at 416 for accepted only worked in this trace because the response had 2 accepted items so grep exited 0.

This bug pre-existed AZ-500. It was previously masked because the perf script never reached PT-08 — it failed at bootstrap (replay #1) due to the SDK mismatch. The .NET 10 migration unmasked it by clearing the bootstrap blocker. PT-01..PT-07 are unaffected (no grep -c/grep -o counts on potentially-empty matches).

The actual perf-relevant data PT-08 captured before crashing (one batch run completed): HTTP 200, batch latency 99ms (well under the AZ-488 2000ms p95 threshold), accepted=2, rejected=0. So the underlying perf is healthy; only the script's failure-counting harness is buggy.

Resolution path (forward)

Two follow-up fixes are needed; both are out of AZ-500 scope per coderule.mdc "scope discipline":

1. scripts/run-performance-tests.sh:416-417 — defensive grep-counting. Replace

   accepted=$(grep -o '"status":"accepted"' "$PERF_TMP_DIR/pt08_resp.json" | wc -l | tr -d ' ')
   rejected=$(grep -o '"status":"rejected"' "$PERF_TMP_DIR/pt08_resp.json" | wc -l | tr -d ' ')
   

   with a pipefail-tolerant variant such as

   accepted=$(grep -c '"status":"accepted"' "$PERF_TMP_DIR/pt08_resp.json" || true)
   rejected=$(grep -c '"status":"rejected"' "$PERF_TMP_DIR/pt08_resp.json" || true)
   

   (grep -c already counts; || true neutralises the exit-1-on-no-match case when summed with set -o pipefail/set -e).

2. Step 15 (Performance Test) of cycle 4 — re-run the full harness (default PERF_REPEAT_COUNT=20 PERF_UAV_BATCH_SIZE=10) after the script fix lands. Only then can the leftover be deleted (per Constraints last bullet of AZ-500: "leftover file is deleted ONLY when the full perf script runs cleanly").

Pre-requisites for full replay

Same as before — env vars must be present (already in .env):

• JWT_SECRET — ≥ 32 bytes
• JWT_ISSUER — DEV-ONLY (AZ-494)
• JWT_AUDIENCE — DEV-ONLY (AZ-494)
• GOOGLE_MAPS_API_KEY

Optionally:

• PERF_REPEAT_COUNT (default 20)
• PERF_UAV_BATCH_SIZE (default 10)

How to replay (after the script fix lands)

docker-compose up -d --build           # bring up API on :18980
./scripts/run-performance-tests.sh     # ~3-5 minutes; full PT-01..PT-08
docker-compose down --remove-orphans

Why this is NOT a hard blocker

• AC-5 of AZ-500 only gates the bootstrap step ("does NOT exit with code 3"). That is met.
• The cycle-3 implementation report and code review verdicts already note that the perf harness was statically verified (script grep + integration-test compile + AZ-492 AC-1/AC-4/AC-5/AC-6 covered).
• The AZ-488 batch-p95 threshold was set in cycle 2; the one PT-08 batch we did capture (99ms) is far below the 2000ms threshold.
• No cycle-3/cycle-4 change altered production hot paths beyond JWT validation (AZ-494 adds two string comparisons per request — sub-microsecond).

Replay obligation

Open a new follow-up PBI for the scripts/run-performance-tests.sh:416-417 grep fix. Once that lands and a full perf run is green, delete this file. Until then, this leftover stays.