[AZ-455] [AZ-456] Decompose Step 1t \u2014 test infrastructure task spec

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-21 18:11:10 +00:00 · 2026-05-11 00:59:46 +03:00
parent 729ad1cac8
commit 83dee6759f
2 changed files with 300 additions and 3 deletions
@@ -0,0 +1,297 @@
+# Test Infrastructure
+
+**Task**: AZ-456_test_infrastructure
+**Name**: Test Infrastructure
+**Description**: Scaffold the Blackbox test project — Vitest + jsdom + MSW for the `fast` profile, Playwright + the suite docker-compose stack for the `e2e` profile, ripgrep + bun scripts for `static`. Includes mock services (`owm-stub`, `tile-stub`), test data fixtures, isolation strategy, and CSV+JUnit reporting.
+**Complexity**: 5 points
+**Dependencies**: None
+**Component**: Blackbox Tests
+**Tracker**: AZ-456
+**Epic**: AZ-455
+
+## Test Project Folder Layout
+
+```
+tests/                          # fast profile (colocated with src/, top-level dir for shared helpers)
+├── setup.ts                    # Vitest global setup — RTL matchers + jsdom polyfills + MSW server.listen
+├── msw/
+│   ├── server.ts               # MSW Node server (Vitest) — used by every fast test
+│   ├── handlers/
+│   │   ├── admin.ts            # /api/admin/* default handlers (auth, users, classes-write, GPS)
+│   │   ├── flights.ts          # /api/flights/* default handlers (CRUD + live-GPS SSE simulator)
+│   │   ├── annotations.ts      # /api/annotations/* default handlers (media + annotations + dataset + SSE)
+│   │   ├── detect.ts           # /api/detect/* default handlers (sync image detect)
+│   │   ├── loader.ts           # /api/loader/* default handlers
+│   │   ├── resource.ts         # /api/resource/* default handlers
+│   │   ├── owm.ts              # owm-stub equivalent for fast (OpenWeatherMap canned wind data)
+│   │   └── tiles.ts            # tile-stub equivalent for fast (returns a 1×1 transparent PNG)
+│   └── helpers.ts              # buildResponse, errorResponse, sse, latency, drop helpers
+├── fixtures/
+│   ├── enum_spec_snapshot.ts   # re-exports the committed _docs/00_problem/input_data/enum_spec_snapshot.json
+│   ├── seed_users.ts           # 4 users matching seed_users (test-data.md)
+│   ├── seed_aircraft.ts        # 3 aircraft, one default
+│   ├── seed_flights.ts         # 5 flights, one with live-GPS wire-up
+│   ├── seed_classes.ts         # [0..N-1, 20..20+N-1, 40..40+N-1] N≥9
+│   ├── seed_media.ts           # 6 media with mediaStatus enum coverage
+│   ├── seed_annotations.ts     # AI + Manual, splitTile valid + malformed
+│   └── seed_user_settings.ts   # known selectedFlightId + panelWidths for op_alice
+├── helpers/
+│   ├── render.tsx              # renderWithProviders (router + auth + i18n)
+│   ├── auth.ts                 # seedBearer / clearBearer (uses setToken from src/api/client.ts)
+│   ├── navigate.ts             # seedNavigateToLogin spy (uses setNavigateToLogin from src/api/client.ts)
+│   └── sse-mock.ts             # EventSource polyfill helper for MSW SSE handlers
+└── (test files colocated alongside source, e.g. src/api/client.test.ts, src/features/flights/FlightsPage.test.tsx)
+
+e2e/                            # e2e profile
+├── docker-compose.suite-e2e.yml    # Suite stack + azaion-ui + owm-stub + tile-stub + playwright-runner
+├── playwright.config.ts        # Chromium + Firefox projects per AC-18; baseURL = http://azaion-ui:80
+├── stubs/
+│   ├── owm/                    # ./testing/stubs/owm — tiny Node/Bun HTTP server, canned /data/2.5/* responses
+│   │   ├── Dockerfile
+│   │   └── server.ts
+│   └── tile/                   # ./testing/stubs/tile — returns 256x256 PNG for /{z}/{x}/{y} and /sat/{z}/{y}/{x}
+│       ├── Dockerfile
+│       └── server.ts
+├── runner/                     # ./testing/runner — Playwright runner container image
+│   ├── Dockerfile
+│   └── entrypoint.sh           # bun run test:e2e + writes reports to /output
+├── tests/
+│   ├── auth.e2e.ts             # Group 1 e2e coverage
+│   ├── flights.e2e.ts          # Groups 4, 11, 16
+│   ├── annotations.e2e.ts      # Groups 3, 10, 16
+│   ├── dataset.e2e.ts          # Group 5
+│   ├── admin.e2e.ts            # Groups 1, 13
+│   ├── i18n.e2e.ts             # Group 8
+│   ├── upload.e2e.ts           # Group 6
+│   └── bundle.e2e.ts           # Group 7 (post-build inspection)
+├── fixtures/
+│   └── seeds.sql               # suite-side seed loader hooked into admin/flights/annotations init
+└── results/                    # mounted to playwright-runner:/output; CSV + JUnit + traces
+
+scripts/
+├── run-tests.sh                # ALREADY EXISTS — extend in Step 6 to actually shell out to vitest / playwright
+└── run-performance-tests.sh    # ALREADY EXISTS — extend in Step 6
+```
+
+### Layout Rationale
+
+- **Colocated `*.test.tsx` next to source** (fast): Vitest convention; one test file per src file keeps the import graph shallow and the change-radius obvious. Shared MSW handlers + fixtures + helpers live under `tests/` so they're discoverable in one place.
+- **Dedicated `e2e/` tree**: Playwright + the docker-compose stack are a hermetic black-box layer; physically separating them from `tests/` prevents accidental cross-imports between fast and e2e (the e2e runner has no access to `src/`).
+- **`testing/stubs/`** (named per environment.md): the suite's existing convention for stub services. Keeps Dockerfiles small and independently buildable.
+- **Reports under `test-output/`**: matches `environment.md` § Reporting and the Docker compose snippet exactly — no path drift.
+
+## Mock Services
+
+| Mock Service | Replaces | Endpoints | Behavior |
+|-------------|----------|-----------|----------|
+| `owm-stub` (e2e Docker) | OpenWeatherMap (E10) | `GET /data/2.5/weather?lat=&lon=&appid=&units=metric` | Returns `{ wind: { speed, deg } }` matching `flightPlanUtils.test.ts` expected wind compute. Two canned response sets keyed by lat,lon. |
+| `tile-stub` (e2e Docker) | OSM + Esri tile servers | `GET /{z}/{x}/{y}.png`, `GET /sat/{z}/{y}/{x}` | Always returns a 256×256 PNG; logs every request so tile-coverage tests can assert. |
+| `tests/msw/handlers/*.ts` (fast) | Every suite service + OWM + tiles | per file | Per-test handler override via `server.use(...)`. Resets between tests. SSE via MSW WebSocket adapter or `EventSourcePolyfill` test double. |
+| Real `admin/`, `flights/`, `annotations/`, `detect/`, `loader/`, `resource/`, `gps-denied-{desktop,onboard}/`, `autopilot/` (e2e Docker) | n/a — they ARE the system in `e2e` | per suite contract | Provided by the parent suite's images. Test-mode endpoints (`POST /test-only/reset`, embedded LiveGPS + annotation-status event generators) enabled via a non-production build flag. |
+
+### Mock Control API
+
+- **Fast (MSW)**: per-test `server.use(...)` for handler overrides; `server.resetHandlers()` in `afterEach`. No HTTP control API needed — it's in-process.
+- **owm-stub** + **tile-stub** (e2e): `POST /mock/config` to swap canned response sets; `GET /mock/log` to read recorded requests. Used by resilience and performance tests.
+- **`admin/` test-only**: `POST /test-only/reset` resets all three DBs to seed state between isolation buckets (per `test-data.md`).
+
+## Docker Test Environment
+
+### docker-compose.suite-e2e.yml Structure
+
+| Service | Image / Build | Purpose | Depends On |
+|---------|--------------|---------|------------|
+| `azaion-ui` | Build `.` (project Dockerfile, ARM64) | SPA under test (the system) | `admin`, `flights`, `annotations`, `detect` |
+| `admin` | `azaion/admin:test` | Auth + RBAC + classes-write | `test-db` |
+| `flights` | `azaion/flights:test` | Flight CRUD + live-GPS SSE simulator | `test-db` |
+| `annotations` | `azaion/annotations:test` | Media + annotations + dataset + status SSE | `test-db` |
+| `detect` | `azaion/detect:test` | Sync image detect | — |
+| `loader`, `resource`, `gps-denied-desktop`, `gps-denied-onboard`, `autopilot` | suite images | Auxiliary services hit by SPA | — |
+| `test-db` | `postgres:16-alpine` | Backs admin/flights/annotations | — |
+| `owm-stub` | Build `./e2e/stubs/owm` | OpenWeatherMap stub | — |
+| `tile-stub` | Build `./e2e/stubs/tile` | OSM + Esri tile stub | — |
+| `playwright-runner` | Build `./e2e/runner` | Test runner (Chromium + Firefox) | `azaion-ui`, `owm-stub`, `tile-stub` |
+
+### Networks and Volumes
+
+- **Network**: `azaion-test-net` — isolated, no internet egress; the two stubs replace the only external hops (OWM, tiles).
+- **Volumes**:
+  - `test-db-data` → `test-db:/var/lib/postgresql/data` (wiped with `down -v` between e2e runs).
+  - `seed-fixtures` → `admin:/seed`, `flights:/seed`, `annotations:/seed` (read-only) loaded at service start from `e2e/fixtures/seeds.sql`.
+  - `test-output` → `playwright-runner:/output` (mounted to `./test-output/` on host).
+
+## Test Runner Configuration
+
+### Fast profile
+
+- **Framework**: Vitest 3.x (Vite 6-compatible; matches S3 pin) under `bun test:fast` (a `package.json` script alias).
+- **Environment**: jsdom (default for component tests); a small subset of node-only tests opt in via `// @vitest-environment node`.
+- **Plugins / libraries**:
+  - `@testing-library/react` ^16.x (React 19 compatible)
+  - `@testing-library/jest-dom` ^6.x
+  - `@testing-library/user-event` ^14.x
+  - `msw` ^2.x (Node server + per-test handler override)
+  - `@vitest/coverage-v8` ^3.x
+- **Entry point**: `bun run test:fast` → `vitest run --coverage --reporter=verbose --reporter=junit --outputFile.junit=./test-output/fast-report.xml`.
+- **Setup file**: `tests/setup.ts` — wires jest-dom matchers, starts MSW server, polyfills `EventSource` if not in jsdom, registers global `afterEach(() => { server.resetHandlers(); cleanup() })`.
+
+### E2E profile
+
+- **Framework**: Playwright ^1.49 with `@playwright/test`. Two browser projects: Chromium and Firefox (per AC-18).
+- **Entry point**: `bun run test:e2e` → `playwright test --reporter=list,junit --output=./test-output/e2e --reporter-output=./test-output/e2e-report.xml`.
+- **Config file**: `e2e/playwright.config.ts` — `baseURL: http://azaion-ui:80`, `workers: 1` for SSE-sensitive tests, `retries: 1` on `dev`/`stage`, traces on first retry.
+- **Network gate**: Playwright `context.route('**/api.openweathermap.org/**', route => route.abort())` + `**/unpkg.com/**` `.abort()` defense-in-depth — fails the run if any external host is hit despite the env-var redirection landed in autodev Step 4.
+
+### Static profile
+
+- **Framework**: existing `scripts/run-tests.sh` (Step 4 deliverable) extended in Step 6 to run:
+  - `tsc -b --noEmit`
+  - `ripgrep` checks: no `unpkg.com`, no banned libraries (ML, signature, persistence, WS/GraphQL/SSR), no service-worker registration, no literal OWM key.
+  - `vite build` + bundle-size assertion.
+- **Entry point**: `./scripts/run-tests.sh --static-only`.
+- **Output**: `./test-output/static-report.csv`.
+
+### Fixture Strategy
+
+| Fixture | Scope | Purpose |
+|---------|-------|---------|
+| `renderWithProviders` | function | Mount component with the same Router + AuthProvider + I18nextProvider tree as the production app |
+| `seedBearer(token)` / `clearBearer()` | function | Set/clear the in-memory bearer via `setToken` (production accessor — no monkey-patching) |
+| `seedNavigateToLogin(spy)` | function | Replace `navigateToLoginImpl` via `setNavigateToLogin` (production accessor from autodev Step 4 / C06) |
+| `mswServer` | module-level | Single MSW Node server shared by all fast tests; reset between each test |
+| `loadEnumSnapshot()` | function | Reads `_docs/00_problem/input_data/enum_spec_snapshot.json` for AC-04 / AC-29 contract assertions |
+| Playwright `page` fixture | per-test | Standard Playwright fixture; extended with `request-log` interceptor that records every outbound request for assertion |
+
+## Test Data Fixtures
+
+| Data Set | Source | Format | Used By |
+|----------|--------|--------|---------|
+| `seed_users` | `e2e/fixtures/seeds.sql` + `tests/fixtures/seed_users.ts` (mirror) | SQL (e2e) / TS (fast) | Groups 1, 13, 14, 16 |
+| `seed_aircraft` | as above | SQL / TS | Group 7, 16 |
+| `seed_flights` | as above | SQL / TS | Group 4, 11, 16 |
+| `seed_classes` | as above | SQL / TS | Group 16 (DetectionClasses + AdminPage class CRUD) |
+| `seed_media` | as above | SQL / TS | Group 3, 5, 16, 17 |
+| `seed_annotations` | as above | SQL / TS | Group 3, 16, 17 |
+| `seed_user_settings` | as above | SQL / TS | Group 4 (AC-21), Group 11 (AC-06) |
+| `enum_spec_snapshot` | `_docs/00_problem/input_data/enum_spec_snapshot.json` (committed) | JSON | Group 2 (AC-04 / AC-29 static contract checks) |
+| `bundle_artifact` | `dist/` from `vite build` | filesystem | Group 7 (AC-11 / AC-31 / AC-33) — static profile only |
+
+### Data Isolation
+
+- **Fast**: MSW handlers are per-test; `server.resetHandlers()` in `afterEach`. React tree unmounted via RTL `cleanup`. In-memory bearer + navigate-to-login spy reset via `afterEach` calls to `setToken(null)` + `setNavigateToLogin(default)`.
+- **E2E**: fresh `docker compose down -v && up -d` per CI run. Within a run, tests grouped into isolation buckets by data set; bucket teardown calls `admin/`'s `POST /test-only/reset` to restore the seed state. Buckets across CI machines run on independent compose stacks. No cross-test order dependencies — any test must boot the bucket's seed snapshot and run alone.
+- **Static**: no shared state; each script run is hermetic.
+
+## Test Reporting
+
+- **Format**: CSV (suite rollup) + JUnit XML (CI ingestion) + Playwright traces (`.zip`) on failure.
+- **Columns** (CSV): `Test ID, Test Name, Profile, Execution Time (ms), Result (PASS|FAIL|SKIP|QUARANTINE), Error Message, Traces to AC, Traces to results_report.md row`.
+- **Output paths**:
+  - `./test-output/fast-report.xml` (JUnit)
+  - `./test-output/fast-report.csv` (rollup written by `scripts/run-tests.sh`)
+  - `./test-output/e2e/` (Playwright artifacts incl. traces)
+  - `./test-output/e2e-report.xml` (JUnit)
+  - `./test-output/static-report.csv`
+  - `./test-output/summary.csv` (suite-level rollup combining the three)
+
+## Acceptance Criteria
+
+**AC-1: Test environment starts (e2e)**
+Given `e2e/docker-compose.suite-e2e.yml`,
+When `docker compose -f e2e/docker-compose.suite-e2e.yml up -d` is executed,
+Then `azaion-ui`, `admin`, `flights`, `annotations`, `detect`, `owm-stub`, `tile-stub`, `test-db`, and `playwright-runner` all become healthy and `http://azaion-ui:80` returns the SPA HTML.
+
+**AC-2: Mock services respond (e2e)**
+Given the test environment is running,
+When the test runner sends `GET http://owm-stub:8081/data/2.5/weather?lat=0&lon=0&appid=test`, and `GET http://tile-stub:8082/1/0/0.png`,
+Then `owm-stub` returns the canned OpenWeatherMap shape and `tile-stub` returns a 256×256 PNG.
+
+**AC-3: MSW intercepts every outbound API call (fast)**
+Given the `fast` profile is configured per this task,
+When any test under the fast profile triggers a fetch to `/api/<service>/...`,
+Then the MSW handler matches, no real network call leaves the process, and `afterEach` resets handlers to defaults.
+
+**AC-4: Fast runner executes**
+Given the fast profile is configured,
+When `bun run test:fast` is executed,
+Then Vitest discovers `**/*.test.ts(x)` files, runs them under jsdom, and produces `./test-output/fast-report.xml` (JUnit) + a coverage summary.
+
+**AC-5: E2E runner executes both browsers**
+Given the e2e environment is running,
+When `bun run test:e2e` is executed inside the `playwright-runner` container,
+Then Playwright runs every `e2e/tests/**/*.e2e.ts` under both Chromium and Firefox projects and writes `./test-output/e2e-report.xml`.
+
+**AC-6: Static runner executes**
+Given the working tree at HEAD,
+When `./scripts/run-tests.sh --static-only` is executed,
+Then it runs `tsc -b --noEmit`, the ripgrep static checks, and `vite build`, and writes `./test-output/static-report.csv`. Exit code is non-zero on any failure.
+
+**AC-7: Reports correctly shaped**
+Given any of the three runners completes,
+When the corresponding JUnit XML or CSV is opened,
+Then it contains the columns listed under "Test Reporting" above and one row per test (or a JUnit `<testcase>` per test).
+
+**AC-8: External-host firewall**
+Given the e2e network has no internet egress,
+When any test inadvertently triggers a fetch to `api.openweathermap.org`, `unpkg.com`, or `*.tile.openstreetmap.org`,
+Then the Playwright route-abort guard fires AND `tile-stub`/`owm-stub` log no requests under those hosts. The test fails with a "leaked external request" assertion.
+
+## Non-Functional Requirements
+
+**Compatibility**
+- Vite 6 (S3), Bun 1.3.11 (S4), React 19, TypeScript 5.7. Vitest 3.x and Playwright 1.49+ are confirmed compatible with this stack via `context7` lookup on Step 6 before installs.
+
+**Security**
+- No production credentials in `seed-fixtures/`. Test bearer + OWM API key are placeholders.
+- Test-only endpoints (`POST /test-only/reset`) gated behind a non-production build flag in the suite — the UI test runner does not embed this concern.
+
+**Operational**
+- Total `fast` suite wall-clock budget ≤ 5 min (per `environment.md` CI/CD Integration).
+- Total `e2e` suite wall-clock budget ≤ 30 min.
+- Each `e2e` test's `Max execution time` is honored per the scenario in `blackbox-tests.md` / etc.
+
+## Constraints
+
+- The runner picks (Vitest, MSW, Playwright) ARE the testability runners environment.md flags as "chosen at Step 5". They MUST match exactly the SPA's stack and the published test profiles. No alternative runner introduced silently.
+- No `src/` import from any test except for the typed wire-contract enum shapes declared in `src/types/index.ts` (per `P9` / `environment.md` § Black-box discipline).
+- `_docs/00_problem/input_data/enum_spec_snapshot.json` is the contract pin for Group 2 (AC-04, AC-29). Any UI drift it carries is asserted by static checks, not silently accepted.
+- The Playwright + MSW pair is the only outbound-network observation surface. No test reads `src/` private state or mounts internal hooks.
+- E2E reports MUST land in `./test-output/` (mounted volume) — never inside the runner container alone.
+
+## Risks & Mitigation
+
+**Risk 1 — MSW Service Worker mode in fast**
+- *Risk*: MSW has two runtimes (Node `setupServer` and browser `setupWorker`); fast tests must use Node (`setupServer`), not the SW mode. Mixing them silently bypasses MSW.
+- *Mitigation*: `tests/msw/server.ts` exports only the Node server. ESLint rule (added in Step 6) bans `import { setupWorker }` from anywhere under `tests/`.
+
+**Risk 2 — Playwright route-abort hides a real regression**
+- *Risk*: AC-8 aborts external requests. If a code-path inadvertently regresses to a hardcoded URL, the abort makes the test still "pass" because the inner promise rejects to a friendly null.
+- *Mitigation*: the abort handler increments a counter; the test fixture asserts `externalHosts.size === 0` in `afterEach`. A leaked request fails the test loudly.
+
+**Risk 3 — SSE testing under MSW**
+- *Risk*: MSW 2.x does not have first-class EventSource support; we lean on a polyfill or a custom stream adapter.
+- *Mitigation*: `tests/helpers/sse-mock.ts` exposes `simulateSseStream(handler, events[])` that returns a fake `EventSource` instance. Tests inject it via `setEventSourceFactory` if `src/api/sse.ts` exports one (NOTE: it does not today — this is flagged for a follow-up "Wrap EventSource constructor in an accessor" testability change in a later refactor run, OR the e2e Playwright path covers SSE definitively while fast uses the polyfill only for narrow cases).
+
+**Risk 4 — Suite-image availability**
+- *Risk*: `azaion/admin:test` etc. tags must exist in the registry; pulling them is the e2e bootstrap. If the parent suite has not published a tagged build, the e2e profile is dead in the water.
+- *Mitigation*: defer to the parent suite's CI; ASK before Step 6 starts if the registry has a published `:test` tag for each service. If absent, gate e2e tests behind a CI-skip flag until the suite publishes.
+
+## Blackbox Tests
+
+| AC Ref | Initial Data / Conditions | What to Test | Expected Behavior | NFR References |
+|--------|--------------------------|--------------|-------------------|----------------|
+| AC-1 | Clean host with Docker 24+ | `docker compose -f e2e/docker-compose.suite-e2e.yml up -d` → wait for healthchecks → `curl http://azaion-ui:80/` | 200 OK with the SPA HTML | Compat |
+| AC-2 | Stubs running | `curl http://owm-stub:8081/data/2.5/weather?...`; `curl http://tile-stub:8082/1/0/0.png` | OWM JSON shape; 256×256 PNG | Compat |
+| AC-3 | Fast profile configured | A trivial component test that triggers `fetch('/api/admin/me')`; no MSW handler registered | The test fails with MSW's "no handler" warning by default; handler override makes it pass | Compat |
+| AC-4 | Fast profile configured | `bun run test:fast` | Exit 0; `./test-output/fast-report.xml` exists with at least one `<testcase>` | Operational |
+| AC-5 | E2E env up | `bun run test:e2e` | Exit 0; `./test-output/e2e-report.xml` exists; Playwright HTML report exists | Operational |
+| AC-6 | Working tree at HEAD | `./scripts/run-tests.sh --static-only` | Exit 0; `./test-output/static-report.csv` exists with one row per check | Compat |
+| AC-7 | All three runners ran | Open each report | Columns match the Test Reporting spec | Operational |
+| AC-8 | E2E env up | Patch a test to `fetch('https://api.openweathermap.org/...')` | Test fails with "leaked external request"; `tile-stub`/`owm-stub` logs show 0 requests under the external host | Security |
+
+## Out of Scope (deferred)
+
+- Writing the actual blackbox / perf / resilience / security / resource-limit test code (that's Step 3 + Step 6 of autodev).
+- Publishing the parent suite's `:test` images (parent suite's CI concern).
+- Visual-regression snapshots — not required by any current AC; can be revisited when AC-40 / AC-25 land in Phase B.
+- Performance-runner bundle-size threshold tuning (already in `run-performance-tests.sh` from Step 4; final thresholds confirmed in Step 7).