mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-23 04:51:12 +00:00
Update demo replay validation and testing documentation
ci/woodpecker/push/02-build-push Pipeline failed
ci/woodpecker/push/02-build-push Pipeline failed
- Modified the autodev state to reflect the current testing phase and details of the new `jetson-e2e` tests. - Enhanced the "How to Test" documentation to provide clearer instructions on the demo replay validation process, including video and tlog alignment steps. - Updated architectural documentation to include the new demo replay operator flow and its dependencies. - Documented the removal of deprecated auto-sync features and clarified the operator-facing UI for replay validation. - Added new entries in the dependencies table for upcoming tasks related to the demo replay flow. These changes improve clarity and usability for operators and developers working with the demo replay system.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: monorepo-e2e
|
||||
description: Syncs the suite-level integration e2e harness (`e2e/docker-compose.suite-e2e.yml`, fixtures, Playwright runner) when component contracts drift in ways that affect the cross-service scenario. Reads `_docs/_repo-config.yaml` to know which suite-e2e artifacts are in play. Touches ONLY suite-e2e files — never per-component CI, docs, or component internals. Use when a component changes a port, env var, public API endpoint, DB schema column, or detection model that the suite e2e exercises.
|
||||
---
|
||||
|
||||
# Monorepo Suite-E2E
|
||||
|
||||
Propagates component changes into the suite-level integration e2e harness. Strictly scoped — never edits docs, component internals, per-component CI configs, or the production deploy compose.
|
||||
|
||||
## Scope — explicit
|
||||
|
||||
| In scope | Out of scope |
|
||||
| -------- | ------------ |
|
||||
| `e2e/docker-compose.suite-e2e.yml` (overlay, healthchecks, seed services) | Production `_infra/deploy/<target>/docker-compose.yml` — `monorepo-cicd` owns it |
|
||||
| `e2e/fixtures/init.sql` (seeded rows that the spec depends on) | Component DB migrations — owned by each component |
|
||||
| `e2e/fixtures/expected_detections.json` (detection baseline) | Detection model itself — owned by `detections/` |
|
||||
| `e2e/runner/tests/*.spec.ts` selector / contract-driven edits | New scenarios (user-driven, not drift-driven) |
|
||||
| `e2e/runner/Dockerfile` / `package.json` Playwright version bumps | Net-new e2e infrastructure (use `monorepo-onboard` or initial scaffolding) |
|
||||
| `.woodpecker/suite-e2e.yml` (suite-level pipeline) | Per-component `.woodpecker/01-test.yml` / `02-build-push.yml` — `monorepo-cicd` owns those |
|
||||
| Suite-e2e leftover entries under `_docs/_process_leftovers/` | Per-component leftovers — owned by each component |
|
||||
|
||||
If a component change needs doc updates too, tell the user to also run `monorepo-document`. If it needs production-deploy or per-component CI updates, run `monorepo-cicd`. This skill **only** updates the suite-e2e surface.
|
||||
|
||||
## Preconditions (hard gates)
|
||||
|
||||
1. `_docs/_repo-config.yaml` exists.
|
||||
2. Top-level `confirmed_by_user: true`.
|
||||
3. `suite_e2e.*` section is populated in config (see "Required config block" below). If absent, abort and ask the user to extend the config via `monorepo-discover`.
|
||||
4. Components-in-scope have confirmed contract mappings (port, public API path, DB tables touched), OR user explicitly approves inferred ones.
|
||||
|
||||
## Required config block
|
||||
|
||||
This skill expects `_docs/_repo-config.yaml` to carry:
|
||||
|
||||
```yaml
|
||||
suite_e2e:
|
||||
overlay: e2e/docker-compose.suite-e2e.yml
|
||||
fixtures:
|
||||
init_sql: e2e/fixtures/init.sql
|
||||
baseline_json: e2e/fixtures/expected_detections.json
|
||||
binary_fixtures:
|
||||
- e2e/fixtures/sample.mp4
|
||||
- e2e/fixtures/model.tar.gz
|
||||
runner:
|
||||
dockerfile: e2e/runner/Dockerfile
|
||||
package_json: e2e/runner/package.json
|
||||
spec_dir: e2e/runner/tests
|
||||
pipeline: .woodpecker/suite-e2e.yml
|
||||
scenario:
|
||||
description: "Upload video → detect → overlays → dataset → DB persistence"
|
||||
components_exercised:
|
||||
- ui
|
||||
- annotations
|
||||
- detections
|
||||
- postgres-local
|
||||
api_contracts:
|
||||
- component: ui
|
||||
path: /api/admin/auth/login
|
||||
- component: annotations
|
||||
path: /api/annotations/media/batch
|
||||
- component: annotations
|
||||
path: /api/annotations/media/{id}/annotations
|
||||
db_tables:
|
||||
- media
|
||||
- annotations
|
||||
- detection
|
||||
- detection_classes
|
||||
model_pin:
|
||||
detections_repo_path: <path-to-model-config-or-classes-source>
|
||||
classes_source: annotations/src/Database/DatabaseMigrator.cs
|
||||
```
|
||||
|
||||
If `suite_e2e:` is missing the skill **stops** — it does not invent a default mapping.
|
||||
|
||||
## Mitigations (M1–M7)
|
||||
|
||||
- **M1** Separation: this skill only touches suite-e2e files; no production deploy compose, no per-component CI, no docs, no component internals.
|
||||
- **M3** Factual vs. interpretive: port, env var, API path, DB column — FACTUAL, read from the components' code. Whether a baseline still matches the model — DEFERRED to the user (the skill flags drift, never silently re-records).
|
||||
- **M4** Batch questions at checkpoints.
|
||||
- **M5** Skip over guess: a component change that doesn't map cleanly to one of the in-scope artifacts → skip and report.
|
||||
- **M6** Assumptions footer + append to `_repo-config.yaml` `assumptions_log`.
|
||||
- **M7** Drift detection: verify every path under `suite_e2e.*` exists on disk; stop if not.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 1: Drift check (M7)
|
||||
|
||||
Verify every file listed under `suite_e2e.*` (excluding `binary_fixtures`, which are gitignored) exists on disk. Missing file → stop and ask:
|
||||
- Run `monorepo-discover` to refresh, OR
|
||||
- Skip the missing artifact (recorded in report)
|
||||
|
||||
For `binary_fixtures` paths that are absent (expected — they live in S3/LFS), check whether `expected_detections.json._meta.video_sha256` is still a `TBD-...` placeholder. If yes, surface this as a known leftover (`_docs/_process_leftovers/2026-04-22_suite-e2e-binary-fixtures.md`) and continue.
|
||||
|
||||
### Phase 2: Determine scope
|
||||
|
||||
Same as `monorepo-cicd` Phase 2 — ask the user, or auto-detect. For **auto-detect**, flag commits that touch suite-e2e-relevant concerns:
|
||||
|
||||
| Commit pattern | Suite-e2e impact |
|
||||
| -------------- | ---------------- |
|
||||
| New port exposed by `<component>` | Healthcheck override may change in `e2e/docker-compose.suite-e2e.yml` |
|
||||
| New required env var on `<component>` | `e2e/docker-compose.suite-e2e.yml` `e2e-runner` env block + `init.sql` seed |
|
||||
| Public API path renamed / removed | Spec selector / API call path in `e2e/runner/tests/*.spec.ts` |
|
||||
| DB schema column renamed in a `db_tables` entry | `init.sql` column reference + spec `pg.query` text |
|
||||
| New required DB table referenced by spec | `init.sql` insert block (skip if owned by component migration) |
|
||||
| Detection model rev change in `detections/` | `expected_detections.json` `_meta.model.revision` + flag baseline as stale |
|
||||
| New canonical detection class added | `expected_detections.json._meta` annotation |
|
||||
|
||||
Present the flagged list; confirm.
|
||||
|
||||
### Phase 3: Classify changes per component
|
||||
|
||||
| Change type | Target suite-e2e files |
|
||||
| ----------- | ---------------------- |
|
||||
| Port / env var change | `e2e/docker-compose.suite-e2e.yml` |
|
||||
| API path / contract change | `e2e/runner/tests/*.spec.ts` |
|
||||
| DB schema reference change | `e2e/fixtures/init.sql` and spec SQL queries |
|
||||
| Model / class catalog change | `e2e/fixtures/expected_detections.json` (mark `_meta.fixture_version` bump + leftover entry for binary refresh) |
|
||||
| Playwright dependency drift | `e2e/runner/package.json` + `e2e/runner/Dockerfile` |
|
||||
| Suite scenario steps gone stale | **Stop and ask** — scenario edits are user-driven, not drift-driven |
|
||||
|
||||
### Phase 4: Apply edits
|
||||
|
||||
Edit each in-scope file. After each batch, run `ReadLints` on touched files. Do NOT run the suite e2e itself — that's a downstream pipeline operation, not a sync-skill responsibility.
|
||||
|
||||
For `expected_detections.json`: when the model revision changes, the skill **does not** re-record the baseline — the binary fixture cannot be regenerated from the dev environment. Instead:
|
||||
1. Set `_meta.model.revision` to the new revision.
|
||||
2. Set `_meta.fixture_version` to a new bumped version with a `-stale` suffix (e.g., `0.2.0-stale`).
|
||||
3. Append a new entry to `_docs/_process_leftovers/` describing the required re-record.
|
||||
4. Leave `expected.by_class` untouched — the spec's tolerance check will fail loudly until the binary refresh lands.
|
||||
|
||||
### Phase 5: Update assumptions log
|
||||
|
||||
Append a new `assumptions_log:` entry to `_docs/_repo-config.yaml` recording:
|
||||
- Date, components in scope, which suite-e2e files were touched
|
||||
- Any inferred contract mappings still tagged `confirmed: false`
|
||||
- Any leftover entries created
|
||||
|
||||
### Phase 6: Report
|
||||
|
||||
Render a Choose-format summary of the synced files, surface any `_process_leftovers/` entries created, and end. Do NOT auto-commit.
|
||||
|
||||
## Self-verification
|
||||
|
||||
- [ ] No file outside `e2e/`, `.woodpecker/suite-e2e.yml`, or `_docs/_process_leftovers/` was edited
|
||||
- [ ] `_docs/_repo-config.yaml` `suite_e2e:` block was not silently mutated except for `assumptions_log` append
|
||||
- [ ] `expected_detections.json` was not re-recorded (only metadata bumped + leftover added)
|
||||
- [ ] Every spec edit traces to a flagged commit pattern in Phase 2
|
||||
- [ ] `ReadLints` clean on every touched file
|
||||
|
||||
## Failure handling
|
||||
|
||||
Same retry / escalation protocol as `monorepo-cicd` — see `protocols.md`. The most common failure mode is the binary-fixture leftover (sample.mp4 missing or SHA-mismatched); this skill does not attempt to resolve it, only surfaces it.
|
||||
Reference in New Issue
Block a user