mirror of
https://github.com/azaion/annotations.git
synced 2026-06-21 11:01:07 +00:00
docs+src: complete Steps 1-3 outcomes + auth re-sync baseline
This commit captures everything produced during autodev existing-code Steps 1 (Document), 2 (Architecture Baseline Scan), and 3 (Test Spec), together with the targeted auth + CORS re-sync triggered on 2026-05-14 when codebase drift was detected at Step 4 entry. None of this work was previously committed. Step 1 (Document) — 50+ _docs/02_document/ files: problem, solution, architecture, system flows, glossary, module-layout, per-component specs (01..06), modules, deployment, diagrams, data model, FINAL report, verification log, discovery. Step 2 (Architecture Baseline) — architecture_compliance_baseline.md. Verdict PASS_WITH_WARNINGS (0 Critical, 0 High, 1 Medium, 2 Low). No High/Critical findings; auto-chained to Step 3 per existing-code flow. Step 3 (Test Spec) — _docs/02_document/tests/* (67 scenarios across blackbox, security, resilience, resource-limit, performance), plus e2e/docker-compose.test.yml, e2e/seed/run.sh, scripts/run-tests.sh, scripts/run-performance-tests.sh. Coverage 88% over the active scope (40 of 45 items covered, 6 RB-deferred, 5 documented-as-uncovered). Targeted auth + CORS re-sync — replaces the deleted in-house token issuer with a JWKS-verifier model. AuthController and TokenService removed; JwtExtensions switched from HS256 symmetric to ES256 over admin's JWKS. ConfigurationResolver and CorsConfigurationValidator added under src/Infrastructure/. ADR-002 and ADR-006 retired; SEC-01, SEC-02, SEC-03 marked Closed. One new testability risk recorded in architecture.md Open Risks Section 6 (JWKS HTTPS gating). Source changes: - src/Auth/JwtExtensions.cs (modified) — ES256, JWKS, alg pinning - src/Program.cs (modified) — DI wiring for ConfigurationResolver and CorsConfigurationValidator - src/Controllers/AuthController.cs (deleted) — no in-service issuance - src/Services/TokenService.cs (deleted) — same - src/Infrastructure/ConfigurationResolver.cs (new) - src/Infrastructure/CorsConfigurationValidator.cs (new) - .env.example (new) — required env var documentation - .gitignore (updated) Cross-repo coordination: _docs/cross-repo/flights_h1_h2_h3_change_spec captures the change-spec for downstream services that consumed the now deleted /auth endpoints. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,208 @@
|
||||
# Codebase discovery — Azaion.Annotations
|
||||
|
||||
## Canonical product documentation (external)
|
||||
|
||||
Suite-level API and integration reference (maintained with the monorepo):
|
||||
|
||||
`/Users/obezdienie001/dev/azaion/suite/_docs/01_annotations.md`
|
||||
(Relative from this repo: `../_docs/01_annotations.md`.)
|
||||
|
||||
This `_docs/02_document/` run is **bottom-up from code**; keep `01_annotations.md` aligned when HTTP contracts or integration behavior change.
|
||||
|
||||
---
|
||||
|
||||
## Directory tree (source)
|
||||
|
||||
```
|
||||
annotations/
|
||||
├── README.md
|
||||
├── src/
|
||||
│ ├── Azaion.Annotations.csproj
|
||||
│ ├── Dockerfile
|
||||
│ ├── GlobalUsings.cs
|
||||
│ ├── Program.cs
|
||||
│ ├── Auth/
|
||||
│ ├── Controllers/
|
||||
│ ├── Database/
|
||||
│ │ ├── AppDataConnection.cs
|
||||
│ │ ├── DatabaseMigrator.cs
|
||||
│ │ └── Entities/
|
||||
│ ├── DTOs/
|
||||
│ ├── Enums/
|
||||
│ ├── Middleware/
|
||||
│ └── Services/
|
||||
└── _docs/
|
||||
```
|
||||
|
||||
Ignored per scan policy: `bin/`, `obj/`, `.git`, `node_modules`, `__pycache__`.
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Area | Choice |
|
||||
|------|--------|
|
||||
| Language | C# / .NET (`net10.0` in csproj) |
|
||||
| Host | ASP.NET Core minimal hosting + controllers |
|
||||
| ORM / DB | Linq2DB + Npgsql → PostgreSQL |
|
||||
| Auth | JWT Bearer (`Microsoft.AspNetCore.Authentication.JwtBearer`) |
|
||||
| Messaging | RabbitMQ Streams (`RabbitMQ.Stream.Client`), MessagePack |
|
||||
| API docs | Swashbuckle (Swagger / OpenAPI) |
|
||||
| Hashing | `System.IO.Hashing` (annotation id / media paths) |
|
||||
|
||||
---
|
||||
|
||||
## Package manifest
|
||||
|
||||
- `src/Azaion.Annotations.csproj` — single web project.
|
||||
|
||||
---
|
||||
|
||||
## Config and operations
|
||||
|
||||
| Artifact | Role |
|
||||
|----------|------|
|
||||
| `src/Dockerfile` | Container build |
|
||||
| `Program.cs` | `DATABASE_URL`, `JWT_ISSUER` / `JWT_AUDIENCE` / `JWT_JWKS_URL`, `CorsConfig:*`, RabbitMQ env vars (`RABBITMQ_*`), migrator on startup. All required vars are resolved through `ConfigurationResolver` (fail-fast). |
|
||||
| `.vscode/launch.json` | Local debugging (if present) |
|
||||
|
||||
No `.github/workflows` in this repository (CI may live in suite/monorepo).
|
||||
|
||||
---
|
||||
|
||||
## Entry points
|
||||
|
||||
- **`Program.cs`** — service registration, JWT, CORS, Swagger, `DatabaseMigrator.Migrate`, middleware pipeline, `MapControllers()`, `/health`.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
No `*.Tests.csproj` or `tests/` tree in this workspace — **no automated test project** discovered.
|
||||
|
||||
---
|
||||
|
||||
## Existing documentation in repo
|
||||
|
||||
- Root `README.md` — points to suite `01_annotations.md`.
|
||||
- `src/README.md` — short service blurb + link to root README.
|
||||
|
||||
---
|
||||
|
||||
## Module boundaries (revised — aligned with `01_annotations.md`)
|
||||
|
||||
The suite file is organized around **annotation lifecycle**, **media**, **settings/camera**, **SSE**, **RabbitMQ sync**, and **auth** (JWT refresh). The codebase splits the same concerns across controllers/services; **dataset** and **detection classes** are additional HTTP surfaces referenced from suite `09_dataset_explorer.md` / UI.
|
||||
|
||||
| # | Module (doc file) | Primary code | Suite `01_annotations.md` anchor |
|
||||
|---|-------------------|--------------|-----------------------------------|
|
||||
| 1 | `wire-enums.md` | `src/Enums/*` | “Wire format”, enum tables |
|
||||
| 2 | `database-layer.md` | `src/Database/*` | Annotation identity, tables, `SilentDetection` / `GenerateAnnotatedImage` columns |
|
||||
| 3 | `common-infrastructure.md` | `PathResolver`, `ErrorHandlingMiddleware`, `PaginatedResponse`, `ErrorResponse`, `GlobalUsings.cs` | File paths for image/label/thumb/results; error JSON shape |
|
||||
| 4 | `auth-identity.md` | `JwtExtensions` (verifier-only over admin's JWKS) | JWT forward only — refresh is admin's responsibility (annotations no longer hosts `/auth/refresh`) |
|
||||
| 5 | `media-service.md` | `MediaService`, `MediaController`, media DTOs | §7–10 POST/GET/DELETE media, batch upload |
|
||||
| 6 | `annotations-service.md` | `AnnotationService`, `AnnotationsController` (REST + static files, not SSE) | §1–6 annotations CRUD/query |
|
||||
| 7 | `dataset-service.md` | `DatasetService`, `DatasetController`, dataset DTOs | Cross-ref §3 note (DATASET); `09_dataset_explorer.md` |
|
||||
| 8 | `settings-metadata-service.md` | `SettingsService`, `SettingsController`, `ClassesController`, settings DTOs | §11–12 camera; directories/system/user settings; GET `/classes` |
|
||||
| 9 | `sse-realtime.md` | `AnnotationEventService`, SSE action on `AnnotationsController` | §SSE `GET /annotations/events`, `AnnotationEvent` |
|
||||
| 10 | `rabbitmq-stream-sync.md` | `FailsafeProducer`, `RabbitMqConfig`, `DTOs/QueueMessages.cs`, queue entity | §Annotation Sync, Failsafe, Stream |
|
||||
| 11 | `composition-program.md` | `Program.cs` | Wiring, env defaults, startup migrate |
|
||||
|
||||
**DTOs** (`src/DTOs/`) are documented **inside the module that owns the HTTP contract**, with cross-links (no separate monolithic “DTOs” module).
|
||||
|
||||
See `modules/README.md` for the same index and file naming.
|
||||
|
||||
---
|
||||
|
||||
## Module dependency graph (revised)
|
||||
|
||||
```mermaid
|
||||
flowchart BT
|
||||
WE[wire-enums]
|
||||
DB[database-layer]
|
||||
CI[common-infrastructure]
|
||||
AUTH[auth-identity]
|
||||
MEDIA[media-service]
|
||||
ANN[annotations-service]
|
||||
DS[dataset-service]
|
||||
SET[settings-metadata-service]
|
||||
SSE[sse-realtime]
|
||||
RMQ[rabbitmq-stream-sync]
|
||||
PRG[composition-program]
|
||||
|
||||
WE --> DB
|
||||
DB --> CI
|
||||
DB --> MEDIA
|
||||
DB --> ANN
|
||||
DB --> DS
|
||||
DB --> SET
|
||||
CI --> MEDIA
|
||||
CI --> ANN
|
||||
AUTH --> MEDIA
|
||||
AUTH --> ANN
|
||||
AUTH --> DS
|
||||
AUTH --> SET
|
||||
MEDIA --> ANN
|
||||
ANN --> SSE
|
||||
ANN --> RMQ
|
||||
SET --> CI
|
||||
SSE --> ANN
|
||||
RMQ --> ANN
|
||||
RMQ --> MEDIA
|
||||
RMQ --> DB
|
||||
MEDIA --> PRG
|
||||
ANN --> PRG
|
||||
DS --> PRG
|
||||
SET --> PRG
|
||||
SSE --> PRG
|
||||
RMQ --> PRG
|
||||
AUTH --> PRG
|
||||
CI --> PRG
|
||||
```
|
||||
|
||||
Edges are “depends on for types, DB, paths, or events” (approximate). `ClassesController` reads DB directly — captured under **settings-metadata-service** for doc cohesion (small surface).
|
||||
|
||||
---
|
||||
|
||||
## Topological order (document skill Step 1)
|
||||
|
||||
1. `wire-enums`
|
||||
2. `database-layer`
|
||||
3. `common-infrastructure`
|
||||
4. `auth-identity`
|
||||
5. `media-service`
|
||||
6. `annotations-service`
|
||||
7. `dataset-service`
|
||||
8. `settings-metadata-service`
|
||||
9. `sse-realtime`
|
||||
10. `rabbitmq-stream-sync`
|
||||
11. `composition-program`
|
||||
|
||||
---
|
||||
|
||||
## Notes for downstream steps
|
||||
|
||||
- **Component assembly (Step 2):** expect components such as “Annotations + sync”, “Media”, “Dataset”, “Settings”, “Platform (auth+db+infra)” — refine with user confirmation (BLOCKING gate).
|
||||
- **RabbitMQ:** `RabbitMqConfig` class lives in `FailsafeProducer.cs`; document in `rabbitmq-stream-sync` module.
|
||||
|
||||
---
|
||||
|
||||
## Suite spec cross-check (`suite/_docs/01_annotations.md`)
|
||||
|
||||
Canonical product/API narrative for this service. Use it when writing module and component docs.
|
||||
|
||||
| Topic in suite doc | Adds context for this repo |
|
||||
|--------------------|------------------------------|
|
||||
| Annotation identity (hash id, image + YOLO label, `Time` / `CreatedDate`) | `annotations-service` + `database-layer` + `common-infrastructure` (`PathResolver`) |
|
||||
| Wire enums as integers | `wire-enums` module |
|
||||
| REST §1–6 vs §7–10 | `annotations-service` vs `media-service` |
|
||||
| Settings §11–12, directories | `settings-metadata-service` |
|
||||
| SSE | `sse-realtime` |
|
||||
| Failsafe + RabbitMQ Stream | `rabbitmq-stream-sync` |
|
||||
| Dataset note (DATASET permission) | `dataset-service` |
|
||||
|
||||
**Drifts spotted (suite vs current code)** — reconcile in suite or in code as you prefer:
|
||||
|
||||
1. **POST /annotations user id:** Suite lists `UserId` on request body; code uses JWT `NameIdentifier` (`annotations-service`).
|
||||
2. **GET /annotations filter:** Suite lists `missionId`; code has `FlightId` and a partial filter — see `annotations-service` module.
|
||||
|
||||
Module docs (`modules/*.md`) carry contract detail per slice; this section stays the cross-file index.
|
||||
@@ -0,0 +1,151 @@
|
||||
# Step 4 — Verification Log
|
||||
|
||||
Verification pass over `_docs/02_document/` against `src/` source.
|
||||
|
||||
## Scope
|
||||
|
||||
Documents verified:
|
||||
|
||||
- `architecture.md`
|
||||
- `system-flows.md`
|
||||
- `data_model.md`
|
||||
- `deployment/{containerization,ci_cd_pipeline,environment_strategy,observability}.md`
|
||||
- `diagrams/flows/{flow_annotation_create,flow_sse_subscription,flow_failsafe_drain}.md`
|
||||
- (sanity re-check only) `module-layout.md`, `components/*/description.md`, `modules/*.md`
|
||||
|
||||
## Method
|
||||
|
||||
For each generated artifact:
|
||||
|
||||
1. Extracted code-entity references (controllers, services, methods, DTOs, env vars, table/column names, route paths).
|
||||
2. Cross-referenced each against the actual source (`src/Program.cs`, `src/Controllers/*`, `src/Services/*`, `src/Database/*`, `src/Enums/*`, `src/DTOs/*`, `.woodpecker/build-arm.yml`, `src/Dockerfile`).
|
||||
3. Re-traced each system flow's mermaid sequence against the corresponding service/controller code.
|
||||
4. Listed corrections, applied them inline to the affected files, and recorded them below.
|
||||
|
||||
## Counts
|
||||
|
||||
| Item | Verified | Corrected | Open question |
|
||||
|------|----------|-----------|----------------|
|
||||
| Controllers + their routes | 6 | 0 | 0 |
|
||||
| Services + their public methods | 8 | 0 | 0 |
|
||||
| DB tables / columns | 9 / ~60 | 0 | 5 (lazy upsert / `media.duration` / class catalog mutability / id collision / outbox JSON shape) |
|
||||
| Enums | 7 | 0 | 0 |
|
||||
| Env vars | 8 | 0 | 0 |
|
||||
| Flows | 8 | 4 (F1, F7, F8, dependencies table) | 6 (consolidated below) |
|
||||
| ADRs | 7 | 1 (ADR-004 hash details) | 0 |
|
||||
|
||||
Module-level coverage: **11 / 11 modules** documented; **6 / 6 components** assembled.
|
||||
|
||||
## Corrections applied inline
|
||||
|
||||
### `architecture.md`
|
||||
|
||||
1. **Internal communication table**: tightened to reflect that SSE publish + outbox enqueue happen **only on `CreateAnnotation`**; outbox enqueue is gated by `system_settings.silent_detection`. Added explicit row noting `DatasetService` writes are silent on SSE/outbox today.
|
||||
2. **ADR-004 (annotation id hash)**: replaced "hash of bytes" with the actual `ComputeHash` strategy — `XxHash64` over a deterministic sample (length prefix + head/middle/tail 1 KB for inputs > 3072 bytes; full bytes otherwise). Documented collision implication.
|
||||
3. **Open Architectural Risks**: rewrote with verified findings — silent Update/Delete/dataset paths, `silent_detection` semantics, F1 non-atomicity, static `EnqueueAsync` vs project rule.
|
||||
4. **Section 4 "Data flow summary"**: split into Create-only / Update-and-friends / read paths, removed the inaccurate claim that thumbnails are produced inline by Create.
|
||||
|
||||
### `system-flows.md`
|
||||
|
||||
1. **F1 sequence + data flow + error scenarios**: replaced with the verified ordering — image file → optional media row → annotation → detections (`BulkCopyAsync`) → label file → SSE publish → conditional outbox enqueue. Removed thumbnail write from the Create path.
|
||||
2. **F7 ("Reset call missed" risk)**: removed — verified that `SettingsService` calls `pathResolver.Reset()` at lines 71 and 85 of `Services/SettingsService.cs`. Replaced with a "Verified" note.
|
||||
3. **F8 (Dataset bulk status)**: rewrote — `DatasetService.UpdateStatus` and `BulkUpdateStatus` issue direct `UPDATE annotations SET status` statements only. **They do NOT publish SSE and do NOT enqueue the outbox.** Updated routes (`PATCH /dataset/{id}/status`, `POST /dataset/bulk-status`) and error scenarios accordingly.
|
||||
4. **Flow Dependencies table**: corrected F1 row (gating + Create-only), F3 row (only F1 Create publishes), F8 row (no SSE / no outbox today).
|
||||
|
||||
### `diagrams/flows/flow_annotation_create.md`
|
||||
|
||||
- Replaced sequence + flowchart to match the verified F1 ordering (image first, optional media, label, SSE, conditional outbox); thumbnail removed.
|
||||
- Added note that Update/UpdateStatus/Delete are silent today.
|
||||
|
||||
### `diagrams/flows/flow_sse_subscription.md`, `flow_failsafe_drain.md`
|
||||
|
||||
- No structural corrections needed; spot-checked sequence vs `AnnotationsController.Events`, `AnnotationEventService`, `FailsafeProducer.EnqueueAsync`. Notes already capture multi-drainer dedupe and channel-unbounded back-pressure concerns.
|
||||
|
||||
### `data_model.md`
|
||||
|
||||
- No structural corrections; verified every column name and default against `Database/DatabaseMigrator.cs` and `Database/Entities/*.cs`. Spot-quirk (`detection_classes` ids 9 + 10 share `#000080`) is pre-existing and noted.
|
||||
|
||||
### `deployment/*`
|
||||
|
||||
- No structural corrections; verified `.woodpecker/build-arm.yml` step-by-step, `Dockerfile` two-stage build, `Program.cs` env-var fallbacks.
|
||||
|
||||
## Confirmed entities (sample — full list traced during the pass)
|
||||
|
||||
Controllers and routes (file:line where attributes were inspected):
|
||||
|
||||
- `AnnotationsController` — `Controllers/AnnotationsController.cs:10–80` — `[Route("annotations")]`, `[Authorize(Policy = "ANN")]`, all listed routes match.
|
||||
- `MediaController` — `Controllers/MediaController.cs:10–55` — `[Route("media")]`, `[Authorize(Policy = "ANN")]`, routes: `POST`, `POST /batch`, `GET`, `GET /{id}/file`, `DELETE /{id}`.
|
||||
- `DatasetController` — `Controllers/DatasetController.cs:9–41` — `[Route("dataset")]`, `[Authorize(Policy = "DATASET")]`, routes: `GET`, `GET /{annotationId}`, `PATCH /{annotationId}/status`, `POST /bulk-status`, `GET /class-distribution`.
|
||||
- `SettingsController` — `Controllers/SettingsController.cs:10–66` — `[Route("settings")]`, segments `system`, `directories`, `camera`, `user` each with GET + PUT.
|
||||
- `ClassesController` — `Controllers/ClassesController.cs:9–13` — `[Route("classes")]`, `[Authorize]`, single `[HttpGet]`.
|
||||
- `AuthController` — **removed** in the auth refactor; annotations no longer mints or refreshes tokens. `JwtExtensions.AddJwtAuth` (verifier-only, ES256 over admin's JWKS) is the sole auth wiring in `Program.cs`.
|
||||
|
||||
Services:
|
||||
|
||||
- `AnnotationService.CreateAnnotation` (`Services/AnnotationService.cs:13–104`) — verified sequence used to rewrite F1.
|
||||
- `AnnotationService.UpdateAnnotation` / `UpdateStatus` / `DeleteAnnotation` — verified that none publish SSE or enqueue outbox.
|
||||
- `DatasetService.UpdateStatus` / `BulkUpdateStatus` (`Services/DatasetService.cs:75–94`) — verified silent on SSE / outbox.
|
||||
- `SettingsService` — verified `pathResolver.Reset()` calls at lines 71, 85.
|
||||
- `FailsafeProducer.EnqueueAsync` — confirmed as the public outbox-write helper, called by `AnnotationService.CreateAnnotation` only.
|
||||
|
||||
Tables / migrator (`Database/DatabaseMigrator.cs`):
|
||||
|
||||
- All 9 tables referenced in `data_model.md` exist with the columns and defaults as documented; idempotent `CREATE TABLE IF NOT EXISTS` + `ALTER TABLE … IF NOT EXISTS`; `detection_classes` seed of 19 rows with `ON CONFLICT DO NOTHING`.
|
||||
|
||||
Env vars (`Program.cs`):
|
||||
|
||||
- Required (fail-fast via `ConfigurationResolver.ResolveRequiredOrThrow`): `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL`.
|
||||
- Optional with defaults: `RABBITMQ_HOST`, `RABBITMQ_STREAM_PORT`, `RABBITMQ_PRODUCER_USER`, `RABBITMQ_PRODUCER_PASS`, `RABBITMQ_STREAM_NAME`.
|
||||
- CORS: `CorsConfig:AllowedOrigins` (string array) + `CorsConfig:AllowAnyOrigin` (bool); `CorsConfigurationValidator.EnsureSafeForEnvironment` blocks startup in `Production` when origins are empty and `AllowAnyOrigin` is not explicitly set.
|
||||
|
||||
CI (`.woodpecker/build-arm.yml`):
|
||||
|
||||
- `event: [push, manual]`, `branch: [dev, stage, main]`, `platform: arm64`, secret refs, `${BRANCH}-arm` tag, OCI image labels — all verified.
|
||||
|
||||
## Stakeholder resolutions (closed 2026-05-14)
|
||||
|
||||
The six open questions surfaced by this pass were resolved with the maintainer. Authoritative wording lives in `architecture.md` (ADR-004, ADR-008..ADR-011 + Refactor Backlog RB-01..RB-06). Quick map:
|
||||
|
||||
| Question | Resolution | Tracked |
|
||||
|----------|------------|---------|
|
||||
| Are silent Update/Delete/dataset-status changes intentional? | No — World B is the design; the drainer (`FailsafeProducer.cs:108–123`) was already plumbed for `Validated` + `Deleted` ops, the producer side was never wired in the new HTTP backend (legacy WPF UI did this directly). Wire all mutations to publish + enqueue. | ADR-009 / RB-01 |
|
||||
| `silent_detection` semantics? | Remove the flag entirely — superseded by the suite e2e harness. | ADR-010 / RB-02 |
|
||||
| F1 atomicity (FS / DB / outbox)? | Adopt a business-transaction wrapper (transactional outbox); FS writes go post-commit. | ADR-008 / RB-03 |
|
||||
| `XxHash64` over sample collision risk? | Switch to `XxHash3.Hash128` over the same sample (file-size-independent — videos can be 3–5 GB). | ADR-004 / RB-04 |
|
||||
| `FailsafeProducer.EnqueueAsync` static + DB I/O? | Accept as-is; documented `coderule.mdc` deviation. | (no refactor) |
|
||||
| `detection_classes` static or admin-managed? | Admin-managed with read-through cache (`PathResolver`-style `Reset()`). | ADR-011 / RB-06 |
|
||||
|
||||
### Additional finding while verifying #1
|
||||
|
||||
- `FailsafeProducer.cs:138` has an empty `catch { }` that swallows `IOException` on image read and emits a stream message with `image = null`. Direct `coderule.mdc` violation ("never suppress errors silently"). Operationally invisible failure mode. Tracked as RB-05 (architecture doc).
|
||||
|
||||
## Step 4.5 follow-on resolutions (closed 2026-05-14)
|
||||
|
||||
Confirmed alongside the Step 4.5 condensed-view approval:
|
||||
|
||||
| Question | Resolution | Tracked |
|
||||
|----------|------------|---------|
|
||||
| Suite vs code: `Flight` (code) vs `mission` (suite spec) | Rename code → `Mission*`; suite stays canonical | ADR-012 / RB-07 |
|
||||
| Stream consumer dedupe contract owner | This service owns it; dedupe by `(annotationId, operation, dateTime)` baked into the wire message | ADR-013 / RB-09 |
|
||||
| Hard-delete vs soft-delete | Soft-delete: status → `Deleted (40)`, files relocated to a new `deleted_dir` | ADR-009 (folded in) / RB-01 |
|
||||
| Tight coupling 04 Dataset ↔ 01 Annotations REST | Decouple — dataset writes flow through `AnnotationService` via a public domain interface | RB-08 |
|
||||
|
||||
## Remaining gaps and uncertainties (carried into Step 6 problem extraction)
|
||||
|
||||
1. **`media.duration` format**: TEXT NOT NULL is permissive; format is unspecified.
|
||||
2. **Lazy-upsert semantics** for `system_settings` / `directory_settings` / `camera_settings` — confirm services initialize defaults vs rely on user-driven inserts.
|
||||
3. **`UserId` body field vs JWT subject** drift — reconcile in suite spec or in code.
|
||||
4. **No automated tests in repo**: addressed by autodev Phase A Steps 3–7.
|
||||
|
||||
## Completeness score
|
||||
|
||||
- 11 / 11 modules documented (`modules/*.md`).
|
||||
- 6 / 6 components assembled (`components/*/description.md`).
|
||||
- 1 / 1 module-layout file (`module-layout.md`).
|
||||
- 1 / 1 architecture file (`architecture.md`).
|
||||
- 1 / 1 system-flows file (`system-flows.md`) covering 8 flows.
|
||||
- 1 / 1 data-model file (`data_model.md`) covering 9 tables.
|
||||
- 4 / 4 deployment files (`deployment/*.md`).
|
||||
- 3 flow diagrams (F1, F3, F4) in `diagrams/flows/`.
|
||||
|
||||
**Score: 100% of modules + components covered.** Remaining open items are behavioral questions, not coverage gaps.
|
||||
@@ -0,0 +1,183 @@
|
||||
# Azaion.Annotations — Documentation Report
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Reverse-engineered the `Azaion.Annotations` codebase bottom-up — 11 module docs → 6 component specs → 1 system architecture + 8 verified flows + ER diagram + deployment + glossary, then synthesized a retrospective `solution.md` and a 5-file problem extraction. Verification surfaced 8 behavioral discrepancies between code and the suite-level `01_annotations.md` narrative; all 8 were resolved with stakeholder decisions, captured as 13 ADRs and 9 Refactor Backlog items (RB-01..RB-09) inside `architecture.md`.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
`Azaion.Annotations` is the suite's annotation lifecycle service. It is the single owner of the `annotations` table, the YOLO label files on disk, and the lifecycle event stream. Three independent consumers (Annotator UI, AI training pipeline, admin sync worker) need the same data shaped differently — push for humans (SSE), durable-pull for machines (RabbitMQ Stream) — and this service is the one place that reconciles those needs. See `_docs/00_problem/problem.md`.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Single .NET 10 ASP.NET Core service, single PostgreSQL state-of-record, content-addressed filesystem cache, in-process SSE channel + transactional outbox drained to RabbitMQ Stream by a hosted background service. JWT bearer with three policies (`ANN`, `DATASET`, `ADM`). Idempotent boot-time DDL migrator removes a separate migration deploy step.
|
||||
|
||||
13 ADRs captured the choices; 9 Refactor Backlog items capture the agreed-upon next moves. Full detail: `_docs/02_document/architecture.md`.
|
||||
|
||||
**Technology stack**: .NET 10 + ASP.NET Core + Linq2DB + Npgsql + JwtBearer + RabbitMQ.Stream.Client + MessagePack + xxHash3 (per RB-04) on PostgreSQL 13+.
|
||||
|
||||
**Deployment**: ARM64 multi-arch Docker image; branch-driven Woodpecker CI emits `${BRANCH}-arm` tags; orchestrator-managed at the suite level.
|
||||
|
||||
## Component Summary
|
||||
|
||||
| # | Component | Purpose | Dependencies | Epic |
|
||||
|---|-----------|---------|--------------|------|
|
||||
| 01 | annotations-rest | Annotation CRUD + image/thumbnail file routes; YOLO label write; lifecycle producer | 02, 06 | TBD (Phase B) |
|
||||
| 02 | annotations-realtime-sync | In-process SSE channel + transactional outbox + `FailsafeProducer` to RabbitMQ Stream | 06 | TBD (Phase B) |
|
||||
| 03 | media | Multipart media upload (single + batch), file download, soft delete | 06 | TBD (Phase B) |
|
||||
| 04 | dataset | Dataset exploration: filters, class distribution, bulk status writes | 06 (today couples 01 — RB-08) | TBD (Phase B) |
|
||||
| 05 | settings-metadata | System / directory / camera / user settings + detection class catalog | 06 | TBD (Phase B) |
|
||||
| 06 | platform | Composition root, JWT, error envelope, path resolver, DB migrator | — | TBD (Phase B) |
|
||||
|
||||
**Implementation order** (logical layer dependency, not "to-build" — the codebase already exists):
|
||||
1. `06 platform` is the foundation; every other component imports from it.
|
||||
2. `02 annotations-realtime-sync` is the lifecycle substrate `01` and (post RB-01) `04` feed into.
|
||||
3. `01 annotations-rest`, `03 media`, `05 settings-metadata` sit on top of `06` directly.
|
||||
4. `04 dataset` reads the storage `01` writes; today via direct DB coupling, post RB-08 via `AnnotationService`.
|
||||
|
||||
**Refactor sequencing** is what Phase B will plan (Steps 8 onward); the Refactor Backlog already orders the items by impact:
|
||||
|
||||
1. RB-01 (lifecycle observability across mutations) — unblocks RB-09 stream contract and most Step 14 audit work.
|
||||
2. RB-03 (transactional outbox wrapper) — required before RB-01 is testable.
|
||||
3. RB-04 (xxHash3.Hash128) — small, isolated, can run parallel.
|
||||
4. RB-02 (drop `silent_detection`) — small cleanup, after RB-01.
|
||||
5. RB-08 (decouple `04 dataset` writes) — unblocks soft-delete read filtering.
|
||||
6. RB-07 (`Flight*` → `Mission*` rename) — high-touch, needs coordination with suite consumers.
|
||||
7. RB-06 (admin-managed detection classes) — feature, can run parallel.
|
||||
8. RB-05 (replace `catch { }` in `FailsafeProducer.cs:138`) — trivial, anytime.
|
||||
9. RB-09 (stream dedupe contract `(annotation_id, operation, date_time)`) — depends on RB-01.
|
||||
|
||||
## System Flows
|
||||
|
||||
| Flow | Description | Key Components |
|
||||
|------|-------------|---------------|
|
||||
| F1 | Annotation create — content-address image, persist, write label, fan-out (SSE + outbox) | 01, 02, 06 |
|
||||
| F2 | Annotation listing / detail | 01, 06 |
|
||||
| F3 | Real-time SSE subscription per mission | 01, 02, 06 |
|
||||
| F4 | Failsafe outbox drain (background loop) → RabbitMQ Stream | 02, 06 |
|
||||
| F5 | Media upload (single + batch) | 03, 06 |
|
||||
| F6 | Auth: login + refresh token rotation | 06 |
|
||||
| F7 | Directory settings change → `pathResolver.Reset()` invariant | 05, 06 |
|
||||
| F8 | Dataset bulk status update | 04, 06 |
|
||||
|
||||
Full sequence diagrams: `_docs/02_document/system-flows.md` and `_docs/02_document/diagrams/flows/`.
|
||||
|
||||
## Risk Summary
|
||||
|
||||
Risks here are the operational / behavioral risks captured during verification + Step 14 candidates from `security_approach.md`. They live in `architecture.md` (Risks + Refactor Backlog) and will be mirrored into a formal risk register during Phase B Step 12.
|
||||
|
||||
| Level | Count | Key Risks |
|
||||
|-------|-------|-----------|
|
||||
| Critical | 0 | — |
|
||||
| High | 3 | (1) silent mutation paths break downstream consumers (RB-01); (2) outbox not transactional with FS+DB write (RB-03); (3) outbox has no row-leasing → multi-instance double-publish (OP-02 / blocked by RB-09 contract). (Former SEC-01 — JWT issuer/audience not validated — closed by the auth refactor.) |
|
||||
| Medium | 3 | xxHash64 collision tolerance (RB-04); `silent_detection` ambiguity (RB-02); Swagger in prod (SEC-04); `04 dataset` direct-DB coupling (RB-08). (Former SEC-03 — CORS wide-open — closed by `CorsConfigurationValidator`.) |
|
||||
| Low | 6 | `Flight` vs `Mission` naming drift (RB-07); empty `catch{}` (RB-05); `detection_classes` not admin-CRUD (RB-06); upload MIME whitelist (SEC-05); rate limiting (SEC-06); audit log substrate (SEC-08). |
|
||||
|
||||
**Iterations completed**: 1 verification pass with stakeholder review.
|
||||
**All Critical/High risks mitigated**: No — High items are tracked as RB-01, RB-03, and OP-02 (multi-instance constraint, time-boxed by current single-instance deployment). SEC-01 / SEC-02 / SEC-03 (the original auth + CORS gaps) were closed by the auth + CORS refactor between Steps 1 and 4. Remaining mitigations are scheduled, not executed.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
The repo currently has **zero automated tests** (`_docs/02_document/00_discovery.md`) and CI runs only build-and-push (`.woodpecker/build-arm.yml`). Test coverage is planned, not measured.
|
||||
|
||||
| Component | Integration | Performance | Security | Acceptance | AC Coverage |
|
||||
|-----------|-------------|-------------|----------|------------|-------------|
|
||||
| 01 annotations-rest | 0 / TBD (Step 3) | 0 / TBD (Step 15) | 0 / TBD (Step 14) | 0 / 8 ACs | 0 / 8 |
|
||||
| 02 realtime-sync | 0 / TBD | 0 / TBD | 0 / TBD | 0 / 4 ACs | 0 / 4 |
|
||||
| 03 media | 0 / TBD | 0 / TBD | 0 / TBD | 0 / 2 ACs | 0 / 2 |
|
||||
| 04 dataset | 0 / TBD | 0 / TBD | 0 / TBD | 0 / 2 ACs | 0 / 2 |
|
||||
| 05 settings-metadata | 0 / TBD | 0 / TBD | 0 / TBD | 0 / 3 ACs | 0 / 3 |
|
||||
| 06 platform | 0 / TBD | 0 / TBD | 0 / TBD | 0 / 5 ACs | 0 / 5 |
|
||||
|
||||
**Overall acceptance criteria coverage**: 0 / 24 functional ACs + 0 / 5 non-functional ACs (0%).
|
||||
|
||||
The autodev existing-code Phase A Steps 3 (Test Spec) and 6 (Implement Tests) own filling this matrix.
|
||||
|
||||
## Epic Roadmap
|
||||
|
||||
The current invocation completed **Phase A Step 1 (Document)** of the autodev existing-code flow. No tracker epics have been opened yet. The work that follows in Phase A produces the test surface and the security/perf baselines:
|
||||
|
||||
| Order | Phase A Step | Output | Effort | Dependencies |
|
||||
|-------|--------------|--------|--------|-------------|
|
||||
| 1 | Step 2 — Documentation Quality Audit | gap log | S | this report |
|
||||
| 2 | Step 3 — Test Spec | per-component `tests.md` (functional + integration shape) | M | step 2 |
|
||||
| 3 | Step 4 — Risk Mitigations | `risk_mitigations.md` | S | step 2 |
|
||||
| 4 | Step 5 — Solution Extraction (already done — see `_docs/01_solution/solution.md`) | — | — | — |
|
||||
| 5 | Step 6 — Implement Tests | actual test project + green CI step | L | step 3 |
|
||||
| 6 | Step 7 — Test Audit | coverage report against AC matrix | S | step 6 |
|
||||
|
||||
Phase B (Feature Cycle) then runs per feature/refactor. The 9 Refactor Backlog items become the first batch of Phase B epics; sizing per the user's Jira complexity rules will be 2–5 points each, except RB-07 (rename across DTOs/controllers/consumers — likely 5).
|
||||
|
||||
**Total estimated effort**: not committed. Phase A Steps 2–7 are scoped against this report; Phase B sizes per epic.
|
||||
|
||||
## Key Decisions Made
|
||||
|
||||
These are the 13 ADRs from `architecture.md`. Eight of them came from the verification stakeholder review.
|
||||
|
||||
| # | Decision | Rationale | Alternatives rejected |
|
||||
|---|----------|-----------|----------------------|
|
||||
| ADR-001 | In-process SSE channel for UI fan-out, separate transactional outbox for durable consumers | Sub-ms UI latency without standing up a broker for the inner loop | Single broker for both (UI latency hit); Postgres LISTEN/NOTIFY for UI (delivery semantics insufficient) |
|
||||
| ADR-002 (RETIRED) | Originally: symmetric HS256 JWT, no issuer/audience validation. Now: ES256 verifier-only over admin's JWKS, with `iss` / `aud` / `exp` / `alg` all enforced. | Identity is centralised in admin; annotations holds no signing material | The original symmetric scheme it replaced |
|
||||
| ADR-003 | Linq2DB + idempotent SQL DDL migrator (no EF, no DbUp/FluentMigrator) | Lighter dependency surface; one less deploy step (ADR-007) | EF Core migrations (heavier); FluentMigrator (separate runner) |
|
||||
| ADR-004 | Annotation id = `XxHash3.Hash128` over a sampled image-bytes window | 128-bit space tolerates the suite's annotation volume; sampled keeps large-frame ingest cheap | Full SHA-256 (CPU); xxHash64 (collision space too small — RB-04 was the upgrade) |
|
||||
| ADR-005 | Swagger UI mounted unconditionally | Internal-only deployment; aids debugging | Gating by env (deferred to SEC-04) |
|
||||
| ADR-006 (RETIRED) | Originally: CORS `AllowAny*`. Now: config-driven allow-list (`CorsConfig:AllowedOrigins` + opt-in `AllowAnyOrigin`) gated by `CorsConfigurationValidator` per environment. | Production cannot start without an explicit origin policy | The original wide-open default |
|
||||
| ADR-007 | DDL applied at boot, not in CI | Single deploy step; matches container-immutable model | Separate migration job (deploy complexity) |
|
||||
| ADR-008 | Business-transaction wrapper (transactional outbox) for annotation lifecycle | Atomicity across DB + outbox; FS write tolerated as best-effort with cleanup | DTC across FS + DB + RabbitMQ (heavyweight, not portable) |
|
||||
| ADR-009 | Every mutation path emits SSE + enqueues outbox row | One observability contract for humans + machines | SSE-only (durability gap for AI/admin worker); outbox-only (UI latency) |
|
||||
| ADR-010 | Remove `silent_detection` flag | Behavior is contradictory once ADR-009 holds | Keep flag and gate on it (forces every consumer to interpret it) |
|
||||
| ADR-011 | Detection class catalog becomes admin-managed (CRUD + cache) | Catalog evolves with deployments; migrator-only is a deploy-time-only escape hatch | Static catalog (RB-06 supersedes) |
|
||||
| ADR-012 | Canonical term is `Mission`; `Flight*` symbols renamed | Single suite-level vocabulary | Keep `Flight` in this service (drift cost grows over time) |
|
||||
| ADR-013 | On-the-wire dedupe key: `(annotation_id, operation, date_time)` | Lets every downstream consumer dedupe re-deliveries safely | Per-consumer offset trust (fragile under outbox replay) |
|
||||
|
||||
## Open Questions
|
||||
|
||||
All 6 verification-pass questions were resolved during stakeholder review. Genuinely-open follow-ups now:
|
||||
|
||||
| # | Question | Impact | Assigned To |
|
||||
|---|----------|--------|-------------|
|
||||
| 1 | Are P50/P95/P99 latency / throughput targets contracted anywhere in the suite? | Bounds NFR ACs (`AC-N-*`) and Step 15 perf-test shape. | Suite ops / product |
|
||||
| 2 | What is the upload format whitelist `/media` should enforce? | Bounds SEC-05 fix scope. | Detections-pipeline owner |
|
||||
| 3 | RPO/RTO contract for `images_dir` and `deleted_dir`? | Bounds the soft-delete restore story (post RB-01). | Suite ops |
|
||||
| 4 | Stream retention window for `azaion-annotations`? | Bounds the consumer replay window the AI pipeline depends on. | Suite ops |
|
||||
| 5 | Is multi-tenancy on the roadmap within the doc horizon? | Decides whether SEC-07 is a Step 14 must-fix or a deferred gap. | Product |
|
||||
|
||||
## Artifact Index
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `_docs/02_document/architecture.md` | Architecture vision, 13 ADRs, refactor backlog, NFRs |
|
||||
| `_docs/02_document/system-flows.md` | F1–F8 verified flow narratives |
|
||||
| `_docs/02_document/data_model.md` | ERD + per-table contract reproduced from `DatabaseMigrator.cs` |
|
||||
| `_docs/02_document/glossary.md` | 36 canonical terms (suite + project + code-level) |
|
||||
| `_docs/02_document/module-layout.md` | Module → component mapping |
|
||||
| `_docs/02_document/04_verification_log.md` | Verification pass corrections + stakeholder resolutions |
|
||||
| `_docs/02_document/components/01_annotations-rest/description.md` | Component 01 spec |
|
||||
| `_docs/02_document/components/02_annotations-realtime-sync/description.md` | Component 02 spec |
|
||||
| `_docs/02_document/components/03_media/description.md` | Component 03 spec |
|
||||
| `_docs/02_document/components/04_dataset/description.md` | Component 04 spec |
|
||||
| `_docs/02_document/components/05_settings-metadata/description.md` | Component 05 spec |
|
||||
| `_docs/02_document/components/06_platform/description.md` | Component 06 spec |
|
||||
| `_docs/02_document/modules/*.md` | 11 module-level deep-dives |
|
||||
| `_docs/02_document/diagrams/components.md` | Component diagram (Mermaid) |
|
||||
| `_docs/02_document/diagrams/flows/flow_annotation_create.md` | F1 sequence (verified) |
|
||||
| `_docs/02_document/diagrams/flows/flow_sse_subscription.md` | F3 sequence |
|
||||
| `_docs/02_document/diagrams/flows/flow_failsafe_drain.md` | F4 sequence |
|
||||
| `_docs/02_document/deployment/containerization.md` | Dockerfile-derived deployment notes |
|
||||
| `_docs/02_document/deployment/ci_cd_pipeline.md` | Woodpecker pipeline-derived notes |
|
||||
| `_docs/02_document/deployment/environment_strategy.md` | Env-var contract + ASPNETCORE_ENVIRONMENT use |
|
||||
| `_docs/02_document/deployment/observability.md` | Logging + `/health` + outbox depth gap |
|
||||
| `_docs/02_document/common-helpers/01_http-error-envelope.md` | Suite error envelope contract |
|
||||
| `_docs/01_solution/solution.md` | Retrospective per-component solution table |
|
||||
| `_docs/00_problem/problem.md` | Retrospective problem statement |
|
||||
| `_docs/00_problem/restrictions.md` | HW / SW / ENV / OP / suite-level restrictions |
|
||||
| `_docs/00_problem/acceptance_criteria.md` | 24 functional + 5 non-functional ACs |
|
||||
| `_docs/00_problem/input_data/data_parameters.md` | REST DTOs + env vars + seed data + wire format |
|
||||
| `_docs/00_problem/security_approach.md` | Auth/AuthZ/secrets posture + 8 SEC-XX gaps |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Suite-level integration narrative: `suite/_docs/01_annotations.md`
|
||||
- Repo-config (monorepo discovery): `_docs/_repo-config.yaml`
|
||||
- Autodev state: `_docs/_autodev_state.md`
|
||||
- Document skill internal state: `_docs/02_document/state.json`
|
||||
@@ -0,0 +1,403 @@
|
||||
# Azaion.Annotations — Architecture
|
||||
|
||||
> **Source of truth for service-internal architecture.** Suite-level integration narrative lives in `../../../suite/_docs/01_annotations.md`. This file documents what the code in `src/` actually implements, derived bottom-up from module and component docs.
|
||||
|
||||
## Architecture Vision
|
||||
|
||||
**Status**: confirmed-by-user 2026-05-14.
|
||||
|
||||
Azaion.Annotations is a single .NET 10 ASP.NET Core service in the Azaion suite that owns the authoritative HTTP + streaming surface for annotation lifecycle, media upload, dataset exploration, and system metadata. State of record is PostgreSQL (Linq2DB + Npgsql) with an idempotent boot-time migrator. Real-time fan-out is in-process SSE; durable cross-service export is a transactional-outbox + RabbitMQ Stream pipeline producing MessagePack frames consumed by the admin sync worker and the AI training pipeline. The runtime is one container per node, ARM64-first via Woodpecker CI, with branch-driven image tags (`dev` | `stage` | `main`).
|
||||
|
||||
### Components & responsibilities
|
||||
|
||||
- **06 Platform** — shared kernel: DB, enums, JWT, error middleware, paths, composition root.
|
||||
- **02 Annotations realtime & sync** — SSE channel + RabbitMQ Stream failsafe drainer.
|
||||
- **01 Annotations REST** — annotation CRUD + image/thumbnail file routes; the lifecycle producer.
|
||||
- **03 Media** — upload (single + batch), list, download, delete.
|
||||
- **04 Dataset** — read-heavy `/dataset` surface + `DATASET`-policy status writes (planned to route through `01 Annotations REST` per RB-08).
|
||||
- **05 Settings & metadata** — system / directory / camera / user settings + `/classes` catalog (becoming admin-managed per RB-06).
|
||||
|
||||
### Major data flows
|
||||
|
||||
- **F1 — Annotation create**: bytes → image file → DB rows → label file → SSE → outbox; will be wrapped in a business transaction (ADR-008).
|
||||
- **F3 — SSE subscription**: UI long-poll on `/annotations/events`.
|
||||
- **F4 — Outbox drain**: `FailsafeProducer` pumps queue rows to the RabbitMQ stream `azaion-annotations`.
|
||||
- **F2 / F5 / F6 / F7 / F8** — read paths, media uploads, auth refresh, directory cache reset, dataset bulk status.
|
||||
|
||||
### Principles / non-negotiables
|
||||
|
||||
- **Wire enums are integer-stable** (suite contract). [inferred-from: `modules/wire-enums.md`, `suite/_docs/01_annotations.md`]
|
||||
- **Annotation id is content-addressed** via a sampled image-bytes hash; remains file-size-independent (videos to ~5 GB). [inferred-from: `AnnotationService.ComputeHash`, ADR-004]
|
||||
- **PostgreSQL is the state of record**; the filesystem is a content-addressed cache. [inferred-from: `data_model.md`, `system-flows.md` F1]
|
||||
- **The transactional outbox is the durability boundary**; SSE is best-effort. [inferred-from: ADR-003 / ADR-008]
|
||||
- **Lifecycle observability is World B**: every mutation publishes SSE and enqueues the outbox. [inferred-from: `FailsafeProducer` drainer plumbing for `Validated`/`Deleted`; maintainer resolution 2026-05-14 → ADR-009 / RB-01]
|
||||
- **Soft-delete with file relocation**: `DeleteAnnotation` flips status to `AnnotationStatus.Deleted = 40` and moves files to a deleted-files directory rather than removing rows. [inferred-from: maintainer resolution 2026-05-14 → ADR-009 / RB-01]
|
||||
- **Stream consumer dedupe contract is owned by this service**: outbox messages must carry enough metadata for downstream consumers to dedupe on `(annotationId, operation, dateTime)`. [inferred-from: maintainer resolution 2026-05-14 → ADR-013 / RB-09]
|
||||
- **Mission is the canonical domain term**: code currently uses `FlightId`; the suite spec uses `missionId`. Code aligns to suite (rename, not the other way). [inferred-from: `00_discovery.md` drift list; maintainer resolution 2026-05-14 → ADR-012 / RB-07]
|
||||
- **Dataset writes flow through the annotation domain service**: `04 Dataset` does not edit `annotations` rows directly. [inferred-from: `module-layout.md` Verification Needed §1; maintainer resolution 2026-05-14 → RB-08]
|
||||
- **DB-driven runtime config**: directory roots and detection classes change at runtime via `ADM` endpoints, not redeploy. [inferred-from: `PathResolver.Reset`, ADR-011]
|
||||
|
||||
### Open questions / drift signals (residual)
|
||||
|
||||
- `UserId` body field vs JWT `NameIdentifier` (suite spec lists `UserId` on `POST /annotations`; code uses JWT subject). Reconcile in suite or code.
|
||||
- The exact dedupe key shape for downstream consumers — `(annotationId, operation, dateTime)` is the working assumption per RB-09; suite consumer doc must be updated to match.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Context
|
||||
|
||||
**Problem being solved**: Provide the canonical HTTP + streaming API for **annotation lifecycle** (create / update / status / delete / list / files), **media** (upload, list, download), **dataset exploration** (`DATASET` policy reads + bulk status writes), and **system metadata** (settings + detection class catalog), with **real-time SSE** push to UI consumers and **failsafe** export to RabbitMQ Stream consumers (admin sync, AI training).
|
||||
|
||||
**System boundaries**:
|
||||
- **Inside**: a single ASP.NET Core process (`Azaion.Annotations.dll`), its embedded migrator, in-memory SSE channel, in-process `BackgroundService` outbox drain, and the on-disk image / label / thumbnail / results layout under `directory_settings`.
|
||||
- **Outside**: PostgreSQL (state of record), RabbitMQ Streams (durable annotation export), the on-disk media/data filesystem (mounted), and every authenticated HTTP / SSE consumer (UIs, detections service, admin sync worker, AI training).
|
||||
|
||||
**External systems**:
|
||||
|
||||
| System | Integration Type | Direction | Purpose |
|
||||
|--------|------------------|-----------|---------|
|
||||
| PostgreSQL | DB (Linq2DB / Npgsql) | Both | State of record (annotations, media, queue, settings, classes) |
|
||||
| RabbitMQ Streams | Stream client (`RabbitMQ.Stream.Client`) | Outbound | Durable export of annotation lifecycle (`azaion-annotations` stream) |
|
||||
| Filesystem (mounted) | File I/O | Both | Annotation images, YOLO label `.txt`, thumbnails, results, GPS routes/sat |
|
||||
| Annotator UI / Dataset Explorer UI | REST + SSE | Inbound | User flows (suite `01_annotations.md`, `09_dataset_explorer.md`) |
|
||||
| Detections service (suite `detections`) | REST | Inbound | POST annotations after model inference; long-running tokens are refreshed against admin (annotations no longer mints tokens) |
|
||||
| Admin sync worker / AI training | RabbitMQ Streams | Outbound | Consume `azaion-annotations` stream offsets (suite `Annotation Sync`) |
|
||||
|
||||
## 2. Technology Stack
|
||||
|
||||
| Layer | Technology | Version | Rationale |
|
||||
|-------|------------|---------|-----------|
|
||||
| Language | C# | `net10.0` (`src/Azaion.Annotations.csproj`) | Single language across suite .NET services |
|
||||
| Framework | ASP.NET Core (minimal hosting + controllers) | net10.0 | Built-in JWT, CORS, Swagger, hosted services |
|
||||
| ORM / DB driver | Linq2DB + Npgsql | per `csproj` | Linq2DB used for `ITable<>` repositories; Npgsql under the hood |
|
||||
| Database | PostgreSQL | not pinned in code (URL-driven) | Suite-wide datastore |
|
||||
| Auth | JWT Bearer (`Microsoft.AspNetCore.Authentication.JwtBearer`) — verifier-only, ES256 over admin's JWKS | net10.0 | Issuer/audience/lifetime/signature all validated; admin is the sole issuer (see Section 7) |
|
||||
| Messaging | RabbitMQ Streams (`RabbitMQ.Stream.Client`) + MessagePack | per `csproj` | Durable, replayable annotation export |
|
||||
| API docs | Swashbuckle (Swagger / Swagger UI) | per `csproj` | Always mounted (see ADR-005) |
|
||||
| Hashing | `System.IO.Hashing` | net10.0 stdlib | Annotation id derived from image bytes hash |
|
||||
| Hosting | `WebApplication` + `IHostedService` | net10.0 | `FailsafeProducer` runs in-process |
|
||||
| Container | `mcr.microsoft.com/dotnet/aspnet:10.0` | linux/arm64 + linux/amd64 | Multi-arch image, ARM-first per Woodpecker |
|
||||
| CI | Woodpecker CI (`.woodpecker/build-arm.yml`) | n/a | Branch-based image tag (`${BRANCH}-arm`) |
|
||||
|
||||
**Key constraints (evidenced in code/config)**:
|
||||
- `DATABASE_URL` is **required** at startup — `ConfigurationResolver.ResolveRequiredOrThrow` throws if not set. The string is auto-converted from `postgresql://user:pass@host:port/db` URI form to Linq2DB's `Host=…;Username=…` form by `Program.ConvertPostgresUrl`.
|
||||
- JWT verification is **required** at startup — `JWT_ISSUER`, `JWT_AUDIENCE`, and `JWT_JWKS_URL` are all resolved by `ConfigurationResolver.ResolveRequiredOrThrow`. There is no insecure fallback. The JWKS URL must be HTTPS (`HttpDocumentRetriever { RequireHttps = true }`).
|
||||
- Default directory roots are `/data/{videos,images,labels,results,thumbnails,gps_sat,gps_route}` (migrator `directory_settings` defaults) → operator must mount or override at the DB level via `PUT /settings/directories`.
|
||||
- CORS is **environment-gated**: `CorsConfigurationValidator.EnsureSafeForEnvironment` refuses to start in `Production` when `CorsConfig:AllowedOrigins` is empty unless `CorsConfig:AllowAnyOrigin=true` is set explicitly. ADR-006 was retired together with the wide-open default.
|
||||
|
||||
## 3. Deployment Model
|
||||
|
||||
**Environments** (evidenced from CI branches): `dev`, `stage`, `main` → image tag `${CI_COMMIT_BRANCH}-arm` pushed to a private registry resolved from `REGISTRY_HOST` secret.
|
||||
|
||||
**Infrastructure**:
|
||||
- Single .NET service container; container exposes port `8080`.
|
||||
- Multi-arch build supported in the Dockerfile (`--platform=$BUILDPLATFORM`, `$TARGETARCH`); the ARM Woodpecker pipeline currently only emits `arm64`.
|
||||
- Scaling is **vertical-only** as written: SSE uses an in-process `Channel<AnnotationEventDto>`, and the `FailsafeProducer` outbox drainer is a per-instance `BackgroundService` — see "Open Architectural Risks".
|
||||
|
||||
**Environment-specific configuration** (defaults vs production):
|
||||
|
||||
| Config | Source | Development default | Production behavior |
|
||||
|--------|--------|---------------------|---------------------|
|
||||
| `DATABASE_URL` | env or `Database:Url` config key | none — fail-fast on missing (`ConfigurationResolver`) | MUST set |
|
||||
| `JWT_ISSUER` | env or `Jwt:Issuer` config key | none — fail-fast | MUST set (matches admin's issuer) |
|
||||
| `JWT_AUDIENCE` | env or `Jwt:Audience` config key | none — fail-fast | MUST set (matches admin's audience for this service) |
|
||||
| `JWT_JWKS_URL` | env or `Jwt:JwksUrl` config key | none — fail-fast; HTTPS required | MUST set to admin's JWKS endpoint |
|
||||
| `RABBITMQ_HOST` / `RABBITMQ_STREAM_PORT` | env | `127.0.0.1` / `5552` | Override per environment |
|
||||
| `RABBITMQ_PRODUCER_USER` / `_PASS` | env | `azaion_producer` / `producer_pass` | Override |
|
||||
| `RABBITMQ_STREAM_NAME` | env | `azaion-annotations` | Usually kept (suite contract) |
|
||||
| `CorsConfig:AllowedOrigins` | `IConfiguration` (string array) | empty | MUST set (or set `AllowAnyOrigin=true` explicitly) — `CorsConfigurationValidator` refuses to start in Production otherwise |
|
||||
| `CorsConfig:AllowAnyOrigin` | `IConfiguration` (bool) | false | Explicit opt-in for permissive policy |
|
||||
| Directory roots (`/data/...`) | DB `directory_settings` | hard-coded SQL defaults | Tune via `PUT /settings/directories` (calls `PathResolver.Reset`) |
|
||||
| Swagger UI | `Program.cs` | mounted | **Also mounted in prod** (ADR-005) |
|
||||
| `AZAION_REVISION` | Dockerfile build arg `CI_COMMIT_SHA` | `unknown` | Stamped per-image |
|
||||
|
||||
## 4. Data Model Overview
|
||||
|
||||
> Detailed ERD, indexes, and migration semantics live in `data_model.md`. This section is the cross-component summary.
|
||||
|
||||
**Core entities** (owned by `06_platform`; consumed by feature components):
|
||||
|
||||
| Entity | Description | Owned by component |
|
||||
|--------|-------------|---------------------|
|
||||
| `media` | Uploaded image/video reference (waypoint-scoped) | `03_media` (writes) / `01_annotations-rest` (reads) |
|
||||
| `annotations` | Annotation row keyed by image-bytes hash, soft-versioned by `created_date`, `time` (BIGINT ticks) | `01_annotations-rest` |
|
||||
| `detection` | YOLO bounding boxes (`center_x/y, width, height`, class, affiliation, combat readiness) per annotation | `01_annotations-rest` |
|
||||
| `annotations_queue_records` | Outbox for failsafe stream sync (`operation`, `annotation_ids` JSON array) | `02_annotations-realtime-sync` (writer) / `01_annotations-rest` (writer side) |
|
||||
| `system_settings` | Singleton-ish org settings + `generate_annotated_image`, `silent_detection` toggles | `05_settings-metadata` |
|
||||
| `directory_settings` | Filesystem roots consumed by `PathResolver` | `05_settings-metadata` |
|
||||
| `detection_classes` | Seeded class catalog for UI label/color (ids 0–18, names + Cyrillic short names + hex colors) | `05_settings-metadata` (read-only `ClassesController`) |
|
||||
| `user_settings` | Per-user UI prefs (panel widths, selected flight) | `05_settings-metadata` |
|
||||
| `camera_settings` | Calibration (altitude, focal length, sensor width) | `05_settings-metadata` |
|
||||
|
||||
**Key relationships**:
|
||||
- `annotations.media_id` → `media.id` (FK).
|
||||
- `detection.annotation_id` → `annotations.id` (FK; cascades on annotation update logic in service layer, not DB).
|
||||
- `annotations_queue_records.annotation_ids` is a **JSON array of TEXT ids** (no FK); single-row outbox entry can reference multiple annotations (bulk).
|
||||
|
||||
**Data flow summary**:
|
||||
- **Inbound write (Create)** — *today*: HTTP body → `AnnotationService.CreateAnnotation` → image bytes to `images_dir/{id}.jpg`, optional `media` row insert, `annotations` + `detection` rows, YOLO label to `labels_dir/{id}.txt`, SSE publish, then (if `silent_detection != true`) outbox row → drained by `FailsafeProducer` → MessagePack frame on RabbitMQ stream. **Thumbnails are not produced by this flow** — they are read-only via `PhysicalFile` and presumed populated out-of-band.
|
||||
- **Inbound write (Update / UpdateStatus / Delete annotations, dataset PATCH / bulk-status)** — *today*: DB-only, silent. *Target* (RB-01): every mutation publishes SSE and enqueues the outbox with the appropriate `QueueOperation` (`Created`, `Validated`, or `Deleted`).
|
||||
- **Lifecycle ordering** — *target* (RB-03): all DB writes plus the outbox row commit inside a single business transaction; FS writes (image / label / future thumbnail generation) and SSE publish are post-commit, with the outbox row as the durable promise.
|
||||
- **Inbound read**: HTTP query → DB joins (`annotations × detection × media`) → JSON list (`PaginatedResponse<AnnotationListItem>`); image/thumbnail served as `PhysicalFile`.
|
||||
|
||||
## 5. Integration Points
|
||||
|
||||
### Internal communication (in-process)
|
||||
|
||||
| From | To | Protocol | Pattern | Notes |
|
||||
|------|----|----------|---------|-------|
|
||||
| `01_annotations-rest` (`AnnotationService`) | `02_annotations-realtime-sync` (`AnnotationEventService`) | C# call | Fire-and-forget publish to `Channel<>` | **Today**: only on Create. **Target (RB-01)**: every mutation publishes (Create, Update, UpdateStatus, Delete) |
|
||||
| `01_annotations-rest` (`AnnotationService`) | `02_annotations-realtime-sync` (`annotations_queue_records` table) | DB INSERT via `FailsafeProducer.EnqueueAsync` (static helper) | Outbox | **Today**: Create only, gated by `silent_detection`. **Target (RB-01 + RB-02)**: every mutation enqueues with the appropriate `QueueOperation`; gating flag removed |
|
||||
| `02_annotations-realtime-sync` (`FailsafeProducer`) | `06_platform` (`AppDataConnection`, `PathResolver`) | C# call | Read-then-delete | Drainer is **already plumbed** for `Created`, `Validated`, and `Deleted` operations (see `FailsafeProducer.cs:108–123`) |
|
||||
| `04_dataset` (`DatasetService.UpdateStatus` / `BulkUpdateStatus`) | `01_annotations-rest` (`AnnotationEventService`) + outbox | shared DB + cross-component call | Direct write today; lifecycle publish + enqueue per RB-01 | Bulk path enqueues a single `Validated` outbox record carrying all ids |
|
||||
| `05_settings-metadata` (directory PUT) | `06_platform` (`PathResolver.Reset`) | C# call | Cache invalidation | Required after directory change |
|
||||
|
||||
### External integrations
|
||||
|
||||
| External system | Protocol | Auth | Rate limits | Failure mode |
|
||||
|-----------------|----------|------|-------------|--------------|
|
||||
| PostgreSQL | TCP / Linq2DB / Npgsql | Conn string | n/a | Surfaced as 500 via `ErrorHandlingMiddleware` |
|
||||
| RabbitMQ Stream `azaion-annotations` | Stream protocol (5552) | Stream user/pass (`azaion_producer` default) | Stream-level | `FailsafeProducer` retries; rows stay in `annotations_queue_records` until drained |
|
||||
| Filesystem (`/data/...`) | POSIX | OS perms | n/a | `IOException` → 500; missing image on GET → 404 |
|
||||
| HTTP clients (UIs, detections, admin) | REST + SSE | JWT Bearer (`ANN`, `DATASET`, `ADM`) | n/a | `401` if invalid; `403` if missing claim |
|
||||
|
||||
## 6. Non-Functional Requirements
|
||||
|
||||
> Pulled only from code-level evidence — config defaults, validators, health checks, idempotent migrator. Anything not evidenced is left blank rather than guessed.
|
||||
|
||||
| Requirement | Target | Measurement | Priority | Source |
|
||||
|-------------|--------|-------------|----------|--------|
|
||||
| Liveness | 200 OK on `GET /health` | route in `Program.cs` | High | `Program.cs` |
|
||||
| Idempotent startup | DB schema applies cleanly on every boot | `DatabaseMigrator.Migrate` uses `CREATE TABLE IF NOT EXISTS` + `ALTER TABLE … IF NOT EXISTS` and `INSERT … ON CONFLICT DO NOTHING` | High | `Database/DatabaseMigrator.cs` |
|
||||
| Recovery: queue durability | Annotation lifecycle events are not lost across pod restarts | DB-backed outbox (`annotations_queue_records`) drained by `FailsafeProducer` | High | `Services/FailsafeProducer.cs` |
|
||||
| Auth lifetime / clock skew | per `JwtExtensions.AddJwtAuth` config | `auth-identity` module | Medium | `Auth/JwtExtensions.cs` |
|
||||
| Pagination defaults | `PaginatedResponse<T>` total/page/pageSize | applied in list endpoints | Medium | `DTOs/PaginatedResponse.cs` |
|
||||
| Thumbnail dimensions | `240×135` with `10` border (defaults) | `system_settings.thumbnail_*` | Low | migrator defaults |
|
||||
| Throughput / latency / availability targets | **not evidenced in code** | — | — | open question, see `00_problem` extraction (Step 6) |
|
||||
|
||||
## 7. Security Architecture
|
||||
|
||||
**Authentication**: JWT Bearer; **ES256 signature** verified against admin's JWKS endpoint (`JWT_JWKS_URL`, default `https://admin.azaion.com/.well-known/jwks.json`). `ValidateIssuer`, `ValidateAudience`, `RequireSignedTokens`, and `RequireExpirationTime` are all enforced; algorithms are pinned to `EcdsaSha256` to block HS256-confusion forgeries. Admin is the sole token issuer for the suite — annotations no longer holds an HMAC secret and no longer mints tokens (`TokenService` and `POST /auth/refresh` were removed; callers refresh against admin).
|
||||
|
||||
**Authorization** (per-endpoint policy claims, all evidenced in controllers):
|
||||
- `ANN` — `AnnotationsController`, `MediaController`.
|
||||
- `DATASET` — `DatasetController` (status writes including bulk).
|
||||
- `ADM` — mutating routes on `SettingsController`.
|
||||
- `[Authorize]` (any authenticated user) — read endpoints on settings, `ClassesController`.
|
||||
- `[AllowAnonymous]` — `/health`.
|
||||
|
||||
**User identity**: server resolves user from JWT `NameIdentifier` (e.g., `AnnotationsController.Create` parses `User.FindFirstValue(ClaimTypes.NameIdentifier)` → `Guid`). Suite spec sometimes lists `UserId` in body — drift recorded in `00_discovery.md`.
|
||||
|
||||
**Data protection**:
|
||||
- **At rest**: nothing in-code — relies on the underlying Postgres deployment + filesystem.
|
||||
- **In transit**: terminated outside the container; service speaks plain HTTP on `:8080`.
|
||||
- **Secrets**: env-driven (`DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL`, `RABBITMQ_*`). `DATABASE_URL` and the three JWT vars now fail-fast on startup if unset (no insecure default). ADR-002 was retired together with `JWT_SECRET`.
|
||||
- **CORS**: config-driven allow-list (`CorsConfig:AllowedOrigins`); `CorsConfigurationValidator.EnsureSafeForEnvironment` refuses to start in `Production` with an empty list unless `CorsConfig:AllowAnyOrigin=true` is explicitly set. ADR-006 was retired together with the wide-open default.
|
||||
|
||||
**Audit logging**: not evidenced beyond ASP.NET Core defaults — open gap; flag in retro/security audit.
|
||||
|
||||
**Input validation**: surfaces through model binding + `ErrorHandlingMiddleware` mapping (`400 / 404 / 409 / 500`); detailed validators per DTO live in `DTOs/Requests/` (component specs to confirm during Step 4 verification).
|
||||
|
||||
## 8. Key Architectural Decisions (inferred from code)
|
||||
|
||||
These ADRs document choices the codebase already evidences. They are descriptive, not prescriptive — call them out so downstream skills can challenge them deliberately.
|
||||
|
||||
### ADR-001: In-process SSE via `Channel<T>`
|
||||
|
||||
**Context**: Real-time annotation activity must reach the Annotator UI within 100ms of a write.
|
||||
|
||||
**Decision**: Use a singleton `AnnotationEventService` exposing an unbounded `Channel<AnnotationEventDto>` and serve subscribers from `AnnotationsController.Events` over `text/event-stream`.
|
||||
|
||||
**Alternatives considered (implicitly rejected)**:
|
||||
1. Broker-backed pub/sub (Redis / RabbitMQ exchange) — rejected because it adds a dependency for what is already a single-process workload, and the failsafe queue covers durable export needs.
|
||||
2. Server-side polling — rejected because it cannot meet sub-second latency cheaply.
|
||||
|
||||
**Consequences**: SSE state is **per-instance only**. Horizontal scaling requires a broker fanout layer or sticky sessions on the LB.
|
||||
|
||||
### ADR-002 (RETIRED): Symmetric JWT, no issuer/audience validation
|
||||
|
||||
**Status**: superseded — annotations is now a JWKS verifier of admin-signed ES256 tokens. `AddJwtAuth(IConfiguration)` pins `ValidAlgorithms = [SecurityAlgorithms.EcdsaSha256]`, enforces `ValidateIssuer`/`ValidateAudience`/`RequireSignedTokens`/`RequireExpirationTime`, and resolves keys through `ConfigurationManager<JsonWebKeySet>` against `JWT_JWKS_URL`. `JWT_SECRET` was removed along with the local refresh path; admin is the sole issuer for the suite. The original ADR is preserved here for historical context only.
|
||||
|
||||
### ADR-003: Failsafe outbox + RabbitMQ Stream (not direct publish)
|
||||
|
||||
**Context**: Annotation lifecycle must reach external consumers (admin sync, AI training) durably even when RabbitMQ is unavailable at the moment of the write.
|
||||
|
||||
**Decision**: Every mutation writes a row to `annotations_queue_records`; the in-process `FailsafeProducer` (`IHostedService`) drains this table and publishes MessagePack frames on the `azaion-annotations` stream, deleting rows after success.
|
||||
|
||||
**Alternatives considered**:
|
||||
1. Direct publish in the request path — rejected because RabbitMQ unavailability would either drop events (`fire-and-forget`) or fail user-visible writes (sync publish).
|
||||
2. Transactional outbox via Debezium / CDC — heavier, deferred.
|
||||
|
||||
**Consequences**: One outbox-drainer per service instance. Multiple instances drain concurrently → safe because the deletion is keyed on `id` and re-reads of disk bytes are idempotent, **but** ordering across consumers is not guaranteed.
|
||||
|
||||
### ADR-004: Annotation id from a sampled `XxHash3.Hash128` of image bytes
|
||||
|
||||
**Context**: Annotation rows must be deduplicated when the same image is re-uploaded (e.g., re-runs of the detection pipeline). The system also serves video media up to **3–5 GB**, so hashing must remain **constant-time with respect to file size** to keep create-path latency stable under load.
|
||||
|
||||
**Decision** (resolved 2026-05-14): Hash a deterministic **fixed-size sample** with `XxHash3.Hash128` (128-bit output, 32-char lower-case hex). Sample composition is unchanged from the current implementation:
|
||||
- For inputs **≤ 3072 bytes**: `[length(8 bytes)] + [full bytes]`.
|
||||
- For inputs **> 3072 bytes**: `[length(8 bytes)] + [first 1024] + [middle 1024 starting at len/2 − 512] + [last 1024]`.
|
||||
|
||||
When `MediaId` is provided instead of bytes, the annotation id is reused from the referenced media row.
|
||||
|
||||
**Why this combination**:
|
||||
- **Sampling preserves file-size independence.** Reading a 5 GB video front-to-back just to derive an id is unacceptable on the hot path.
|
||||
- **`XxHash3.Hash128` over the same sample** keeps the hashing itself O(1) in file size while moving the collision space from 2^64 to 2^128. Distinct large images that happen to share `(length, head 1 KB, middle 1 KB, tail 1 KB)` still collide deterministically — but the practical collision probability among such samples is now negligible at any realistic volume.
|
||||
|
||||
**Migration consequences**:
|
||||
- The annotation `id` column is `TEXT PRIMARY KEY`; switching from 16-char (`XxHash64`) to 32-char (`XxHash3.Hash128`) hex requires no schema change.
|
||||
- Existing rows keep their 16-char ids; new rows get 32-char ids. Re-create of an image whose original id was generated under `XxHash64` will produce a **different** new id under `XxHash3.Hash128` — i.e., re-creates after the upgrade no longer collide with their pre-upgrade row. Acceptable (and expected): old ids are stable, the deduplication property is preserved going forward, and the upgrade is irreversible by design.
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-04).
|
||||
|
||||
### ADR-005: Swagger UI mounted in all environments
|
||||
|
||||
**Context**: Internal debugging / partner integration friction.
|
||||
|
||||
**Decision**: `app.UseSwagger()` and `app.UseSwaggerUI()` are unconditional in `Program.cs`.
|
||||
|
||||
**Consequences**: Schema is publicly readable wherever the service is reachable. If the perimeter is not closed, this leaks endpoint surface — treat as a security finding for production-internet exposure.
|
||||
|
||||
### ADR-006 (RETIRED): Wide-open CORS
|
||||
|
||||
**Status**: superseded — the default policy now reads `CorsConfig:AllowedOrigins` (string array) and `CorsConfig:AllowAnyOrigin` (boolean opt-in). `CorsConfigurationValidator.EnsureSafeForEnvironment` refuses to start in `Production` when origins are empty and `AllowAnyOrigin` is not explicitly set; a `LogWarning` is emitted in non-production when running with the permissive default. The original ADR is preserved here for historical context only.
|
||||
|
||||
### ADR-007: Embedded SQL migrator (not EF migrations / Flyway)
|
||||
|
||||
**Context**: Suite values single-binary deploys; the team prefers idempotent boot-time DDL over a separate migration tool.
|
||||
|
||||
**Decision**: `DatabaseMigrator.Migrate` runs a single multi-statement script via Linq2DB on every startup. Schema evolution is additive (`ALTER … ADD COLUMN IF NOT EXISTS`).
|
||||
|
||||
**Consequences**: Backwards-only, no down migrations. Renames or destructive changes need an explicit out-of-band script. Drift detection requires diffing live DB against `Database/DatabaseMigrator.cs`.
|
||||
|
||||
### ADR-008: Annotation lifecycle wrapped in a business transaction (planned)
|
||||
|
||||
**Context**: `CreateAnnotation` today touches the filesystem, three DB tables, an in-memory channel, and an outbox row, with no atomicity. World B (lifecycle is observable — see ADR-009) widens this surface to Update / Delete / status-change paths. A naive DB transaction does not wrap the FS writes; we want a single conceptual transactional boundary for the lifecycle, not just for the DB rows.
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented): introduce a **business-transaction wrapper** for annotation lifecycle operations. Concretely the chosen pattern is the **transactional outbox**:
|
||||
|
||||
1. Write all relevant DB rows (annotation / detection / annotations_queue_records) inside a single `db.BeginTransaction` scope.
|
||||
2. Commit. The outbox row is the durable promise that the post-commit work is owed.
|
||||
3. **Post-commit**, perform side effects: write image / label / thumbnail files, publish SSE event. These steps are idempotent on retry; the outbox row stays until the drainer succeeds.
|
||||
4. The drainer (`FailsafeProducer`) is unchanged in role — it consumes the outbox.
|
||||
|
||||
**Implications**:
|
||||
- FS write order shifts: today image is first, before any DB row; after the refactor, DB rows + outbox commit first, then FS writes execute (with the outbox row as the recovery anchor).
|
||||
- A new abstraction (e.g., `AnnotationLifecycleTransaction` or a thin extension on `AppDataConnection`) is the right place to centralize this. Implementation deferred to RB-03.
|
||||
|
||||
**Alternatives considered**:
|
||||
1. Pure DB transaction wrapping current order — rejected: doesn't cover FS, leaves orphan-file risk.
|
||||
2. Saga / compensation steps with explicit rollback handlers — rejected: overkill for the linear lifecycle here.
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-03).
|
||||
|
||||
### ADR-009: Lifecycle observability — World B (planned)
|
||||
|
||||
**Context**: Today only `CreateAnnotation` publishes SSE and enqueues the outbox. Update / UpdateStatus / Delete (annotations) and UpdateStatus / BulkUpdateStatus (dataset) are silent. The `QueueOperation` enum already declares `Validated` and `Deleted`, and `FailsafeProducer.cs:108–123` has a dedicated drainer branch for both — strong evidence that the design always intended every lifecycle change to be observable. The producer side simply was never wired (the prior WPF codebase blended UI + backend; lifecycle calls likely came from the UI directly, which the new HTTP backend has not replicated).
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented): every annotation mutation publishes SSE and enqueues the outbox.
|
||||
|
||||
Mapping (initial; sub-questions to be resolved at implementation time):
|
||||
|
||||
| Mutation | SSE | Outbox `QueueOperation` |
|
||||
|----------|-----|--------------------------|
|
||||
| `AnnotationService.CreateAnnotation` | yes (today) | `Created` (today) |
|
||||
| `AnnotationService.UpdateAnnotation` (replace detections, status → `Edited`) | yes | open: re-enqueue as `Created` (richer payload) **or** add `QueueOperation.Updated` + corresponding drainer branch |
|
||||
| `AnnotationService.UpdateStatus` (status → `Validated (30)` or `Deleted (40)`) | yes | `Validated` |
|
||||
| `AnnotationService.UpdateStatus` (other transitions) | yes | open: skip outbox, or always enqueue `Validated`? |
|
||||
| `AnnotationService.DeleteAnnotation` | yes | `Deleted` — **soft-delete**: status flips to `AnnotationStatus.Deleted = 40`, the row stays, image / label / thumbnail files relocate to a `deleted_dir` (new `directory_settings` column added by RB-01) |
|
||||
| `DatasetService.UpdateStatus` / `BulkUpdateStatus` | yes (per-id for bulk) | `Validated` (single record covers the whole bulk via `AnnotationIds`) |
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-01).
|
||||
|
||||
### ADR-010: Remove `system_settings.silent_detection`
|
||||
|
||||
**Context**: `silent_detection` was a debug-time switch to keep the RabbitMQ stream clean while a developer iterated locally. Now that the suite has e2e tests with isolated queues (per `_docs/_repo-config.yaml` suite-e2e), the in-product flag is dead code — debug isolation belongs in the test harness, not in `system_settings`.
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented):
|
||||
- Remove the gating block in `AnnotationService.CreateAnnotation:100–102` (always enqueue).
|
||||
- Drop `silent_detection` from `system_settings` (column, entity, migrator `CREATE TABLE`, migrator `ALTER` line, any DTO references).
|
||||
- Remove the field from `UpdateSystemSettingsRequest` if present.
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-02). Schema column removal is a destructive change explicitly authorized by the maintainer.
|
||||
|
||||
### ADR-012: Rename `Flight` → `Mission` to align with suite canonical (planned)
|
||||
|
||||
**Context**: The suite product spec (`suite/_docs/01_annotations.md`) calls the domain concept `mission` / `missionId`. The code uses `Flight` / `FlightId` (table `media.waypoint_id` + DTO `FlightId` filter). This drift was flagged in `00_discovery.md`.
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented): align code to the suite. `Flight*` → `Mission*` rename across DTOs, controllers, services, and the relevant query-parameter names. The `media.waypoint_id` column stays (it is the underlying physical identifier; mission is the logical grouping concept above it).
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-07). Schema column changes are scoped to renames in DTOs and code only — no DB column rename is required for this ADR.
|
||||
|
||||
### ADR-013: Stream consumer dedupe contract is owned by this service (planned)
|
||||
|
||||
**Context**: The failsafe outbox + RabbitMQ Stream pipeline can produce duplicate stream entries when (a) the drainer retries after a partial publish or (b) two service instances both pick up the same outbox row before either deletes it. Today there is no documented dedupe contract; consumers (admin sync, AI training) silently accept whatever they get.
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented): publish a documented dedupe contract owned by this service. Working shape: consumers MUST dedupe by `(annotationId, operation, dateTime)`. The outbox row's `DateTime` (already populated by `EnqueueAsync`) becomes part of the on-the-wire stream message, alongside the `annotationId` and `operation` already in `AnnotationQueueMessage` / `AnnotationBulkQueueMessage`.
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-09).
|
||||
|
||||
### ADR-011: Detection class catalog is admin-managed with in-memory cache (planned)
|
||||
|
||||
**Context**: `detection_classes` is currently seeded by the migrator (19 rows) and read-only via `GET /classes`. Operators have no way to add or correct classes (e.g., the `Smoke`/`Plane` color clash on `#000080`) without a code change and redeploy.
|
||||
|
||||
**Decision** (resolved 2026-05-14, to-be-implemented):
|
||||
- `ClassesController` exposes `POST /classes`, `PUT /classes/{id}`, `DELETE /classes/{id}` under `[Authorize(Policy = "ADM")]`. `GET /classes` stays `[Authorize]`.
|
||||
- Reads go through a new `DetectionClassCache` (DI singleton) modeled on `PathResolver`: lazy-load on first read, `Reset()` after any write.
|
||||
- Migrator-seeded rows remain as the bootstrap state; admin writes overwrite them per id.
|
||||
|
||||
**Status**: agreed. Implementation lives in the Refactor Backlog (RB-06). Adds a new feature surface; must land before any UI change relying on dynamic class management.
|
||||
|
||||
## Resolved Architectural Decisions (Step 4 verification)
|
||||
|
||||
The following items were surfaced during verification and resolved with the maintainer on 2026-05-14. Each one either becomes an ADR above or maps to a refactor backlog entry below.
|
||||
|
||||
| # | Concern | Resolution | Tracked as |
|
||||
|---|---------|------------|------------|
|
||||
| 1 | Update / Delete / dataset-status changes are silent on SSE + outbox | Treat as gap; lifecycle is observable (World B) — every mutation publishes + enqueues | ADR-009 / RB-01 |
|
||||
| 2 | `system_settings.silent_detection` semantics | Remove the flag; e2e harness covers debug isolation now | ADR-010 / RB-02 |
|
||||
| 3 | F1 not transactional across FS + DB + outbox | Wrap lifecycle in a business-transaction (transactional outbox); FS writes happen post-commit | ADR-008 / RB-03 |
|
||||
| 4 | `XxHash64` over sampled bytes — collision risk | Switch to `XxHash3.Hash128` over the same sample (file-size-independent + 128-bit space) | ADR-004 / RB-04 |
|
||||
| 5 | `FailsafeProducer.EnqueueAsync` static method does DB I/O — violates `coderule.mdc` | Accept as-is; documented deviation from rule | (no refactor) |
|
||||
| 6 | `detection_classes` schema-mutable but no controller writes | Admin-managed CRUD with read-through cache (modeled on `PathResolver`) | ADR-011 / RB-06 |
|
||||
| 7 | `Flight` (code) vs `mission` (suite spec) drift | Rename code → `Mission*`; suite spec stays canonical | ADR-012 / RB-07 |
|
||||
| 8 | Dataset writes coupled directly to annotation rows via shared `AppDataConnection` | Route dataset writes through `AnnotationService` (via a public domain interface) | RB-08 |
|
||||
| 9 | Stream consumer dedupe contract owner | This service owns it; dedupe by `(annotationId, operation, dateTime)` baked into the wire message | ADR-013 / RB-09 |
|
||||
| 10 | Hard-delete vs soft-delete on `DeleteAnnotation` | Soft-delete: status → `Deleted (40)`, files moved to a `deleted_dir` | ADR-009 (folded in) / RB-01 |
|
||||
|
||||
## Remaining Open Architectural Risks
|
||||
|
||||
These are residual risks that still need attention from later autodev steps (Test Spec, Refactor, Security Audit). Items previously listed here that have been resolved as of 2026-05-14 (Flight/mission drift, dataset coupling, hard-vs-soft delete, JWT issuer/audience validation, CORS environment gating, dev secret fallback) moved to the Resolved Architectural Decisions table above and the Refactor Backlog below.
|
||||
|
||||
1. **Horizontal scaling**: SSE channel is per-instance (singleton `AnnotationEventService`); the failsafe outbox uses no leasing/locking. Two pods will independently drain rows, with deletion keyed on `id`; under high concurrency the same row can be picked by both before either deletes — duplicate stream entries possible. Consumers must dedupe per ADR-013. (Touched by RB-03 / RB-09 indirectly but not solved by them.)
|
||||
2. **Swagger exposure** in production: see ADR-005. Belongs to Step 14 (Security Audit). (CORS exposure was resolved by `CorsConfigurationValidator`; ADR-006 retired.)
|
||||
3. **`UserId` body field vs JWT `NameIdentifier`** drift (suite spec lists `UserId` on `POST /annotations`; code uses JWT subject). Reconcile in the suite spec.
|
||||
4. **No automated tests**: addressed by autodev Phase A Steps 3–7 (Test Spec → Implement Tests → Run Tests).
|
||||
5. **`FailsafeProducer.cs:138` swallows `IOException` on image read silently** (`catch { }`). Direct `coderule.mdc` violation. Symptom in product: a missing or unreadable image yields a stream message with `image = null` and no log/metric — the gap is invisible to operators. Track on Refactor Backlog (RB-05).
|
||||
6. **JWKS HTTPS-only retrieval blocks containerised test harnesses** that would otherwise serve a static JWKS over plain HTTP. Tests must either run a TLS-terminating sidecar in the test compose stack or rely on test-only configuration that relaxes `RequireHttps`. Not a production risk; a Step 4 (Code Testability Revision) item.
|
||||
|
||||
## Refactor Backlog
|
||||
|
||||
These items are the implementation work for the resolved decisions above. They are **not** part of Step 4 (Verification) corrections — they will be picked up by the autodev existing-code flow at Step 8 (Refactor) and/or new feature tasks in Phase B.
|
||||
|
||||
| ID | Scope | Source ADR / Risk | Notes |
|
||||
|----|-------|--------------------|-------|
|
||||
| RB-01 | Wire lifecycle publish + outbox enqueue across Update / UpdateStatus / Delete (annotations + dataset). Includes the soft-delete behavior: `DeleteAnnotation` flips `AnnotationStatus → Deleted (40)`, leaves the row, and moves image / label / thumbnail files to a new `deleted_dir` (added to `directory_settings`). Read paths must filter `Status = Deleted (40)` from default lists. | ADR-009 | Open sub-questions: (a) `UpdateAnnotation` mapping — re-enqueue as `Created` or add `QueueOperation.Updated` + drainer branch; (b) which non-Validated/Deleted status transitions enqueue at all |
|
||||
| RB-02 | Remove `silent_detection` (schema column, entity field, gating logic, DTOs) | ADR-010 | Destructive schema change explicitly authorized |
|
||||
| RB-03 | Introduce business-transaction wrapper (transactional outbox) for annotation lifecycle | ADR-008 | Reorders FS writes to post-commit; covers all mutation paths |
|
||||
| RB-04 | Switch annotation id hashing to `XxHash3.Hash128` over the same sampled buffer | ADR-004 | Existing 16-char ids stay; new ids are 32-char hex |
|
||||
| RB-05 | Replace `catch { }` at `FailsafeProducer.cs:138` with logged failure path; surface as a metric | Open Risk §6 | Downstream consumer should know an image-less message means a real disk error |
|
||||
| RB-06 | Admin-managed `detection_classes` (CRUD endpoints `[ADM]`, in-memory cache with `Reset()`) | ADR-011 | Migrator seed remains as bootstrap; admin overrides per id; fix `Smoke`/`Plane` color collision while at it |
|
||||
| RB-07 | Rename `Flight*` → `Mission*` across DTOs, controllers, services, and query-parameter names. `media.waypoint_id` column is unchanged (it's the physical id; mission is the logical concept). | ADR-012 | Code-only rename to align with suite spec; suite stays canonical |
|
||||
| RB-08 | Decouple `04 Dataset` writes from direct `annotations` row mutations — route status writes through a public `AnnotationService` interface. Reads can stay direct for now (read coupling is lower-risk than write coupling). | Open Risks (former §4) | Likely introduces an `IAnnotationLifecycle` (or similar) interface owned by `01 Annotations REST` that `04 Dataset` consumes via DI |
|
||||
| RB-09 | Bake `(annotationId, operation, dateTime)` into the on-the-wire stream message; document the dedupe contract in `suite/_docs/01_annotations.md`. | ADR-013 | Coordinate the suite-doc update with admin sync + AI training maintainers |
|
||||
|
||||
## References
|
||||
|
||||
- Suite product spec: `../../../suite/_docs/01_annotations.md` (REST contracts, SSE, Annotation Sync, camera, classes).
|
||||
- Suite dataset narrative: `../../../suite/_docs/09_dataset_explorer.md`.
|
||||
- Component specs: `components/01..06_*/description.md`.
|
||||
- Module docs: `modules/*.md`.
|
||||
- File ownership (downstream skills): `module-layout.md`.
|
||||
- Component diagram: `diagrams/components.md`.
|
||||
- Per-flow diagrams: `diagrams/flows/`.
|
||||
@@ -0,0 +1,105 @@
|
||||
# Architecture Compliance Baseline — Azaion.Annotations
|
||||
|
||||
**Mode**: code-review baseline (Phase 1 + Phase 7 only)
|
||||
**Scope**: full codebase under `src/` (57 C# files)
|
||||
**Date**: 2026-05-14
|
||||
**Verdict**: PASS_WITH_WARNINGS
|
||||
|
||||
This is the **one-time architecture baseline** for the existing-code flow's Step 2. Future per-batch code-review runs partition findings against this baseline (carried over / resolved / newly introduced) per `.cursor/skills/code-review/SKILL.md` → "Baseline delta".
|
||||
|
||||
## Inputs (verified loaded)
|
||||
|
||||
- `_docs/02_document/architecture.md` — layering rules, ADRs, refactor backlog
|
||||
- `_docs/02_document/module-layout.md` — per-component file ownership, Public API, Allowed Dependencies table
|
||||
- `_docs/02_document/components/*/description.md` — six component specs
|
||||
- `_docs/00_problem/restrictions.md` — operational constraints
|
||||
- `_docs/01_solution/solution.md` — solution overview
|
||||
|
||||
## Method
|
||||
|
||||
Per Phase 7 of the code-review skill:
|
||||
|
||||
1. Mapped all 57 C# files to one of six logical components per `module-layout.md`.
|
||||
2. Parsed every `using Azaion.Annotations.*` directive and every constructor-injection / static reference between domain types (`AnnotationService`, `AnnotationEventService`, `FailsafeProducer`, `MediaService`, `DatasetService`, `SettingsService`, `PathResolver`, `AppDataConnection`). Note: `TokenService` and `AuthController` were removed in the auth refactor and are no longer part of the reference graph.
|
||||
3. Resolved each cross-file reference against the Allowed Dependencies table (Layer 1 → Layer 2 → Layer 3) and the per-component Public API list.
|
||||
4. Applied the five Phase 7 checks: layer direction, Public API respect, cyclic dependencies, duplicate symbols, cross-cutting concerns.
|
||||
|
||||
## Findings summary
|
||||
|
||||
| Severity | Count | Categories |
|
||||
|----------|-------|------------|
|
||||
| Critical | 0 | — |
|
||||
| High | 0 | — |
|
||||
| Medium | 1 | Architecture |
|
||||
| Low | 2 | Architecture, Maintainability |
|
||||
|
||||
**No new High or Critical findings.** Per the existing-code flow Step 2 auto-chain rule, this allows direct progression to Step 3 (Test Spec).
|
||||
|
||||
## Findings detail
|
||||
|
||||
### F1 — `04 dataset` writes directly to the annotation domain (Medium / Architecture)
|
||||
|
||||
- **Location**: `src/Services/DatasetService.cs:75-94` (`UpdateStatus`, `BulkUpdateStatus`).
|
||||
- **Description**: `DatasetService` mutates `db.Annotations.Set(a => a.Status, …)` directly. The `annotations` row is part of `01 annotations-rest`'s domain (per `module-layout.md` → DTO ownership table and component spec). Today this is technically allowed because the only cross-component reference is `AppDataConnection` (a `06_platform` foundation type), but it duplicates ownership of the annotation lifecycle: there are now two paths that mutate `annotations.status` — `AnnotationService.UpdateStatus` (in 01) and `DatasetService.UpdateStatus` / `BulkUpdateStatus` (in 04) — and only the former is wired (post RB-01) into the lifecycle observability contract (SSE + outbox).
|
||||
- **Architecture vision impact**: ADR-009 + RB-01 require every mutation to emit lifecycle events. As long as 04 has its own DB write path, that contract cannot be enforced from one place — RB-08 fixes this by routing 04's status writes through `AnnotationService`.
|
||||
- **Suggestion**: Track via RB-08; no inline action required at this baseline. Confirms the refactor backlog item is well-grounded.
|
||||
- **Module-layout reference**: section "Allowed Dependencies → Rules" — "today: only via shared AppDataConnection in same assembly — acceptable but treat as tight coupling; prefer domain services for new code."
|
||||
|
||||
### F2 — `ClassesController` bypasses the service layer (Low / Architecture)
|
||||
|
||||
- **Location**: `src/Controllers/ClassesController.cs:11`.
|
||||
- **Description**: `ClassesController` injects `AppDataConnection` directly and queries the `detection_classes` table inline. Every other controller in the codebase (`AnnotationsController`, `MediaController`, `DatasetController`, `SettingsController`) routes through a service. Allowed by the layering rules (06 is a foundation), but inconsistent with the project convention.
|
||||
- **Architecture vision impact**: RB-06 introduces admin-managed CRUD for detection classes plus an in-memory cache with `Reset()`. That work will naturally land a `DetectionClassService` (or extend `SettingsService` to own this surface), retiring the direct-DB pattern.
|
||||
- **Suggestion**: Defer to RB-06; do not address inline.
|
||||
|
||||
### F3 — `FailsafeProducer.EnqueueAsync` is a static method that performs DB I/O (Low / Maintainability)
|
||||
|
||||
- **Location**: `src/Services/FailsafeProducer.cs:195` (the static helper) and the call site `src/Services/AnnotationService.cs:102`.
|
||||
- **Description**: `FailsafeProducer.EnqueueAsync(AppDataConnection db, string annotationId, QueueOperation operation)` is a static method on the same type as the hosted-service producer, called from `AnnotationService` to insert a row into `annotations_queue_records`. `coderule.mdc` discourages static methods that touch resources; the user has explicitly accepted this as technical debt during the verification stakeholder review (no RB item — keep as-is).
|
||||
- **Architecture impact**: The Public API of `02 annotations-realtime-sync` includes both the running `FailsafeProducer` *and* this static helper; that is documented in `module-layout.md` Component 02. So there is no boundary violation — it is a deliberate API choice the user has confirmed.
|
||||
- **Suggestion**: No action. Recorded for future reviewers so the pattern is not flagged again as a finding.
|
||||
|
||||
## Phase 7 checklist (full traversal)
|
||||
|
||||
| Check | Result | Notes |
|
||||
|-------|--------|-------|
|
||||
| 1. Layer direction (Layer 3 → Layer 2 → Layer 1 only) | PASS | All Layer 3 components import only `06` (and `01` additionally `02`'s `AnnotationEventService` + `FailsafeProducer.EnqueueAsync`). No reverse imports detected. |
|
||||
| 2. Public API respect | PASS | Every cross-component constructor injection or static call targets a type listed in `module-layout.md` → Public API. No internal-file imports across components. |
|
||||
| 3. No cyclic module dependencies | PASS | DI graph: `06 ← {01, 02, 03, 04, 05}`, `02 ← 01`. No cycles. |
|
||||
| 4. Duplicate symbols across components | PASS (with F1) | No class/function name collisions. F1 is a *logical* duplication of write authority over `annotations.status`, captured separately. |
|
||||
| 5. Cross-cutting concerns not locally re-implemented | PASS | Logging via `ILogger<T>`; auth in `06_platform/Auth`; error envelope in `06_platform/Middleware/ErrorHandlingMiddleware`; config / env reading concentrated in `Program.cs`. No per-component re-implementations. |
|
||||
|
||||
## Static reference graph (verified)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
P06[06 platform]
|
||||
P02[02 realtime sync]
|
||||
P01[01 annotations rest]
|
||||
P03[03 media]
|
||||
P04[04 dataset]
|
||||
P05[05 settings metadata]
|
||||
|
||||
P02 --> P06
|
||||
P01 --> P06
|
||||
P01 --> P02
|
||||
P03 --> P06
|
||||
P04 --> P06
|
||||
P05 --> P06
|
||||
```
|
||||
|
||||
Edges represent constructor-injection or static-call dependencies. `Program.cs` (composition root, in 06) is excluded — it touches everything by definition.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- ADR-008 (transactional outbox) and RB-01, RB-03, RB-08 in `_docs/02_document/architecture.md` cover the lifecycle / coupling concerns.
|
||||
- `module-layout.md` → "Allowed Dependencies" table is the source of truth for layer membership.
|
||||
- `_docs/02_document/04_verification_log.md` documents the stakeholder acceptance of `FailsafeProducer.EnqueueAsync` as tech debt (F3).
|
||||
|
||||
## Auto-chain decision
|
||||
|
||||
Per `.cursor/skills/autodev/flows/existing-code.md` → Step 2 auto-chain rule:
|
||||
|
||||
> If the baseline report contains High or Critical Architecture findings: append to Step 4 testability inputs OR surface to user. If clean (no High/Critical): auto-chain directly to Step 3.
|
||||
|
||||
This baseline contains **0 Critical, 0 High, 1 Medium, 2 Low** Architecture findings. → **Auto-chain to Step 3 (Test Spec)** without user gate.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Shared: HTTP error envelope
|
||||
|
||||
**Used by:** All controllers (via pipeline).
|
||||
|
||||
**Implementation:** `ErrorHandlingMiddleware` — lives under **06 Platform** for ownership; feature components rely on it without duplication.
|
||||
|
||||
See `modules/common-infrastructure.md` and `components/06_platform/description.md`.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Annotations (REST & files)
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** HTTP API for annotation CRUD, status, listing, and serving **annotation image / thumbnail** bytes — the surface described in `suite/_docs/01_annotations.md` §1–6 (excluding the SSE stream).
|
||||
|
||||
**Architectural pattern:** Layered API — controller → application service → database + filesystem.
|
||||
|
||||
**Upstream dependencies:** Platform (DB, auth, paths), Media (annotation create may reference existing `MediaId`).
|
||||
|
||||
**Downstream consumers:** Annotator UI, Detections pipeline (HTTP POST annotations), Dataset (read-only overlap on entities).
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
Primary application API: `AnnotationService` — create/update/status/delete/query/get-by-id; uses `PathResolver`, `AppDataConnection`, hashing, label files, thumbnails, and triggers **real-time publish** (calls into `AnnotationEventService`) and **queue enqueue** (via failsafe path — see component 02).
|
||||
|
||||
Controller: `AnnotationsController` — **REST and file routes**; the `GET …/events` SSE action is **specified and operated** from the **Annotations realtime & sync** component (same source file, split responsibility for docs).
|
||||
|
||||
### Representative HTTP
|
||||
|
||||
| Area | Routes (policy `ANN`) |
|
||||
|------|------------------------|
|
||||
| CRUD | `POST/PUT/PATCH/DELETE/GET` under `/annotations` |
|
||||
| Files | `GET /annotations/{id}/thumbnail`, `GET /annotations/{id}/image` |
|
||||
|
||||
## 3. External API specification
|
||||
|
||||
See `01_annotations.md` §1–6; confirm drift notes in `00_discovery.md` (JWT user id, `FlightId` vs suite `missionId`).
|
||||
|
||||
## 4. Data access patterns
|
||||
|
||||
Heavy use of `annotations`, `detection`, `media` joins for list filters; writes cascade detections on update.
|
||||
|
||||
## 5. Implementation notes
|
||||
|
||||
- Annotation id from image hash when bytes provided; else copy from existing media path.
|
||||
- `TimeSpan` persisted as ticks on `Annotation`.
|
||||
|
||||
## 6. Dependency graph (relative to other components)
|
||||
|
||||
**Imports from:** Platform, Media (logical). **Consumed by:** Dataset (read), UI, external Detections service.
|
||||
|
||||
## 7. Modules included
|
||||
|
||||
`annotations-service` (module doc); **shared file** `AnnotationsController.cs` with component 02 for SSE action only.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Annotations (realtime & stream sync)
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** **SSE** push for annotation changes and **RabbitMQ Stream** failsafe export — `01_annotations.md` sections *SSE Communication* and *Annotation Sync* / *Failsafe* / *RabbitMQ Stream*.
|
||||
|
||||
**Architectural pattern:** Event channel + background outbox producer.
|
||||
|
||||
**Upstream dependencies:** Platform (DB, config, paths), Annotations REST (domain mutations enqueue/publish).
|
||||
|
||||
**Downstream consumers:** Browser UI (SSE); Admin sync worker; AI Training consumer (external).
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
- `AnnotationEventService` — in-process `Channel<AnnotationEventDto>`; `PublishAsync` / `Reader`.
|
||||
- `FailsafeProducer` + `RabbitMqConfig` — stream client, MessagePack payloads, drains `annotations_queue_records`.
|
||||
- **HTTP:** `AnnotationsController.Events` — `text/event-stream` subscription (same controller file as REST component; **doc ownership** here for SSE).
|
||||
|
||||
## 3. External API / integration
|
||||
|
||||
| Surface | Notes |
|
||||
|---------|--------|
|
||||
| `GET /annotations/events` | SSE; see suite SSE section |
|
||||
| RabbitMQ stream `azaion-annotations` | Env `RABBITMQ_*` from `Program` |
|
||||
|
||||
## 4. Data access patterns
|
||||
|
||||
Queue table buffering; stream send on connectivity; image bytes in create messages per suite.
|
||||
|
||||
## 5. Caveats
|
||||
|
||||
MessagePack key stability; stream consumer offsets independent per consumer type.
|
||||
|
||||
## 6. Dependency graph
|
||||
|
||||
**Imports from:** Platform, annotations domain (via service calls / shared types). **Consumed by:** External infrastructure.
|
||||
|
||||
## 7. Modules included
|
||||
|
||||
`sse-realtime`, `rabbitmq-stream-sync`.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Media
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** Upload, batch-upload, list, delete, and download **media** files for missions/waypoints — `01_annotations.md` §7–10 and *Media Browsing*.
|
||||
|
||||
**Architectural pattern:** Service + controller; filesystem + DB.
|
||||
|
||||
**Upstream dependencies:** Platform (auth, DB, paths for storage roots).
|
||||
|
||||
**Downstream consumers:** Annotator UI; Annotations REST (references `MediaId`).
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
`MediaService`, `MediaController` — JSON create, **multipart** batch with `waypointId`, paged list, file download, delete.
|
||||
|
||||
## 3. External API
|
||||
|
||||
| Policy | Base |
|
||||
|--------|------|
|
||||
| `ANN` | `/media` |
|
||||
|
||||
## 4. Data access
|
||||
|
||||
`media` table; files on disk under configured video/image roots.
|
||||
|
||||
## 5. Modules included
|
||||
|
||||
`media-service`.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Dataset Explorer (API)
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** Backend for **Dataset Explorer** — grid, detail, status PATCH, bulk status — `DATASET` policy; cross-ref `suite/_docs/09_dataset_explorer.md`.
|
||||
|
||||
**Architectural pattern:** Read-heavy + controlled writes on annotation status.
|
||||
|
||||
**Upstream dependencies:** Platform, shared annotation entities/status with Annotations REST.
|
||||
|
||||
**Downstream consumers:** Dataset Explorer UI.
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
`DatasetService`, `DatasetController` — `/dataset` routes.
|
||||
|
||||
## 3. External API
|
||||
|
||||
| Policy | Base |
|
||||
|--------|------|
|
||||
| `DATASET` | `/dataset` |
|
||||
|
||||
## 4. Modules included
|
||||
|
||||
`dataset-service`.
|
||||
@@ -0,0 +1,23 @@
|
||||
# Settings & metadata
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** System, directory, camera, and per-user UI settings; **detection class catalog** for labels/colors — `01_annotations.md` §11–12 (camera) plus settings/directory narratives; `GET /classes` for annotator.
|
||||
|
||||
**Architectural pattern:** CRUD settings services + thin metadata read controller.
|
||||
|
||||
**Upstream dependencies:** Platform (DB, auth — `ADM` on mutating settings).
|
||||
|
||||
**Downstream consumers:** All UIs; `PathResolver` after directory updates (reset).
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
`SettingsService`, `SettingsController`, `ClassesController`.
|
||||
|
||||
## 3. External API
|
||||
|
||||
Mixed `[Authorize]` and policy `ADM` for writes under `/settings`; `GET /classes` authenticated read.
|
||||
|
||||
## 4. Modules included
|
||||
|
||||
`settings-metadata-service`.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Platform foundation
|
||||
|
||||
## 1. High-Level Overview
|
||||
|
||||
**Purpose:** **Wire enums**, **PostgreSQL schema/mapping**, **cross-cutting HTTP error handling**, **path resolution**, **JWT policies + token refresh**, and **application bootstrap** — no standalone product feature; enables all other components.
|
||||
|
||||
**Architectural pattern:** Shared kernel / infrastructure.
|
||||
|
||||
**Upstream dependencies:** None (root).
|
||||
|
||||
**Downstream consumers:** All feature components.
|
||||
|
||||
## 2. Internal interfaces
|
||||
|
||||
- `src/Enums/*`
|
||||
- `src/Database/*` (`AppDataConnection`, `DatabaseMigrator`, entities)
|
||||
- `ErrorHandlingMiddleware`, `PathResolver`, `PaginatedResponse`, `ErrorResponse`, `GlobalUsings.cs`
|
||||
- `JwtExtensions` (JWKS verifier), `ConfigurationResolver`, `CorsConfigurationValidator`
|
||||
- `Program.cs`
|
||||
|
||||
## 3. External API
|
||||
|
||||
`/health` (AllowAnonymous); Swagger in development configuration. Token refresh is no longer hosted here — callers refresh against admin's `POST /token/refresh`.
|
||||
|
||||
## 4. Modules included
|
||||
|
||||
`wire-enums`, `database-layer`, `common-infrastructure`, `auth-identity`, `composition-program`.
|
||||
@@ -0,0 +1,254 @@
|
||||
# Azaion.Annotations — Data Model
|
||||
|
||||
> Source-of-truth: `src/Database/DatabaseMigrator.cs` and `src/Database/Entities/*.cs`. Every column name and type below is reproduced from migrator SQL.
|
||||
|
||||
## Schema overview
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
media ||--o{ annotations : "media_id"
|
||||
annotations ||--o{ detection : "annotation_id"
|
||||
annotations_queue_records }o..o{ annotations : "annotation_ids JSON (no FK)"
|
||||
detection_classes ||..o{ detection : "class_num (logical, no FK)"
|
||||
|
||||
media {
|
||||
TEXT id PK
|
||||
TEXT name
|
||||
TEXT path
|
||||
INTEGER media_type "MediaType enum"
|
||||
INTEGER media_status "MediaStatus enum"
|
||||
UUID waypoint_id
|
||||
UUID user_id
|
||||
TEXT duration "added later (ALTER)"
|
||||
}
|
||||
|
||||
annotations {
|
||||
TEXT id PK "image-bytes hash (ADR-004)"
|
||||
TEXT media_id FK
|
||||
BIGINT time "ticks of TimeSpan"
|
||||
TIMESTAMP created_date
|
||||
UUID user_id
|
||||
INTEGER source "AnnotationSource enum"
|
||||
INTEGER status "AnnotationStatus enum"
|
||||
BOOLEAN is_split "added via ALTER"
|
||||
TEXT split_tile "added via ALTER"
|
||||
}
|
||||
|
||||
detection {
|
||||
UUID id PK
|
||||
REAL center_x
|
||||
REAL center_y
|
||||
REAL width
|
||||
REAL height
|
||||
INTEGER class_num
|
||||
TEXT label
|
||||
TEXT description
|
||||
REAL confidence
|
||||
INTEGER affiliation "AffiliationEnum"
|
||||
INTEGER combat_readiness "CombatReadiness"
|
||||
TEXT annotation_id FK
|
||||
}
|
||||
|
||||
annotations_queue_records {
|
||||
UUID id PK
|
||||
TIMESTAMP date_time
|
||||
INTEGER operation "QueueOperation enum"
|
||||
TEXT annotation_ids "JSON array of TEXT ids"
|
||||
}
|
||||
|
||||
system_settings {
|
||||
UUID id PK
|
||||
TEXT name
|
||||
TEXT military_unit
|
||||
INTEGER default_camera_width
|
||||
NUMERIC default_camera_fov
|
||||
INTEGER thumbnail_width "default 240"
|
||||
INTEGER thumbnail_height "default 135"
|
||||
INTEGER thumbnail_border "default 10"
|
||||
BOOLEAN generate_annotated_image "default false"
|
||||
BOOLEAN silent_detection "default false"
|
||||
}
|
||||
|
||||
directory_settings {
|
||||
UUID id PK
|
||||
TEXT videos_dir "default /data/videos"
|
||||
TEXT images_dir "default /data/images"
|
||||
TEXT labels_dir "default /data/labels"
|
||||
TEXT results_dir "default /data/results"
|
||||
TEXT thumbnails_dir "default /data/thumbnails"
|
||||
TEXT gps_sat_dir "default /data/gps_sat"
|
||||
TEXT gps_route_dir "default /data/gps_route"
|
||||
}
|
||||
|
||||
detection_classes {
|
||||
SERIAL id PK
|
||||
TEXT name
|
||||
TEXT short_name
|
||||
TEXT color "hex e.g. #FF0000"
|
||||
INTEGER max_size_m
|
||||
INTEGER photo_mode
|
||||
}
|
||||
|
||||
user_settings {
|
||||
UUID id PK
|
||||
UUID user_id "UNIQUE (ix_user_settings_user_id)"
|
||||
UUID selected_flight_id
|
||||
NUMERIC annotations_left_panel_width
|
||||
NUMERIC annotations_right_panel_width
|
||||
NUMERIC dataset_left_panel_width
|
||||
NUMERIC dataset_right_panel_width
|
||||
}
|
||||
|
||||
camera_settings {
|
||||
UUID id PK
|
||||
NUMERIC altitude "default 100"
|
||||
NUMERIC focal_length "default 50"
|
||||
NUMERIC sensor_width "default 36"
|
||||
}
|
||||
```
|
||||
|
||||
> Mermaid `erDiagram` does not represent JSON-array references; the dotted line for `annotations_queue_records ↔ annotations` is logical only — there is **no FK** in the schema.
|
||||
|
||||
## Tables
|
||||
|
||||
### `media`
|
||||
|
||||
Owned writes: `03_media`. Reads: `01_annotations-rest`, `04_dataset`.
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | TEXT PK | Application-generated |
|
||||
| `name` | TEXT NOT NULL | |
|
||||
| `path` | TEXT NOT NULL | Filesystem path under media dir |
|
||||
| `media_type` | INTEGER NOT NULL DEFAULT 0 | `MediaType` enum (numeric wire — see `wire-enums.md`) |
|
||||
| `media_status` | INTEGER NOT NULL DEFAULT 0 | `MediaStatus` enum |
|
||||
| `waypoint_id` | UUID | Indexed `ix_media_waypoint_id` |
|
||||
| `user_id` | UUID NOT NULL | |
|
||||
| `duration` | TEXT | Added via `ALTER`; nullable |
|
||||
|
||||
### `annotations`
|
||||
|
||||
Owned writes: `01_annotations-rest`. Status writes: `04_dataset` (bulk + single PATCH).
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | TEXT PK | **Hash of image bytes** (ADR-004); collision implication noted |
|
||||
| `media_id` | TEXT NOT NULL FK → `media.id` | |
|
||||
| `time` | BIGINT NOT NULL DEFAULT 0 | Ticks of `TimeSpan` (suite spec stores `time` as ticks) |
|
||||
| `created_date` | TIMESTAMP NOT NULL DEFAULT NOW() | Indexed `ix_annotations_created_date` |
|
||||
| `user_id` | UUID NOT NULL | Indexed `ix_annotations_user_id` |
|
||||
| `source` | INTEGER NOT NULL DEFAULT 0 | `AnnotationSource` enum (AI=0, Manual=1) |
|
||||
| `status` | INTEGER NOT NULL DEFAULT 0 | `AnnotationStatus` enum (Created=10, Edited=20, …) |
|
||||
| `is_split` | BOOLEAN NOT NULL DEFAULT false | Added via `ALTER`; tile-splitting flag |
|
||||
| `split_tile` | TEXT | Tile id reference |
|
||||
|
||||
Indexes: `ix_annotations_media_id`, `ix_annotations_created_date`, `ix_annotations_user_id`.
|
||||
|
||||
### `detection`
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | UUID PK | |
|
||||
| `center_x`, `center_y`, `width`, `height` | REAL NOT NULL | YOLO-normalized box |
|
||||
| `class_num` | INTEGER NOT NULL | Logical reference to `detection_classes.id` |
|
||||
| `label` | TEXT NOT NULL DEFAULT '' | |
|
||||
| `description` | TEXT | |
|
||||
| `confidence` | REAL NOT NULL DEFAULT 0 | |
|
||||
| `affiliation` | INTEGER NOT NULL DEFAULT 0 | `AffiliationEnum` |
|
||||
| `combat_readiness` | INTEGER NOT NULL DEFAULT 0 | `CombatReadiness` enum |
|
||||
| `annotation_id` | TEXT NOT NULL FK → `annotations.id` | Indexed `ix_detection_annotation_id` |
|
||||
|
||||
### `annotations_queue_records` (failsafe outbox)
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `id` | UUID PK | |
|
||||
| `date_time` | TIMESTAMP NOT NULL DEFAULT NOW() | |
|
||||
| `operation` | INTEGER NOT NULL DEFAULT 0 | `QueueOperation` enum |
|
||||
| `annotation_ids` | TEXT NOT NULL DEFAULT '[]' | JSON array of annotation ids — single or bulk |
|
||||
|
||||
No FK to `annotations` — by design, since rows can survive an annotation deletion if export is in flight.
|
||||
|
||||
### `system_settings`
|
||||
|
||||
Singleton-ish (one row in practice). Includes:
|
||||
|
||||
- `generate_annotated_image` (BOOLEAN) — emits a baked-in annotated image alongside YOLO label when true (suite spec).
|
||||
- `silent_detection` (BOOLEAN) — suppresses SSE / sync for detection events.
|
||||
- `thumbnail_*` — defaults 240×135 with 10 border.
|
||||
|
||||
### `directory_settings`
|
||||
|
||||
Roots consumed by `PathResolver` (`06_platform`). Defaults: `/data/{videos,images,labels,results,thumbnails,gps_sat,gps_route}`. Updates require `PathResolver.Reset` (Flow F7 invariant).
|
||||
|
||||
### `detection_classes`
|
||||
|
||||
Seeded with 19 rows (ids 0–18) on first run via `INSERT ... ON CONFLICT (id) DO NOTHING`. Names + Cyrillic short names + hex colors + `max_size_m` + `photo_mode`.
|
||||
|
||||
| id | name | short_name | color | max_size_m |
|
||||
|----|------|------------|-------|-------------|
|
||||
| 0 | ArmorVehicle | Броня | `#FF0000` | 7 |
|
||||
| 1 | Truck | Вантаж. | `#00FF00` | 8 |
|
||||
| 2 | Vehicle | Машина | `#0000FF` | 7 |
|
||||
| 3 | Artillery | Арта | `#FFFF00` | 14 |
|
||||
| 4 | Shadow | Тінь | `#FF00FF` | 9 |
|
||||
| 5 | Trenches | Окопи | `#00FFFF` | 10 |
|
||||
| 6 | MilitaryMan | Військов | `#188021` | 2 |
|
||||
| 7 | TyreTracks | Накати | `#800000` | 5 |
|
||||
| 8 | AdditionArmoredTank | Танк.захист | `#008000` | 7 |
|
||||
| 9 | Smoke | Дим | `#000080` | 8 |
|
||||
| 10 | Plane | Літак | `#000080` | 12 |
|
||||
| 11 | Moto | Мото | `#808000` | 3 |
|
||||
| 12 | CamouflageNet | Сітка | `#800080` | 14 |
|
||||
| 13 | CamouflageBranches | Гілки | `#2f4f4f` | 8 |
|
||||
| 14 | Roof | Дах | `#1e90ff` | 15 |
|
||||
| 15 | Building | Будівля | `#ffb6c1` | 20 |
|
||||
| 16 | Caponier | Капонір | `#ffb6c1` | 10 |
|
||||
| 17 | Ammo | БК | `#33658a` | 2 |
|
||||
| 18 | Protect.Struct | Зуби.драк | `#969647` | 2 |
|
||||
|
||||
Note: ids 9 and 10 (`Smoke`, `Plane`) share `#000080` — a pre-existing data quirk, not a bug introduced by this skill.
|
||||
|
||||
### `user_settings`
|
||||
|
||||
Per-user UI prefs. Unique index on `user_id` (`ix_user_settings_user_id`). Carries selected flight + four panel widths (annotator left/right, dataset left/right).
|
||||
|
||||
### `camera_settings`
|
||||
|
||||
Calibration triple `(altitude, focal_length, sensor_width)` with defaults `(100, 50, 36)`.
|
||||
|
||||
## Migration strategy
|
||||
|
||||
- **Tool**: hand-rolled embedded SQL in `DatabaseMigrator.Migrate`, executed at every startup via Linq2DB.
|
||||
- **Safety**: every statement is idempotent — `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE … ADD COLUMN IF NOT EXISTS`, seed `INSERT … ON CONFLICT DO NOTHING`.
|
||||
- **Direction**: forward-only. No down migrations or `DROP` operations; renames or destructive changes require an out-of-band migration.
|
||||
- **Drift**: the only authoritative schema definition is `Database/DatabaseMigrator.cs`. Live DBs should be diffed against it on cadence; suite-level monitoring is out of scope here.
|
||||
|
||||
## Seed data observations
|
||||
|
||||
Only `detection_classes` has seeded data; all other tables start empty. `system_settings`, `directory_settings`, and `camera_settings` are inserted **lazily** by their respective services on first read/write — confirm exact upsert semantics in Step 4 verification.
|
||||
|
||||
## Backward compatibility
|
||||
|
||||
- Wire enums are **integer-stable** (suite contract). Renaming an enum case does not break wire compatibility because numeric values are the contract.
|
||||
- Annotation id format is the hash of image bytes — changing the hashing algorithm would invalidate cross-build references; treat as a contract.
|
||||
- MessagePack key order in `DTOs/QueueMessages.cs` is the export contract for RabbitMQ stream consumers — changing it breaks downstream.
|
||||
|
||||
## Cross-component data ownership
|
||||
|
||||
| Component | Writes | Reads |
|
||||
|-----------|--------|-------|
|
||||
| `01_annotations-rest` | `annotations`, `detection`, files on disk, `annotations_queue_records` (Created/Updated/Deleted) | `media` |
|
||||
| `02_annotations-realtime-sync` | drains `annotations_queue_records` | `annotations`, `detection`, file bytes |
|
||||
| `03_media` | `media`, files on disk | — |
|
||||
| `04_dataset` | `annotations.status` (single + bulk) → also writes `annotations_queue_records`, publishes SSE | `annotations`, `detection`, `media` |
|
||||
| `05_settings-metadata` | all `*_settings` tables | `detection_classes` (read-through for UI) |
|
||||
| `06_platform` | none (pure infra) | `directory_settings` (via `PathResolver`) |
|
||||
|
||||
## Open data-model questions (Step 4 verification)
|
||||
|
||||
1. **`annotations.id` collisions**: behavior under same-bytes re-upload (insert vs noop vs error) is implicit — confirm in `AnnotationService`.
|
||||
2. **`annotations_queue_records.annotation_ids` shape**: confirm consistent JSON formatting (escaped strings vs raw) across `Created`, `Updated`, `StatusChanged`, `Deleted`, bulk variants.
|
||||
3. **`detection_classes` mutability**: schema permits inserts via `ALTER`/seed, but no controller exposes writes today — confirm whether class catalog is intended to be DB-managed or static.
|
||||
4. **`media.duration`**: nullable TEXT — confirm format (`hh:mm:ss` vs ISO 8601 vs ticks).
|
||||
5. **Lazy upsert** of `system_settings` / `directory_settings` / `camera_settings` first-row creation — confirm services initialize defaults vs rely on user-driven inserts.
|
||||
@@ -0,0 +1,72 @@
|
||||
# CI / CD Pipeline
|
||||
|
||||
Source of truth: `.woodpecker/build-arm.yml`.
|
||||
|
||||
## Engine
|
||||
|
||||
Woodpecker CI. No GitHub Actions / GitLab CI / Azure Pipelines configured in this repo — `.github/workflows/` is absent (`00_discovery.md`). Suite-wide CI may layer on top of this; that lives outside the workspace.
|
||||
|
||||
## Trigger
|
||||
|
||||
```yaml
|
||||
when:
|
||||
event: [push, manual]
|
||||
branch: [dev, stage, main]
|
||||
```
|
||||
|
||||
- Builds run on push to **`dev`**, **`stage`**, or **`main`**, plus manual triggers.
|
||||
- Other branches do **not** build images.
|
||||
|
||||
## Runner constraint
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
platform: arm64
|
||||
```
|
||||
|
||||
Pipeline pins to ARM64 runners. The Dockerfile is multi-arch capable but this pipeline only builds `arm64`.
|
||||
|
||||
## Steps (single step `build-push`)
|
||||
|
||||
1. Login to private registry using secrets `registry_host`, `registry_user`, `registry_token`.
|
||||
2. Compute `TAG=${CI_COMMIT_BRANCH}-arm` and `BUILD_DATE` (`date -u +%Y-%m-%dT%H:%M:%SZ`).
|
||||
3. `docker build -f src/Dockerfile` with build args + OCI labels:
|
||||
- `--build-arg CI_COMMIT_SHA=$CI_COMMIT_SHA`
|
||||
- `--label org.opencontainers.image.revision=$CI_COMMIT_SHA`
|
||||
- `--label org.opencontainers.image.created=$BUILD_DATE`
|
||||
- `--label org.opencontainers.image.source=$CI_REPO_URL`
|
||||
- tag: `$REGISTRY_HOST/azaion/annotations:$TAG`
|
||||
4. `docker push` of that tag.
|
||||
5. Mounts `/var/run/docker.sock` into the build container (Docker-out-of-Docker pattern).
|
||||
|
||||
## Image tagging
|
||||
|
||||
Per branch:
|
||||
|
||||
| Branch | Image tag |
|
||||
|--------|-----------|
|
||||
| `dev` | `dev-arm` |
|
||||
| `stage` | `stage-arm` |
|
||||
| `main` | `main-arm` |
|
||||
|
||||
Tags are **mutable** — every push to a branch overwrites the prior image at that tag. No immutable revision-tagged images are produced today (`main-arm-${SHA}` is not pushed). Adding immutable tags would simplify rollback and trace-back from a running image to a commit.
|
||||
|
||||
## Secrets
|
||||
|
||||
| Secret | Purpose |
|
||||
|--------|---------|
|
||||
| `registry_host` | Registry hostname (also used in pushed image FQN) |
|
||||
| `registry_user` | Registry login user |
|
||||
| `registry_token` | Registry login token (used via `--password-stdin`) |
|
||||
|
||||
Secrets are referenced via `from_secret:` and never echoed.
|
||||
|
||||
## What CI does NOT do today
|
||||
|
||||
- No tests run (no test project exists in repo per `00_discovery.md`).
|
||||
- No linters / format checks (`dotnet format`).
|
||||
- No `amd64` image.
|
||||
- No scan (Trivy / Grype) on the produced image.
|
||||
- No automated rollback on failed deploy (deploy itself is out of pipeline scope).
|
||||
|
||||
These are gaps to track when the test project is added in autodev Phase A Step 6.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Containerization
|
||||
|
||||
Source of truth: `src/Dockerfile`.
|
||||
|
||||
## Build
|
||||
|
||||
Two-stage build:
|
||||
|
||||
1. **build stage** — `mcr.microsoft.com/dotnet/sdk:10.0`, `--platform=$BUILDPLATFORM`. Reads `$TARGETARCH` and runs `dotnet publish -c Release -o /app --os linux --arch $arch` (mapping `amd64 → x64`, otherwise `$TARGETARCH`).
|
||||
2. **runtime stage** — `mcr.microsoft.com/dotnet/aspnet:10.0`. Copies the published output, exposes port `8080`, sets `ENTRYPOINT ["dotnet", "Azaion.Annotations.dll"]`.
|
||||
|
||||
## Build arguments
|
||||
|
||||
| Arg | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `BUILDPLATFORM` | provided by Buildx | Multi-arch host platform |
|
||||
| `TARGETARCH` | provided by Buildx | Output arch (`amd64` / `arm64`) |
|
||||
| `CI_COMMIT_SHA` | `unknown` | Stamped into `AZAION_REVISION` env at runtime |
|
||||
|
||||
## Runtime
|
||||
|
||||
| Aspect | Value |
|
||||
|--------|-------|
|
||||
| Base image | `mcr.microsoft.com/dotnet/aspnet:10.0` |
|
||||
| Working dir | `/app` |
|
||||
| Exposed port | `8080` (HTTP) |
|
||||
| Entry point | `dotnet Azaion.Annotations.dll` |
|
||||
| Runtime env stamped at build | `AZAION_REVISION = $CI_COMMIT_SHA` |
|
||||
|
||||
## Multi-arch
|
||||
|
||||
Dockerfile is multi-arch capable via Buildx. The current Woodpecker pipeline emits **`arm64` only** (label `platform: arm64`, tag `${BRANCH}-arm`). Producing `amd64` requires an additional pipeline (or extending the existing one to a matrix).
|
||||
|
||||
## Image size & caching
|
||||
|
||||
- Layers: SDK install → `COPY . .` → publish → runtime copy. The final layer is the published `/app` directory only — no SDK in runtime image.
|
||||
- Cache hit on `COPY . .` is wide (entire `src/`); finer caching (e.g., `COPY *.csproj` first, then `dotnet restore`, then sources) is **not configured** — improvement candidate.
|
||||
|
||||
## Image labels
|
||||
|
||||
Set in CI (`.woodpecker/build-arm.yml`), not in the Dockerfile:
|
||||
|
||||
- `org.opencontainers.image.revision = $CI_COMMIT_SHA`
|
||||
- `org.opencontainers.image.created = $BUILD_DATE`
|
||||
- `org.opencontainers.image.source = $CI_REPO_URL`
|
||||
|
||||
These follow the OCI standard so the registry surfaces them correctly.
|
||||
|
||||
## Open items
|
||||
|
||||
- Add `amd64` build target if non-ARM hosts are required.
|
||||
- Consider non-root user inside the runtime image (none configured today).
|
||||
- Consider `dotnet restore` cache layer split for faster CI builds.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Environment Strategy
|
||||
|
||||
Source of truth: `src/Program.cs` + `src/Database/DatabaseMigrator.cs` + `.woodpecker/build-arm.yml`.
|
||||
|
||||
## Environments
|
||||
|
||||
Branch-driven from CI:
|
||||
|
||||
| Branch | Image tag | Intended environment |
|
||||
|--------|-----------|----------------------|
|
||||
| `dev` | `dev-arm` | Development (shared) |
|
||||
| `stage` | `stage-arm` | Pre-production |
|
||||
| `main` | `main-arm` | Production |
|
||||
|
||||
The service binary is identical across environments — all variation is **runtime configuration via env vars** (no per-environment build flags).
|
||||
|
||||
## Configuration sources (priority order, per `Program.cs`)
|
||||
|
||||
1. `Environment.GetEnvironmentVariable("KEY")`.
|
||||
2. ASP.NET Core `IConfiguration` (`builder.Configuration["KEY"]`) — covers `appsettings.json`, command-line args, etc.
|
||||
3. **No hard-coded fallback for security-sensitive values.** `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL`, and (in `Production`) a non-empty `CorsConfig:AllowedOrigins` are required; missing values cause startup to fail fast via `ConfigurationResolver.ResolveRequiredOrThrow` / `CorsConfigurationValidator.EnsureSafeForEnvironment`.
|
||||
|
||||
## Required environment variables
|
||||
|
||||
| Variable | Purpose | Default | Production action |
|
||||
|----------|---------|---------|---------------------|
|
||||
| `DATABASE_URL` | Postgres connection (URL or LinqToDB conn string) | — (required, fail-fast) | **MUST set** |
|
||||
| `JWT_ISSUER` | Expected `iss` claim; must match admin's `JwtConfig:Issuer` | — (required, fail-fast) | **MUST set** |
|
||||
| `JWT_AUDIENCE` | Expected `aud` claim; must match admin's `JwtConfig:Audience` | — (required, fail-fast) | **MUST set** |
|
||||
| `JWT_JWKS_URL` | Admin's JWKS endpoint (HTTPS) | — (required, fail-fast) | `https://admin.azaion.com/.well-known/jwks.json` |
|
||||
| `CorsConfig__AllowedOrigins__0` | First allowed CORS origin (array via `__N` indices) | — | **MUST set** (or `CorsConfig__AllowAnyOrigin=true`) in Production |
|
||||
| `CorsConfig__AllowAnyOrigin` | Opt-in to permissive CORS (non-production only) | `false` | Leave `false` in Production |
|
||||
| `RABBITMQ_HOST` | Stream host | `127.0.0.1` | Override |
|
||||
| `RABBITMQ_STREAM_PORT` | Stream port | `5552` | Override if non-default |
|
||||
| `RABBITMQ_PRODUCER_USER` | Stream user | `azaion_producer` | Override |
|
||||
| `RABBITMQ_PRODUCER_PASS` | Stream password | `producer_pass` | Override |
|
||||
| `RABBITMQ_STREAM_NAME` | Stream name | `azaion-annotations` | Usually keep (suite contract) |
|
||||
|
||||
`JWT_SECRET` was removed in this cycle — annotations no longer mints HS256 tokens; admin is the sole token issuer (ES256).
|
||||
|
||||
## URL format conversion
|
||||
|
||||
`Program.cs` accepts `DATABASE_URL` either as a Linq2DB connection string or as a `postgresql://user:pass@host:port/db` URL. The `ConvertPostgresUrl` helper rewrites the URL form into LinqToDB conn-string form. This means operators can use either ENV-style URLs (kubectl/Postgres operator output) or `Host=...` directly.
|
||||
|
||||
## DB-driven configuration
|
||||
|
||||
Several runtime concerns are stored in **database tables**, not env:
|
||||
|
||||
- **Filesystem roots** — `directory_settings` (defaults `/data/...`). Updated via `PUT /settings/directories`; **must trigger** `PathResolver.Reset` for the change to take effect (Flow F7).
|
||||
- **System settings** — `system_settings` (`generate_annotated_image`, `silent_detection`, thumbnail dimensions).
|
||||
- **User settings** — `user_settings` (per UI session prefs).
|
||||
|
||||
Operators changing filesystem layout in production need an `ADM` JWT and the right cluster connectivity, **not** a redeploy.
|
||||
|
||||
## Filesystem mounts
|
||||
|
||||
The container expects `/data/` (or whatever `directory_settings` points at) to be a **writable persistent mount**:
|
||||
|
||||
- `/data/images` — annotation full images
|
||||
- `/data/labels` — YOLO `.txt` files
|
||||
- `/data/thumbnails` — thumbnails
|
||||
- `/data/results` — annotated images (when `generate_annotated_image=true`)
|
||||
- `/data/videos` — media uploads
|
||||
- `/data/gps_sat`, `/data/gps_route` — GPS overlays
|
||||
|
||||
Without these mounts, every annotation-create / media-upload flow returns 500 from `ErrorHandlingMiddleware` (FS write fails).
|
||||
|
||||
## Config drift between environments
|
||||
|
||||
Today, environment-specific config is held wherever the deployment platform places env vars (Helm values / Kustomize overlays / Compose files in `_infra/`). This repo intentionally does not commit per-environment values; the only environment-aware file in-repo is `.woodpecker/build-arm.yml`.
|
||||
|
||||
## Open items
|
||||
|
||||
- No `appsettings.Production.json` — all env-specific config is operator-supplied.
|
||||
- `Swagger UI` is mounted in all environments (ADR-005); production exposure must be controlled at the perimeter.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Observability
|
||||
|
||||
Source of truth: `src/Program.cs` (no dedicated logging config files exist in repo).
|
||||
|
||||
## Health check
|
||||
|
||||
```csharp
|
||||
app.MapGet("/health", () => Results.Ok(new { status = "healthy" }));
|
||||
```
|
||||
|
||||
- Path: `GET /health`
|
||||
- Auth: none (`MapGet` bypasses controller-level `[Authorize]`).
|
||||
- Response: `200 { "status": "healthy" }`
|
||||
- **Liveness only**: this endpoint does not probe the DB, RabbitMQ, or filesystem. A pod can return healthy while the failsafe outbox is unable to publish or while DB connectivity is broken.
|
||||
|
||||
## API documentation
|
||||
|
||||
- `app.UseSwagger()` and `app.UseSwaggerUI()` mounted unconditionally (ADR-005).
|
||||
- Endpoints: `/swagger/v1/swagger.json` (OpenAPI), `/swagger/index.html` (UI).
|
||||
- No version pinning of the OpenAPI document (Swashbuckle defaults).
|
||||
|
||||
## Logging
|
||||
|
||||
- Default ASP.NET Core console logger. No `appsettings.json` overrides in repo.
|
||||
- No structured logger (Serilog / NLog) configured.
|
||||
- No correlation id middleware in repo (`X-Request-Id` not propagated).
|
||||
|
||||
## Metrics
|
||||
|
||||
None configured today. Possible additions:
|
||||
- `prometheus-net` exporter on `/metrics`.
|
||||
- ASP.NET Core `MetricsCollector` (built-in HTTP / runtime counters).
|
||||
|
||||
## Traces
|
||||
|
||||
None configured. OpenTelemetry SDK is not referenced in `csproj`.
|
||||
|
||||
## Image revision stamp
|
||||
|
||||
The runtime container has `AZAION_REVISION = $CI_COMMIT_SHA` set as an env var (Dockerfile + Woodpecker pipeline). This makes "what's running?" diagnosable from inside the container with `printenv AZAION_REVISION` or by surfacing it in a future `/info` endpoint.
|
||||
|
||||
## Error visibility to clients
|
||||
|
||||
`ErrorHandlingMiddleware` maps exceptions to JSON `{ statusCode, message }` with HTTP 400 / 404 / 409 / 500. Internal exception details are not leaked beyond the `message` string (confirm during Step 4 verification — make sure 500 paths do not echo stack traces).
|
||||
|
||||
## Open observability items
|
||||
|
||||
- **Readiness vs liveness split**: today there is one endpoint that does not check dependencies. A `GET /ready` that pings DB and (optionally) RabbitMQ would unblock proper rolling-update gates.
|
||||
- **Structured logs** with request id correlation across HTTP + outbox drain + SSE.
|
||||
- **Outbox depth metric** (`COUNT(*)` on `annotations_queue_records`) — surfaces stuck-failsafe scenarios early.
|
||||
- **SSE subscriber count metric** — visibility into connected UIs.
|
||||
- **Stream publish lag** — time from outbox row insertion to RabbitMQ publish.
|
||||
- **Failure injection / chaos hooks** — none today.
|
||||
|
||||
These are candidates for the deploy-and-retro phase of autodev (Steps 14–17 once the project enters Phase B).
|
||||
@@ -0,0 +1,44 @@
|
||||
# Component diagram (Azaion.Annotations)
|
||||
|
||||
Derived from the **six-component** breakdown (user choice **B**: Annotations REST split from realtime + RabbitMQ sync).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph platform [06 Platform]
|
||||
DB[(PostgreSQL)]
|
||||
AUTH[JWT / refresh]
|
||||
PATH[Paths + errors]
|
||||
end
|
||||
|
||||
subgraph media [03 Media]
|
||||
MAPI["/media"]
|
||||
end
|
||||
|
||||
subgraph annRest [01 Annotations REST]
|
||||
ARAPI["/annotations REST + files"]
|
||||
end
|
||||
|
||||
subgraph annRT [02 Annotations realtime and sync]
|
||||
SSE["SSE /events"]
|
||||
RMQ[RabbitMQ stream]
|
||||
end
|
||||
|
||||
subgraph dataset [04 Dataset]
|
||||
DAPI["/dataset DATASET"]
|
||||
end
|
||||
|
||||
subgraph settings [05 Settings and metadata]
|
||||
SAPI["/settings /classes"]
|
||||
end
|
||||
|
||||
platform --> media
|
||||
platform --> annRest
|
||||
platform --> annRT
|
||||
platform --> dataset
|
||||
platform --> settings
|
||||
media --> annRest
|
||||
annRest --> annRT
|
||||
annRest --> dataset
|
||||
```
|
||||
|
||||
**Shared source file:** `AnnotationsController.cs` is **split by concern** between **01** (REST + static files) and **02** (SSE `Events` action).
|
||||
@@ -0,0 +1,83 @@
|
||||
# Flow F1 — Annotation Create
|
||||
|
||||
Cross-reference: `system-flows.md` → Flow F1.
|
||||
|
||||
## Sequence (verified against `Services/AnnotationService.cs`)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Caller as Detections / UI
|
||||
participant Ctrl as AnnotationsController (01)
|
||||
participant Svc as AnnotationService (01)
|
||||
participant Path as PathResolver (06)
|
||||
participant DB as PostgreSQL (06)
|
||||
participant FS as Filesystem
|
||||
participant Evt as AnnotationEventService (02)
|
||||
participant Q as annotations_queue_records (DB / 02)
|
||||
|
||||
Caller->>Ctrl: POST /annotations (CreateAnnotationRequest, JWT ANN)
|
||||
Ctrl->>Svc: CreateAnnotation(request, userIdFromJwt)
|
||||
|
||||
alt request.Image bytes provided
|
||||
Svc->>Svc: ComputeHash (XxHash64 over sampled bytes) -> id
|
||||
Svc->>FS: write {id}.jpg under images_dir
|
||||
Svc->>DB: SELECT media WHERE id = :id
|
||||
opt media row missing
|
||||
Svc->>DB: INSERT media (Image, MediaStatus.New, ...)
|
||||
end
|
||||
else MediaId provided
|
||||
Svc->>DB: SELECT media WHERE id = :MediaId (404 if missing)
|
||||
opt source media file exists & target image missing
|
||||
Svc->>FS: copy media.Path -> images_dir/{id}.jpg
|
||||
end
|
||||
end
|
||||
|
||||
Svc->>DB: INSERT annotations
|
||||
Svc->>DB: BulkCopy detection rows
|
||||
Svc->>FS: write {id}.txt (YOLO label) under labels_dir
|
||||
Svc->>Evt: PublishAsync(AnnotationEventDto)
|
||||
Svc->>DB: SELECT system_settings (FirstOrDefault)
|
||||
alt SilentDetection != true
|
||||
Svc->>Q: FailsafeProducer.EnqueueAsync(db, id, QueueOperation.Created)
|
||||
end
|
||||
Svc-->>Ctrl: Annotation
|
||||
Ctrl-->>Caller: 201 Created (Location: /annotations/{id})
|
||||
```
|
||||
|
||||
## Flowchart
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
start([POST /annotations]) --> auth{JWT valid + ANN claim?}
|
||||
auth -->|no| rej401([401 / 403])
|
||||
auth -->|yes| input{bytes or MediaId?}
|
||||
input -->|neither| arg([400 ArgumentException])
|
||||
input -->|bytes| hash[ComputeHash sampled XxHash64 -> id]
|
||||
input -->|MediaId| lookupMedia[SELECT media WHERE id = MediaId]
|
||||
lookupMedia -->|missing| nf404([404 KeyNotFound])
|
||||
lookupMedia -->|exists| copyImg[copy media.Path to images dir if missing]
|
||||
hash --> writeImg[write {id}.jpg]
|
||||
writeImg --> mediaRow[INSERT media if absent]
|
||||
mediaRow --> writeDb
|
||||
copyImg --> writeDb[INSERT annotations + BulkCopy detections]
|
||||
writeDb --> writeLabel[write {id}.txt YOLO label]
|
||||
writeLabel --> sse[PublishAsync SSE event]
|
||||
sse --> readSettings[SELECT system_settings]
|
||||
readSettings --> silentChk{SilentDetection?}
|
||||
silentChk -->|yes| ok([201 Created])
|
||||
silentChk -->|no| outbox[FailsafeProducer.EnqueueAsync Created]
|
||||
outbox --> ok
|
||||
|
||||
writeImg -->|IOException| err500([500 via ErrorHandlingMiddleware])
|
||||
writeDb -->|DB error| err500
|
||||
writeLabel -->|IOException| err500
|
||||
outbox -->|DB error| err500
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Image hashing is `XxHash64` over a **sampled** input (length prefix + head/middle/tail 1KB) for inputs > 3072 bytes. See ADR-004 in `architecture.md` for collision implications.
|
||||
- The implementation is **not transactional across FS + DB + outbox**. Partial failure can leave orphan files or unsent outbox rows. Captured in `system-flows.md` → Open Behavioral Questions §4.
|
||||
- `Update`, `UpdateStatus`, `DeleteAnnotation` paths do **NOT** publish SSE or enqueue outbox today. Captured in `system-flows.md` → Open Behavioral Questions §1.
|
||||
- Outbox row is consumed asynchronously by Flow F4 (`FailsafeProducer`).
|
||||
@@ -0,0 +1,52 @@
|
||||
# Flow F4 — Failsafe Outbox Drain → RabbitMQ Stream
|
||||
|
||||
Cross-reference: `system-flows.md` → Flow F4.
|
||||
|
||||
## Sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant FP as FailsafeProducer (02, IHostedService)
|
||||
participant DB
|
||||
participant Path as PathResolver (06)
|
||||
participant FS as Filesystem
|
||||
participant RMQ as RabbitMQ Stream "azaion-annotations"
|
||||
|
||||
loop while host running
|
||||
FP->>DB: SELECT FROM annotations_queue_records
|
||||
DB-->>FP: pending rows (operation, annotation_ids JSON)
|
||||
loop per row
|
||||
alt operation = Created
|
||||
FP->>Path: GetImagePath(annotationId)
|
||||
FP->>FS: read bytes
|
||||
end
|
||||
FP->>FP: serialize MessagePack (Annotation*QueueMessage)
|
||||
FP->>RMQ: publish stream entry
|
||||
alt publish ok
|
||||
FP->>DB: DELETE annotations_queue_records WHERE id = :id
|
||||
else stream unavailable
|
||||
FP->>FP: log + backoff
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## State
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Idle
|
||||
Idle --> Draining: queue rows present
|
||||
Draining --> Publishing: row picked
|
||||
Publishing --> Acked: stream publish ok
|
||||
Acked --> Idle: row deleted
|
||||
Publishing --> Backoff: stream unavailable
|
||||
Backoff --> Idle: backoff elapsed
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- See ADR-003 in `architecture.md` for rationale.
|
||||
- Multi-instance drain: no leasing in DB → duplicate stream entries possible. Suite consumer contract should dedupe.
|
||||
- Bulk message (`AnnotationBulkQueueMessage`) carries multiple annotation ids; `Created` semantics on bulk are out of scope here — confirm during Step 4 verification.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Flow F3 — Real-time SSE Subscription
|
||||
|
||||
Cross-reference: `system-flows.md` → Flow F3.
|
||||
|
||||
## Sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant Ctrl as AnnotationsController.Events (component 02 doc-ownership)
|
||||
participant Evt as AnnotationEventService (02)
|
||||
participant ProducerF1 as Flow F1 (annotation create)
|
||||
participant ProducerF8 as Flow F8 (dataset bulk status)
|
||||
|
||||
UI->>Ctrl: GET /annotations/events (Accept: text/event-stream, JWT ANN)
|
||||
Ctrl->>Ctrl: set Content-Type: text/event-stream, no-cache
|
||||
Ctrl->>Evt: ReadAllAsync(cancellationToken)
|
||||
par event sources
|
||||
ProducerF1->>Evt: PublishAsync(eventDto)
|
||||
ProducerF8->>Evt: PublishAsync(eventDto)
|
||||
end
|
||||
Evt-->>Ctrl: yield AnnotationEventDto
|
||||
Ctrl-->>UI: data: {json}\n\n
|
||||
UI--xCtrl: client disconnects
|
||||
Ctrl->>Ctrl: cancellation token fires; loop exits
|
||||
```
|
||||
|
||||
## State
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Subscribing
|
||||
Subscribing --> Streaming: header sent + reader attached
|
||||
Streaming --> Streaming: PublishAsync -> data frame
|
||||
Streaming --> Closed: client cancel / process restart
|
||||
Closed --> [*]
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Channel is **unbounded**: a slow client cannot back-pressure the producer. If a client stalls indefinitely, memory growth is bounded by per-publisher cancellation tokens at the controller level. Step 4 verification candidate.
|
||||
- Cross-pod fan-out is **not provided** — each pod has its own channel. Sticky sessions or a broker-backed bus required for horizontal scale.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Glossary
|
||||
|
||||
**Status**: confirmed-by-user 2026-05-14.
|
||||
|
||||
System-wide terminology for `Azaion.Annotations`. Generic CS / industry terms (HTTP, JWT mechanics, REST, etc.) are excluded — only project-specific or domain-specific terms are listed. Each entry cites the doc or source file that establishes it.
|
||||
|
||||
---
|
||||
|
||||
**Annotation** — Hash-keyed record carrying detections, status, source, user, and time, attached to a media row. Central object of the service. *source: `data_model.md`, `modules/annotations-service.md`.*
|
||||
|
||||
**Annotation event** — SSE payload (`AnnotationEventDto`) describing a lifecycle change broadcast to UI subscribers. *source: `modules/sse-realtime.md`, `DTOs/AnnotationEventDto.cs`.*
|
||||
|
||||
**AnnotationSource** — Wire enum: `AI = 0`, `Manual = 1`. *source: `Enums/AnnotationSource.cs`.*
|
||||
|
||||
**AnnotationStatus** — Wire enum: `None = 0`, `Created = 10`, `Edited = 20`, `Validated = 30`, `Deleted = 40`. Soft-delete uses value 40 (per ADR-009). *source: `Enums/AnnotationStatus.cs`.*
|
||||
|
||||
**Annotator UI** — Operator-facing client of `01 Annotations REST` + SSE. Active editing surface. *source: `components/01_annotations-rest/description.md`.*
|
||||
|
||||
**Bulk status** — Multi-id status update via `POST /dataset/bulk-status` carrying `BulkStatusRequest { AnnotationIds, Status }`. *source: `Controllers/DatasetController.cs:34`.*
|
||||
|
||||
**Business transaction** — The lifecycle-level transactional boundary planned per ADR-008: DB rows + outbox commit atomically; FS writes and SSE publish run post-commit using the outbox row as the durable promise. *source: `architecture.md` ADR-008.*
|
||||
|
||||
**Camera settings** — Per-camera calibration (`altitude`, `focal_length`, `sensor_width`) used by detection geometry. *source: `data_model.md`, `Database/Entities/CameraSettings.cs`.*
|
||||
|
||||
**Combat readiness** — Wire enum on a detection (`CombatReadiness`). *source: `Enums/CombatReadiness.cs`, `modules/wire-enums.md`.*
|
||||
|
||||
**Dataset Explorer** — Read-heavy UI exposed under `/dataset` (policy `DATASET`). *source: `components/04_dataset/description.md`, suite `09_dataset_explorer.md`.*
|
||||
|
||||
**Detection** — Bounding box (`center_x/y, width, height`) + class number + label + affiliation + combat readiness, child of an annotation. *source: `data_model.md`, `Database/Entities/Detection.cs`.*
|
||||
|
||||
**Detection class** — Row in `detection_classes` (id, name, short_name, color, max_size_m, photo_mode). 19 rows seeded by the migrator; becoming admin-managed per RB-06. *source: `data_model.md`, `Database/DatabaseMigrator.cs`.*
|
||||
|
||||
**Directory settings** — DB-driven filesystem roots (`videos_dir`, `images_dir`, `labels_dir`, `thumbnails_dir`, `results_dir`, `gps_sat_dir`, `gps_route_dir`). Consumed via `PathResolver`. RB-01 will add `deleted_dir` for soft-delete relocation. *source: `data_model.md`, `Database/DatabaseMigrator.cs`, `modules/common-infrastructure.md`.*
|
||||
|
||||
**Failsafe outbox** — `annotations_queue_records` table; the durable bridge between local writes and the RabbitMQ stream. Drained by `FailsafeProducer`. *source: `architecture.md` ADR-003, `modules/rabbitmq-stream-sync.md`.*
|
||||
|
||||
**Flight** — *Deprecated synonym for Mission.* The codebase currently uses `FlightId` (DTOs and service queries) but will rename to `MissionId` per RB-07 to align with the suite spec. *source: `00_discovery.md` drift list, ADR-012.*
|
||||
|
||||
**JWT policies** — Authorization claims `ANN`, `DATASET`, `ADM` checked by `[Authorize(Policy = ...)]` on controllers. *source: `modules/auth-identity.md`, `Auth/JwtExtensions.cs`.*
|
||||
|
||||
**Media** — Uploaded image / video reference, waypoint-scoped, written via `MediaController`. *source: `data_model.md`, `components/03_media/description.md`.*
|
||||
|
||||
**MessagePack** — Wire encoding for outbox messages on the RabbitMQ stream (`AnnotationQueueMessage`, `AnnotationBulkQueueMessage`). Gzip-compressed at the producer. *source: `modules/rabbitmq-stream-sync.md`, `Services/FailsafeProducer.cs`.*
|
||||
|
||||
**Mission** — *Canonical domain term* per the suite spec — the logical grouping that the codebase currently calls "Flight" and that physically backs onto `media.waypoint_id`. The code → suite alignment is RB-07 / ADR-012; the suite remains canonical. *source: `suite/_docs/01_annotations.md`, `00_discovery.md`.*
|
||||
|
||||
**PathResolver** — DI singleton that lazy-loads filesystem roots from `directory_settings` and exposes per-annotation paths (image / label / thumbnail / result). Calls `Reset()` after directory updates. *source: `modules/common-infrastructure.md`, `Services/PathResolver.cs`.*
|
||||
|
||||
**QueueOperation** — Outbox enum: `Created = 0`, `Validated = 1`, `Deleted = 2`. RB-01 may add `Updated` for `UpdateAnnotation` semantics. *source: `Enums/QueueOperation.cs`.*
|
||||
|
||||
**RabbitMQ Stream `azaion-annotations`** — Durable export channel consumed by the admin sync worker and the AI training pipeline. Default port `5552`. *source: `architecture.md` ADR-003, `Program.cs:43`.*
|
||||
|
||||
**Refresh token** — Long-lived credential issued and rotated by the **admin** service. Annotations is a verifier only — it neither mints nor refreshes tokens. Long-running callers (e.g. the detections service) refresh against admin's `POST /token/refresh` and pass the resulting ES256 access token to annotations. *source: `modules/auth-identity.md`.*
|
||||
|
||||
**Silent detection** — *Deprecated.* Boolean flag on `system_settings` that gated outbox enqueue during development debugging. Scheduled for removal per ADR-010 / RB-02 — the suite e2e harness covers this need now. *source: `architecture.md` ADR-010.*
|
||||
|
||||
**Soft-delete** — `DeleteAnnotation` semantics agreed on 2026-05-14: status flips to `AnnotationStatus.Deleted = 40`, the annotation row stays, and image / label / thumbnail files relocate to `deleted_dir`. RB-01 implements this; today's code is hard-delete. *source: `architecture.md` ADR-009 / RB-01.*
|
||||
|
||||
**SSE (Server-Sent Events)** — `text/event-stream` channel on `GET /annotations/events` carrying `AnnotationEventDto` payloads. In-process, per-instance; no cross-pod fan-out. *source: `modules/sse-realtime.md`, `Controllers/AnnotationsController.cs`.*
|
||||
|
||||
**System settings** — Singleton-ish service-config row (`thumbnail_*`, `generate_annotated_image`, etc.). *source: `data_model.md`.*
|
||||
|
||||
**Thumbnail** — Per-annotation small image at `thumbnails_dir/{id}.jpg`. **Not produced by `CreateAnnotation`** — read-only via `PhysicalFile`; populated out-of-band today. *source: `system-flows.md` Flow F1, F2.*
|
||||
|
||||
**Transactional outbox** — Pattern adopted in ADR-008: a queue table populated inside a DB transaction, drained asynchronously by a background worker (`FailsafeProducer`), used to bridge local commits to a remote stream durably. *source: `architecture.md` ADR-003, ADR-008.*
|
||||
|
||||
**User settings** — Per-user UI prefs (selected flight / mission, panel widths). Unique on `user_id`. *source: `data_model.md`, `Database/Entities/UserSettings.cs`.*
|
||||
|
||||
**Waypoint** — UUID associated with media uploads, used for mission-scoped grouping. Physical foreign key under the logical "Mission" concept. *source: `Database/Entities/Media.cs`.*
|
||||
|
||||
**World B** — Internal label for the agreed lifecycle-observability stance: every annotation mutation publishes SSE and enqueues the outbox, not just `Create`. *source: `architecture.md` ADR-009.*
|
||||
|
||||
**YOLO label** — Plain-text format used in `{id}.txt` files: one detection per line, fields `class cx cy w h` (normalized box). *source: `Services/AnnotationService.cs:243–249`, `modules/annotations-service.md`.*
|
||||
@@ -0,0 +1,178 @@
|
||||
# Module Layout
|
||||
|
||||
**Status**: derived-from-code
|
||||
**Language**: csharp
|
||||
**Layout Convention**: single-assembly (`Azaion.Annotations`), vertical slices expressed as **logical components** under flat `src/` folders (`Controllers/`, `Services/`, `DTOs/`).
|
||||
**Root**: `src/`
|
||||
**Last Updated**: 2026-05-14
|
||||
|
||||
## Verification Needed
|
||||
|
||||
1. **No per-component physical root** — all components share `src/Controllers/`, `src/Services/`, `src/DTOs/`. File ownership below is **exclusive for implementation planning**; merges touching the same file need explicit batch ordering.
|
||||
2. **`AnnotationsController.cs` split (user-approved six-component model)** — **REST + static files** belong to **01 Annotations REST**; **SSE** (`Events`) belongs to **02 Annotations realtime & sync**. For `/implement`, treat **`src/Controllers/AnnotationsController.cs` as owned by 01**; tasks that **only** change SSE must still edit this file — flag as **cross-component** (01 + 02) or split into partial classes in a future refactor.
|
||||
3. **`src/DTOs/`** — no subfolders; each file has a **primary owner** in [Shared: DTO files](#shared-dto-files-primary-ownership) to resolve FORBIDDEN vs OWNED during tasks.
|
||||
4. **`FailsafeProducer.cs`** contains `RabbitMqConfig` and `FailsafeProducer` — fully owned by **02** (even though `Program.cs` registers the config as singleton).
|
||||
|
||||
---
|
||||
|
||||
## Layout Rules (adapted for this repo)
|
||||
|
||||
1. Components map to **logical** slices from `_docs/02_document/components/*/description.md`, not to separate top-level directories.
|
||||
2. **Foundation** (`06_platform`) owns schema, enums, auth registration helpers, middleware, path resolution, and **composition** (`Program.cs`, csproj, Dockerfile).
|
||||
3. **Feature** components own listed **service + controller** files; they **read** foundation public APIs and **shared DTO** types per the DTO table.
|
||||
4. Tests are **not present** in-repo; future test project should follow `tests/Azaion.Annotations.Tests/` (conventional) — not owned by feature slices until created.
|
||||
|
||||
---
|
||||
|
||||
## Per-Component Mapping
|
||||
|
||||
### Component: `01_annotations-rest`
|
||||
|
||||
- **Epic**: (assign per change; layout is structural)
|
||||
- **Directory (primary)**: `src/Services/` (partial), `src/Controllers/` (partial)
|
||||
- **Public API** (types other components may reference through DI / same assembly):
|
||||
- `Azaion.Annotations.Services.AnnotationService`
|
||||
- `Azaion.Annotations.Controllers.AnnotationsController` (REST + image/thumbnail actions only — see Verification)
|
||||
- **Internal**: private methods inside owned types; do not reach into other components’ services from new code without updating this layout.
|
||||
- **Owns (exclusive write scope)**:
|
||||
- `src/Services/AnnotationService.cs`
|
||||
- `src/Controllers/AnnotationsController.cs` — **primary file owner** (REST, `GetThumbnail`, `GetImage`; coordinate with 02 for `Events`)
|
||||
- **Imports from**: `06_platform` (Database, Enums, DTOs used here, PathResolver, Middleware indirectly), `02_annotations-realtime-sync` (`AnnotationEventService` for publish)
|
||||
- **Consumed by**: `04_dataset` (read paths share DB entities; no direct `AnnotationService` reference required), external callers
|
||||
|
||||
### Component: `02_annotations-realtime-sync`
|
||||
|
||||
- **Epic**: (assign per change)
|
||||
- **Directory (primary)**: `src/Services/` (partial)
|
||||
- **Public API**:
|
||||
- `Azaion.Annotations.Services.AnnotationEventService`
|
||||
- `Azaion.Annotations.Services.RabbitMqConfig`
|
||||
- `Azaion.Annotations.Services.FailsafeProducer`
|
||||
- `FailsafeProducer.EnqueueAsync` (static helper on same type as producer)
|
||||
- **Owns**:
|
||||
- `src/Services/FailsafeProducer.cs` (includes `RabbitMqConfig` + `FailsafeProducer`)
|
||||
- `src/Services/AnnotationEventService.cs`
|
||||
- **SSE contract**: `AnnotationsController.Events` (same `.cs` as 01 — see Verification)
|
||||
- **Imports from**: `06_platform` (Database, Entities, PathResolver, Enums, `DTOs/QueueMessages.cs`), `DTOs/AnnotationEventDto.cs` (see Shared)
|
||||
- **Consumed by**: `01_annotations-rest` (publishes events), external RabbitMQ consumers
|
||||
|
||||
### Component: `03_media`
|
||||
|
||||
- **Epic**: (assign per change)
|
||||
- **Public API**: `MediaService`, `MediaController`
|
||||
- **Owns**:
|
||||
- `src/Services/MediaService.cs`
|
||||
- `src/Controllers/MediaController.cs`
|
||||
- **Imports from**: `06_platform`
|
||||
- **Consumed by**: `01_annotations-rest` (domain: media rows referenced by annotations), UI
|
||||
|
||||
### Component: `04_dataset`
|
||||
|
||||
- **Epic**: (assign per change)
|
||||
- **Public API**: `DatasetService`, `DatasetController`
|
||||
- **Owns**:
|
||||
- `src/Services/DatasetService.cs`
|
||||
- `src/Controllers/DatasetController.cs`
|
||||
- **Imports from**: `06_platform`
|
||||
- **Consumed by**: Dataset Explorer UI
|
||||
|
||||
### Component: `05_settings-metadata`
|
||||
|
||||
- **Epic**: (assign per change)
|
||||
- **Public API**: `SettingsService`, `SettingsController`, `ClassesController`
|
||||
- **Owns**:
|
||||
- `src/Services/SettingsService.cs`
|
||||
- `src/Controllers/SettingsController.cs`
|
||||
- `src/Controllers/ClassesController.cs`
|
||||
- **Imports from**: `06_platform`
|
||||
- **Consumed by**: UI; `PathResolver` / directory settings (via DB) interact with **06** cache reset when dirs change
|
||||
|
||||
### Component: `06_platform`
|
||||
|
||||
- **Epic**: (assign per change)
|
||||
- **Public API** (representative; all `public` types in these areas are the integration surface):
|
||||
- `Azaion.Annotations.Database.AppDataConnection`, `DatabaseMigrator`, `Azaion.Annotations.Database.Entities.*`
|
||||
- `Azaion.Annotations.Enums.*`
|
||||
- `Azaion.Annotations.Middleware.ErrorHandlingMiddleware`
|
||||
- `Azaion.Annotations.Auth.JwtExtensions`
|
||||
- `Azaion.Annotations.Infrastructure.ConfigurationResolver`, `CorsConfigurationValidator`
|
||||
- `Azaion.Annotations.Services.PathResolver`
|
||||
- `Program` (implicit entry), `src/Program.cs`
|
||||
- **Owns**:
|
||||
- `src/Enums/**`
|
||||
- `src/Database/**`
|
||||
- `src/Middleware/**`
|
||||
- `src/Auth/**`
|
||||
- `src/Infrastructure/**`
|
||||
- `src/Services/PathResolver.cs`
|
||||
- `src/GlobalUsings.cs`
|
||||
- `src/Program.cs`
|
||||
- `src/Azaion.Annotations.csproj`
|
||||
- `src/Dockerfile`
|
||||
- **Imports from**: (none — foundation)
|
||||
- **Consumed by**: all other components
|
||||
|
||||
---
|
||||
|
||||
## Shared: DTO files (primary ownership)
|
||||
|
||||
| File under `src/DTOs/` | Primary component | Notes |
|
||||
|------------------------|-------------------|--------|
|
||||
| `PaginatedResponse.cs` | `06_platform` | Generic list wrapper — cross-cutting |
|
||||
| `ErrorResponse.cs` | `06_platform` | Shared error shape |
|
||||
| `CreateAnnotationRequest.cs` | `01_annotations-rest` | |
|
||||
| `UpdateAnnotationRequest.cs` | `01_annotations-rest` | |
|
||||
| `UpdateStatusRequest.cs` | `01_annotations-rest` / `04_dataset` | **Shared type** — **01** owns file edits; `04_dataset` uses for PATCH |
|
||||
| `GetAnnotationsQuery.cs` | `01_annotations-rest` | |
|
||||
| `AnnotationListItem.cs` | `01_annotations-rest` | |
|
||||
| `DetectionDto.cs` | `01_annotations-rest` | |
|
||||
| `AnnotationEventDto.cs` | `02_annotations-realtime-sync` | SSE payload |
|
||||
| `QueueMessages.cs` | `02_annotations-realtime-sync` | MessagePack stream payloads |
|
||||
| `CreateMediaRequest.cs` | `03_media` | |
|
||||
| `GetMediaQuery.cs` | `03_media` | |
|
||||
| `MediaListItem.cs` | `03_media` | |
|
||||
| `GetDatasetQuery.cs` | `04_dataset` | |
|
||||
| `DatasetItem.cs` | `04_dataset` | |
|
||||
| `ClassDistributionItem.cs` | `04_dataset` | |
|
||||
| `BulkStatusRequest.cs` | `04_dataset` | |
|
||||
| `UpdateSystemSettingsRequest.cs` | `05_settings-metadata` | |
|
||||
| `UpdateDirectoriesRequest.cs` | `05_settings-metadata` | |
|
||||
| `UpdateCameraSettingsRequest.cs` | `05_settings-metadata` | |
|
||||
| `UpdateUserSettingsRequest.cs` | `05_settings-metadata` | |
|
||||
|
||||
---
|
||||
|
||||
## Shared / Cross-Cutting (non-DTO)
|
||||
|
||||
### `common-helpers/01_http-error-envelope.md`
|
||||
|
||||
- **Purpose**: Documents middleware as cross-cutting (see `_docs/02_document/common-helpers/`).
|
||||
- **Owned by**: tasks touching **06_platform** (`ErrorHandlingMiddleware`).
|
||||
- **Consumed by**: all HTTP components.
|
||||
|
||||
---
|
||||
|
||||
## Allowed Dependencies (layering)
|
||||
|
||||
Higher layers may depend on lower; **not** the reverse. Same-layer components should not introduce compile-time cycles (current codebase: none detected).
|
||||
|
||||
| Layer | Components | May import from (namespaces / types from) |
|
||||
|-------|------------|---------------------------------------------|
|
||||
| 1 — Foundation | `06_platform` | *(none)* |
|
||||
| 2 — Realtime infra | `02_annotations-realtime-sync` | `06_platform` |
|
||||
| 3 — Application features | `01_annotations-rest`, `03_media`, `04_dataset`, `05_settings-metadata` | `06_platform`, and **`01` additionally `02`** (`AnnotationEventService`) |
|
||||
|
||||
**Rules**
|
||||
|
||||
- `03`, `04`, `05` → **must not** reference `AnnotationService` / `MediaService` across features without an explicit API (today: only via shared `AppDataConnection` in same assembly — acceptable but treat as **tight coupling**; prefer domain services for new code).
|
||||
- `02` → **must not** reference `01` service types (no reverse dependency today).
|
||||
|
||||
Violations are **Architecture** findings for code-review Phase 7.
|
||||
|
||||
---
|
||||
|
||||
## Layout Conventions (reference)
|
||||
|
||||
| Language | Root | This repo |
|
||||
|----------|------|-----------|
|
||||
| C# (.NET) | `src/` | Single web project; vertical slices = **logical** component rows above + DTO table |
|
||||
@@ -0,0 +1,19 @@
|
||||
# Module documentation index
|
||||
|
||||
Modules follow **`suite/_docs/01_annotations.md`**: annotations vs media, SSE, auth/JWT refresh, DB, RabbitMQ sync, plus **dataset** (DATASET) and **settings / detection classes** as implemented in this repo.
|
||||
|
||||
| Order | File | Scope |
|
||||
|------:|------|--------|
|
||||
| 1 | [wire-enums.md](./wire-enums.md) | `src/Enums/*` |
|
||||
| 2 | [database-layer.md](./database-layer.md) | `src/Database/*` |
|
||||
| 3 | [common-infrastructure.md](./common-infrastructure.md) | `PathResolver`, `ErrorHandlingMiddleware`, shared small types |
|
||||
| 4 | [auth-identity.md](./auth-identity.md) | `JwtExtensions` (JWKS verifier), `ConfigurationResolver`, `CorsConfigurationValidator` |
|
||||
| 5 | [media-service.md](./media-service.md) | `MediaService`, `MediaController`, media DTOs |
|
||||
| 6 | [annotations-service.md](./annotations-service.md) | `AnnotationService`, `AnnotationsController` (REST + files) |
|
||||
| 7 | [dataset-service.md](./dataset-service.md) | `DatasetService`, `DatasetController`, dataset DTOs |
|
||||
| 8 | [settings-metadata-service.md](./settings-metadata-service.md) | `SettingsService`, `SettingsController`, `ClassesController`, settings DTOs |
|
||||
| 9 | [sse-realtime.md](./sse-realtime.md) | `AnnotationEventService`, SSE endpoint |
|
||||
| 10 | [rabbitmq-stream-sync.md](./rabbitmq-stream-sync.md) | `FailsafeProducer`, `RabbitMqConfig`, `QueueMessages` |
|
||||
| 11 | [composition-program.md](./composition-program.md) | `Program.cs` |
|
||||
|
||||
`src/DTOs/` types are described in the module that exposes them on the wire.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Module: Annotations service
|
||||
|
||||
## Purpose
|
||||
|
||||
Core **annotation CRUD**, listing, static image/thumbnail delivery, and coordination with **media** and **files on disk**. Maps to **`01_annotations.md` §1–6** (not SSE — see `sse-realtime.md`).
|
||||
|
||||
## Code
|
||||
|
||||
- `AnnotationService` — create/update/status/delete/query/get one; uses `PathResolver`, hashing, label/thumbnail generation, queue handoff to failsafe path as implemented.
|
||||
- `AnnotationsController` — `[Route("annotations")]`, `[Authorize(Policy = "ANN")]` except where noted.
|
||||
- REST: `POST`, `PUT/{id}`, `PATCH/{id}/status`, `DELETE/{id}`, `GET`, `GET/{id}`.
|
||||
- Files: `GET/{id}/thumbnail`, `GET/{id}/image`.
|
||||
- **SSE** `GET/events` documented in `sse-realtime.md` (same controller type).
|
||||
|
||||
## DTOs (this module)
|
||||
|
||||
- `CreateAnnotationRequest`, `UpdateAnnotationRequest`, `UpdateStatusRequest`, `GetAnnotationsQuery`, `AnnotationListItem`, `DetectionDto` (annotation payloads).
|
||||
|
||||
## Dependencies
|
||||
|
||||
Database, `PathResolver`, optional integration with queue/SSE services.
|
||||
|
||||
## Suite vs code (maintain in suite or code)
|
||||
|
||||
- **UserId:** suite pseudo-code shows `UserId` on create; **implementation** uses JWT subject (`AnnotationsController`).
|
||||
- **GET filter:** suite `missionId` vs code `FlightId` + filter behavior — track as open alignment.
|
||||
|
||||
## Suite doc
|
||||
|
||||
§1–6; annotation identity at top of `01_annotations.md`.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Module: Auth & identity
|
||||
|
||||
## Purpose
|
||||
|
||||
JWT validation for API policies. Tokens are minted exclusively by the admin service (ES256-signed); annotations is a **verifier only**.
|
||||
|
||||
## Components
|
||||
|
||||
### `JwtExtensions` (`Auth/JwtExtensions.cs`)
|
||||
|
||||
- `AddJwtAuth(IConfiguration)` — pulls `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL` via `ConfigurationResolver.ResolveRequiredOrThrow` (fail-fast at startup if any is missing).
|
||||
- `TokenValidationParameters` mirrors admin's verifier contract:
|
||||
- `ValidateIssuer = true` / `ValidateAudience = true` / `ValidateLifetime = true`.
|
||||
- `ValidAlgorithms = [SecurityAlgorithms.EcdsaSha256]` — pinned so an HS256-forgery using the public key as the HMAC secret cannot pass.
|
||||
- `RequireSignedTokens = true`, `RequireExpirationTime = true`.
|
||||
- `ClockSkew = 30s`.
|
||||
- Signing keys are fetched from admin's `/.well-known/jwks.json` via a `ConfigurationManager<JsonWebKeySet>` backed by a minimal `IConfigurationRetriever<JsonWebKeySet>` (admin exposes JWKS but not the full OIDC discovery document). The manager honours admin's `Cache-Control: public, max-age=3600` and refreshes on the default schedule. During key rotation both `kid`s are present in JWKS so in-flight tokens still verify.
|
||||
- Policies: `ANN`, `DATASET`, `ADM` — each requires claim `permissions` with that code (matches suite "Required permission: ANN" and Dataset Explorer `DATASET`).
|
||||
|
||||
## Dependencies
|
||||
|
||||
Configuration (all required, no defaults):
|
||||
|
||||
- `JWT_ISSUER` (alt `Jwt:Issuer`) — must match admin's `JwtConfig:Issuer`.
|
||||
- `JWT_AUDIENCE` (alt `Jwt:Audience`) — must match admin's `JwtConfig:Audience`.
|
||||
- `JWT_JWKS_URL` (alt `Jwt:JwksUrl`) — `https://admin.azaion.com/.well-known/jwks.json` in production.
|
||||
|
||||
## Consumers
|
||||
|
||||
All `[Authorize]` controllers.
|
||||
|
||||
## Removed in this cycle
|
||||
|
||||
- `Services/TokenService.cs` (HS256 minting of access tokens from refresh tokens) — deleted; refresh is now the admin service's responsibility (`POST /token/refresh`).
|
||||
- `Controllers/AuthController.cs` and the `POST /auth/refresh` endpoint — deleted along with `TokenService`. Detections (and any other client) must call admin's refresh endpoint and pass the returned access token to annotations.
|
||||
- `JWT_SECRET` env var — no longer read.
|
||||
|
||||
## Suite doc
|
||||
|
||||
`01_annotations.md` §Annotation Sync (verifier role); suite `10_auth.md` for full auth story (admin = issuer, satellite-provider / annotations / flights / ui = verifiers).
|
||||
@@ -0,0 +1,38 @@
|
||||
# Module: Common infrastructure
|
||||
|
||||
## Purpose
|
||||
|
||||
Cross-cutting **filesystem layout** (annotation images, labels, thumbnails, results), **global error JSON**, and trivial shared API types.
|
||||
|
||||
## Components
|
||||
|
||||
### `PathResolver` (`Services/PathResolver.cs`)
|
||||
|
||||
- Lazy-loads paths from `directory_settings` via `AppDataConnection`.
|
||||
- Methods: `GetImagePath`, `GetLabelPath`, `GetThumbnailPath`, `GetResultPath`, `GetMediaDir` — paths under configured dirs with `{annotationId}.jpg` / `.txt` patterns.
|
||||
- `Reset()` clears cache after directory settings change.
|
||||
|
||||
### `ErrorHandlingMiddleware` (`Middleware/`)
|
||||
|
||||
Maps exceptions to `{ statusCode, message }` JSON (400/404/409/500). Aligns HTTP outcomes with `01_annotations.md` status tables.
|
||||
|
||||
### Shared DTOs (`DTOs/`)
|
||||
|
||||
- `PaginatedResponse<T>` — list + `totalCount` / `page` / `pageSize` (annotations list, media list, dataset).
|
||||
- `ErrorResponse` — available for explicit error contracts where used.
|
||||
|
||||
### `GlobalUsings.cs`
|
||||
|
||||
Project-wide usings only.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `PathResolver` → `AppDataConnection`, `DirectorySettings` entity.
|
||||
|
||||
## Consumers
|
||||
|
||||
All services and controllers that touch disk or return paged lists.
|
||||
|
||||
## Suite doc
|
||||
|
||||
File cleanup on DELETE annotation (`GetImgPath` / label / thumb) in `01_annotations.md` §4.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Module: Composition (`Program.cs`)
|
||||
|
||||
## Purpose
|
||||
|
||||
Single **composition root**: configuration, PostgreSQL `AppDataConnection`, service registrations, **JWT**, **CORS**, Swagger, **migrator** on startup, **middleware** order, `MapControllers`, `/health`, `WebApplication.Run`.
|
||||
|
||||
## Notable wiring
|
||||
|
||||
- `DATABASE_URL` (required, no fallback — startup fails fast via `ConfigurationResolver.ResolveRequiredOrThrow`) → Npgsql connection string helper.
|
||||
- `JWT_ISSUER` / `JWT_AUDIENCE` / `JWT_JWKS_URL` for `AddJwtAuth` (all required; resolved by `ConfigurationResolver`). The validator pulls public ES256 keys from admin's JWKS endpoint; this service no longer holds an HMAC secret.
|
||||
- `CorsConfig:AllowedOrigins` / `CorsConfig:AllowAnyOrigin` for the default CORS policy; `CorsConfigurationValidator` refuses to start with a permissive policy in `Production`.
|
||||
- `RabbitMqConfig` from env + `AddHostedService<FailsafeProducer>()`.
|
||||
- Scoped services: `AnnotationService`, `MediaService`, `DatasetService`, `SettingsService`, `PathResolver`; singletons: `AnnotationEventService`, `RabbitMqConfig`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
All modules; documented last after slices are understood.
|
||||
|
||||
## Suite doc
|
||||
|
||||
Operational/env story complements `01_annotations.md` deployment sections in suite architecture docs.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Module: Database layer (`src/Database`)
|
||||
|
||||
## Purpose
|
||||
|
||||
PostgreSQL schema and Linq2DB mapping for annotations, media, detections, queue buffer, settings, and `detection_classes`. Underpins every HTTP module in `01_annotations.md`.
|
||||
|
||||
## Public interface
|
||||
|
||||
- `AppDataConnection` — `ITable<>` for all mapped entities.
|
||||
- `DatabaseMigrator.Migrate` — embedded SQL: `CREATE TABLE IF NOT EXISTS` / `ALTER … IF NOT EXISTS`, seed detection classes.
|
||||
|
||||
## Entities (summary)
|
||||
|
||||
- `Annotation`, `Media`, `Detection` — core annotation + YOLO row model (`time` stored as BIGINT ticks).
|
||||
- `AnnotationsQueueRecord` — failsafe outbox (`operation`, `annotation_ids`).
|
||||
- `SystemSettings` — includes `GenerateAnnotatedImage`, `SilentDetection` (suite §Annotated Image / Silent Detection).
|
||||
- `DirectorySettings` — `/data/...` roots consumed by `PathResolver`.
|
||||
- `DetectionClass`, `UserSettings`, `CameraSettings`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Wire enums on columns.
|
||||
|
||||
## Consumers
|
||||
|
||||
All services and `ClassesController`.
|
||||
|
||||
## Suite doc
|
||||
|
||||
Annotation identity and ER-level behavior; cross-check `00_database_schema.md` in suite when entities evolve.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Module: Dataset service
|
||||
|
||||
## Purpose
|
||||
|
||||
**Dataset Explorer** backend: paginated grid, detail, status updates, bulk status — **`[Authorize(Policy = "DATASET")]`** per suite note on PATCH status (`01_annotations.md` §3 points to `09_dataset_explorer.md`).
|
||||
|
||||
## Code
|
||||
|
||||
- `DatasetService` — queries tuned for dataset views; may reuse annotation entities.
|
||||
- `DatasetController` — `[Route("dataset")]`.
|
||||
|
||||
## DTOs (this module)
|
||||
|
||||
- `GetDatasetQuery`, `DatasetItem`, `ClassDistributionItem`, `BulkStatusRequest` — and shared `UpdateStatusRequest` where used for PATCH.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Database, same status enums as annotator.
|
||||
|
||||
## Suite doc
|
||||
|
||||
Primary behavioral spec: `suite/_docs/09_dataset_explorer.md`; permission cross-ref in `01_annotations.md` §3.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Module: Media service
|
||||
|
||||
## Purpose
|
||||
|
||||
HTTP surface and domain logic for **§7–10** in `01_annotations.md`: create media, batch upload, list, delete, and download raw media file.
|
||||
|
||||
## Code
|
||||
|
||||
- `MediaService` — persistence + disk writes, batch from `IFormFileCollection`.
|
||||
- `MediaController` — `[Route("media")]`, `[Authorize(Policy = "ANN")]`.
|
||||
- `POST /media`, `POST /media/batch` (form: `waypointId` + files), `GET /media`, `GET /media/{id}/file`, delete route as implemented.
|
||||
|
||||
## DTOs (this module)
|
||||
|
||||
- `CreateMediaRequest`, `GetMediaQuery`, `MediaListItem` — plus any media-specific shapes used only here.
|
||||
|
||||
## Dependencies
|
||||
|
||||
`AppDataConnection`, `PathResolver` (media dir), JWT user id from claims.
|
||||
|
||||
## Consumers
|
||||
|
||||
Annotator UI / React upload flows described in suite §Media Browsing.
|
||||
|
||||
## Suite doc
|
||||
|
||||
`01_annotations.md` §7–10; accepted formats table in same doc.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Module: RabbitMQ stream sync (failsafe)
|
||||
|
||||
## Purpose
|
||||
|
||||
**Annotation Sync** outbox and **RabbitMQ Stream** producer — `01_annotations.md` §Annotation Sync, Failsafe Queue, RabbitMQ Stream.
|
||||
|
||||
## Code
|
||||
|
||||
- `RabbitMqConfig` + `FailsafeProducer` (`Services/FailsafeProducer.cs`) — `BackgroundService`; builds `StreamSystem`, drains `annotations_queue_records`, serializes **MessagePack** payloads (`AnnotationQueueMessage`, `AnnotationBulkQueueMessage` in `DTOs/QueueMessages.cs`), gzip as implemented.
|
||||
- Entity `AnnotationsQueueRecord` — see `database-layer.md`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
`AppDataConnection`, `PathResolver` (for image bytes on create), env-driven `RABBITMQ_*` from `Program`.
|
||||
|
||||
## Consumers (downstream, external)
|
||||
|
||||
Admin `AnnotationSyncWorker`, AI Training consumer — described in suite doc.
|
||||
|
||||
## Suite doc
|
||||
|
||||
Full sync topology and stream semantics in `01_annotations.md`; keep MessagePack key layout stable.
|
||||
@@ -0,0 +1,23 @@
|
||||
# Module: Settings & metadata
|
||||
|
||||
## Purpose
|
||||
|
||||
**System / directory / camera / user** settings and **detection class** list for UI color maps (`01_annotations.md` §11–12, “GET /classes” narrative).
|
||||
|
||||
## Code
|
||||
|
||||
- `SettingsService` + `SettingsController` — `[Route("settings")]`, mixed `[Authorize]` and `ADM` for writes.
|
||||
- System, directories, camera, user settings endpoints (see controller for full list).
|
||||
- `ClassesController` — `[Route("classes")]`, `GET` all `detection_classes` via Linq2DB (thin read-through).
|
||||
|
||||
## DTOs (this module)
|
||||
|
||||
- `UpdateSystemSettingsRequest`, `UpdateDirectoriesRequest`, `UpdateCameraSettingsRequest`, `UpdateUserSettingsRequest`, etc.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Database entities `SystemSettings`, `DirectorySettings`, `CameraSettings`, `UserSettings`, `DetectionClass`.
|
||||
|
||||
## Suite doc
|
||||
|
||||
`01_annotations.md` camera §11–12; directory defaults align with `PathResolver` / migrator defaults.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Module: SSE (real-time)
|
||||
|
||||
## Purpose
|
||||
|
||||
**Server-Sent Events** for annotation activity — `01_annotations.md` §“GET /annotations/events (SSE)” and `AnnotationEvent` shape.
|
||||
|
||||
## Code
|
||||
|
||||
- `AnnotationEventService` — unbounded `Channel<AnnotationEventDto>`; `PublishAsync` / `Reader` for subscribers.
|
||||
- `AnnotationsController.Events` — sets `text/event-stream`, subscribes readers, pushes JSON events (implementation detail in source).
|
||||
|
||||
## DTOs
|
||||
|
||||
- `AnnotationEventDto` — ids, `Status`, `Source`, `Detections`, `CreatedDate`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
`AnnotationService` (or controller) calls `PublishAsync` after mutations.
|
||||
|
||||
## Suite doc
|
||||
|
||||
SSE section + `DetectionEvent` vs `AnnotationEvent` distinction (detection progress may be separate pipeline).
|
||||
@@ -0,0 +1,29 @@
|
||||
# Module: Wire enums (`src/Enums`)
|
||||
|
||||
## Purpose
|
||||
|
||||
Integer-backed enums for JSON and MessagePack. **`01_annotations.md`** states all listed enums serialize as **numbers**, not names.
|
||||
|
||||
## Types
|
||||
|
||||
| Enum | File | Suite |
|
||||
|------|------|-------|
|
||||
| `AnnotationSource` | `AnnotationSource.cs` | Suite table (AI=0, Manual=1) |
|
||||
| `AnnotationStatus` | `AnnotationStatus.cs` | Created=10, Edited=20, etc. |
|
||||
| `MediaStatus` | `MediaStatus.cs` | SSE / media lifecycle |
|
||||
| `MediaType` | `MediaType.cs` | Image vs video |
|
||||
| `AffiliationEnum` | `AffiliationEnum.cs` | Detection payload |
|
||||
| `CombatReadiness` | `CombatReadiness.cs` | Detection payload |
|
||||
| `QueueOperation` | `QueueOperation.cs` | Failsafe / bulk queue |
|
||||
|
||||
## Dependencies
|
||||
|
||||
None (leaf).
|
||||
|
||||
## Consumers
|
||||
|
||||
Entities, DTOs, `FailsafeProducer`, services.
|
||||
|
||||
## Suite doc
|
||||
|
||||
Keep enum **numeric** contracts in sync with `01_annotations.md` and consuming UIs.
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"current_step": "complete",
|
||||
"completed_steps": ["discovery", "modules", "components", "module-layout", "architecture-synthesis", "verification-pass", "verification-accepted", "glossary-architecture-vision", "solution-extraction", "problem-extraction", "problem-extraction-accepted", "final-report"],
|
||||
"focus_dir": null,
|
||||
"modules_total": 11,
|
||||
"modules_documented": [
|
||||
"wire-enums",
|
||||
"database-layer",
|
||||
"common-infrastructure",
|
||||
"auth-identity",
|
||||
"media-service",
|
||||
"annotations-service",
|
||||
"dataset-service",
|
||||
"settings-metadata-service",
|
||||
"sse-realtime",
|
||||
"rabbitmq-stream-sync",
|
||||
"composition-program"
|
||||
],
|
||||
"modules_remaining": [],
|
||||
"module_batch": 2,
|
||||
"components_written": [
|
||||
"01_annotations-rest",
|
||||
"02_annotations-realtime-sync",
|
||||
"03_media",
|
||||
"04_dataset",
|
||||
"05_settings-metadata",
|
||||
"06_platform"
|
||||
],
|
||||
"system_synthesis_artifacts": [
|
||||
"architecture.md",
|
||||
"system-flows.md",
|
||||
"data_model.md",
|
||||
"deployment/containerization.md",
|
||||
"deployment/ci_cd_pipeline.md",
|
||||
"deployment/environment_strategy.md",
|
||||
"deployment/observability.md",
|
||||
"diagrams/flows/flow_annotation_create.md",
|
||||
"diagrams/flows/flow_sse_subscription.md",
|
||||
"diagrams/flows/flow_failsafe_drain.md"
|
||||
],
|
||||
"step_4_5_glossary_vision": "confirmed",
|
||||
"last_updated": "2026-05-14T08:37:00Z"
|
||||
}
|
||||
@@ -0,0 +1,460 @@
|
||||
# Azaion.Annotations — System Flows
|
||||
|
||||
> Bottom-up: traces in this document are derived from `components/*/description.md`, `modules/*.md`, and the source under `src/`. Mermaid diagrams per flow are linked under `diagrams/flows/`.
|
||||
|
||||
## Flow Inventory
|
||||
|
||||
| # | Flow Name | Trigger | Primary Components | Criticality |
|
||||
|---|-----------|---------|---------------------|-------------|
|
||||
| F1 | Annotation Create (with image bytes) | `POST /annotations` from detections service or UI | 01 + 02 + 06 + 03 | High |
|
||||
| F2 | Annotation Listing / Read | `GET /annotations`, `GET /annotations/{id}/{thumbnail|image}` | 01 + 06 + 03 | High |
|
||||
| F3 | Real-time SSE Subscription | `GET /annotations/events` from UI | 01 + 02 + 06 | High |
|
||||
| F4 | Failsafe Outbox Drain → RabbitMQ Stream | `FailsafeProducer` background loop | 02 + 06 | High |
|
||||
| F5 | Media Upload (single + batch) | `POST /media`, `POST /media/batch` | 03 + 06 | High |
|
||||
| F6 | Auth Refresh (out-of-process) | Long-running callers refresh against admin's `POST /token/refresh`; annotations only verifies the resulting access token | 06 (verifier) + admin (issuer, out-of-scope) | Medium |
|
||||
| F7 | Directory Settings Change → Path Cache Reset | `PUT /settings/directories` | 05 + 06 | Medium |
|
||||
| F8 | Dataset Bulk Status | `PATCH /dataset/.../status`, bulk variant | 04 + 06 | Medium |
|
||||
|
||||
## Flow Dependencies
|
||||
|
||||
| Flow | Depends on | Shares data with |
|
||||
|------|------------|-------------------|
|
||||
| F1 | F5 (media must exist for create-with-`MediaId`) | F2 (read-after-write), F3 (Create-only event publish), F4 (Create-only queue insert, gated by `silent_detection`) |
|
||||
| F2 | F1 (writes data being read), F5 | F3 (consistency window) |
|
||||
| F3 | F1 (SSE stream is fed by F1 Create publishes only) | — |
|
||||
| F4 | F1 (reads outbox written by F1 Create only) | downstream consumers (admin sync, AI training) |
|
||||
| F5 | — | F1 |
|
||||
| F6 | — | all `[Authorize]` flows (refreshes the token they use) |
|
||||
| F7 | — | F1, F2, F4, F5 (all paths via `PathResolver`) |
|
||||
| F8 | F1 | **none today** — F8 does not feed F3 or F4 (open question) |
|
||||
|
||||
---
|
||||
|
||||
## Flow F1: Annotation Create (with image bytes)
|
||||
|
||||
### Description
|
||||
|
||||
Detections service or UI POSTs an annotation payload with image bytes (or a `MediaId` for an existing media row). The service hashes the bytes, derives the annotation id, writes the image to disk, ensures a `media` row exists, persists annotation + detection rows, writes the YOLO label file, publishes an in-process SSE event, and — unless `system_settings.silent_detection` is true — enqueues an outbox row for downstream RabbitMQ stream export. **Thumbnails are not generated in this flow** (they are read-only via `PhysicalFile` from a separately populated path).
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Caller holds a JWT with `permissions: ANN`.
|
||||
- `directory_settings` row exists (seeded by migrator with `/data/...` defaults).
|
||||
- Postgres reachable (errors otherwise surfaced as 500 by `ErrorHandlingMiddleware`).
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
See `diagrams/flows/flow_annotation_create.md` for the full sequence + flowchart.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Caller as Detections / UI
|
||||
participant Ctrl as AnnotationsController (01)
|
||||
participant Svc as AnnotationService (01)
|
||||
participant Path as PathResolver (06)
|
||||
participant DB as PostgreSQL (06)
|
||||
participant FS as Filesystem
|
||||
participant Evt as AnnotationEventService (02)
|
||||
participant Q as annotations_queue_records (DB / 02)
|
||||
|
||||
Caller->>Ctrl: POST /annotations (CreateAnnotationRequest, JWT)
|
||||
Ctrl->>Svc: CreateAnnotation(request, userId from JWT)
|
||||
|
||||
alt request.Image bytes provided
|
||||
Svc->>Svc: ComputeHash (XxHash64 over sampled bytes) -> id
|
||||
Svc->>Path: GetImagePath(id)
|
||||
Svc->>FS: write {id}.jpg
|
||||
Svc->>DB: SELECT media WHERE id=id
|
||||
opt media row missing
|
||||
Svc->>DB: INSERT media (Image, MediaStatus.New, ...)
|
||||
end
|
||||
else request.MediaId provided
|
||||
Svc->>DB: SELECT media WHERE id=MediaId (404 if missing)
|
||||
Svc->>Path: GetImagePath(id)
|
||||
opt source media file exists & target image missing
|
||||
Svc->>FS: copy media.Path -> {id}.jpg
|
||||
end
|
||||
end
|
||||
|
||||
Svc->>DB: INSERT annotations
|
||||
Svc->>DB: BulkCopy detection rows
|
||||
Svc->>Path: GetLabelPath(id)
|
||||
Svc->>FS: write {id}.txt (YOLO)
|
||||
Svc->>Evt: PublishAsync(AnnotationEventDto)
|
||||
Svc->>DB: SELECT system_settings (FirstOrDefault)
|
||||
alt SilentDetection != true
|
||||
Svc->>Q: FailsafeProducer.EnqueueAsync(db, id, QueueOperation.Created)
|
||||
end
|
||||
Svc-->>Ctrl: Annotation
|
||||
Ctrl-->>Caller: 201 Created (Location: /annotations/{id})
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
| Step | From | To | Data | Format |
|
||||
|------|------|----|------|--------|
|
||||
| 1 | Caller | `AnnotationsController` | `CreateAnnotationRequest` + JWT | JSON / Bearer |
|
||||
| 2 | `AnnotationService` | Filesystem | image bytes | `{id}.jpg` under `images_dir` |
|
||||
| 3 | `AnnotationService` | DB | `media` row (insert if absent) | SQL via Linq2DB |
|
||||
| 4 | `AnnotationService` | DB | `annotations` row | SQL |
|
||||
| 5 | `AnnotationService` | DB | `detection` rows | `BulkCopyAsync` |
|
||||
| 6 | `AnnotationService` | Filesystem | YOLO label `{id}.txt` | text lines `class cx cy w h` |
|
||||
| 7 | `AnnotationService` | `AnnotationEventService` | `AnnotationEventDto` | in-memory `Channel<>` |
|
||||
| 8 | `AnnotationService` | DB outbox | `annotations_queue_records` (operation=Created) | row, only if `SilentDetection != true` |
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Neither bytes nor MediaId provided | request validation | `ArgumentException` in service | mapped to 400 by middleware |
|
||||
| Referenced `MediaId` not found | media lookup | `KeyNotFoundException` | 404 |
|
||||
| Filesystem write fails (no perms / disk full) | step 2 / 6 | IOException | 500 via middleware; **NOT transactional with DB** — risk of orphan files on partial failure |
|
||||
| DB write fails after FS success | steps 3–5 | Linq2DB exception | 500; orphan image / label may remain (open risk) |
|
||||
| SSE publish fails | step 7 | unbounded channel — failure unlikely | logged via default ASP.NET Core logger |
|
||||
| Outbox insert fails after SSE publish | step 8 | exception | 500; UI saw the event but downstream stream consumers will not — **observable inconsistency** |
|
||||
| RabbitMQ unavailable | n/a here | — | F4 handles drain offline — F1 itself is unaffected |
|
||||
|
||||
### Performance Expectations
|
||||
|
||||
| Metric | Target | Notes |
|
||||
|--------|--------|-------|
|
||||
| End-to-end latency | not specified in code | dominant cost: hashing + 3 disk writes; flag for `00_problem` extraction |
|
||||
| Throughput | not specified | single instance bounded by DB + disk FS |
|
||||
|
||||
---
|
||||
|
||||
## Flow F2: Annotation Listing / Read
|
||||
|
||||
### Description
|
||||
|
||||
UIs and dataset consumers list annotations with filters (e.g., `FlightId`, status) and fetch image / thumbnail bytes. Read path is read-only against Postgres + `PhysicalFile` from the configured directories.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Caller holds JWT with `ANN` (or `DATASET` for the dataset variant in F8).
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant Ctrl as AnnotationsController (01)
|
||||
participant Svc as AnnotationService (01)
|
||||
participant DB
|
||||
participant Path as PathResolver (06)
|
||||
participant FS as Filesystem
|
||||
|
||||
UI->>Ctrl: GET /annotations?filters
|
||||
Ctrl->>Svc: GetAnnotations(query)
|
||||
Svc->>DB: SELECT annotations × detection × media
|
||||
DB-->>Svc: rows
|
||||
Svc-->>Ctrl: PaginatedResponse<AnnotationListItem>
|
||||
Ctrl-->>UI: 200 OK (JSON)
|
||||
|
||||
UI->>Ctrl: GET /annotations/{id}/thumbnail
|
||||
Ctrl->>Path: GetThumbnailPath(id)
|
||||
Path-->>Ctrl: /data/thumbnails/{id}.jpg
|
||||
Ctrl->>FS: File.Exists?
|
||||
alt exists
|
||||
Ctrl-->>UI: 200 OK (image/jpeg, PhysicalFile)
|
||||
else missing
|
||||
Ctrl-->>UI: 404 NotFound
|
||||
end
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
| Step | From | To | Data | Format |
|
||||
|------|------|----|------|--------|
|
||||
| 1 | UI | controller | `GetAnnotationsQuery` | query string |
|
||||
| 2 | service | DB | filtered join | SQL |
|
||||
| 3 | service | UI | list + paging metadata | `PaginatedResponse<AnnotationListItem>` |
|
||||
| 4 | controller | UI | image / thumbnail bytes | `image/jpeg` |
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Missing image file | thumbnail / image route | `File.Exists` false | 404 |
|
||||
| Auth failure | model binding | JWT pipeline | 401 / 403 |
|
||||
| DB error | listing | Linq2DB | 500 via middleware |
|
||||
|
||||
---
|
||||
|
||||
## Flow F3: Real-time SSE Subscription
|
||||
|
||||
### Description
|
||||
|
||||
UI opens a long-lived `text/event-stream` connection and receives JSON-serialized `AnnotationEventDto` payloads as they are published by F1, F8, and any other annotation mutation.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant Ctrl as AnnotationsController.Events (02 doc-ownership)
|
||||
participant Evt as AnnotationEventService (02)
|
||||
participant Producer as Other flows (F1/F8)
|
||||
|
||||
UI->>Ctrl: GET /annotations/events (Accept: text/event-stream, JWT ANN)
|
||||
Ctrl->>Evt: subscribe(Reader)
|
||||
loop until cancelled
|
||||
Producer->>Evt: PublishAsync(AnnotationEventDto)
|
||||
Evt-->>Ctrl: ReadAllAsync yields event
|
||||
Ctrl-->>UI: data: {json}\n\n
|
||||
end
|
||||
UI--xCtrl: client disconnect / cancel
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
| Step | From | To | Data | Format |
|
||||
|------|------|----|------|--------|
|
||||
| 1 | UI | controller | upgrade to SSE | HTTP/1.1 |
|
||||
| 2 | producer | service | `AnnotationEventDto` | in-memory message |
|
||||
| 3 | controller | UI | `data: {json}\n\n` | SSE frame |
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Auth failure | request | JWT pipeline | 401 |
|
||||
| Client disconnect | streaming | `CancellationToken` | controller exits cleanly |
|
||||
| Process restart | streaming | n/a | UI must reconnect; **buffered events between disconnect and restart are lost** (intentional — durability handled by F4) |
|
||||
|
||||
### Performance Expectations
|
||||
|
||||
In-process channel; latency is bounded by `Channel<>` + write-flush — sub-millisecond locally.
|
||||
|
||||
---
|
||||
|
||||
## Flow F4: Failsafe Outbox Drain → RabbitMQ Stream
|
||||
|
||||
### Description
|
||||
|
||||
`FailsafeProducer` is a singleton `BackgroundService` that polls `annotations_queue_records`, re-reads image bytes for `Created` operations, packs `AnnotationQueueMessage` / `AnnotationBulkQueueMessage` (MessagePack), and publishes to the `azaion-annotations` RabbitMQ stream. After a successful publish, the row is deleted.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant FP as FailsafeProducer (02)
|
||||
participant DB
|
||||
participant Path as PathResolver (06)
|
||||
participant FS as Filesystem
|
||||
participant RMQ as RabbitMQ Stream
|
||||
|
||||
loop while host running
|
||||
FP->>DB: SELECT annotations_queue_records
|
||||
DB-->>FP: pending rows
|
||||
loop per row
|
||||
alt operation = Created
|
||||
FP->>Path: GetImagePath(annotationId)
|
||||
FP->>FS: read bytes
|
||||
end
|
||||
FP->>FP: serialize MessagePack (Annotation* QueueMessage)
|
||||
FP->>RMQ: publish stream entry
|
||||
alt publish ok
|
||||
FP->>DB: DELETE annotations_queue_records WHERE id = ...
|
||||
else stream unavailable
|
||||
FP->>FP: backoff + retry next loop
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
| Step | From | To | Data | Format |
|
||||
|------|------|----|------|--------|
|
||||
| 1 | DB | producer | outbox rows | SQL |
|
||||
| 2 | filesystem | producer | image bytes | binary |
|
||||
| 3 | producer | RabbitMQ stream | `AnnotationQueueMessage` / `AnnotationBulkQueueMessage` | MessagePack (gzip per impl) |
|
||||
| 4 | producer | DB | DELETE | SQL |
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| RabbitMQ unreachable | publish | client exception | row stays in outbox; retried next tick |
|
||||
| Image file missing for `Created` | step 2 | FS read fails | open question — current behavior should be confirmed in code-review (skip vs retry) |
|
||||
| Concurrent drainers (multiple instances) | step 4 | no leasing | rows may be picked up twice → duplicate stream entries; consumers must dedupe |
|
||||
|
||||
### Performance Expectations
|
||||
|
||||
Bounded by RabbitMQ stream throughput + disk read for `Created`; durability is the priority (see ADR-003).
|
||||
|
||||
---
|
||||
|
||||
## Flow F5: Media Upload (single + batch)
|
||||
|
||||
### Description
|
||||
|
||||
UI uploads media files. `MediaController` accepts a single JSON-described upload (`POST /media`) or a multipart batch (`POST /media/batch` with `waypointId` + `IFormFileCollection`). `MediaService` writes the file under the configured media directory and persists a `media` row.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI
|
||||
participant Ctrl as MediaController (03)
|
||||
participant Svc as MediaService (03)
|
||||
participant Path as PathResolver (06)
|
||||
participant DB
|
||||
participant FS as Filesystem
|
||||
|
||||
UI->>Ctrl: POST /media[/batch] (multipart or JSON, JWT ANN)
|
||||
Ctrl->>Svc: CreateMedia / CreateBatch
|
||||
Svc->>Path: GetMediaDir(...)
|
||||
Svc->>FS: write file(s) under media dir
|
||||
Svc->>DB: INSERT media row(s)
|
||||
Svc-->>Ctrl: created media id(s)
|
||||
Ctrl-->>UI: 201 Created
|
||||
```
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Filesystem write fails | service | IOException | 500 |
|
||||
| Unsupported format | service | format check | 400 (per service validation; confirm during Step 4 verification) |
|
||||
|
||||
---
|
||||
|
||||
## Flow F6: Auth Refresh — REMOVED
|
||||
|
||||
Annotations no longer mints tokens. The legacy `POST /auth/refresh` endpoint and its backing `TokenService` were removed; admin (`POST /token/refresh`) is now the sole refresh issuer for the suite. Detections and any other long-running caller must refresh against admin and pass the resulting access token to annotations.
|
||||
|
||||
This service is a **verifier only**: it validates the `Authorization: Bearer …` header against admin's JWKS (`JWT_JWKS_URL`) on every `[Authorize]` route — see `JwtExtensions` in `_docs/02_document/modules/auth-identity.md`.
|
||||
|
||||
---
|
||||
|
||||
## Flow F7: Directory Settings Change → Path Cache Reset
|
||||
|
||||
### Description
|
||||
|
||||
Admin updates filesystem roots (`videos_dir`, `images_dir`, `labels_dir`, `thumbnails_dir`, `results_dir`, `gps_*`) via `PUT /settings/directories`. `SettingsService` persists the row and **must call** `PathResolver.Reset()` so subsequent reads see the new roots.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Admin
|
||||
participant Ctrl as SettingsController (05)
|
||||
participant Svc as SettingsService (05)
|
||||
participant DB
|
||||
participant Path as PathResolver (06)
|
||||
|
||||
Admin->>Ctrl: PUT /settings/directories (UpdateDirectoriesRequest, JWT ADM)
|
||||
Ctrl->>Svc: UpdateDirectories(request)
|
||||
Svc->>DB: UPDATE directory_settings
|
||||
Svc->>Path: Reset()
|
||||
Svc-->>Ctrl: ok
|
||||
Ctrl-->>Admin: 204 NoContent
|
||||
```
|
||||
|
||||
### Verified
|
||||
|
||||
`SettingsService` calls `pathResolver.Reset()` on directory updates (lines 71 and 85 of `Services/SettingsService.cs`). The invariant holds today.
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Multi-instance deployments | n/a | each instance caches independently in its own `PathResolver` singleton | each pod re-loads on next miss; no cross-pod fan-out — flagged for horizontal scale planning |
|
||||
|
||||
---
|
||||
|
||||
## Flow F8: Dataset Bulk Status
|
||||
|
||||
### Description
|
||||
|
||||
Dataset Explorer changes annotation status one at a time or in bulk. `DatasetService.UpdateStatus` / `BulkUpdateStatus` issue a direct `UPDATE annotations SET status = ...` via `AppDataConnection`. **Today this flow does NOT publish SSE and does NOT enqueue the failsafe outbox** — the Annotator UI will not see dataset-driven status changes in real time, and downstream stream consumers will not see the lifecycle event. Open behavioral question (see Open Items below).
|
||||
|
||||
### Routes
|
||||
|
||||
- `PATCH /dataset/{annotationId}/status` (single)
|
||||
- `POST /dataset/bulk-status` with `BulkStatusRequest { AnnotationIds, Status }` (bulk)
|
||||
|
||||
Both require `[Authorize(Policy = "DATASET")]`.
|
||||
|
||||
### Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant UI as Dataset Explorer
|
||||
participant Ctrl as DatasetController (04)
|
||||
participant Svc as DatasetService (04)
|
||||
participant DB
|
||||
|
||||
UI->>Ctrl: PATCH /dataset/{id}/status OR POST /dataset/bulk-status (JWT DATASET)
|
||||
Ctrl->>Svc: UpdateStatus(id, status) OR BulkUpdateStatus(request)
|
||||
alt single
|
||||
Svc->>DB: UPDATE annotations SET status WHERE id = :id
|
||||
DB-->>Svc: rowcount
|
||||
opt rowcount = 0
|
||||
Svc-->>Ctrl: KeyNotFoundException
|
||||
Ctrl-->>UI: 404
|
||||
end
|
||||
else bulk
|
||||
Svc->>Svc: validate ids list non-empty (else 400)
|
||||
Svc->>DB: UPDATE annotations SET status WHERE id IN (:ids)
|
||||
end
|
||||
Svc-->>Ctrl: ok
|
||||
Ctrl-->>UI: 200 / 204
|
||||
```
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
| Error | Where | Detection | Recovery |
|
||||
|-------|-------|-----------|----------|
|
||||
| Empty bulk list | `BulkUpdateStatus` | `ArgumentException` | 400 via middleware |
|
||||
| Annotation not found (single) | `UpdateStatus` | `updated == 0` | 404 |
|
||||
| Partial bulk failure under DB error | service | exception mid-update | UPDATE is a single SQL statement (`Set` + `UpdateAsync`) — atomic at the statement level; either all listed rows update or none |
|
||||
|
||||
### Open behavioral questions
|
||||
|
||||
- Should this flow publish SSE so the Annotator UI updates live?
|
||||
- Should this flow enqueue the outbox so AI training / admin sync reflect dataset status decisions?
|
||||
- Today the answer to both is "no" — confirm with stakeholders.
|
||||
|
||||
---
|
||||
|
||||
## Stakeholder Resolutions (Step 4 outcome)
|
||||
|
||||
These were the open behavioral questions raised by the verification pass; resolved with the maintainer on 2026-05-14. The architecture doc carries the full ADRs (ADR-008..ADR-011) and the Refactor Backlog (RB-01..RB-06). Summary here:
|
||||
|
||||
1. **Silent Update / Delete / dataset-status changes** — confirmed real gap, not intent. World B is the design (drainer is already plumbed for `Validated` and `Deleted` per `FailsafeProducer.cs:108–123`; the producer side was simply never wired in the new HTTP backend after the WPF split). Tracked: ADR-009 / RB-01.
|
||||
2. **`system_settings.silent_detection`** — debug-time switch superseded by the suite e2e harness. Remove the flag and gating logic. Tracked: ADR-010 / RB-02.
|
||||
3. **F1 atomicity** — adopt a business-transaction wrapper (transactional outbox): DB rows + outbox commit first, FS writes execute post-commit. Tracked: ADR-008 / RB-03.
|
||||
4. **Annotation id collision risk** — switch to `XxHash3.Hash128` over the same sampled buffer to keep the hash file-size-independent (videos can be 3–5 GB) while moving from 64-bit to 128-bit collision space. Tracked: ADR-004 / RB-04.
|
||||
5. **`FailsafeProducer.EnqueueAsync` static method doing DB I/O** — accepted as-is despite the `coderule.mdc` deviation; documented exception, no refactor.
|
||||
6. **`detection_classes` static catalog** — promote to admin-managed (`POST/PUT/DELETE /classes` under `[ADM]`) with a read-through cache modeled on `PathResolver.Reset()`. Tracked: ADR-011 / RB-06.
|
||||
|
||||
### Sub-questions deferred to RB-01 implementation
|
||||
|
||||
- `UpdateAnnotation` (replaces detections, sets `Status=Edited`) → re-enqueue as `Created` (rich payload) or add `QueueOperation.Updated` and a new drainer branch?
|
||||
- Status transitions other than `→ Validated` / `→ Deleted` — should they enqueue at all?
|
||||
- `DeleteAnnotation` is hard-delete today even though `AnnotationStatus.Deleted = 40` exists. Confirm hard- vs soft-delete semantics.
|
||||
|
||||
### Verified during Step 4
|
||||
|
||||
- F7 (`PathResolver.Reset` on directory change) — invariant holds; `SettingsService` calls `Reset` on lines 71 + 85.
|
||||
- All endpoint routes / policies match controller attributes.
|
||||
- `AnnotationService.CreateAnnotation` exact sequence (image file → media row → annotation → detections → label file → SSE → outbox).
|
||||
- `BulkUpdateStatus` empty-list rejection (`ArgumentException`).
|
||||
- Whole `src/` tree has exactly **two** producer call sites: `AnnotationService.cs:90` (`PublishAsync`) and `:102` (`EnqueueAsync`). All other paths are silent today.
|
||||
|
||||
### Open at flow level (residual)
|
||||
|
||||
- **F4 missing-file behavior** for `Created` operations: `FailsafeProducer.cs:138` swallows `IOException` silently and emits a stream message with `image = null`. Tracked as RB-05 (architecture doc).
|
||||
- **F4 multi-drainer dedupe**: still required — outbox uses no leasing. Suite consumer contract should dedupe by `(annotationId, operation)`.
|
||||
|
||||
Mermaid renderings of each flow are kept simple (no styling) per the template convention.
|
||||
@@ -0,0 +1,572 @@
|
||||
# Blackbox Tests
|
||||
|
||||
## Positive Scenarios
|
||||
|
||||
### FT-P-01: Annotation create — single detection, small image
|
||||
|
||||
**Summary**: A `POST /annotations` with a small frame and one synthetic detection persists the row, writes the YOLO label file, and returns the persisted DTO.
|
||||
**Traces to**: AC-F-01, AC-F-03, AC-F-04
|
||||
**Category**: Annotation lifecycle — Create
|
||||
|
||||
**Preconditions**:
|
||||
- SUT healthy (`/health` returns 200)
|
||||
- DB clean (no rows in `annotations`, `detection`, `media`, `annotations_queue_records`)
|
||||
- Runner has minted an ES256 token with the `ANN` claim (see `test-data.md` → "Bearer token harness")
|
||||
|
||||
**Input data**: `image_small.jpg` + `F1_001_request.json` (1 detection, `class_num=10` Plane, normalized bbox)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with the request body | HTTP 200; body matches `AnnotationDto` schema; `body.id =~ /^[0-9a-f]{32}$/`; `body.detections.length == 1` |
|
||||
| 2 | Out-of-band: assert `<images_dir>/<id>.jpg` exists with same bytes as `image_small.jpg` | file present, byte-for-byte match |
|
||||
| 3 | Out-of-band: assert `<images_dir>/<id>.txt` exists with one line `10 0.45 0.32 0.08 0.12` (or whatever the request supplied, formatted) | file present, line matches regex `^10 \d+\.\d+ \d+\.\d+ \d+\.\d+ \d+\.\d+$` |
|
||||
| 4 | `GET /annotations/{id}` | HTTP 200; same body as step 1 |
|
||||
|
||||
**Expected outcome**: the persisted entity round-trips through `GET /annotations/{id}` byte-for-byte, the image file is on disk, and the label file format is YOLO.
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-02: Annotation create — idempotency on identical re-POST
|
||||
|
||||
**Summary**: Re-POSTing the same image bytes + same detections does not create a new row; the second response carries the same `id`.
|
||||
**Traces to**: AC-F-01, AC-F-02
|
||||
|
||||
**Preconditions**:
|
||||
- FT-P-01 has just succeeded, so an annotation for `image_small.jpg` already exists.
|
||||
|
||||
**Input data**: same as FT-P-01
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with the same body | HTTP 200; `body.id == <id from FT-P-01>` |
|
||||
| 2 | Out-of-band: count rows in `annotations WHERE id = <id>` | `count == 1` |
|
||||
|
||||
**Expected outcome**: idempotent write — same hash → same id → same row.
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-03: Annotation create — empty scene, 0 detections
|
||||
|
||||
**Summary**: An empty-scene image with 0 detections creates an annotation row with no detection rows; the YOLO label file is empty.
|
||||
**Traces to**: AC-F-03 (label-file format with 0 detections)
|
||||
|
||||
**Preconditions**: clean state.
|
||||
|
||||
**Input data**: `image_empty_scene.jpg` + `F1_003_request.json` (0 detections)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` | HTTP 200; `body.detections.length == 0` |
|
||||
| 2 | Out-of-band: read `<images_dir>/<id>.txt` | file exists; content is empty (0 bytes) or whitespace-only |
|
||||
| 3 | Out-of-band: count rows in `detection WHERE annotation_id = <id>` | `count == 0` |
|
||||
|
||||
**Expected outcome**: persisted annotation with empty detections; label file present and empty.
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-04: Annotation create — dense scene, 5 mixed-class detections
|
||||
|
||||
**Summary**: A dense frame with 5 detections across multiple seeded classes persists 5 detection rows, writes a 5-line YOLO label, and returns a DTO with all 5 detections.
|
||||
**Traces to**: AC-F-03, AC-F-04
|
||||
|
||||
**Preconditions**: clean state.
|
||||
|
||||
**Input data**: `image_dense01.jpg` + `F1_004_request.json` (5 detections, class_num ∈ {0=ArmorVehicle, 1=Truck, 2=Vehicle, 9=Smoke, 10=Plane})
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` | HTTP 200; `body.detections.length == 5` |
|
||||
| 2 | Out-of-band: read `<images_dir>/<id>.txt` | exactly 5 lines; each matches `^\d+ \d+\.\d+ \d+\.\d+ \d+\.\d+ \d+\.\d+$` |
|
||||
| 3 | Compare line set against expected | line set equals the 5 detections in `F1_004_request.json` (order may differ — test uses set equality) |
|
||||
|
||||
**Expected outcome**: 5 detections round-trip through both DB and YOLO label.
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-05: Annotation listing — paginated read
|
||||
|
||||
**Summary**: After several creates, `GET /annotations` returns a paginated list with the correct shape and count.
|
||||
**Traces to**: AC-F-04 (read path)
|
||||
|
||||
**Preconditions**: FT-P-01..FT-P-04 have run; 4 annotations exist.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations?limit=10` | HTTP 200; `body.length == 4`; each item conforms to `AnnotationListItem` schema |
|
||||
| 2 | `GET /annotations?limit=2&offset=0` | HTTP 200; `body.length == 2` |
|
||||
| 3 | `GET /annotations?limit=2&offset=2` | HTTP 200; `body.length == 2`; ids disjoint from step 2's response |
|
||||
|
||||
**Expected outcome**: paginated read works; results are stable across paging windows.
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-06: Annotation detail by id
|
||||
|
||||
**Summary**: `GET /annotations/{id}` returns the full DTO including detections.
|
||||
**Traces to**: AC-F-04
|
||||
|
||||
**Preconditions**: FT-P-04 has run.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations/<id from FT-P-04>` | HTTP 200; body matches `AnnotationDto`; `body.detections.length == 5` |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-07: SSE delivery — event for new annotation
|
||||
|
||||
**Summary**: A subscriber connected to `/annotations/events?missionId=<m>` receives the lifecycle event for a `POST /annotations` against that mission within 1 second.
|
||||
**Traces to**: AC-F-10
|
||||
|
||||
**Preconditions**: clean state.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Open SSE connection to `/annotations/events?missionId=<m>` | HTTP 200; `Content-Type: text/event-stream` |
|
||||
| 2 | `POST /annotations` against mission `<m>` | HTTP 200 |
|
||||
| 3 | Read next event from the SSE stream | event arrives within 1000ms; `event.data` parses as `AnnotationEventDto`; `event.operation == "Created"`; `event.annotationId == <id from step 2>` |
|
||||
|
||||
**Expected outcome**: real-time delivery of the lifecycle event.
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-08: Outbox row on create
|
||||
|
||||
**Summary**: A successful `POST /annotations` inserts exactly one row into `annotations_queue_records` with `operation == 10` (Created).
|
||||
**Traces to**: AC-F-12 (outbox drain), AC-F-05 (`[after RB-01]` for non-Created paths)
|
||||
|
||||
**Preconditions**: clean state; RabbitMQ broker reachable but the test does not consume from the stream yet.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` (any valid payload) | HTTP 200 |
|
||||
| 2 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id> AND operation = 10` immediately after step 1 | `count == 1` (within 500ms — outbox insert happens before the response returns) |
|
||||
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-09: Stream message round-trip
|
||||
|
||||
**Summary**: After the outbox drain interval, a message arrives on the `azaion-annotations` stream that decodes to the documented schema.
|
||||
**Traces to**: AC-F-12
|
||||
|
||||
**Preconditions**: FT-P-08 just succeeded; outbox row present.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Connect a stream consumer to `azaion-annotations` at offset `next` | consumer alive |
|
||||
| 2 | Wait up to `drain_interval + 2s` for one message | one message arrives |
|
||||
| 3 | gzip-decompress + MessagePack-deserialize the body | object matches the documented stream schema |
|
||||
| 4 | Out-of-band: re-query `annotations_queue_records WHERE annotation_id = <id>` | `count == 0` (drainer deleted the row) |
|
||||
|
||||
**Max execution time**: 30s (depends on configured drain interval)
|
||||
|
||||
---
|
||||
|
||||
### FT-P-10: Media single upload
|
||||
|
||||
**Summary**: `POST /media` (multipart) persists the file and a media row.
|
||||
**Traces to**: AC-F-20
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /media` with `image_small.jpg`, `mediaType=Image`, `waypointId=<m>` (multipart) | HTTP 200; body matches `MediaListItem` schema |
|
||||
| 2 | Out-of-band: `<media_dir>/<media_id>.jpg` exists | file present |
|
||||
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-11: Media batch upload
|
||||
|
||||
**Summary**: `POST /media/batch` with N files persists N rows + N files in one request.
|
||||
**Traces to**: AC-F-21
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /media/batch` with 3 distinct files (`image_small`, `image_dense01`, `image_dense02`) | HTTP 200; `body.length == 3`; 3 distinct media ids |
|
||||
| 2 | Out-of-band: 3 distinct files exist on disk | 3 files present |
|
||||
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-12: Bearer token verification — happy path
|
||||
|
||||
**Summary**: A request bearing an ES256 access token whose `iss`, `aud`, signature, and `exp` are all valid is accepted by every authenticated endpoint reached.
|
||||
**Traces to**: AC-F-50
|
||||
|
||||
**Preconditions**: A test-only ES256 key pair is published at the `JWT_JWKS_URL` fetched by the service at boot (see `test-data.md` → "Bearer token harness"). The runner mints an access token signed with the matching private key.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations` with `Authorization: Bearer <ES256 token, iss=$JWT_ISSUER, aud=$JWT_AUDIENCE, exp=now+5m, ANN claim present>` | HTTP 200; valid `PaginatedResponse<AnnotationListItem>` body |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-13: Bearer token verification — alg pinning
|
||||
|
||||
**Summary**: A token signed with `alg=HS256` (using the public ES256 key as the HMAC secret) is rejected — `JwtExtensions.AddJwtAuth` pins `ValidAlgorithms = [EcdsaSha256]`.
|
||||
**Traces to**: AC-F-50
|
||||
|
||||
**Preconditions**: Same harness as FT-P-12. Runner additionally produces a forged HS256 token using the public ES256 key bytes as the HMAC key.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations` with `Authorization: Bearer <forged HS256 token>` | HTTP 401; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-14: Detection class catalog read
|
||||
|
||||
**Summary**: `GET /classes` returns the 19 seeded classes with stable ids.
|
||||
**Traces to**: AC-F-41
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /classes` | HTTP 200; `body.length == 19`; ids `[0..18]` present (set equality); entry where `id==9` has `name=="Smoke"`; entry where `id==10` has `name=="Plane"` |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-15: Directory settings → PathResolver invariant
|
||||
|
||||
**Summary**: `PUT /settings/directories` updates the values; the next annotation create writes to the new path.
|
||||
**Traces to**: AC-F-40
|
||||
|
||||
**Preconditions**: ADM JWT in hand. Volume mounts include both old and new paths so the SUT can write to either.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /settings/directories` | HTTP 200; record current `imagesDir` |
|
||||
| 2 | `PUT /settings/directories` with `imagesDir = /data/images-alt` | HTTP 200 |
|
||||
| 3 | `GET /settings/directories` | HTTP 200; `imagesDir == "/data/images-alt"` |
|
||||
| 4 | `POST /annotations` for a fresh image | HTTP 200; out-of-band: image lands at `/data/images-alt/<id>.jpg`, NOT at the original `imagesDir` |
|
||||
|
||||
**Expected outcome**: `pathResolver.Reset()` has fired and the next write uses the new directory.
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-16: Dataset filter by status
|
||||
|
||||
**Summary**: `GET /dataset?status=10` (Pending) returns only Pending rows.
|
||||
**Traces to**: AC-F-30
|
||||
|
||||
**Preconditions**: FT-P-04 just ran; one Pending annotation exists.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /dataset?status=10` | HTTP 200; every item in `body` has `status == 10` |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-17: Dataset class distribution
|
||||
|
||||
**Summary**: `GET /dataset/class-distribution` returns counts grouped by class with the expected shape.
|
||||
**Traces to**: AC-F-30 (read path), AC-F-41 (class metadata)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /dataset/class-distribution` after FT-P-04 (5 detections of mixed classes) | HTTP 200; body is an array; entry for `classNum=10` has `count >= 1`; sum of all `count` values equals total detection rows |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-18: Dataset bulk status
|
||||
|
||||
**Summary**: `POST /dataset/status/bulk` flips status atomically on N rows.
|
||||
**Traces to**: AC-F-31
|
||||
|
||||
**Preconditions**: 2+ Pending annotations from FT-P-01 and FT-P-04 (now FT-P-04 has 1; need at least 2).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /dataset/status/bulk` with `{annotationIds: [<id1>, <id2>], status: 20}` | HTTP 200 |
|
||||
| 2 | `GET /annotations/<id1>` | HTTP 200; `body.status == 20` |
|
||||
| 3 | `GET /annotations/<id2>` | HTTP 200; `body.status == 20` |
|
||||
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-19: Health check
|
||||
|
||||
**Summary**: `GET /health` returns 200 with low latency at any time post-boot.
|
||||
**Traces to**: AC-F-54
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /health` | HTTP 200 within 200ms |
|
||||
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-20: Migrator idempotence
|
||||
|
||||
**Summary**: Restarting the SUT against the same DB makes 0 schema changes.
|
||||
**Traces to**: AC-N-02
|
||||
|
||||
**Preconditions**: SUT booted once; DB schema captured (e.g., `pg_dump --schema-only`).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Capture schema-only dump → `dump_a.sql` | non-empty dump |
|
||||
| 2 | `docker compose restart annotations` | SUT comes back healthy |
|
||||
| 3 | Capture schema-only dump → `dump_b.sql` | non-empty dump |
|
||||
| 4 | Diff `dump_a.sql` and `dump_b.sql` | zero meaningful differences (whitespace / SERIAL counters tolerated) |
|
||||
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-21 `[after RB-01]`: Lifecycle event on update
|
||||
|
||||
**Summary**: `PUT /annotations/{id}` emits an `Updated` SSE event AND inserts an outbox row.
|
||||
**Traces to**: AC-F-05 (post RB-01)
|
||||
|
||||
**Note**: this test stays disabled (skipped with reason `"awaiting RB-01"`) until the refactor lands.
|
||||
|
||||
---
|
||||
|
||||
### FT-P-22 `[after RB-01]`: Lifecycle event on delete + soft-delete file relocation
|
||||
|
||||
**Summary**: `DELETE /annotations/{id}` flips status to `40`, relocates files to `deleted_dir`, and emits a `Deleted` SSE event + outbox row.
|
||||
**Traces to**: AC-F-06, AC-F-07
|
||||
|
||||
**Note**: skipped until RB-01 + RB-08 land.
|
||||
|
||||
---
|
||||
|
||||
## Negative Scenarios
|
||||
|
||||
### FT-N-01: Create without image bytes
|
||||
|
||||
**Summary**: `POST /annotations` with no `image` field is rejected.
|
||||
**Traces to**: AC-F-04 (negative)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with body missing `image` | HTTP 400 or 422; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-02: Create without mediaType
|
||||
|
||||
**Summary**: Missing required enum field is rejected.
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with no `mediaType` | HTTP 400 or 422; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-03: Create without ANN policy
|
||||
|
||||
**Summary**: A token with policy `DATASET` cannot create annotations.
|
||||
**Traces to**: AC-F-52
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with an ES256 token carrying only the `DATASET` claim | HTTP 403; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-04: Create unauthenticated
|
||||
|
||||
**Summary**: Missing `Authorization` header → 401.
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with no `Authorization` header | HTTP 401; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-05: Out-of-range bbox value (current lenient behavior)
|
||||
|
||||
**Summary**: `centerX = 1.5` is accepted today; the test asserts the **current** behavior. Will flip to expecting 400/422 after SEC-05 lands.
|
||||
**Traces to**: documented gap in `security_approach.md` SEC-05
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /annotations` with `detections[0].centerX = 1.5` | HTTP 200 today (lenient); test will be inverted post-SEC-05 |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-06: GET nonexistent annotation
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations/00000000000000000000000000000000` | HTTP 404; error envelope; `error.code` matches `/not.?found/i` |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-07: Filter by unknown mission
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations?missionId=<unknown-guid>` | HTTP 200; `body.length == 0` |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-08: SSE without auth
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | Open SSE to `/annotations/events?missionId=<m>` with no `Authorization` | HTTP 401 on connection establishment |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-09: Bearer token — expired
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations` with `Authorization: Bearer <token with exp=now-1m, otherwise valid>` | HTTP 401; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-10: Bearer token — wrong issuer
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations` with `Authorization: Bearer <token with iss="https://other.example.com" but otherwise valid>` | HTTP 401; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-11: Bearer token — wrong audience
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `GET /annotations` with `Authorization: Bearer <token with aud="some-other-service" but otherwise valid>` | HTTP 401; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-12: Mutating settings without ADM
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `PUT /settings/system` with an ES256 token carrying only the `ANN` claim | HTTP 403; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-13: PUT directories without ADM
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `PUT /settings/directories` with non-ADM JWT | HTTP 403; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-14: Media upload missing waypoint
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /media` multipart without `waypointId` | HTTP 400 or 422; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-15: Media upload without ANN
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /media` with non-ANN JWT | HTTP 403; error envelope |
|
||||
|
||||
**Max execution time**: 3s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-16: Bulk status with empty list
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | `POST /dataset/status/bulk` with `annotationIds: []` | HTTP 400; error envelope (verified: `DatasetService.BulkUpdateStatus` throws `ArgumentException`) |
|
||||
|
||||
**Max execution time**: 3s
|
||||
@@ -0,0 +1,218 @@
|
||||
# Test Environment
|
||||
|
||||
## Overview
|
||||
|
||||
**System under test**: `Azaion.Annotations` HTTP API on port 8080 (REST + SSE) plus its RabbitMQ Stream producer (`azaion-annotations` stream).
|
||||
**Consumer app purpose**: A standalone test runner that exercises the system through its public HTTP / SSE / Stream interfaces only — no in-process imports, no direct DB queries against the system's main DB, no shared filesystem.
|
||||
|
||||
## Docker Environment
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Image / Build | Purpose | Ports |
|
||||
|---------|--------------|---------|-------|
|
||||
| `annotations` | Built from `src/Dockerfile` (ARM64) with `AZAION_REVISION=test-<sha>` | System under test | `8080:8080` |
|
||||
| `postgres` | `postgres:13` | DB for the system under test | `5432:5432` (private to test net) |
|
||||
| `rabbitmq` | `rabbitmq:3.13-management` with the **streams plugin** enabled | Stream broker the SUT publishes to | `5552:5552` (stream listener), `15672:15672` (mgmt UI, optional) |
|
||||
| `e2e-runner` | Built from `tests/Azaion.Annotations.E2E/Dockerfile` | Black-box test runner (xUnit + HttpClient + RabbitMQ.Stream.Client consumer); also holds the ES256 private key used to mint per-test bearer tokens | — |
|
||||
| `e2e-issuer` | `python:3.12-alpine` running `tests/harness/mock_issuer.py` (≈40 lines, serves a static JWKS over HTTP) | Mock JWKS endpoint stand-in for admin's real issuer; publishes the public ES256 key the SUT validates against | `8080` (on `e2e-net`; not exposed to host) |
|
||||
| `dataseed` | One-shot job: `psql` only | Boot-time seed of any required reference data (no users — annotations has no `users` table) | — |
|
||||
|
||||
The fixture binaries (frame images, videos) are mounted from `../detections/_docs/00_problem/input_data/` (suite-relative path, see `_docs/00_problem/input_data/fixtures.md`) into both the `annotations` service (read-only, for direct file ingestion paths) and the `e2e-runner` (read-only, for upload-as-multipart paths).
|
||||
|
||||
### Networks
|
||||
|
||||
| Network | Services | Purpose |
|
||||
|---------|----------|---------|
|
||||
| `e2e-net` | `annotations`, `postgres`, `rabbitmq`, `e2e-issuer`, `e2e-runner`, `dataseed` | Isolated bridge network — services reach each other by container hostname |
|
||||
|
||||
### Volumes
|
||||
|
||||
| Volume | Mounted to | Purpose |
|
||||
|--------|-----------|---------|
|
||||
| `annotations-images` | `annotations:/data/images` | `images_dir` — content-addressed image bytes + YOLO label files |
|
||||
| `annotations-videos` | `annotations:/data/videos` | `videos_dir` |
|
||||
| `annotations-deleted` | `annotations:/data/deleted` | `deleted_dir` (post RB-01 soft-delete relocation) |
|
||||
| `pg-data` | `postgres:/var/lib/postgresql/data` | DB durability across container restart (resilience scenarios) |
|
||||
| `fixtures-ro` (bind) | `annotations:/fixtures:ro`, `e2e-runner:/fixtures:ro` | Reuse of detections corpus binaries |
|
||||
|
||||
### docker-compose structure
|
||||
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:13
|
||||
environment:
|
||||
POSTGRES_DB: annotations
|
||||
POSTGRES_USER: annotations
|
||||
POSTGRES_PASSWORD: annotations
|
||||
volumes:
|
||||
- pg-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U annotations"]
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:3.13-management
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: annotations
|
||||
RABBITMQ_DEFAULT_PASS: annotations
|
||||
RABBITMQ_PLUGINS: rabbitmq_stream rabbitmq_management
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "ping"]
|
||||
|
||||
e2e-issuer:
|
||||
image: python:3.12-alpine
|
||||
command: ["python", "/harness/mock_issuer.py"]
|
||||
volumes:
|
||||
- ../tests/harness:/harness:ro
|
||||
- jwt-keys:/keys
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://localhost:8080/.well-known/jwks.json"]
|
||||
|
||||
annotations:
|
||||
build:
|
||||
context: ../src
|
||||
environment:
|
||||
ASPNETCORE_ENVIRONMENT: E2ETest
|
||||
DATABASE_URL: postgresql://annotations:annotations@postgres:5432/annotations
|
||||
JWT_ISSUER: https://e2e-issuer.test
|
||||
JWT_AUDIENCE: annotations-e2e
|
||||
JWT_JWKS_URL: http://e2e-issuer:8080/.well-known/jwks.json
|
||||
CorsConfig__AllowedOrigins__0: http://e2e-runner.test
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_STREAM_PORT: 5552
|
||||
RABBITMQ_PRODUCER_USER: annotations
|
||||
RABBITMQ_PRODUCER_PASS: annotations
|
||||
AZAION_REVISION: test-${GIT_SHA:-local}
|
||||
volumes:
|
||||
- annotations-images:/data/images
|
||||
- annotations-videos:/data/videos
|
||||
- annotations-deleted:/data/deleted
|
||||
- ../../detections/_docs/00_problem/input_data:/fixtures:ro
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
e2e-issuer:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
|
||||
|
||||
dataseed:
|
||||
image: postgres:13
|
||||
depends_on:
|
||||
annotations:
|
||||
condition: service_healthy
|
||||
entrypoint: ["/bin/sh", "/seed/run.sh"]
|
||||
volumes:
|
||||
- ./seed:/seed:ro
|
||||
|
||||
e2e-runner:
|
||||
build:
|
||||
context: ../tests/Azaion.Annotations.E2E
|
||||
depends_on:
|
||||
dataseed:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
ANNOTATIONS_BASE_URL: http://annotations:8080
|
||||
JWT_ISSUER: https://e2e-issuer.test
|
||||
JWT_AUDIENCE: annotations-e2e
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_STREAM_PORT: 5552
|
||||
RABBITMQ_USER: annotations
|
||||
RABBITMQ_PASS: annotations
|
||||
FIXTURES_DIR: /fixtures
|
||||
volumes:
|
||||
- ../../detections/_docs/00_problem/input_data:/fixtures:ro
|
||||
- jwt-keys:/keys:ro
|
||||
|
||||
volumes:
|
||||
pg-data: {}
|
||||
annotations-images: {}
|
||||
annotations-videos: {}
|
||||
annotations-deleted: {}
|
||||
jwt-keys: {}
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: e2e-net
|
||||
```
|
||||
|
||||
## Consumer Application
|
||||
|
||||
**Tech stack**: .NET 10 + xUnit (matches the SUT runtime to avoid a second toolchain in CI). Uses `HttpClient` for REST, raw `HttpClient` with `text/event-stream` for SSE, and `RabbitMQ.Stream.Client` for stream-consumer scenarios.
|
||||
**Entry point**: `dotnet test --logger "console;verbosity=normal" --logger "trx" --results-directory /results`
|
||||
|
||||
### Communication with system under test
|
||||
|
||||
| Interface | Protocol | Endpoint / Topic | Authentication |
|
||||
|-----------|----------|-----------------|----------------|
|
||||
| Annotations REST | HTTP/1.1 JSON | `http://annotations:8080/annotations/*`, `/media/*`, `/dataset/*`, `/settings/*`, `/classes`, `/health` | `Authorization: Bearer <jwt>` (ES256 JWT minted on demand by the runner using the in-stack mock-issuer key) |
|
||||
| Annotations SSE | HTTP/1.1 `text/event-stream` | `http://annotations:8080/annotations/events?missionId=<guid>` | Same ES256 bearer token |
|
||||
| Mock JWKS | HTTP/1.1 JSON | `http://e2e-issuer:8080/.well-known/jwks.json` | None (test-net only) |
|
||||
| RabbitMQ Stream | AMQP 1.0 / streams (port 5552) | Stream `azaion-annotations` | Username + password env vars; consumer offset starts at `next` for fresh test runs |
|
||||
| Postgres (test-only, read-only assertions on DB state) | direct (out-of-band) | `postgresql://postgres:5432/annotations` | DB user; **only the test runner uses this and only for blackbox-allowed assertions** (e.g., F4-001 verifying the outbox row was inserted). Tests that need DB introspection are clearly marked. |
|
||||
|
||||
### What the consumer does NOT have access to
|
||||
|
||||
- No in-process import of the `Azaion.Annotations` assembly.
|
||||
- No direct write to the SUT's `annotations`, `media`, `detection`, `annotations_queue_records` tables (DB read access only, for outbox-state assertions documented in `test-data.md`). Annotations has no `users` table.
|
||||
- No shared memory or filesystem with the SUT (volumes are mounted read-only).
|
||||
- No mocking of internal services (`AnnotationService`, `FailsafeProducer`, etc.) — all interactions go through the public surface.
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
**When to run**: on every push to `dev` and on every PR; nightly full run including the long-running performance + resilience scenarios.
|
||||
**Pipeline stage**: after Woodpecker `build` step; new step `test-e2e` invoking `docker compose -f e2e/docker-compose.test.yml up --abort-on-container-exit --exit-code-from e2e-runner` (or, equivalently, `scripts/run-tests.sh`).
|
||||
**Gate behavior**: any failed scenario blocks the merge; nightly perf failures emit a warning but do not block a green PR.
|
||||
**Timeout**: 30 min for the standard suite (functional + smoke perf); 2 hours for the nightly full perf + resilience suite.
|
||||
|
||||
## Reporting
|
||||
|
||||
**Format**: CSV (xUnit's `trx` output is converted by the runner into a flat CSV).
|
||||
**Columns**: `test_id`, `test_name`, `category`, `traces_to`, `execution_time_ms`, `result`, `error_message`.
|
||||
**Output path**: `e2e-results/report.csv` inside the `e2e-runner` container, mounted out to `./e2e-results/report.csv` on the host.
|
||||
|
||||
In addition, raw xUnit `.trx` is preserved at `e2e-results/results.trx` for human inspection / IDE integration.
|
||||
|
||||
## Dependencies on the existing stack
|
||||
|
||||
This environment intentionally **does not** re-use the suite's running development DB or RabbitMQ — it stands up its own. The only suite-level dependency is the read-only mount of `detections/_docs/00_problem/input_data/` for fixtures.
|
||||
|
||||
## Test Execution
|
||||
|
||||
**Decision**: Docker only.
|
||||
|
||||
**Rationale** (from Hardware-Dependency Assessment, run between test-spec Phase 3 and Phase 4):
|
||||
|
||||
- **Documentation scan** — `restrictions.md` lists HW-01 (ARM64-only image), HW-02 (writable filesystem dirs), HW-03 (memory pressure on `FailsafeProducer`). None of these are accelerator / sensor / OS-feature dependencies; they are generic infrastructure constraints satisfiable in any Linux container.
|
||||
- **Code scan** — zero hits across `src/` for CUDA, TensorRT, CoreML, OpenCL, Vulkan, TPU, V4L2, GPIO, `cv2.VideoCapture`, `sys.platform`-style branches, or `platform.machine()` checks. The Dockerfile's `TARGETARCH` branch (line 5) is a buildplatform-aware Node toolchain selector, not a runtime hardware gate — the running binary uses managed .NET 10 with no native acceleration paths.
|
||||
- **Dependency files** — `Azaion.Annotations.csproj` references only managed NuGet packages (Linq2DB, Npgsql, JwtBearer, RabbitMQ.Stream.Client, MessagePack, Swashbuckle, System.IO.Hashing). No native-binding libraries, no hardware-specific packages.
|
||||
|
||||
**Classification**: not hardware-dependent. Docker is the preferred default and the only chosen mode.
|
||||
|
||||
### Docker mode — execution instructions
|
||||
|
||||
Run from the suite root (parent of `annotations/` and `detections/`) so the fixture bind-mount path resolves:
|
||||
|
||||
```bash
|
||||
# From the annotations repo root:
|
||||
./scripts/run-tests.sh # functional + smoke perf
|
||||
./scripts/run-performance-tests.sh # full perf scenarios
|
||||
|
||||
# Equivalent without the wrapper:
|
||||
docker compose -f e2e/docker-compose.test.yml up \
|
||||
--abort-on-container-exit \
|
||||
--exit-code-from e2e-runner
|
||||
```
|
||||
|
||||
Results land at `e2e/e2e-results/report.csv` (host path), and at `test-results/` for any JUnit/CTRX outputs. The exit code of `e2e-runner` becomes the suite's exit code; CI uses it as the gate.
|
||||
|
||||
### Why not local mode
|
||||
|
||||
The xUnit test runner CAN execute against a SUT bound to `localhost:8080` if a developer wants to iterate inside the IDE. That path is not the supported test environment for CI; it is a developer convenience. Phase 4 produces only the Docker runner script.
|
||||
|
||||
### CI image arch
|
||||
|
||||
The Docker test stack runs on the same ARM64 hosts the Woodpecker pipeline already targets (HW-01). If a future CI runner family is x86_64-only, the same docker-compose works because every service in `e2e-net` is multi-arch (`postgres:13`, `rabbitmq:3.13-management`, the SUT itself if rebuilt with `--platform linux/amd64`).
|
||||
@@ -0,0 +1,147 @@
|
||||
# Performance Tests
|
||||
|
||||
> **Calibration note**: no contracted SLAs exist anywhere in the codebase or `acceptance_criteria.md`. The thresholds below are **inferred starting points** anchored to the documented system properties. Step 15 (Performance Test) of the autodev existing-code flow will tune them against real targets. A test that fails the threshold is a *signal*, not a release-blocker, until the targets are contracted.
|
||||
|
||||
### NFT-PERF-LATENCY-01: Annotation create — p95 latency, small image
|
||||
|
||||
**Summary**: Sequential `POST /annotations` with a small frame stays under a per-call threshold at p95.
|
||||
**Traces to**: implicit NFR; documented gap on AC-N-* (no contracted target)
|
||||
**Metric**: end-to-end response latency in ms (consumer wall-clock from request start to body close).
|
||||
|
||||
**Preconditions**:
|
||||
- SUT freshly started; warmup loop of 10 sequential calls discarded.
|
||||
- Clean state; clean outbox; RabbitMQ stream consumer not connected (writes fan out via channel + outbox only).
|
||||
- Single in-process consumer (no concurrent load).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Warmup: 10× `POST /annotations` with `image_small.jpg` | discarded |
|
||||
| 2 | Measure: 50× `POST /annotations` with `image_small.jpg`, sequential, single consumer | record latency per call |
|
||||
| 3 | Compute p50, p95, p99 | summary stats |
|
||||
|
||||
**Pass criteria**: p95 ≤ 1500ms, p99 ≤ 3000ms (single-instance dev DB, no concurrent load).
|
||||
**Duration**: ~2 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-LATENCY-02: Annotation create — large image
|
||||
|
||||
**Summary**: Same shape as -01 with a 7 MB image.
|
||||
**Traces to**: same as -01.
|
||||
**Metric**: end-to-end latency.
|
||||
|
||||
**Preconditions**: same as -01.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Warmup: 5× `POST /annotations` with `image_large.JPG` | discarded |
|
||||
| 2 | Measure: 20× `POST /annotations` with `image_large.JPG`, sequential | record latency per call |
|
||||
| 3 | p50, p95, p99 | summary stats |
|
||||
|
||||
**Pass criteria**: p95 ≤ 5000ms, p99 ≤ 8000ms.
|
||||
**Duration**: ~2 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-THROUGHPUT-01: Annotation create — sustained writes
|
||||
|
||||
**Summary**: 5-minute sustained `POST /annotations` traffic at 5 RPS does not degrade response latency.
|
||||
**Metric**: response latency over time + total successful responses.
|
||||
|
||||
**Preconditions**: SUT warm; clean state; clean outbox; RabbitMQ broker reachable.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Warmup: 30s at 5 RPS with `image_small.jpg` | discarded |
|
||||
| 2 | Measure: 5 minutes at 5 RPS, 1 consumer | record per-second latency p50/p95 |
|
||||
| 3 | Compare windows | p95 in last minute ≤ 1.5× p95 in first minute |
|
||||
|
||||
**Pass criteria**: 0 HTTP 5xx; p95 latency in last minute ≤ 1.5× p95 in first minute.
|
||||
**Duration**: ~6 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-OUTBOX-DRAIN-01: FailsafeProducer drain rate
|
||||
|
||||
**Summary**: Under sustained writes, the outbox queue depth stays bounded.
|
||||
**Traces to**: AC-N-03
|
||||
**Metric**: `SELECT COUNT(*) FROM annotations_queue_records` sampled every 5s during the run.
|
||||
|
||||
**Preconditions**: NFT-PERF-THROUGHPUT-01 running; RabbitMQ broker reachable; no stream consumer back-pressure.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | While -THROUGHPUT-01 is running, sample queue depth every 5s for the full duration | record samples |
|
||||
| 2 | Compute max queue depth + average drain interval | summary stats |
|
||||
|
||||
**Pass criteria**: max queue depth ≤ 100 rows; depth at end-of-run ≤ depth at start-of-run + 10.
|
||||
**Duration**: 5 minutes (overlaid on -THROUGHPUT-01).
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-SSE-FANOUT-01: SSE delivery latency under modest fan-out
|
||||
|
||||
**Summary**: 10 simultaneous SSE subscribers receive every event for their mission within the latency budget.
|
||||
**Traces to**: AC-F-10
|
||||
**Metric**: per-subscriber event-arrival latency (consumer wall-clock from `POST /annotations` returning to SSE event arrival).
|
||||
|
||||
**Preconditions**: SUT warm; clean state.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Open 10 SSE connections to `/annotations/events?missionId=<m>` | all 10 alive |
|
||||
| 2 | `POST /annotations` once for mission `<m>` | record post-return timestamp |
|
||||
| 3 | Each subscriber records its event-arrival timestamp | per-subscriber latency |
|
||||
| 4 | Compute max latency across the 10 subscribers | summary |
|
||||
|
||||
**Pass criteria**: every subscriber receives the event; max latency ≤ 1000ms.
|
||||
**Duration**: 30s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-LIST-01: Annotation listing on populated DB
|
||||
|
||||
**Summary**: `GET /annotations?limit=100` against a DB with 10,000 rows responds within budget.
|
||||
**Metric**: end-to-end response latency.
|
||||
|
||||
**Preconditions**: DB pre-seeded with 10,000 annotations + 50,000 detections (use `dataseed` to insert via direct SQL, bypassing the public API for population speed — the test still queries via the public API).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Warmup: 5× `GET /annotations?limit=100&offset=0` | discarded |
|
||||
| 2 | Measure: 20× `GET /annotations?limit=100&offset=<random 0..9000>` | record per-call latency |
|
||||
| 3 | p95 | summary |
|
||||
|
||||
**Pass criteria**: p95 ≤ 1000ms (read-only path; index `ix_annotations_created_date` should keep it fast).
|
||||
**Duration**: ~1 minute.
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-DATASET-01: Dataset class distribution at scale
|
||||
|
||||
**Summary**: `GET /dataset/class-distribution` against the populated DB.
|
||||
**Metric**: end-to-end latency.
|
||||
|
||||
**Preconditions**: same populated DB as NFT-PERF-LIST-01.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Measurement |
|
||||
|------|----------------|-------------|
|
||||
| 1 | Warmup: 3 calls | discarded |
|
||||
| 2 | Measure: 10 calls | record latency |
|
||||
|
||||
**Pass criteria**: p95 ≤ 2000ms.
|
||||
**Duration**: ~30s.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Resilience Tests
|
||||
|
||||
### NFT-RES-01: RabbitMQ broker outage during create
|
||||
|
||||
**Summary**: `POST /annotations` succeeds (HTTP 200) when the RabbitMQ broker is unreachable; the outbox row is preserved; `FailsafeProducer` does not crash; on broker recovery the message is delivered.
|
||||
**Traces to**: AC-F-12, OP-02 (single-instance baseline)
|
||||
|
||||
**Preconditions**: SUT healthy; broker initially reachable; clean outbox.
|
||||
|
||||
**Fault injection**:
|
||||
- `docker exec rabbitmq rabbitmqctl stop_app` mid-test (stops AMQP/streams listeners; container stays up).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Stop RabbitMQ app | broker unreachable on 5552 |
|
||||
| 2 | `POST /annotations` once | HTTP 200; outbox row inserted |
|
||||
| 3 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | `count == 1` (row not deleted because drain failed) |
|
||||
| 4 | `GET /health` | HTTP 200 (SUT not crashed) |
|
||||
| 5 | `docker exec rabbitmq rabbitmqctl start_app` | broker recovers |
|
||||
| 6 | Wait `drain_interval × 3` | drainer publishes the queued message |
|
||||
| 7 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | `count == 0` (drained) |
|
||||
| 8 | Stream consumer (started before step 5 at offset `next`) reads one message | message body matches the documented schema |
|
||||
|
||||
**Pass criteria**: zero 5xx during outage; outbox preserves the row; recovery delivers the deferred message; total recovery time ≤ 60s after broker comes back.
|
||||
**Duration**: ~2 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-02: Postgres restart between writes
|
||||
|
||||
**Summary**: Killing and restarting Postgres during a quiet period does not corrupt state; subsequent writes succeed.
|
||||
**Traces to**: AC-N-02 (idempotent migrator), implicit data-integrity NFR
|
||||
|
||||
**Fault injection**: `docker compose restart postgres` while no in-flight requests.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | `POST /annotations` once (FT-P-01-shape) | HTTP 200; row in DB |
|
||||
| 2 | `docker compose restart postgres` | DB up after ~5s |
|
||||
| 3 | Wait for SUT `/health` to return 200 | SUT recovers connection pool (or restarts itself) |
|
||||
| 4 | `POST /annotations` again | HTTP 200; row in DB |
|
||||
| 5 | `GET /annotations/<id from step 1>` | HTTP 200; original row intact |
|
||||
|
||||
**Pass criteria**: original row intact after restart; new write succeeds within 30s of DB recovery; zero data loss.
|
||||
**Duration**: ~2 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-03: Postgres unreachable during create
|
||||
|
||||
**Summary**: When DB is unreachable mid-request, the SUT returns a structured error envelope (no 500 with stack trace); the SUT recovers when DB returns.
|
||||
**Traces to**: AC-N-04 (zero unhandled exceptions to clients)
|
||||
|
||||
**Fault injection**: `docker pause postgres` between request start and request end (race-y; use a delay-injecting test proxy if needed).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | `docker pause postgres` | DB connections hang |
|
||||
| 2 | `POST /annotations` once with timeout 30s | HTTP 5xx OR HTTP 503; **error envelope present**; **no raw exception text in body** |
|
||||
| 3 | `docker unpause postgres` | DB responsive |
|
||||
| 4 | `POST /annotations` again | HTTP 200; SUT recovered |
|
||||
|
||||
**Pass criteria**: under-DB-outage response uses the error envelope; SUT recovers within 30s of DB recovery.
|
||||
**Duration**: ~2 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-04: SSE subscriber disconnect mid-stream
|
||||
|
||||
**Summary**: A subscriber that disconnects mid-stream does not crash the SUT or block other subscribers.
|
||||
**Traces to**: AC-F-10, OP-01 (per-instance SSE state)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Open 3 SSE connections to `/annotations/events?missionId=<m>` | all 3 alive |
|
||||
| 2 | Abruptly close subscriber #2 (TCP RST) | SUT cleans up its channel slot |
|
||||
| 3 | `POST /annotations` for mission `<m>` | HTTP 200 |
|
||||
| 4 | Subscribers #1 and #3 each receive the event | both receive within 1000ms |
|
||||
| 5 | `GET /health` | HTTP 200 |
|
||||
|
||||
**Pass criteria**: surviving subscribers still receive events; no SUT memory growth visible (channel slots reclaimed); `/health` stays green.
|
||||
**Duration**: ~1 minute.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-05: Repeated FailsafeProducer empty-catch path
|
||||
|
||||
**Summary**: When the image referenced by an outbox row no longer exists on disk, the drainer logs and proceeds (post RB-05). Tests today's behavior (empty catch) AND, after RB-05 lands, asserts the logged failure path.
|
||||
**Traces to**: RB-05
|
||||
|
||||
**Fault injection**: insert an outbox row whose `annotation_id` references a missing image (manually delete the file after `POST /annotations` returned 200, before the drain interval fires).
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | `POST /annotations` (FT-P-01) | HTTP 200; outbox row + image file present |
|
||||
| 2 | Delete `<images_dir>/<id>.jpg` | image gone |
|
||||
| 3 | Wait `drain_interval × 2` | drainer runs |
|
||||
| 4 | Out-of-band: `SELECT COUNT(*) FROM annotations_queue_records WHERE annotation_id = <id>` | today's behavior: row may be deleted or stuck (empty catch swallows IOException) — **document actual behavior here** |
|
||||
| 5 `[after RB-05]` | Inspect SUT logs for an `ERROR` entry mentioning the missing image | one log entry present; metric counter `failsafe_drain_errors` incremented |
|
||||
|
||||
**Pass criteria today**: SUT does not crash; `/health` stays 200.
|
||||
**Pass criteria after RB-05**: as above + the logged failure path is exercised.
|
||||
**Duration**: ~1 minute.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-06: Stream consumer reconnect
|
||||
|
||||
**Summary**: A stream consumer that drops and reconnects with offset `last_committed` reads only post-disconnect messages.
|
||||
**Traces to**: implicit (consumer-side concern, but documents the contract Annotations producer expects)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Start consumer at offset `next`; record current end-of-stream offset `O0` | consumer up |
|
||||
| 2 | `POST /annotations` 5 times | 5 outbox rows; 5 stream messages produced shortly after |
|
||||
| 3 | Consumer reads all 5; commits offset after each | consumer offset = `O0 + 5` |
|
||||
| 4 | Disconnect consumer | done |
|
||||
| 5 | `POST /annotations` 3 more times | 3 more stream messages |
|
||||
| 6 | Reconnect consumer at `last_committed = O0 + 5` | consumer reads only messages 6..8 |
|
||||
|
||||
**Pass criteria**: re-attached consumer sees no duplicates and no gaps.
|
||||
**Duration**: ~1 minute.
|
||||
@@ -0,0 +1,123 @@
|
||||
# Resource Limit Tests
|
||||
|
||||
### NFT-RES-LIM-01: Sustained-load process memory
|
||||
|
||||
**Summary**: Process memory stays bounded under sustained `POST /annotations` traffic.
|
||||
**Traces to**: AC-N-03 (outbox depth bounded → memory bounded), HW-03 (memory pressure on `FailsafeProducer`'s image re-read)
|
||||
**Preconditions**: SUT freshly started; clean state; a stream consumer connected so the outbox actually drains.
|
||||
|
||||
**Monitoring**:
|
||||
- `docker stats annotations` polled every 10s for `MemUsage` (RSS) and `MemPerc`.
|
||||
- Sample at the 0s / 60s / 600s marks.
|
||||
|
||||
**Duration**: 10 minutes at 5 RPS.
|
||||
**Pass criteria**: RSS at the 600s mark ≤ 1.5× RSS at the 60s mark; no OOMKilled events; container stays healthy.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-02: Single-file upload boundary
|
||||
|
||||
**Summary**: Determine the maximum single-file upload size accepted by `POST /media`.
|
||||
**Traces to**: documented gap (no explicit limit in code; ASP.NET form-options apply)
|
||||
|
||||
**Monitoring**: HTTP status code per uploaded size.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Size | Expected Result |
|
||||
|------|-----------------|
|
||||
| 1 MB | HTTP 200 |
|
||||
| 10 MB | HTTP 200 |
|
||||
| 50 MB | HTTP 200 |
|
||||
| 100 MB | HTTP 200 (probable, depends on ASP.NET defaults) |
|
||||
| 256 MB | HTTP 200 OR 400 (test the boundary) |
|
||||
| 512 MB | likely HTTP 400 / form-options reject |
|
||||
|
||||
**Duration**: ~5 minutes (one upload per size).
|
||||
**Pass criteria**: a clear cutoff size is documented; below it the SUT accepts; at or above it the SUT returns the error envelope (NOT a 500 with no body, NOT a hang).
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-03: Outbox depth under broker outage
|
||||
|
||||
**Summary**: With RabbitMQ stopped for an extended period, the outbox `annotations_queue_records` table grows linearly with traffic AND does not exceed disk capacity / DB connection pool limits within the test window.
|
||||
**Traces to**: NFT-RES-01 (extended), AC-N-03
|
||||
|
||||
**Monitoring**:
|
||||
- `SELECT COUNT(*) FROM annotations_queue_records` every 30s.
|
||||
- Disk usage of the Postgres data volume every minute.
|
||||
- `docker stats postgres` for memory.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | `docker exec rabbitmq rabbitmqctl stop_app` | broker down |
|
||||
| 2 | Run 10 RPS of `POST /annotations` for 5 minutes | 3000 outbox rows written |
|
||||
| 3 | Sample queue depth and disk usage | depth grows linearly; disk grows linearly with image bytes (since `images_dir` is also written) |
|
||||
| 4 | `docker exec rabbitmq rabbitmqctl start_app` | broker recovers |
|
||||
| 5 | Wait for queue to drain | depth goes to 0 within 5 minutes of recovery |
|
||||
|
||||
**Duration**: 15 minutes total.
|
||||
**Pass criteria**:
|
||||
- During outage: SUT does not return 5xx; queue depth is exactly equal to total successful POSTs since the outage started.
|
||||
- During recovery: queue drains to 0 within 5 minutes.
|
||||
- No DB connection pool exhaustion (no `connection refused` from Postgres).
|
||||
- No SUT crashes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-04: Disk usage by `images_dir` over many distinct uploads
|
||||
|
||||
**Summary**: Each distinct `image_bytes` POST consumes O(image-size) disk; identical re-uploads consume zero additional disk (idempotent).
|
||||
**Traces to**: AC-F-01, AC-F-02
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Capture `du -sb $images_dir` baseline | non-empty path |
|
||||
| 2 | `POST /annotations` 100× with `image_small.jpg` (same bytes) | 1 file added, ~1.5 MB delta from step 1 |
|
||||
| 3 | `POST /annotations` 100× with random distinct image bytes (synthetic) | 100 new files; delta ≈ 100 × avg-size |
|
||||
|
||||
**Pass criteria**: identical uploads do not duplicate disk; distinct uploads scale linearly.
|
||||
**Duration**: ~5 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-05: Concurrent SSE subscribers — process-memory boundary
|
||||
|
||||
**Summary**: 100 simultaneous SSE subscribers do not exhaust the SUT's memory or thread pool.
|
||||
**Traces to**: AC-N-05 (idle-channel memory bounded), OP-01 (per-instance SSE state)
|
||||
|
||||
**Preconditions**: SUT freshly started.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Open 100 SSE connections to `/annotations/events?missionId=<m>` | all 100 alive |
|
||||
| 2 | Sample `docker stats annotations` immediately after connection | RSS recorded |
|
||||
| 3 | Idle for 10 minutes; sample every 60s | RSS stays within ± 10% of step 2 |
|
||||
| 4 | `POST /annotations` once for mission `<m>` | all 100 subscribers receive the event within 1500ms |
|
||||
|
||||
**Pass criteria**: RSS bounded; all subscribers receive the event; no `connection refused` or thread-pool starvation.
|
||||
**Duration**: ~12 minutes.
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-06: Migration on cold-start cost
|
||||
|
||||
**Summary**: Boot-time `DatabaseMigrator.MigrateAsync()` adds bounded latency to cold start (`/health` returns 200 within `<budget>` after container start).
|
||||
**Traces to**: AC-N-01
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | `docker compose down annotations && docker compose up -d annotations` | container starting |
|
||||
| 2 | Poll `/health` every 200ms; record time-to-first-200 | record time |
|
||||
| 3 | Repeat with a fresh DB (cold migrator) and a populated DB (warm migrator) | both runs measured |
|
||||
|
||||
**Pass criteria** (until contracted): time-to-first-200 ≤ 30s on cold migrator; ≤ 10s on warm migrator. **Step 15 will tune.**
|
||||
**Duration**: ~2 minutes.
|
||||
@@ -0,0 +1,179 @@
|
||||
# Security Tests
|
||||
|
||||
> Blackbox-level only. Code-level vulnerabilities (e.g., SQL injection at the source level) are out of scope here — they belong to Step 14 (Security Audit). The SEC-XX gap list in `security_approach.md` is the broader inventory; the tests below are the ones that can be exercised through the public surface.
|
||||
>
|
||||
> **Auth model assumed by these tests**: annotations is a verifier-only service. Tokens are minted by the e2e harness's mock issuer (see `test-data.md` → "Bearer token harness") using an ES256 key pair whose public half is published at the JWKS URL the service fetches at boot. There is no `/auth/login`, `/auth/refresh`, or `/auth/register` endpoint on this service.
|
||||
|
||||
### NFT-SEC-01: JWT signature mismatch
|
||||
|
||||
**Summary**: A token signed with a key not published in the SUT's JWKS is rejected.
|
||||
**Traces to**: AC-F-50, AC-F-52
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Mint an ES256 token with valid `iss` / `aud` / `exp` and an `ANN` claim, but signed with a private key whose public half is **not** in the JWKS | token is well-formed |
|
||||
| 2 | `POST /annotations` with that bearer token | HTTP 401; error envelope; `error.code` matches `/auth|unauthor/i` |
|
||||
|
||||
**Pass criteria**: 401, no 500, no leaking which key the SUT expected.
|
||||
**Duration**: 3s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-02: JWT expired
|
||||
|
||||
**Summary**: An expired ES256 JWT is rejected.
|
||||
**Traces to**: AC-F-50
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Mint a JWT with `exp` 1 hour in the past, signed with a key in the JWKS, otherwise valid (`iss`, `aud`, `ANN` claim) | token is well-formed |
|
||||
| 2 | `POST /annotations` with that bearer token | HTTP 401; error envelope |
|
||||
|
||||
**Pass criteria**: 401; SUT does not honor the expired token.
|
||||
**Duration**: 3s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-03: Cross-policy attempt — DATASET token cannot create annotations
|
||||
|
||||
**Summary**: Policy `DATASET` cannot reach `/annotations` POST.
|
||||
**Traces to**: AC-F-52
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Mint an ES256 token with the `DATASET` claim and `ANN` claim absent | token is well-formed |
|
||||
| 2 | `POST /annotations` with that bearer | HTTP 403; error envelope |
|
||||
|
||||
**Pass criteria**: 403, request rejected before any DB / FS write.
|
||||
**Duration**: 3s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-04: Cross-policy attempt — ANN token cannot mutate settings
|
||||
|
||||
**Summary**: Policy `ANN` cannot reach an `[Authorize(Policy = "ADM")]` route.
|
||||
**Traces to**: AC-F-52
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | `PUT /settings/system` with an ES256 token carrying only the `ANN` claim | HTTP 403; error envelope |
|
||||
|
||||
**Pass criteria**: 403.
|
||||
**Duration**: 3s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-05: Anonymous access to non-public endpoints
|
||||
|
||||
**Summary**: Every endpoint other than `/health` requires authentication.
|
||||
**Traces to**: AC-F-50, AC-F-52, security_approach.md surface inventory
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | `GET /annotations` with no `Authorization` | HTTP 401 |
|
||||
| 2 | `GET /dataset` with no `Authorization` | HTTP 401 |
|
||||
| 3 | `GET /classes` with no `Authorization` | HTTP 401 |
|
||||
| 4 | `GET /settings/system` with no `Authorization` | HTTP 401 |
|
||||
| 5 | `GET /health` with no `Authorization` | HTTP 200 |
|
||||
|
||||
**Pass criteria**: every authenticated endpoint returns 401; only `/health` is anonymous.
|
||||
**Duration**: 5s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-06: Error envelope leaks no stack trace in production-mode env
|
||||
|
||||
**Summary**: Triggering a 500 path returns an error envelope with no `stackTrace` / `innerException` fields.
|
||||
**Traces to**: AC-N-04
|
||||
|
||||
**Preconditions**: SUT started with `ASPNETCORE_ENVIRONMENT=Production`.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Trigger a path that produces a 500 (e.g., NFT-RES-03 step 2 in DB-paused state) OR a malformed multipart body | HTTP 5xx; body is the error envelope |
|
||||
| 2 | Inspect body | no key matches `/stack/i`, `/inner/i`, `/trace/i` (case-insensitive) |
|
||||
|
||||
**Pass criteria**: error envelope present; no stack-trace leakage.
|
||||
**Duration**: 30s (depends on fault induction).
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-07: Path traversal in image / thumbnail GET routes
|
||||
|
||||
**Summary**: Path traversal sequences in the `id` segment do not escape `images_dir`.
|
||||
**Traces to**: implicit; SEC-05 broader scope
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | `GET /annotations/%2E%2E%2Fetc%2Fpasswd/image` (encoded `../etc/passwd`) | HTTP 400 OR HTTP 404 (NOT 200, NOT containing `/etc/passwd` content) |
|
||||
| 2 | `GET /annotations/..%2F..%2Fetc%2Fpasswd/thumbnail` | same |
|
||||
|
||||
**Pass criteria**: SUT rejects or returns 404; no host file content in the response body.
|
||||
**Duration**: 5s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-08: Token claim modification (signature breaks)
|
||||
|
||||
**Summary**: An attacker who edits a JWT payload to elevate to ADM but cannot resign sees 401, not 200.
|
||||
**Traces to**: AC-F-52
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Mint an ES256 token with the `ANN` claim | original token |
|
||||
| 2 | Decode payload; replace policy claim with `ADM`; re-encode payload but **keep the original signature** | tampered token |
|
||||
| 3 | `PUT /settings/system` with the tampered token | HTTP 401; error envelope |
|
||||
|
||||
**Pass criteria**: 401 — signature validation catches the tamper.
|
||||
**Duration**: 5s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-09: CORS preflight respects configured allow-list
|
||||
|
||||
**Summary**: With the SUT booted under `ASPNETCORE_ENVIRONMENT=Production` and `CorsConfig:AllowedOrigins=["https://app.azaion.local"]`, a preflight from an arbitrary origin is not given a wildcard ACAO header. `CorsConfigurationValidator` already prevents the wide-open default in Production.
|
||||
**Traces to**: AC-N-CORS (see `restrictions.md` ENV-06)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | `OPTIONS /annotations` with `Origin: https://attacker.example`, `Access-Control-Request-Method: POST` | HTTP 204 with **no** `Access-Control-Allow-Origin: *`; either no ACAO at all, or an ACAO matching the configured allow-list (which the attacker origin is not in) |
|
||||
| 2 | `OPTIONS /annotations` with `Origin: https://app.azaion.local` | HTTP 204; `Access-Control-Allow-Origin: https://app.azaion.local` |
|
||||
|
||||
**Pass criteria**: only configured origins receive a permissive ACAO; arbitrary origins do not.
|
||||
**Duration**: 3s.
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-10: Algorithm confusion — `alg=HS256` over the public ES256 key
|
||||
|
||||
**Summary**: Annotations pins `ValidAlgorithms = [EcdsaSha256]` to block the classic JWKS-confusion attack where an attacker forges an HS256 token using the published ES256 public key as the HMAC secret.
|
||||
**Traces to**: AC-F-50
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected Response |
|
||||
|------|----------------|------------------|
|
||||
| 1 | Fetch the SUT's published JWKS; export the ES256 public key bytes | bytes obtained |
|
||||
| 2 | Mint a JWT with `alg=HS256` and the public key bytes as the HMAC key, with otherwise-valid `iss` / `aud` / `exp` / `ANN` claim | forged token |
|
||||
| 3 | `GET /annotations` with that bearer token | HTTP 401; error envelope |
|
||||
|
||||
**Pass criteria**: 401 — algorithm pinning rejects the forged token.
|
||||
**Duration**: 5s.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Test Data Management
|
||||
|
||||
## Seed Data Sets
|
||||
|
||||
| Data Set | Description | Used by Tests | How Loaded | Cleanup |
|
||||
|----------|-------------|---------------|-----------|---------|
|
||||
| `tokens-test` | 3 ES256 access tokens minted on demand by the runner: `ann-token` (claim `ANN`), `dataset-token` (`DATASET`), `adm-token` (`ADM`). All carry `iss=$JWT_ISSUER`, `aud=$JWT_AUDIENCE`, `exp=now+5m`, and a deterministic `sub` GUID per role. | F1-N-003, F1-N-004, F5-004, F6-001..006, F7-004, F8-*, NFT-SEC-01..10, FT-N-10..12 | The harness runs a **mock JWKS issuer** (Python script `tests/harness/mock_issuer.py` or the equivalent .NET fixture) that publishes the public ES256 key at `JWT_JWKS_URL`. The runner imports the matching private key as a fixture and mints tokens per test. | Tokens are short-lived (5m) and never persisted; key pair regenerates on `docker compose down -v` |
|
||||
| `mission-test` | One canonical waypoint id `00000000-0000-0000-0000-000000000aaa` used as `WaypointId` / `MissionId` in every annotation create. | All F1, F2, F3, F4, F5, F8 | Implicit — no FK enforcement; the GUID is just a column value. | N/A |
|
||||
| `classes-baseline` | The 19 detection classes seeded by `DatabaseMigrator` (ids 0–18, names per `data_parameters.md`). | F7-001 (catalog read), F1-* (class_num references) | Auto, by the SUT's boot-time migrator. | N/A — schema-managed |
|
||||
| `clean-state` | Empty `annotations`, `media`, `detection`, `annotations_queue_records` tables at the start of each test class. | every test class that asserts on count / depth | xUnit class fixture: `TRUNCATE annotations, media, detection, annotations_queue_records RESTART IDENTITY CASCADE;` via direct DB connection (out-of-band, runner-only). | Fixture's `Dispose()` truncates again |
|
||||
|
||||
## Data Isolation Strategy
|
||||
|
||||
- **Per-class truncation** — each xUnit test class declares an `IClassFixture<CleanStateFixture>` that truncates the four mutable tables before the first test in the class and again after the last.
|
||||
- **Per-test token** — every test mints its own ES256 token via the mock issuer fixture (see "Bearer token harness" below); tokens never cross test boundaries.
|
||||
- **Per-test mission id** — tests that need fan-out isolation (e.g., F3 SSE subscribers) generate a fresh `WaypointId` GUID per test so concurrent test runs don't leak events into each other.
|
||||
- **Per-test stream consumer** — F4 stream-consumer scenarios use a fresh consumer name per test and start at offset `next` (current end of stream). They consume only messages produced after the test starts.
|
||||
- **Filesystem isolation** — `annotations-images`, `annotations-videos`, `annotations-deleted` volumes are recreated by `docker compose down -v` between full runs. Per-test cleanup removes only files the test wrote (matching `<id>` patterns).
|
||||
|
||||
## Input Data Mapping
|
||||
|
||||
| Input Data File | Source Location | Description | Covers Scenarios |
|
||||
|-----------------|----------------|-------------|-----------------|
|
||||
| `image_small.jpg` | `<fixtures>/image_small.jpg` | 1280×720 frame, ~1.5 MB | F1-001, F1-002, F1-N-003..005, F2-001/002, F3-001/002, F4-001/002, F5-001/002, F8-* |
|
||||
| `image_dense01.jpg` | `<fixtures>/image_dense01.jpg` | small dense frame (~230 KB) | F1-004, F5-002, F8-002 |
|
||||
| `image_dense02.jpg` | `<fixtures>/image_dense02.jpg` | larger dense frame (~2.8 MB) | F5-002 |
|
||||
| `image_different_types.jpg` | `<fixtures>/image_different_types.jpg` | multi-class scene (900×1600) | F8-002 (class filter) |
|
||||
| `image_empty_scene.jpg` | `<fixtures>/image_empty_scene.jpg` | 1920×1080 empty scene | F1-003 (zero detections), NFT-PERF-* warmup |
|
||||
| `image_large.JPG` | `<fixtures>/image_large.JPG` | 6252×4168, ~7 MB | F1-005 (large payload), NFT-PERF-LATENCY |
|
||||
| `video_short01.mp4` | `<fixtures>/video_short01.mp4` | ~150 MB video | F1-006 (video annotation), F1-007 |
|
||||
| `video_short02.mp4` | `<fixtures>/video_short02.mp4` | distinct-bytes second video | F1-007 (distinct bytes → distinct ids) |
|
||||
|
||||
`<fixtures>` resolves to `/fixtures` inside the test runner / SUT container, bound to `../detections/_docs/00_problem/input_data/` per `_docs/00_problem/input_data/fixtures.md`.
|
||||
|
||||
## Synthetic request payloads
|
||||
|
||||
JSON request bodies for `POST /annotations`, `PUT /annotations/{id}`, `POST /dataset/status/bulk`, and the auth flows live under `_docs/00_problem/input_data/requests/`. Each test references a request file by id (`F1_001_request.json`). Class numbers in detections come from the seeded `detection_classes` (ids 0–18); coordinates are normalized 0..1 floats.
|
||||
|
||||
## Expected Results Mapping
|
||||
|
||||
(Full table is `_docs/00_problem/input_data/expected_results/results_report.md` — 44 rows. Selected entries here for cross-reference.)
|
||||
|
||||
| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Source |
|
||||
|-----------------|------------|-----------------|-------------------|-----------|--------|
|
||||
| FT-P-01 (=F1-001) | `image_small.jpg` + `F1_001_request.json` | HTTP 200 + `AnnotationDto`; `id =~ /^[0-9a-f]{32}$/`; `detections.length == 1` | exact, schema_match, regex | N/A | `expected_results/F1_001_response.json` |
|
||||
| FT-P-02 (=F1-002) | Same input, second POST | Same `id` as FT-P-01; no duplicate row | exact | N/A | inline |
|
||||
| FT-P-04 (=F1-004) | `image_dense01.jpg` + `F1_004_request.json` | HTTP 200; `detections.length == 5`; YOLO label file with 5 lines | exact, file_content | N/A | `expected_results/F1_004_response.json` |
|
||||
| FT-P-10 (=F3-001) | F1-001 fires, SSE subscriber connected | event with `operation == "Created"`, `latency ≤ 1000ms` | exact, threshold_max | ± 200ms | inline |
|
||||
| FT-N-04 (=F1-N-004) | F1-001 with no `Authorization` header | HTTP 401 + error envelope | exact, schema_match | N/A | inline |
|
||||
| NFT-PERF-LATENCY-01 | `image_small.jpg` × 50 sequential calls | p95 latency ≤ 1500ms | threshold_max | N/A | inline |
|
||||
| NFT-RES-01 | RabbitMQ stopped, F1-001 fires | HTTP 200 returned to caller; outbox row stays; SUT stays alive | exact | N/A | inline |
|
||||
| NFT-SEC-01 | F1-001 with JWT signed by **wrong** key | HTTP 401 | exact | N/A | inline |
|
||||
| NFT-RES-LIM-01 | F4 outbox under sustained load | queue depth ≤ 10× steady-state for ≥ 30 min | threshold_max | N/A | inline |
|
||||
|
||||
## External Dependency Mocks
|
||||
|
||||
| External Service | Mock/Stub | How Provided | Behavior |
|
||||
|-----------------|-----------|-------------|----------|
|
||||
| RabbitMQ Stream broker | Real `rabbitmq:3.13-management` with the streams plugin | Docker service in `e2e-net` | Real broker; resilience tests (NFT-RES-01..03) restart it mid-test using `docker exec rabbitmq rabbitmqctl stop_app && start_app` |
|
||||
| Postgres | Real `postgres:13` | Docker service | Real DB; resilience tests (NFT-RES-04) crash and restart it |
|
||||
| Detections service | Not run | N/A | The annotations service does not call the detections service; tests bypass it by hand-authoring synthetic `detections[]` payloads in `requests/`. |
|
||||
| Suite-level reverse proxy / TLS terminator | Not run | N/A | Tests speak directly to `http://annotations:8080`. SEC-tests for HTTPS / HSTS therefore explicitly skip with reason "out-of-process for SUT". |
|
||||
|
||||
## Data Validation Rules
|
||||
|
||||
| Data Type | Validation | Invalid Examples | Expected System Behavior |
|
||||
|-----------|-----------|-----------------|------------------------|
|
||||
| `image_bytes` (POST /annotations) | non-null, non-empty byte array | empty array `[]`, missing field | HTTP 400/422; error envelope |
|
||||
| `mediaType` (POST /annotations) | enum `Image=10` or `Video=20` | `5`, `100`, missing | HTTP 400/422; error envelope |
|
||||
| `detections[].class_num` | int, no range validator today | `-1`, `999` | HTTP 200 today (lenient); flagged as gap (SEC-05) |
|
||||
| `detections[].centerX/Y/width/height` | float, no range validator today | `1.5`, `-0.1`, `NaN` | HTTP 200 today (lenient); flagged as gap (SEC-05) |
|
||||
| `Authorization` header | bearer ES256 JWT issued by the mock issuer; validated for issuer / audience / signature / expiry, with `alg` pinned to ES256 | missing, wrong issuer, wrong audience, wrong signature, expired, `alg=HS256` forgery | HTTP 401; error envelope |
|
||||
| Caller policy | `ANN`, `DATASET`, or `ADM` per endpoint | mismatched policy | HTTP 403; error envelope |
|
||||
| `WaypointId` (POST /annotations, /media) | GUID format | not a GUID | HTTP 400/422 from model binder |
|
||||
| File-upload size (POST /media) | no explicit limit visible at controller; underlying ASP.NET form-options apply | >256 MB single file | likely HTTP 400 from form-options; verify in NFT-RES-LIM-02 |
|
||||
|
||||
## Runtime-generated test data
|
||||
|
||||
Two scenario groups consume **synthetic test data generated by the runner at execution time** rather than static files on disk. This is intentional and explicitly allowed by `templates/expected-results.md` ("Test data may be generated programmatically — note this in test-data.md"):
|
||||
|
||||
| Scenario | Generated data | How |
|
||||
|----------|----------------|-----|
|
||||
| NFT-RES-LIM-02 (single-file upload boundary) | Synthetic JPEG-prefixed binary blobs at sizes 1, 10, 50, 100, 256, 512 MB | Runner xUnit fixture writes a temp file: 4-byte JPEG magic header + pseudo-random bytes filling to the target size; uploaded once, deleted after. Files NOT committed to the repo. |
|
||||
| NFT-PERF-LIST-01, NFT-PERF-DATASET-01 | 10,000 `annotations` rows + 50,000 `detection` rows in the test DB | `dataseed` job runs a parameterised SQL script that bulk-inserts rows with `media_id` referencing 100 distinct seeded media rows; uses `CROSS JOIN generate_series` for speed. Cleared by `clean-state` truncation between test classes. |
|
||||
|
||||
The generated data still satisfies Phase 3 quantifiability: every generated input has a deterministic shape (size, count) AND a quantifiable expected result (HTTP code, latency threshold, returned row count).
|
||||
|
||||
## Bearer token harness
|
||||
|
||||
Annotations is verifier-only — there is no `/auth/login` to call from a test. The harness reproduces the production model in miniature:
|
||||
|
||||
1. **Key pair** — a fresh ES256 key pair is generated when the test stack starts (`docker compose up`). The private key is mounted into the runner container; the public key is mounted into a tiny **mock issuer** sidecar that serves `/.well-known/jwks.json` over HTTP **inside the docker-compose network**.
|
||||
2. **JWKS URL configuration** — the SUT is started with `JWT_ISSUER=https://e2e-issuer.test`, `JWT_AUDIENCE=annotations-e2e`, and `JWT_JWKS_URL=http://e2e-issuer:8080/.well-known/jwks.json`. The HTTPS-only constraint of `HttpDocumentRetriever { RequireHttps = true }` is relaxed for tests by either (a) overriding `RequireHttps=false` via test-only configuration, or (b) running a TLS-terminating proxy in front of the issuer. Option (a) is preferred for simplicity; the relaxation is gated on `ASPNETCORE_ENVIRONMENT=E2ETest` and never applied in production builds. (This is the testability item flagged in `architecture.md` Open Risks §6.)
|
||||
3. **Token minting** — the runner exposes a per-test helper `mintToken(claim: "ANN" | "DATASET" | "ADM", overrides?)` that builds an ES256 JWT from the in-process private key with the configured `iss`/`aud`, `exp = now + 5m`, a per-role deterministic `sub` GUID, and the requested policy claim. `overrides` lets a test produce expired / wrong-iss / wrong-aud / forged-`alg=HS256` variants for the security suite.
|
||||
4. **No persisted users** — there is no `users` table in this service. Each test mints exactly the token it needs.
|
||||
|
||||
## Notes for the runner
|
||||
|
||||
- **Boot order**: `postgres` → `rabbitmq` → `e2e-issuer` (mock JWKS) → `annotations` (waits for postgres, rabbitmq, and a successful JWKS fetch) → `dataseed` → `e2e-runner`.
|
||||
- **Fresh-state vs. carry-over**: the suite truncates per class, so test ordering inside a class matters; ordering across classes does not.
|
||||
- **Stream consumption**: every test that reads from `azaion-annotations` records the offset before the test acts, then consumes from `start_offset = recorded_offset + 1` to ignore historical messages.
|
||||
- **Conditional probes**: tests that depend on SUT behavior decisions (e.g., specific 4xx code on a corner case) include a fixture step that probes the SUT once at class-init, records the actual behavior, then asserts that branch consistently within the test class. Mismatch on a subsequent run flags as a behavior-drift test failure.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Traceability Matrix
|
||||
|
||||
## Acceptance Criteria Coverage
|
||||
|
||||
### Functional ACs
|
||||
|
||||
| AC ID | Acceptance Criterion (short) | Test IDs | Coverage |
|
||||
|-------|-----------------------------|----------|----------|
|
||||
| AC-F-01 | Same image bytes → same id | FT-P-01, FT-P-02 | Covered |
|
||||
| AC-F-02 | Re-POST is no-op | FT-P-02 | Covered |
|
||||
| AC-F-03 | YOLO label file format | FT-P-03, FT-P-04 | Covered |
|
||||
| AC-F-04 | POST /annotations returns persisted DTO | FT-P-01, FT-N-01, FT-N-02, FT-N-06, FT-N-07 | Covered |
|
||||
| AC-F-05 | `[after RB-01]` Every mutation emits SSE + outbox | FT-P-21, FT-P-22, NFT-RES-01 | Deferred — gated on RB-01 |
|
||||
| AC-F-06 | `[after RB-01]` DELETE is soft + relocates files | FT-P-22 | Deferred — gated on RB-01 |
|
||||
| AC-F-07 | `[after RB-01+RB-08]` soft-deleted hidden from reads | (test added in cycle-update once RB-01+RB-08 land) | Deferred |
|
||||
| AC-F-08 | `[after RB-02]` no `silent_detection` artifacts | (covered by RB-02 implementation tests) | Deferred — gated on RB-02 |
|
||||
| AC-F-10 | SSE delivery < 1s | FT-P-07, NFT-PERF-SSE-FANOUT-01 | Covered |
|
||||
| AC-F-11 | No SSE backfill | FT-P-07 (step 2), inline assertion | Partial — add explicit test in cycle-update |
|
||||
| AC-F-12 | Outbox drain → stream | FT-P-08, FT-P-09, NFT-RES-01 | Covered |
|
||||
| AC-F-13 | `[after RB-09]` `(annotation_id, operation, date_time)` on the wire | (added in cycle-update once RB-09 lands) | Deferred — gated on RB-09 |
|
||||
| AC-F-20 | POST /media single | FT-P-10, FT-N-14, FT-N-15 | Covered |
|
||||
| AC-F-21 | POST /media/batch | FT-P-11 | Covered |
|
||||
| AC-F-30 | GET /dataset filter | FT-P-16, FT-P-17 | Covered |
|
||||
| AC-F-31 | POST /dataset/status/bulk | FT-P-18, FT-N-16 | Covered |
|
||||
| AC-F-40 | PUT /settings/directories triggers Reset() | FT-P-15, FT-N-13 | Covered |
|
||||
| AC-F-41 | GET /classes returns 19 rows | FT-P-14 | Covered |
|
||||
| AC-F-42 | `[after RB-06]` admin CRUD on /classes | (added once RB-06 lands) | Deferred — gated on RB-06 |
|
||||
| AC-F-50 | Bearer token verification (iss/aud/exp/sig/alg) | FT-P-12, FT-P-13, FT-N-10, FT-N-11, NFT-SEC-01, NFT-SEC-02, NFT-SEC-10 | Covered |
|
||||
| AC-F-51 | Annotations does not host token-issuance/refresh | (asserted by NFT-SEC-05 — only `/health` is anonymous) | Covered (negative) |
|
||||
| AC-F-52 | Policy boundaries | FT-N-03, FT-N-04, FT-N-08, FT-N-12, FT-N-13, FT-N-15, NFT-SEC-03, NFT-SEC-04, NFT-SEC-05, NFT-SEC-08 | Covered |
|
||||
| AC-F-53 | Error envelope shape | covered as global invariant; FT-N-* assert envelope | Covered |
|
||||
| AC-F-54 | GET /health returns 200 | FT-P-19, NFT-PERF-* warmup | Covered |
|
||||
|
||||
### Non-Functional ACs
|
||||
|
||||
| AC ID | Acceptance Criterion (short) | Test IDs | Coverage |
|
||||
|-------|-----------------------------|----------|----------|
|
||||
| AC-N-01 | Container boot to /health 200 within healthcheck budget | FT-P-19, NFT-RES-LIM-06 | Covered (threshold inferred — Step 15 contracts it) |
|
||||
| AC-N-02 | Migrator is idempotent | FT-P-20 | Covered |
|
||||
| AC-N-03 | Outbox queue depth bounded | NFT-PERF-OUTBOX-DRAIN-01, NFT-RES-LIM-01, NFT-RES-LIM-03 | Covered |
|
||||
| AC-N-04 | Zero unhandled exceptions to clients | NFT-RES-03, NFT-SEC-06 | Covered |
|
||||
| AC-N-05 | SSE longevity ≥ 30 min | NFT-RES-LIM-05 | Covered (10-min run is a smoke proxy; 30-min is the nightly variant) |
|
||||
|
||||
## Restrictions Coverage
|
||||
|
||||
| Restriction ID | Restriction (short) | Test IDs | Coverage |
|
||||
|----------------|---------------------|----------|----------|
|
||||
| HW-01 | ARM64 only | covered by build pipeline (the test image IS ARM64) | Covered (environment-level) |
|
||||
| HW-02 | Writable `images_dir` / `videos_dir` / `deleted_dir` | FT-P-01, FT-P-15, FT-P-22 | Covered |
|
||||
| HW-03 | Memory pressure on `FailsafeProducer` image re-read | NFT-RES-LIM-01, NFT-RES-LIM-04 | Covered |
|
||||
| SW-01 | .NET 10 | environment-level (Dockerfile) | Covered (deployment) |
|
||||
| SW-02 | Postgres 13+ semantics | FT-P-20 (idempotent migrator exercises `CREATE TYPE` etc.) | Covered |
|
||||
| SW-03 | RabbitMQ streams plugin | FT-P-09, NFT-RES-01, NFT-RES-06 | Covered |
|
||||
| SW-04 | Linq2DB + MessagePack + gzip wire | FT-P-09 (decodes the wire format) | Covered |
|
||||
| SW-05 | JWT verifier-only (ES256 over admin's JWKS, alg pinned) | NFT-SEC-01, NFT-SEC-02, NFT-SEC-08, NFT-SEC-10, FT-N-10, FT-N-11 | Covered |
|
||||
| ENV-01 | Env vars required | environment.md docker-compose | Covered (environment-level) |
|
||||
| ENV-02 | Service on port 8080 HTTP, no in-image TLS | environment.md | Covered (environment-level) |
|
||||
| ENV-03 | `AZAION_REVISION` boot stamp | not exposed via API today; covered by inspecting `docker logs` (test runner asserts log line `AZAION_REVISION=test-...` appears within 5s of boot) | Partial — add log-assertion test in cycle-update |
|
||||
| ENV-04 | Branch-driven `${BRANCH}-arm` tags | CI-pipeline concern; not a runtime test | Not covered (CI-level) |
|
||||
| ENV-05 | Swagger UI mounted always | NFT-SEC (verifier in Step 14 catches this); not a hard test today | Not covered — Step 14 |
|
||||
| ENV-06 | Config-driven CORS gated by `CorsConfigurationValidator` | NFT-SEC-09 | Covered (asserts allow-list-only ACAO in `Production`) |
|
||||
| ENV-07 | DDL applied at boot | FT-P-20 | Covered |
|
||||
| OP-01 | Per-instance SSE state | NFT-RES-LIM-05, NFT-RES-04 | Covered |
|
||||
| OP-02 | No outbox row leasing | NFT-RES-01 (single-instance baseline); multi-instance double-publish is **not tested today** because the test stack runs a single SUT — flagged | Not covered (multi-instance) |
|
||||
| OP-03 | No automated test suite | this matrix IS the contract; the implementation lands in Step 6 | N/A (meta) |
|
||||
| OP-04 | No lint / formatter step in CI | CI concern | Not covered (CI-level) |
|
||||
| OP-05 | `HEALTHCHECK` calls `/health` | FT-P-19, environment.md (Dockerfile has `HEALTHCHECK`) | Covered |
|
||||
| OP-06 | `annotations_queue_records` is a private outbox | enforced by code ownership; test asserts no public endpoint allows writing to it (negative coverage via NFT-SEC-05) | Covered (negative) |
|
||||
| OP-07 | DB connection string in `jdbc:postgresql://…` form | Boot succeeds with this format → FT-P-19 implicitly checks it | Covered (implicit) |
|
||||
|
||||
## Coverage Summary
|
||||
|
||||
| Category | Total Items | Covered | Deferred (RB) | Not Covered | Coverage % (excl. deferred) |
|
||||
|----------|-----------|---------|--------------|-------------|----------------------------|
|
||||
| Functional ACs | 24 | 18 | 6 | 0 | 18 / 18 = 100% (active scope) |
|
||||
| Non-Functional ACs | 5 | 5 | 0 | 0 | 100% |
|
||||
| HW restrictions | 3 | 3 | 0 | 0 | 100% |
|
||||
| SW restrictions | 5 | 5 | 0 | 0 | 100% |
|
||||
| ENV restrictions | 7 | 4 | 0 | 3 (ENV-04, ENV-05; OP-04 noted) | 57% — gaps are CI-level / Step-14 |
|
||||
| OP restrictions | 7 | 5 | 0 | 2 (OP-02 multi-instance, OP-04 CI lint) | 71% |
|
||||
| **Total (active scope)** | **51** | **40** | **6** | **5** | 88% covered, 12% NOT_COVERED with reasons |
|
||||
|
||||
## Uncovered Items Analysis
|
||||
|
||||
| Item | Reason Not Covered | Risk | Mitigation |
|
||||
|------|-------------------|------|-----------|
|
||||
| AC-F-05, AC-F-06, AC-F-07, AC-F-08, AC-F-13, AC-F-42 | Gated on Refactor Backlog items (RB-01, RB-02, RB-06, RB-08, RB-09) | Until those refactors land, the lifecycle observability + soft-delete + dedupe contract are not in code | The corresponding tests are authored in advance (FT-P-21, FT-P-22, NFT-RES-01) and remain `skipped` until the RB items move; the cycle-update mode of the test-spec skill (per `.cursor/skills/test-spec/modes/cycle-update.md`) flips them to `enabled` when Phase B implements the RB items |
|
||||
| ENV-06 (post-refactor) | CORS test now exercises the validator-enforced allow-list rather than the legacy wide-open default | None — the test asserts current behavior | NFT-SEC-09 covers it; no follow-up needed |
|
||||
| ENV-04 | Branch-driven CI tag scheme is a CI concern, not a runtime contract | Wrong tag could deploy the wrong revision | Covered by Woodpecker pipeline tests (separate harness) — not a Step 6 deliverable |
|
||||
| ENV-05 | Swagger UI exposure is a Step 14 (Security Audit) item | Information disclosure | Step 14 produces a SEC-XX item; test added once the gating decision is made |
|
||||
| OP-02 | Multi-instance double-publish requires the test harness to spin up ≥ 2 SUT instances; current harness is single-instance | Two-pod deploy could double-publish | Documented as a pre-deployment constraint; full multi-instance testing waits for either RB-09 dedupe contract OR a horizontal-scale design decision |
|
||||
| OP-04 | "No lint / formatter in CI" is a meta-restriction (about CI), not a runtime contract | Style drift, dead code accumulating | Step 14 / Step 17 retrospective will set this up; not a runtime test |
|
||||
|
||||
## Notes
|
||||
|
||||
- The `[after RB-XX]` rows in `results_report.md` correspond directly to the **Deferred** column above. The implementation skill (Step 6) is instructed to author these tests with `[Skip(Reason = "awaiting RB-01")]` etc., so they show in the test discovery surface and flip to active automatically when the gating refactor lands.
|
||||
- The `Not covered` rows under ENV / OP are intentional — they are CI-pipeline or environment-level concerns that do NOT belong in the Step 6 blackbox suite. They are listed here so reviewers see the full restriction inventory.
|
||||
- Per the test-spec Phase 3 hard gate threshold (≥ 75% coverage), the active-scope coverage of **88%** clears the bar with a wide margin.
|
||||
Reference in New Issue
Block a user