annotations/_docs/01_solution/solution.md

# Azaion.Annotations — Solution (retrospective)

> Retrospective view, derived from `_docs/02_document/`. Mirrors the artifact the `research` skill produces, but synthesized from verified code rather than user interview. Read alongside `_docs/02_document/architecture.md` (which carries the confirmed Architecture Vision and the ADR list) and the agreed Refactor Backlog (RB-01..RB-09).

## 1. Product solution description

`Azaion.Annotations` is the suite-internal HTTP + streaming service that owns the **annotation lifecycle**: ingest a video frame (or pre-existing media), record the YOLO detections produced by the upstream detection pipeline, expose CRUD over those annotations, and broadcast every lifecycle change to (a) the Annotator UI in real time via SSE and (b) downstream durable consumers (admin sync worker, AI training pipeline) via a transactional-outbox + RabbitMQ Stream pipeline. It also serves the dataset exploration surface, the media upload pipeline, and the system-metadata catalog (settings + detection classes).

Single .NET 10 binary, single Postgres state-of-record, content-addressed filesystem cache, ARM64 container deployed by branch via Woodpecker CI.

```mermaid
flowchart LR
    subgraph clients [Clients]
        UI[Annotator UI]
        DSE[Dataset Explorer UI]
        DET[Detections service]
        ADM[Admin sync worker]
        AI[AI training]
    end

    subgraph svc [Azaion.Annotations]
        REST[01 Annotations REST]
        RT[02 Realtime and sync]
        MEDIA[03 Media]
        DS[04 Dataset]
        SET[05 Settings and metadata]
        PLAT[06 Platform]
    end

    subgraph store [Stores]
        DB[(PostgreSQL)]
        FS[(Filesystem /data)]
        STREAM[(RabbitMQ Stream)]
    end

    UI -- "REST + SSE" --> REST
    UI -- "REST + SSE" --> RT
    DSE -- "REST DATASET" --> DS
    DET -- "POST + auth refresh" --> REST
    DET -- "POST" --> MEDIA

    REST --> RT
    REST --> PLAT
    RT --> PLAT
    MEDIA --> PLAT
    DS --> PLAT
    SET --> PLAT

    PLAT --> DB
    PLAT --> FS
    RT --> STREAM
    STREAM --> ADM
    STREAM --> AI
```

## 2. Architecture (as implemented)

The implemented architecture per component, with the agreed near-term direction (Refactor Backlog RB-01..RB-09) flagged in **Limitations** and **Requirements** rows.

### 2.1 — `06 Platform` (foundation)

| Field | Value |
|-------|-------|
| Solution | Shared kernel: `AppDataConnection` (Linq2DB), `DatabaseMigrator` (idempotent boot-time DDL), JWT verifier (`JwtExtensions.AddJwtAuth` — ES256 over admin's JWKS, no local minting), `Infrastructure/ConfigurationResolver` (fail-fast required-config resolution), `Infrastructure/CorsConfigurationValidator` (env-aware safety check), `PathResolver`, `ErrorHandlingMiddleware`, composition root (`Program.cs`). |
| Tools | .NET 10, ASP.NET Core, Linq2DB, Npgsql, JwtBearer (verifier-only), `Microsoft.IdentityModel.Protocols.OpenIdConnect` for JWKS resolution, Swashbuckle. |
| Advantages | Single composition root; idempotent migrator removes a separate migration tool from the deployment story (ADR-007); error envelope is uniform across all controllers; identity is fully out-sourced to admin (no HMAC secret to leak, no token-issuance code path to attack). |
| Limitations | Swagger UI mounted in all environments (ADR-005); JWKS retrieval requires HTTPS — test harnesses need a TLS-terminating sidecar or test-only relaxation. (ADR-002 / ADR-006 retired by the auth + CORS refactor.) |
| Requirements | `DATABASE_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `JWT_JWKS_URL` are required at startup (fail-fast); `CorsConfig:AllowedOrigins` (or explicit `AllowAnyOrigin=true`) required in `Production`; `directory_settings` row reachable; Postgres 13+ behavior assumed by `ON CONFLICT` and `IF NOT EXISTS` clauses. |
| Security | JWT bearer with policies `ANN`, `DATASET`, `ADM`. `[AllowAnonymous]` only on `/health`; refresh is admin's responsibility. |
| Cost | Negligible — pure in-process plumbing. |
| Fit | Platform mandate is satisfied (single state-of-record, single secret family, single error envelope). Hardening items belong to the Security Audit step. |

### 2.2 — `02 Annotations realtime & sync`

| Field | Value |
|-------|-------|
| Solution | In-process SSE channel (`AnnotationEventService`, unbounded `Channel<AnnotationEventDto>`) + transactional outbox (`annotations_queue_records`) drained by `FailsafeProducer` (`IHostedService`) into the `azaion-annotations` RabbitMQ stream as MessagePack-gzip frames. |
| Tools | `System.Threading.Channels`, `RabbitMQ.Stream.Client`, `MessagePack`, `System.IO.Compression` (gzip). |
| Advantages | Sub-millisecond fan-out for UI (channel) without standing up a broker for the inner loop; durability for external consumers via the outbox even when RabbitMQ is unreachable; producer + drainer co-located removes a deployment unit. |
| Limitations | Per-instance SSE state — no cross-pod fan-out; no leasing on outbox rows (multi-instance can double-publish); empty `catch { }` at `FailsafeProducer.cs:138` swallows IOException on image read — RB-05. |
| Requirements | RabbitMQ Stream reachable on `RABBITMQ_HOST:RABBITMQ_STREAM_PORT`; `pathResolver` must resolve `images_dir/{id}.jpg` for `Created` operations; world-B mutation paths still TODO (RB-01). |
| Security | RabbitMQ stream auth via `RABBITMQ_PRODUCER_USER` / `_PASS`. SSE inherits `[Authorize(Policy = "ANN")]`. |
| Cost | Channel = O(1) memory per pending message; outbox row + drained delete = ~1 round-trip per lifecycle event. Stream send is gzip-batched. |
| Fit | Strong fit for current scale; horizontal-scale constraints surface at >1 instance and need to be designed before that point. |

### 2.3 — `01 Annotations REST`

| Field | Value |
|-------|-------|
| Solution | `AnnotationsController` (REST + image/thumbnail file routes) → `AnnotationService` → DB + filesystem; lifecycle producer for SSE + outbox. |
| Tools | ASP.NET Core controllers, Linq2DB, `System.IO.Hashing.XxHash64` (today; `XxHash3.Hash128` per RB-04), `System.IO.File`. |
| Advantages | Content-addressed annotation id deduplicates re-uploads; YOLO label written deterministically next to the image; SSE event carries the full detection payload so UIs can render without an extra round-trip. |
| Limitations | Today only `Create` publishes / enqueues — Update / UpdateStatus / Delete are silent (RB-01); not transactional across FS + DB + outbox (RB-03); sampled `XxHash64` collision domain is small (RB-04); thumbnails not generated inline. |
| Requirements | World-B publish + enqueue per RB-01; business-transaction wrapper per RB-03; switch to `XxHash3.Hash128` per RB-04; rename `FlightId` → `MissionId` per RB-07. |
| Security | `[Authorize(Policy = "ANN")]` on the controller; user identity derived from JWT `NameIdentifier`. |
| Cost | Per-create: 1 image write + (optional) 1 image copy + 3 DB INSERTs (media optional, annotation, detections via BulkCopy) + 1 label write + 1 SSE channel write + 1 outbox INSERT. |
| Fit | Solid current shape; the four RB items above bring it in line with the agreed direction. |

### 2.4 — `03 Media`

| Field | Value |
|-------|-------|
| Solution | `MediaController` (single + batch upload, list, file download, delete) → `MediaService` → DB + filesystem under media dir. |
| Tools | ASP.NET Core multipart binding (`IFormFileCollection`), Linq2DB, `System.IO.File`. |
| Advantages | Batch path takes a single waypoint id + multiple files in one request — avoids N round-trips for bulk video frame uploads. |
| Limitations | No format whitelist enforcement is visible at the controller layer (verify during Step 14 Security Audit); no per-tenant quota enforcement. |
| Requirements | `videos_dir` / `images_dir` writable; `MediaType` correctly set per upload. |
| Security | `[Authorize(Policy = "ANN")]`. User from JWT `NameIdentifier`. |
| Cost | One disk write per file + one DB INSERT per row. |
| Fit | Adequate for the current upload volumes. |

### 2.5 — `04 Dataset`

| Field | Value |
|-------|-------|
| Solution | Read-heavy `/dataset` surface (filtered queries, class distribution, single + bulk status updates). |
| Tools | Linq2DB queries against `annotations × media × detection`. |
| Advantages | Bulk status update collapses N row updates into a single `UPDATE … WHERE id IN (…)` — atomic at the SQL level. |
| Limitations | Tight coupling to the annotation domain via shared `AppDataConnection` (RB-08); writes are silent (no SSE / outbox today — fixed by RB-01); reads do not yet filter soft-deleted annotations (will need to once RB-01 lands). |
| Requirements | Decouple writes per RB-08 (route through `AnnotationService`); honor soft-delete filter on read paths once status `Deleted=40` becomes a soft-delete marker. |
| Security | `[Authorize(Policy = "DATASET")]`. |
| Cost | Read paths perform LINQ `EXISTS` subqueries (`db.Detections.Any(...)`) — acceptable for current data volume; revisit during Step 15 Performance Test if dataset grows substantially. |
| Fit | Fits the Dataset Explorer UI; the coupling fix will improve maintainability without changing user-visible behavior. |

### 2.6 — `05 Settings & metadata`

| Field | Value |
|-------|-------|
| Solution | CRUD endpoints for system / directory / camera / user settings under `/settings`; read-only `/classes` for the detection class catalog (becoming admin-managed per RB-06). |
| Tools | Linq2DB, ASP.NET Core controllers. |
| Advantages | Directory cache reset is wired (verified — `SettingsService.cs:71, 85`); single-row settings model keeps the surface simple. |
| Limitations | `system_settings.silent_detection` is a debug remnant scheduled for removal (RB-02); detection classes are migrator-only today, no admin write path (RB-06); `Smoke` and `Plane` share color `#000080` — fixed as part of RB-06. |
| Requirements | Add `[ADM]` CRUD on `/classes` + read-through cache (RB-06); drop `silent_detection` (RB-02). |
| Security | Mixed `[Authorize]` reads / `[ADM]` writes. |
| Cost | One Postgres row family per concern; cache reset is O(1). |
| Fit | Good fit; the two RB items above complete the surface. |

## 3. Testing strategy

**Current state (verified)**: there is **no automated test project** in this workspace (`00_discovery.md`). CI runs only the build + image push (`.woodpecker/build-arm.yml`) — no test step, no lint step. There is no Postman / Bruno collection in-repo either.

**Implication for the autodev existing-code flow**: Step 3 (Test Spec) and Step 6 (Implement Tests) of Phase A produce the missing test surface. The shape required is:

- **Functional / integration tests** — happy-path and error-path coverage for every controller endpoint listed in `system-flows.md` (F1–F8), exercised against a real Postgres + RabbitMQ stack (test-environment parity is a `coderule.mdc` mandate).
- **Lifecycle-observability tests** (post-RB-01) — every mutation path emits an SSE event AND inserts the expected outbox row with the right `QueueOperation`.
- **Soft-delete contract tests** (post-RB-01) — `DELETE /annotations/{id}` flips status to `Deleted (40)`, leaves the row, and relocates files to `deleted_dir`.
- **Stream consumer dedupe tests** (post-RB-09) — outbox messages carry `(annotationId, operation, dateTime)` and a synthetic dedupe consumer collapses a deliberately re-published message.
- **Hash collision regression** (post-RB-04) — same image bytes still hash to the same 32-char hex id; two distinct images do not collide on the sampled `XxHash3.Hash128` domain at scale.
- **Auth boundary tests** — unauthenticated, wrong-policy, expired-token, and refresh-flow scenarios for every policy (`ANN`, `DATASET`, `ADM`, `[Authorize]`).
- **Migrator idempotence** — boot, boot, boot — schema converges to the same shape; seed rows respect `ON CONFLICT DO NOTHING`.
- **Path resolver invariants** — `PUT /settings/directories` triggers `Reset()` and subsequent path lookups reflect the change.

Non-functional ones to layer on once the functional surface is green:

- **Throughput / latency** for `POST /annotations` with image bytes — service must handle the suite's current detections-pipeline cadence without queue backpressure surfacing as 5xx.
- **SSE longevity** — single connection survives 30+ minutes idle without buffer growth.
- **Outbox drain throughput** — `FailsafeProducer` keeps queue depth ~constant under steady-state lifecycle traffic.

## 4. References

| Source | Relevance |
|--------|-----------|
| `src/Program.cs` | Composition root: services, JWT, CORS, Swagger, migrator, middleware, `/health`. |
| `src/Database/DatabaseMigrator.cs` | Authoritative DB schema + seeded rows. |
| `src/Services/AnnotationService.cs` | F1 lifecycle producer; the only producer call site for SSE + outbox. |
| `src/Services/FailsafeProducer.cs` | Outbox drainer + `EnqueueAsync` static helper; contains the empty-catch RB-05 finding. |
| `src/Services/SettingsService.cs:71,85` | `pathResolver.Reset()` invariant (verified). |
| `src/Dockerfile` | Multi-arch build; `EXPOSE 8080`; `AZAION_REVISION` stamp. |
| `.woodpecker/build-arm.yml` | CI: branch-driven `${BRANCH}-arm` tags; OCI labels; ARM64 only. |
| `_docs/02_document/architecture.md` | Architecture Vision + 13 ADRs + 9-item Refactor Backlog. |
| `_docs/02_document/system-flows.md` | F1–F8 traces with verified sequences. |
| `_docs/02_document/data_model.md` | ERD, tables, columns, seed data, migration semantics. |
| `_docs/02_document/glossary.md` | Project-specific terminology, with code → suite term alignment. |
| `_docs/02_document/04_verification_log.md` | Step 4 corrections + stakeholder resolutions. |
| `suite/_docs/01_annotations.md` | Suite-level product/integration narrative; canonical for `Mission`, wire enums, REST contract. |