annotations/_docs/01_solution/solution.md

Azaion.Annotations — Solution (retrospective)

Retrospective view, derived from _docs/02_document/. Mirrors the artifact the research skill produces, but synthesized from verified code rather than user interview. Read alongside _docs/02_document/architecture.md (which carries the confirmed Architecture Vision and the ADR list) and the agreed Refactor Backlog (RB-01..RB-09).

1. Product solution description

Azaion.Annotations is the suite-internal HTTP + streaming service that owns the annotation lifecycle: ingest a video frame (or pre-existing media), record the YOLO detections produced by the upstream detection pipeline, expose CRUD over those annotations, and broadcast every lifecycle change to (a) the Annotator UI in real time via SSE and (b) downstream durable consumers (admin sync worker, AI training pipeline) via a transactional-outbox + RabbitMQ Stream pipeline. It also serves the dataset exploration surface, the media upload pipeline, and the system-metadata catalog (settings + detection classes).

Single .NET 10 binary, single Postgres state-of-record, content-addressed filesystem cache, ARM64 container deployed by branch via Woodpecker CI.

flowchart LR
    subgraph clients [Clients]
        UI[Annotator UI]
        DSE[Dataset Explorer UI]
        DET[Detections service]
        ADM[Admin sync worker]
        AI[AI training]
    end

    subgraph svc [Azaion.Annotations]
        REST[01 Annotations REST]
        RT[02 Realtime and sync]
        MEDIA[03 Media]
        DS[04 Dataset]
        SET[05 Settings and metadata]
        PLAT[06 Platform]
    end

    subgraph store [Stores]
        DB[(PostgreSQL)]
        FS[(Filesystem /data)]
        STREAM[(RabbitMQ Stream)]
    end

    UI -- "REST + SSE" --> REST
    UI -- "REST + SSE" --> RT
    DSE -- "REST DATASET" --> DS
    DET -- "POST + auth refresh" --> REST
    DET -- "POST" --> MEDIA

    REST --> RT
    REST --> PLAT
    RT --> PLAT
    MEDIA --> PLAT
    DS --> PLAT
    SET --> PLAT

    PLAT --> DB
    PLAT --> FS
    RT --> STREAM
    STREAM --> ADM
    STREAM --> AI

2. Architecture (as implemented)

The implemented architecture per component, with the agreed near-term direction (Refactor Backlog RB-01..RB-09) flagged in Limitations and Requirements rows.

2.1 — 06 Platform (foundation)

Field	Value
Solution	Shared kernel: AppDataConnection (Linq2DB), DatabaseMigrator (idempotent boot-time DDL), JWT verifier (JwtExtensions.AddJwtAuth — ES256 over admin's JWKS, no local minting), Infrastructure/ConfigurationResolver (fail-fast required-config resolution), Infrastructure/CorsConfigurationValidator (env-aware safety check), PathResolver, ErrorHandlingMiddleware, composition root (Program.cs).
Tools	.NET 10, ASP.NET Core, Linq2DB, Npgsql, JwtBearer (verifier-only), Microsoft.IdentityModel.Protocols.OpenIdConnect for JWKS resolution, Swashbuckle.
Advantages	Single composition root; idempotent migrator removes a separate migration tool from the deployment story (ADR-007); error envelope is uniform across all controllers; identity is fully out-sourced to admin (no HMAC secret to leak, no token-issuance code path to attack).
Limitations	Swagger UI mounted in all environments (ADR-005); JWKS retrieval requires HTTPS — test harnesses need a TLS-terminating sidecar or test-only relaxation. (ADR-002 / ADR-006 retired by the auth + CORS refactor.)
Requirements	DATABASE_URL, JWT_ISSUER, JWT_AUDIENCE, JWT_JWKS_URL are required at startup (fail-fast); CorsConfig:AllowedOrigins (or explicit AllowAnyOrigin=true) required in Production; directory_settings row reachable; Postgres 13+ behavior assumed by ON CONFLICT and IF NOT EXISTS clauses.
Security	JWT bearer with policies ANN, DATASET, ADM. [AllowAnonymous] only on /health; refresh is admin's responsibility.
Cost	Negligible — pure in-process plumbing.
Fit	Platform mandate is satisfied (single state-of-record, single secret family, single error envelope). Hardening items belong to the Security Audit step.

2.2 — 02 Annotations realtime & sync

Field	Value
Solution	In-process SSE channel (AnnotationEventService, unbounded Channel<AnnotationEventDto>) + transactional outbox (annotations_queue_records) drained by FailsafeProducer (IHostedService) into the azaion-annotations RabbitMQ stream as MessagePack-gzip frames.
Tools	System.Threading.Channels, RabbitMQ.Stream.Client, MessagePack, System.IO.Compression (gzip).
Advantages	Sub-millisecond fan-out for UI (channel) without standing up a broker for the inner loop; durability for external consumers via the outbox even when RabbitMQ is unreachable; producer + drainer co-located removes a deployment unit.
Limitations	Per-instance SSE state — no cross-pod fan-out; no leasing on outbox rows (multi-instance can double-publish); empty catch { } at FailsafeProducer.cs:138 swallows IOException on image read — RB-05.
Requirements	RabbitMQ Stream reachable on RABBITMQ_HOST:RABBITMQ_STREAM_PORT; pathResolver must resolve images_dir/{id}.jpg for Created operations; world-B mutation paths still TODO (RB-01).
Security	RabbitMQ stream auth via RABBITMQ_PRODUCER_USER / _PASS. SSE inherits [Authorize(Policy = "ANN")].
Cost	Channel = O(1) memory per pending message; outbox row + drained delete = ~1 round-trip per lifecycle event. Stream send is gzip-batched.
Fit	Strong fit for current scale; horizontal-scale constraints surface at >1 instance and need to be designed before that point.

2.3 — 01 Annotations REST

Field	Value
Solution	AnnotationsController (REST + image/thumbnail file routes) → AnnotationService → DB + filesystem; lifecycle producer for SSE + outbox.
Tools	ASP.NET Core controllers, Linq2DB, System.IO.Hashing.XxHash64 (today; XxHash3.Hash128 per RB-04), System.IO.File.
Advantages	Content-addressed annotation id deduplicates re-uploads; YOLO label written deterministically next to the image; SSE event carries the full detection payload so UIs can render without an extra round-trip.
Limitations	Today only Create publishes / enqueues — Update / UpdateStatus / Delete are silent (RB-01); not transactional across FS + DB + outbox (RB-03); sampled XxHash64 collision domain is small (RB-04); thumbnails not generated inline.
Requirements	World-B publish + enqueue per RB-01; business-transaction wrapper per RB-03; switch to XxHash3.Hash128 per RB-04; rename FlightId → MissionId per RB-07.
Security	[Authorize(Policy = "ANN")] on the controller; user identity derived from JWT NameIdentifier.
Cost	Per-create: 1 image write + (optional) 1 image copy + 3 DB INSERTs (media optional, annotation, detections via BulkCopy) + 1 label write + 1 SSE channel write + 1 outbox INSERT.
Fit	Solid current shape; the four RB items above bring it in line with the agreed direction.

2.4 — 03 Media

Field	Value
Solution	MediaController (single + batch upload, list, file download, delete) → MediaService → DB + filesystem under media dir.
Tools	ASP.NET Core multipart binding (IFormFileCollection), Linq2DB, System.IO.File.
Advantages	Batch path takes a single waypoint id + multiple files in one request — avoids N round-trips for bulk video frame uploads.
Limitations	No format whitelist enforcement is visible at the controller layer (verify during Step 14 Security Audit); no per-tenant quota enforcement.
Requirements	videos_dir / images_dir writable; MediaType correctly set per upload.
Security	[Authorize(Policy = "ANN")]. User from JWT NameIdentifier.
Cost	One disk write per file + one DB INSERT per row.
Fit	Adequate for the current upload volumes.

2.5 — 04 Dataset

Field	Value
Solution	Read-heavy /dataset surface (filtered queries, class distribution, single + bulk status updates).
Tools	Linq2DB queries against annotations × media × detection.
Advantages	Bulk status update collapses N row updates into a single UPDATE … WHERE id IN (…) — atomic at the SQL level.
Limitations	Tight coupling to the annotation domain via shared AppDataConnection (RB-08); writes are silent (no SSE / outbox today — fixed by RB-01); reads do not yet filter soft-deleted annotations (will need to once RB-01 lands).
Requirements	Decouple writes per RB-08 (route through AnnotationService); honor soft-delete filter on read paths once status Deleted=40 becomes a soft-delete marker.
Security	[Authorize(Policy = "DATASET")].
Cost	Read paths perform LINQ EXISTS subqueries (db.Detections.Any(...)) — acceptable for current data volume; revisit during Step 15 Performance Test if dataset grows substantially.
Fit	Fits the Dataset Explorer UI; the coupling fix will improve maintainability without changing user-visible behavior.

2.6 — 05 Settings & metadata

Field	Value
Solution	CRUD endpoints for system / directory / camera / user settings under /settings; read-only /classes for the detection class catalog (becoming admin-managed per RB-06).
Tools	Linq2DB, ASP.NET Core controllers.
Advantages	Directory cache reset is wired (verified — SettingsService.cs:71, 85); single-row settings model keeps the surface simple.
Limitations	system_settings.silent_detection is a debug remnant scheduled for removal (RB-02); detection classes are migrator-only today, no admin write path (RB-06); Smoke and Plane share color #000080 — fixed as part of RB-06.
Requirements	Add [ADM] CRUD on /classes + read-through cache (RB-06); drop silent_detection (RB-02).
Security	Mixed [Authorize] reads / [ADM] writes.
Cost	One Postgres row family per concern; cache reset is O(1).
Fit	Good fit; the two RB items above complete the surface.

3. Testing strategy

Current state (verified): there is no automated test project in this workspace (00_discovery.md). CI runs only the build + image push (.woodpecker/build-arm.yml) — no test step, no lint step. There is no Postman / Bruno collection in-repo either.

Implication for the autodev existing-code flow: Step 3 (Test Spec) and Step 6 (Implement Tests) of Phase A produce the missing test surface. The shape required is:

• Functional / integration tests — happy-path and error-path coverage for every controller endpoint listed in system-flows.md (F1–F8), exercised against a real Postgres + RabbitMQ stack (test-environment parity is a coderule.mdc mandate).
• Lifecycle-observability tests (post-RB-01) — every mutation path emits an SSE event AND inserts the expected outbox row with the right QueueOperation.
• Soft-delete contract tests (post-RB-01) — DELETE /annotations/{id} flips status to Deleted (40), leaves the row, and relocates files to deleted_dir.
• Stream consumer dedupe tests (post-RB-09) — outbox messages carry (annotationId, operation, dateTime) and a synthetic dedupe consumer collapses a deliberately re-published message.
• Hash collision regression (post-RB-04) — same image bytes still hash to the same 32-char hex id; two distinct images do not collide on the sampled XxHash3.Hash128 domain at scale.
• Auth boundary tests — unauthenticated, wrong-policy, expired-token, and refresh-flow scenarios for every policy (ANN, DATASET, ADM, [Authorize]).
• Migrator idempotence — boot, boot, boot — schema converges to the same shape; seed rows respect ON CONFLICT DO NOTHING.
• Path resolver invariants — PUT /settings/directories triggers Reset() and subsequent path lookups reflect the change.

Non-functional ones to layer on once the functional surface is green:

• Throughput / latency for POST /annotations with image bytes — service must handle the suite's current detections-pipeline cadence without queue backpressure surfacing as 5xx.
• SSE longevity — single connection survives 30+ minutes idle without buffer growth.
• Outbox drain throughput — FailsafeProducer keeps queue depth ~constant under steady-state lifecycle traffic.

4. References

Source	Relevance
src/Program.cs	Composition root: services, JWT, CORS, Swagger, migrator, middleware, /health.
src/Database/DatabaseMigrator.cs	Authoritative DB schema + seeded rows.
src/Services/AnnotationService.cs	F1 lifecycle producer; the only producer call site for SSE + outbox.
src/Services/FailsafeProducer.cs	Outbox drainer + EnqueueAsync static helper; contains the empty-catch RB-05 finding.
src/Services/SettingsService.cs:71,85	pathResolver.Reset() invariant (verified).
src/Dockerfile	Multi-arch build; EXPOSE 8080; AZAION_REVISION stamp.
.woodpecker/build-arm.yml	CI: branch-driven ${BRANCH}-arm tags; OCI labels; ARM64 only.
_docs/02_document/architecture.md	Architecture Vision + 13 ADRs + 9-item Refactor Backlog.
_docs/02_document/system-flows.md	F1–F8 traces with verified sequences.
_docs/02_document/data_model.md	ERD, tables, columns, seed data, migration semantics.
_docs/02_document/glossary.md	Project-specific terminology, with code → suite term alignment.
_docs/02_document/04_verification_log.md	Step 4 corrections + stakeholder resolutions.
suite/_docs/01_annotations.md	Suite-level product/integration narrative; canonical for Mission, wire enums, REST contract.