# Module group: `src/features/annotations/`

> Compact doc covering the 4 annotations-feature modules. `classColors.ts` was carved out of this directory to its own component (`src/class-colors/`) by AZ-511 on 2026-05-13 — see `src__class-colors__classColors.md`; consumers in this feature now import via the `../../class-colors` barrel. The annotations feature is the **central legacy concern** of the codebase per `_docs/legacy/wpf-era.md §4` (`Azaion.Annotator` window) — what's documented here is the React port. For the canonical product spec see `_docs/ui_design/README.md` (Annotations Tab Layout, Annotation Quality Guidelines, Affiliation Icons, Combat Readiness, Annotation Row Gradient, Keyboard Shortcuts, Video Annotation Time-Window Display) and parent suite `../../../../_docs/01_annotations.md` for the API contract.

## Scope

Owns the `/annotations` route. Lets the user:
1. Browse media (video / image) for the currently selected flight, with a 300-ms debounced name filter, drag-and-drop / file-picker / folder-picker upload, and right-click delete.
2. Play / pause / step a video, scrub the timeline, mute, with frame stepping at 1 / 5 / 10 / 30 / 60 frames in both directions (assumed 30 FPS — see Findings).
3. Draw / move / resize / multi-select bounding boxes on a custom canvas overlay, click-and-drag with Ctrl-modifier behaviours, snap to image-normalized coordinates 0–1.
4. Pick the active detection class (1–9 hotkeys) and PhotoMode (Regular / Winter / Night) via the shared `DetectionClasses` component.
5. Save the per-frame detection set back to `POST endpoints.annotations.annotations()` (= `/api/annotations/annotations`), with **fall-back to in-memory storage** when the file is a local blob: URL or the backend is unreachable.
6. Stream the annotations sidebar from the `GET endpoints.annotations.annotationEvents()` (= `/api/annotations/annotations/events`) SSE feed; show each row with the gradient defined in `_docs/ui_design/README.md`.
7. Trigger AI detection via `POST endpoints.detect.media(mediaId)` (= `/api/detect/{mediaId}`) — modal log overlay.
8. Download an annotation as YOLO `.txt` + a PNG of the frame with rectangles burned in.

> All path strings produced by `endpoints.*` builders from `src/api/endpoints.ts` (since AZ-486 / F7).

## Module map

| Module | Layer | Responsibility |
|---|---|---|
| ~~`classColors.ts`~~ | (moved) | Carved out by AZ-511 to `src/class-colors/`; imported via the `class-colors` barrel by `CanvasEditor.tsx`, `AnnotationsSidebar.tsx`, `AnnotationsPage.tsx`. |
| `MediaList.tsx` | sub-component | Left panel media browser. Owns `media[]` state, debounced filter, dropzone upload, blob: local-mode fallback when backend POST fails. Calls `endpoints.annotations.media(qs)`, `endpoints.annotations.mediaItem(id)` (DELETE), `endpoints.annotations.mediaBatch()` (POST). |
| `VideoPlayer.tsx` | sub-component | Native `<video>` wrapper. `forwardRef` exposes `seek(seconds)` and `getVideoElement()`. Custom progress slider + frame-step toolbar. Global `keydown` handler for Space / ←/→ (Ctrl=±150) / M. Image / video bytes via `endpoints.annotations.mediaFile(id)`. |
| `AnnotationsSidebar.tsx` | sub-component | Right panel: SSE-driven annotation list (`endpoints.annotations.annotationEvents()` filtered by `mediaId`), AI detect button (`endpoints.detect.media(mediaId)`), gradient row background built from per-detection class colour + confidence-modulated alpha, download button (delegates to page). |
| `CanvasEditor.tsx` | sub-component | Canvas overlay used in both image-only and video-overlay modes. `forwardRef` exposes `deleteSelected / deleteAll / hasSelection`. Owns zoom (Ctrl+wheel, 0.1–10×), pan (held in `pan` state — there is no Ctrl+drag pan code, only an unused `pan` state — see Findings), 8-handle resize, multi-select, draw-on-Ctrl-mousedown. Renders detections + a "time-window" preview of *other* annotations during video playback. |
| `AnnotationsPage.tsx` | page | Orchestrator: 3-panel layout via `useResizablePanel` (left 250 / right 200, min/max bounds). Owns `selectedMedia`, `annotations`, `detections`, `selectedClassNum`, `photoMode`, `currentTime`. Wires `MediaList` → `CanvasEditor` ↔ `VideoPlayer` ↔ `AnnotationsSidebar`. Handles the `Save` POST + local-fallback. Builds the YOLO + PNG download. |

## Key contracts

- **`Detection`** (from `src/types/index.ts`): `{ id, classNum, label, confidence ∈ [0,1], affiliation: 0|1|2, combatReadiness, centerX, centerY, width, height }` — all coords normalized 0–1. Matches `_docs/legacy/wpf-era.md §10` and parent suite `_docs/01_annotations.md` Detection DTO.
- **`AnnotationListItem`**: `{ id, mediaId, time: 'HH:MM:SS.mmm' | null, createdDate, source: AnnotationSource, status: AnnotationStatus, isSplit, splitTile, detections[] }`. Matches `Annotations` table in parent `_docs/00_database_schema.md` modulo client-side `isSplit / splitTile`.
- **AI detect endpoint**: `endpoints.detect.media(mediaId)` → `POST /api/detect/{mediaId}` — matches parent `../../../../_docs/03_detections.md §2` after nginx strips `/api/detect/`. NOTE: UI does NOT forward the `X-Refresh-Token` header that the spec requires for long-running video detection (`_docs/03_detections.md §2 request headers`). Carried as a finding.
- **Save body**: `{ mediaId, time: 'HH:MM:SS.mmm' | null, detections: Detection[] }`. .NET `TimeSpan.Parse` accepts that format so the round-trip works for `time → VideoTime`. **Body is missing required `Source` and optional `WaypointId`** required by parent spec `CreateAnnotationRequest` — see Findings.

## External integrations

| Builder → Path | Where | Direction | Notes |
|---|---|---|---|
| `endpoints.annotations.media(qs)` → `GET /api/annotations/media?flightId&name&pageSize=1000` | `MediaList.fetchMedia` | egress | Hardcoded ceiling 1000 (in caller). |
| `endpoints.annotations.mediaFile(id)` → `GET /api/annotations/media/{id}/file` | `CanvasEditor`, `VideoPlayer`, `AnnotationsPage.handleDownload` | egress | Image / video bytes. |
| `endpoints.annotations.mediaBatch()` → `POST /api/annotations/media/batch` | `MediaList.uploadFiles` | egress | `multipart/form-data`. |
| `endpoints.annotations.mediaItem(id)` → `DELETE /api/annotations/media/{id}` | `MediaList.handleDelete` | egress | Skipped for local blob: entries. |
| `endpoints.annotations.annotationsByMedia(mediaId, 1000)` → `GET /api/annotations/annotations?mediaId&pageSize=1000` | `MediaList.handleSelect`, `AnnotationsSidebar`, `AnnotationsPage.handleSave` | egress | Refresh after save / SSE event. |
| `endpoints.annotations.annotations()` → `POST /api/annotations/annotations` | `AnnotationsPage.handleSave` | egress | Body: `{ mediaId, time, detections }`. Falls back to in-memory on 4xx/5xx. |
| `endpoints.annotations.annotationImage(id)` → `GET /api/annotations/annotations/{id}/image` | `CanvasEditor.loadImage` | egress | Per-annotation hashed image; preferred over the raw media for image annotations. |
| `endpoints.annotations.annotationEvents()` → `GET /api/annotations/annotations/events` (SSE) | `AnnotationsSidebar` | egress | Listens for `{ annotationId, mediaId, status }` events; filters client-side by `media.id`. |
| `endpoints.detect.media(mediaId)` → `POST /api/detect/{mediaId}` | `AnnotationsSidebar.handleDetect` | egress | Triggers AI inference — endpoint shape unverified vs `_docs/03_detections.md`. |
| `URL.createObjectURL(File)` | `MediaList.uploadFiles`, `AnnotationsPage.handleDownload` | browser API | Local-mode blob URLs are revoked on delete or unmount. |

Path builders live in `src/api/endpoints.ts` (since AZ-486 / F7); `STC-ARCH-02` forbids re-introducing literal `/api/...` strings in `src/`.

## Findings carried into Step 4 / 6 / 8

1. **`VideoPlayer.stepFrames` hardcodes `fps = 30`** — frame stepping is wrong for any other fps (most drone footage is 25 / 30 / 60). UI spec says "Frame duration = 1 / video FPS" (`_docs/ui_design/README.md`). Should read from `video.getVideoPlaybackQuality()` / metadata. Step 4.
2. **`VideoPlayer` Ctrl+Left/Right is documented in `_docs/ui_design/README.md` as "skip 5 seconds"** but here it does `±150 frames` (`= 5s @ 30fps` only). Same fps coupling as #1.
3. **`VideoPlayer.error` state has no setter call beyond initial `null`/onLoadedMetadata reset** — but the `setError` call on `onError` only sets a string when source fails to load; spec says status messages should appear in a status bar (`_docs/ui_design/README.md`). The status-bar pattern is not implemented.
4. **`CanvasEditor` Ctrl+drag pan is documented (`_docs/ui_design/README.md`)** but not implemented — `pan` state exists, only `setPan(...)` calls are inside the (no-op) zoom flow. The canvas always renders at `pan = {0,0}`. Step 4 / Step 6 problem extraction.
5. **`CanvasEditor.useEffect[draw]` has missing deps** (`isVideo`, `imgSize` dependency tracked indirectly through other deps; `currentTime`/`annotations` are listed but `getTimeWindowDetections` is recreated each render and is closed-over). Specifically: `draw`'s closure reads `getTimeWindowDetections()` and the inner `getTimeWindowDetections` reads `media`, `currentTime`, `annotations` — those *are* in the dep list, but `media` itself is missing. Step 4 a11y / correctness.
6. **`CanvasEditor` time-window threshold is `< 2_000_000` ticks** in `getTimeWindowDetections` — at 10 000 000 ticks/second this is **±200 ms (400 ms total window)**. Spec is **asymmetric 50 ms before + 150 ms after (200 ms total)** per `../../ui_design/README.md`. Implementation window is 4× too wide and centred on the wrong offset. Step 4.
7. **`CanvasEditor` `ctx.fillStyle = color; ctx.globalAlpha = 0.1; ctx.fillRect(...)`** — the box "fill" is class colour at 10% opacity, but the **affiliation icon** (NATO MIL-STD-2525 shape) and **combat-readiness indicator** are missing. The current code only renders a tiny green dot for `combatReadiness === 1` (Ready). UI spec demands Friendly/Hostile/Unknown/None affiliation icons and Ready/NotReady/Unknown CR. Step 4 vs `_docs/ui_design/README.md` — significant gap.
8. **`CanvasEditor.AFFILIATION_COLORS` constant is dead code** — defined but never used. Likely the seed of the missing affiliation rendering. Step 4.
9. **Annotation row gradient cap is wrong by ~9 percentage points**: `AnnotationsSidebar.getRowGradient` uses `Math.round(alpha * 40).toString(16)` — `40` is **decimal**, not hex, so the maximum alpha byte is `0x28` (decimal 40 ≈ **16% opacity**). Spec wireframe `../../ui_design/annotations.html` uses `rgba(...,0.25)` = `0x40` hex (25%). Almost certainly a typo: should be `* 64` (decimal) or `* 0x40` to match the wireframe. Step 4.
10. **`AnnotationsSidebar` AI-detect modal**: `setDetectLog` writes "Detection complete" immediately after `await api.post(...)` returns — there is no progress streaming despite the spec ("scrolling log of detection progress" via `DetectionEvent` SSE per `_docs/03_detections.md`). The Detection SSE feed is not subscribed. Step 6.
11. **`AnnotationsSidebar` SSE refresh** is fire-and-forget (`.catch(() => {})`) — silent error suppression, violates `coderule.mdc`. Step 4.
12. **`AnnotationsPage.formatTicks(seconds)`** produces `HH:MM:SS.mmm` (string). Parent suite `_docs/01_annotations.md` carries `TimeSpan VideoTime` — .NET `TimeSpan.Parse` accepts that format, so the round-trip works, but the API contract is `TimeSpan` ticks, not a string. Verify in Step 4.
13. **`AnnotationsPage.handleDownload` never sets `crossOrigin` on the video element** (only on the standalone image fallback) — Canvas `toBlob` will throw "tainted canvas" if the video served from `/api/annotations/media/.../file` doesn't include `Access-Control-Allow-Origin`. Same-origin via the dev proxy and nginx is fine, but cross-origin (CDN / direct) breaks download. Step 4.
14. **`AnnotationsPage.handleSelect` annotation flow** does NOT expose `Validate` (V key per UI spec). The Annotations tab shouldn't validate (that's Dataset Explorer), but UI spec keyboard shortcuts list V on Annotations tab too — confirm scope in Step 6.
15. **No `R` key for AI detect** in any module here despite UI spec (`R = Trigger AI detection`). The AI Detect button is only reachable by mouse. Step 4.
16. **No PageUp / PageDown for prev/next media file** despite UI spec.
17. **No Camera config side panel** (altitude / FocalLength / SensorWidth) per `_docs/ui_design/README.md` — completely missing. Documented in Findings of `AdminPage.tsx` too: aircraft camera defaults are global, but per-session override UI is not built. Step 6.
18. **`MediaList` uses `alert(...)` for "Unsupported file type"** — not the project modal/toast pattern. Step 4.
19. **`MediaList` blob: local-mode is a graceful degradation** that lets the page work without backend — useful for demos but the user has no indication that "you're working offline, save will be lost on reload". Document explicitly in Step 6.
20. **`MediaList.fetchMedia` always merges blob: locals on top of backend results** even when filtering — local entries ignore the name filter. Step 4.
21. **`AnnotationsSidebar` `try { ... } catch (e: any)`** — `any` cast bypasses TS strict; `e.message` may be `undefined`. Step 4.
22. **`AnnotationsPage` left/right panels resize but widths NOT persisted** to `UserSettings` — UI spec says they should be restored per-user across sessions. The `useResizablePanel` hook only owns runtime state. Step 6 / Step 8.
23. **`CanvasEditor.handleMouseDown` Ctrl-modifier semantics**: Ctrl+click on a box should multi-select (per spec). Code does this. Ctrl+click on empty space should NOT start a draw — but the current branch starts `dragState = 'draw'` either way, then differentiates inside `handleMouseUp` by drawRect size. Slight UX cost only. Step 4.
24. **Tile / split-image annotation rendering**: spec mentions "Tile zoom" — auto-zoom to a tile region when opening a split-image detection. `AnnotationListItem.splitTile` exists but no consumer code reads it. Step 6 problem extraction.
25. **`AnnotationsPage.handleSave` 4xx/5xx fallback creates an in-memory `local-${uuid}` annotation** that looks identical to a saved one. The user can't distinguish the two — risk of data loss on reload. Step 6 problem extraction.
26. **`AnnotationsPage` does not subscribe to the inference detect SSE** — no AI progress visualization, no update on detect completion, no error if inference returns 5xx. Step 6.
27. **[STEP 4 — UI FIX, spec is canonical, user 2026-05-10]** `AnnotationStatus` enum diverges from spec. UI declares `Created=0, Edited=1, Validated=2` (`src/types/index.ts:23`). Spec (`../../../../_docs/09_dataset_explorer.md`) is `None=0, Created=10, Edited=20, Validated=30, Deleted=40`. Action: change `src/types/index.ts` to the spec values. Cascades through every status filter, every PATCH/POST that sends a status, every render. **PRIORITY** — without this fix every dataset status filter and every PATCH `/dataset/{id}/status` from the UI sends a wrong integer.
28. **[STEP 4 — UI FIX, spec is canonical, user 2026-05-10]** `Affiliation` enum diverges from spec. UI has 3 values `Unknown=0, Friendly=1, Hostile=2`. Spec (`../../ui_design/README.md` Affiliation Icons + parent `../../../../_docs/03_detections.md`) requires four values `None, Friendly, Hostile, Unknown`. Action: change `src/types/index.ts` to the four-value set; integer values to be confirmed once with the .NET service before patching (likely `None=0, Friendly=1, Hostile=2, Unknown=3`). Without this fix the UI cannot send/render the **None** (no-icon) case.
29. **[STEP 4 — UI FIX, spec is canonical, user 2026-05-10]** `CombatReadiness` enum diverges from spec. UI has `NotReady=0, Ready=1`. Spec adds `Unknown` (no indicator rendered). Action: add `Unknown` to `src/types/index.ts`; confirm integer values with the .NET service.
30. **[STEP 4 — UI FIX, spec is canonical, user 2026-05-10]** `MediaStatus` enum diverges from spec. UI has `New=0, AiProcessing=1, AiProcessed=2, ManualCreated=3`. Spec (`../../../../_docs/01_annotations.md` SSE + `_docs/03_detections.md §2`) is `None, New, AIProcessing, AIProcessed, ManualCreated, Confirmed, Error`. Action: change `src/types/index.ts` to the seven-value set; confirm integer values with the .NET service. Without this fix the UI cannot render the **Error** SSE event when inference fails.
31. **[RESOLVED 2026-05-10 — PARENT-DOC FIX APPLIED]** `AnnotationSource` numeric serialization is canonical; UI is correct (`AI=0, Manual=1`). Parent `../../../../_docs/01_annotations.md` §5 response example and AnnotationSource enum table updated to integer values; `../../../../_docs/09_dataset_explorer.md §1`, §2, §4 examples likewise. Both files also got a "wire format is numeric" header note. No UI change needed.
32. **[STEP 4 — UI FIX, user 2026-05-10]** `AnnotationsPage.handleSave` must add `Source` and `WaypointId` to the request body and rename `time` → `videoTime`. Required body shape per parent `../../../../_docs/01_annotations.md §1` `CreateAnnotationRequest`:

    ```json
    {
      "mediaId":    "<string>",
      "waypointId": "<guid | null>",
      "source":     1,
      "videoTime":  "HH:MM:SS.mmm",
      "detections": [ ... ]
    }
    ```

    Notes: (a) `userId` is supplied server-side from JWT — do not send. (b) `image` is omitted in the save flow; the server reuses the media's frame at `videoTime`. (c) `source` is `Manual` (= `1`) for hand-edited save; AI inference posts use `AI` (= `0`) and are sent server-to-server by the Detections service. (d) `waypointId` must come from `Media.waypointId` (already typed in `src/types/index.ts:16`); the UI currently does not thread waypoint association through to save. (e) Field name `time` must become `videoTime` (.NET PascalCase `VideoTime` → camelCase on the wire).
33. **[RESOLVED 2026-05-10 — PARENT-DOC FIX APPLIED]** `DatasetItem.isSplit` is correct on the UI side; parent spec response now includes it. `../../../../_docs/09_dataset_explorer.md §1` `items[]` shape updated with `isSplit: boolean` plus a description tying it to `_docs/03_detections.md §4` Tile-Based Detection. No UI change needed.
34. **[STEP 4 — UI FIX, user 2026-05-10]** `Detect` endpoint needs conditional `X-Refresh-Token` header. Spec `../../../../_docs/03_detections.md §2` lists it as required for long-running video detection. Access tokens expire in 3600 s per `../../../../_docs/10_auth.md §1`, so any video that takes >1 h to process loses auth on the server-to-server `POST /annotations` call. Action: when `media.mediaType === MediaType.Video`, attach `X-Refresh-Token: <localStorage refreshToken>` to the `POST /api/detect/{mediaId}` request. Caveat: `/auth/refresh` rotates the refresh token (per `_docs/10_auth.md §2`), so the value can go stale if the UI also refreshes its own token mid-flight. Step 4 must decide whether to (i) cache the refresh token at detect-start, (ii) use a long-lived service token, or (iii) accept the failure mode and surface it to the user. For images and short videos the header is unnecessary — but the UI cannot tell upfront, so default to "always send for video".

## Tests

None.

## Cross-doc references

- Parent suite Annotations API: `../../../../_docs/01_annotations.md`
- Parent suite Detections (AI inference): `../../../../_docs/03_detections.md`
- Parent suite Database schema: `../../../../_docs/00_database_schema.md`
- Legacy WPF reference: `../../legacy/wpf-era.md` §4 (Annotator window) and §10 (what survived).
- UI spec: `../../ui_design/README.md` (Annotations Tab Layout + keyboard shortcuts + time-window + annotation row gradient + affiliation icons + combat readiness).
- Shared component used here: `src/components/DetectionClasses.tsx` (already documented).