ui/_docs/02_document/modules/src__features__annotations.md

Module group: src/features/annotations/

Compact doc covering all 5 annotations modules (classColors.ts is a shared leaf — see existing src__features__annotations__classColors.md). The annotations feature is the central legacy concern of the codebase per _docs/legacy/wpf-era.md §4 (Azaion.Annotator window) — what's documented here is the React port. For the canonical product spec see _docs/ui_design/README.md (Annotations Tab Layout, Annotation Quality Guidelines, Affiliation Icons, Combat Readiness, Annotation Row Gradient, Keyboard Shortcuts, Video Annotation Time-Window Display) and parent suite ../../../../_docs/01_annotations.md for the API contract.

Scope

Owns the /annotations route. Lets the user:

1. Browse media (video / image) for the currently selected flight, with a 300-ms debounced name filter, drag-and-drop / file-picker / folder-picker upload, and right-click delete.
2. Play / pause / step a video, scrub the timeline, mute, with frame stepping at 1 / 5 / 10 / 30 / 60 frames in both directions (assumed 30 FPS — see Findings).
3. Draw / move / resize / multi-select bounding boxes on a custom canvas overlay, click-and-drag with Ctrl-modifier behaviours, snap to image-normalized coordinates 0–1.
4. Pick the active detection class (1–9 hotkeys) and PhotoMode (Regular / Winter / Night) via the shared DetectionClasses component.
5. Save the per-frame detection set back to POST /api/annotations/annotations, with fall-back to in-memory storage when the file is a local blob: URL or the backend is unreachable.
6. Stream the annotations sidebar from the GET /api/annotations/annotations/events SSE feed; show each row with the gradient defined in _docs/ui_design/README.md.
7. Trigger AI detection via POST /api/detect/{mediaId} (modal log overlay).
8. Download an annotation as YOLO .txt + a PNG of the frame with rectangles burned in.

Module map

Module	Layer	Responsibility
classColors.ts	leaf	(already documented separately) Class-number → colour + photoMode-suffix lookup.
MediaList.tsx	sub-component	Left panel media browser. Owns media[] state, debounced filter, dropzone upload, blob: local-mode fallback when backend POST fails. Calls GET /api/annotations/media, DELETE /api/annotations/media/{id}, POST /api/annotations/media/batch.
VideoPlayer.tsx	sub-component	Native <video> wrapper. forwardRef exposes seek(seconds) and getVideoElement(). Custom progress slider + frame-step toolbar. Global keydown handler for Space / ←/→ (Ctrl=±150) / M.
AnnotationsSidebar.tsx	sub-component	Right panel: SSE-driven annotation list (/api/annotations/annotations/events filtered by mediaId), AI detect button, gradient row background built from per-detection class colour + confidence-modulated alpha, download button (delegates to page).
CanvasEditor.tsx	sub-component	Canvas overlay used in both image-only and video-overlay modes. forwardRef exposes deleteSelected / deleteAll / hasSelection. Owns zoom (Ctrl+wheel, 0.1–10×), pan (held in pan state — there is no Ctrl+drag pan code, only an unused pan state — see Findings), 8-handle resize, multi-select, draw-on-Ctrl-mousedown. Renders detections + a "time-window" preview of other annotations during video playback.
AnnotationsPage.tsx	page	Orchestrator: 3-panel layout via useResizablePanel (left 250 / right 200, min/max bounds). Owns selectedMedia, annotations, detections, selectedClassNum, photoMode, currentTime. Wires MediaList → CanvasEditor ↔ VideoPlayer ↔ AnnotationsSidebar. Handles the Save POST + local-fallback. Builds the YOLO + PNG download.

Key contracts

• Detection (from src/types/index.ts): { id, classNum, label, confidence ∈ [0,1], affiliation: 0|1|2, combatReadiness, centerX, centerY, width, height } — all coords normalized 0–1. Matches _docs/legacy/wpf-era.md §10 and parent suite _docs/01_annotations.md Detection DTO.
• AnnotationListItem: { id, mediaId, time: 'HH:MM:SS.mmm' | null, createdDate, source: AnnotationSource, status: AnnotationStatus, isSplit, splitTile, detections[] }. Matches Annotations table in parent _docs/00_database_schema.md modulo client-side isSplit / splitTile.
• AI detect endpoint: POST /api/detect/{mediaId} — matches parent ../../../../_docs/03_detections.md §2 after nginx strips /api/detect/. NOTE: UI does NOT forward the X-Refresh-Token header that the spec requires for long-running video detection (_docs/03_detections.md §2 request headers). Carried as a finding.
• Save body: { mediaId, time: 'HH:MM:SS.mmm' | null, detections: Detection[] }. .NET TimeSpan.Parse accepts that format so the round-trip works for time → VideoTime. Body is missing required Source and optional WaypointId required by parent spec CreateAnnotationRequest — see Findings.

External integrations

Endpoint / origin	Where	Direction	Notes
GET /api/annotations/media?flightId&name&pageSize=1000	MediaList.fetchMedia	egress	Hardcoded ceiling 1000.
GET /api/annotations/media/{id}/file	CanvasEditor, VideoPlayer, AnnotationsPage.handleDownload	egress	Image / video bytes.
POST /api/annotations/media/batch	MediaList.uploadFiles	egress	multipart/form-data.
DELETE /api/annotations/media/{id}	MediaList.handleDelete	egress	Skipped for local blob: entries.
GET /api/annotations/annotations?mediaId&pageSize=1000	MediaList.handleSelect, AnnotationsSidebar, AnnotationsPage.handleSave	egress	Refresh after save / SSE event.
POST /api/annotations/annotations	AnnotationsPage.handleSave	egress	Body: { mediaId, time, detections }. Falls back to in-memory on 4xx/5xx.
GET /api/annotations/annotations/{id}/image	CanvasEditor.loadImage	egress	Per-annotation hashed image; preferred over the raw media for image annotations.
GET /api/annotations/annotations/events (SSE)	AnnotationsSidebar	egress	Listens for { annotationId, mediaId, status } events; filters client-side by media.id.
POST /api/detect/{mediaId}	AnnotationsSidebar.handleDetect	egress	Triggers AI inference — endpoint shape unverified vs _docs/03_detections.md.
URL.createObjectURL(File)	MediaList.uploadFiles, AnnotationsPage.handleDownload	browser API	Local-mode blob URLs are revoked on delete or unmount.

Findings carried into Step 4 / 6 / 8

1. VideoPlayer.stepFrames hardcodes fps = 30 — frame stepping is wrong for any other fps (most drone footage is 25 / 30 / 60). UI spec says "Frame duration = 1 / video FPS" (_docs/ui_design/README.md). Should read from video.getVideoPlaybackQuality() / metadata. Step 4.

2. VideoPlayer Ctrl+Left/Right is documented in _docs/ui_design/README.md as "skip 5 seconds" but here it does ±150 frames (= 5s @ 30fps only). Same fps coupling as #1.

3. VideoPlayer.error state has no setter call beyond initial null/onLoadedMetadata reset — but the setError call on onError only sets a string when source fails to load; spec says status messages should appear in a status bar (_docs/ui_design/README.md). The status-bar pattern is not implemented.

4. CanvasEditor Ctrl+drag pan is documented (_docs/ui_design/README.md) but not implemented — pan state exists, only setPan(...) calls are inside the (no-op) zoom flow. The canvas always renders at pan = {0,0}. Step 4 / Step 6 problem extraction.

5. CanvasEditor.useEffect[draw] has missing deps (isVideo, imgSize dependency tracked indirectly through other deps; currentTime/annotations are listed but getTimeWindowDetections is recreated each render and is closed-over). Specifically: draw's closure reads getTimeWindowDetections() and the inner getTimeWindowDetections reads media, currentTime, annotations — those are in the dep list, but media itself is missing. Step 4 a11y / correctness.

6. CanvasEditor time-window threshold is < 2_000_000 ticks in getTimeWindowDetections — at 10 000 000 ticks/second this is ±200 ms (400 ms total window). Spec is asymmetric 50 ms before + 150 ms after (200 ms total) per ../../ui_design/README.md. Implementation window is 4× too wide and centred on the wrong offset. Step 4.

7. CanvasEditor ctx.fillStyle = color; ctx.globalAlpha = 0.1; ctx.fillRect(...) — the box "fill" is class colour at 10% opacity, but the affiliation icon (NATO MIL-STD-2525 shape) and combat-readiness indicator are missing. The current code only renders a tiny green dot for combatReadiness === 1 (Ready). UI spec demands Friendly/Hostile/Unknown/None affiliation icons and Ready/NotReady/Unknown CR. Step 4 vs _docs/ui_design/README.md — significant gap.

8. CanvasEditor.AFFILIATION_COLORS constant is dead code — defined but never used. Likely the seed of the missing affiliation rendering. Step 4.

9. Annotation row gradient cap is wrong by ~9 percentage points: AnnotationsSidebar.getRowGradient uses Math.round(alpha * 40).toString(16) — 40 is decimal, not hex, so the maximum alpha byte is 0x28 (decimal 40 ≈ 16% opacity). Spec wireframe ../../ui_design/annotations.html uses rgba(...,0.25) = 0x40 hex (25%). Almost certainly a typo: should be * 64 (decimal) or * 0x40 to match the wireframe. Step 4.

10. AnnotationsSidebar AI-detect modal: setDetectLog writes "Detection complete" immediately after await api.post(...) returns — there is no progress streaming despite the spec ("scrolling log of detection progress" via DetectionEvent SSE per _docs/03_detections.md). The Detection SSE feed is not subscribed. Step 6.

11. AnnotationsSidebar SSE refresh is fire-and-forget (.catch(() => {})) — silent error suppression, violates coderule.mdc. Step 4.

12. AnnotationsPage.formatTicks(seconds) produces HH:MM:SS.mmm (string). Parent suite _docs/01_annotations.md carries TimeSpan VideoTime — .NET TimeSpan.Parse accepts that format, so the round-trip works, but the API contract is TimeSpan ticks, not a string. Verify in Step 4.

13. AnnotationsPage.handleDownload never sets crossOrigin on the video element (only on the standalone image fallback) — Canvas toBlob will throw "tainted canvas" if the video served from /api/annotations/media/.../file doesn't include Access-Control-Allow-Origin. Same-origin via the dev proxy and nginx is fine, but cross-origin (CDN / direct) breaks download. Step 4.

14. AnnotationsPage.handleSelect annotation flow does NOT expose Validate (V key per UI spec). The Annotations tab shouldn't validate (that's Dataset Explorer), but UI spec keyboard shortcuts list V on Annotations tab too — confirm scope in Step 6.

15. No R key for AI detect in any module here despite UI spec (R = Trigger AI detection). The AI Detect button is only reachable by mouse. Step 4.

16. No PageUp / PageDown for prev/next media file despite UI spec.

17. No Camera config side panel (altitude / FocalLength / SensorWidth) per _docs/ui_design/README.md — completely missing. Documented in Findings of AdminPage.tsx too: aircraft camera defaults are global, but per-session override UI is not built. Step 6.

18. MediaList uses alert(...) for "Unsupported file type" — not the project modal/toast pattern. Step 4.

19. MediaList blob: local-mode is a graceful degradation that lets the page work without backend — useful for demos but the user has no indication that "you're working offline, save will be lost on reload". Document explicitly in Step 6.

20. MediaList.fetchMedia always merges blob: locals on top of backend results even when filtering — local entries ignore the name filter. Step 4.

21. AnnotationsSidebar try { ... } catch (e: any) — any cast bypasses TS strict; e.message may be undefined. Step 4.

22. AnnotationsPage left/right panels resize but widths NOT persisted to UserSettings — UI spec says they should be restored per-user across sessions. The useResizablePanel hook only owns runtime state. Step 6 / Step 8.

23. CanvasEditor.handleMouseDown Ctrl-modifier semantics: Ctrl+click on a box should multi-select (per spec). Code does this. Ctrl+click on empty space should NOT start a draw — but the current branch starts dragState = 'draw' either way, then differentiates inside handleMouseUp by drawRect size. Slight UX cost only. Step 4.

24. Tile / split-image annotation rendering: spec mentions "Tile zoom" — auto-zoom to a tile region when opening a split-image detection. AnnotationListItem.splitTile exists but no consumer code reads it. Step 6 problem extraction.

25. AnnotationsPage.handleSave 4xx/5xx fallback creates an in-memory local-${uuid} annotation that looks identical to a saved one. The user can't distinguish the two — risk of data loss on reload. Step 6 problem extraction.

26. AnnotationsPage does not subscribe to the inference detect SSE — no AI progress visualization, no update on detect completion, no error if inference returns 5xx. Step 6.

27. [STEP 4 — UI FIX, spec is canonical, user 2026-05-10] AnnotationStatus enum diverges from spec. UI declares Created=0, Edited=1, Validated=2 (src/types/index.ts:23). Spec (../../../../_docs/09_dataset_explorer.md) is None=0, Created=10, Edited=20, Validated=30, Deleted=40. Action: change src/types/index.ts to the spec values. Cascades through every status filter, every PATCH/POST that sends a status, every render. PRIORITY — without this fix every dataset status filter and every PATCH /dataset/{id}/status from the UI sends a wrong integer.

28. [STEP 4 — UI FIX, spec is canonical, user 2026-05-10] Affiliation enum diverges from spec. UI has 3 values Unknown=0, Friendly=1, Hostile=2. Spec (../../ui_design/README.md Affiliation Icons + parent ../../../../_docs/03_detections.md) requires four values None, Friendly, Hostile, Unknown. Action: change src/types/index.ts to the four-value set; integer values to be confirmed once with the .NET service before patching (likely None=0, Friendly=1, Hostile=2, Unknown=3). Without this fix the UI cannot send/render the None (no-icon) case.

29. [STEP 4 — UI FIX, spec is canonical, user 2026-05-10] CombatReadiness enum diverges from spec. UI has NotReady=0, Ready=1. Spec adds Unknown (no indicator rendered). Action: add Unknown to src/types/index.ts; confirm integer values with the .NET service.

30. [STEP 4 — UI FIX, spec is canonical, user 2026-05-10] MediaStatus enum diverges from spec. UI has New=0, AiProcessing=1, AiProcessed=2, ManualCreated=3. Spec (../../../../_docs/01_annotations.md SSE + _docs/03_detections.md §2) is None, New, AIProcessing, AIProcessed, ManualCreated, Confirmed, Error. Action: change src/types/index.ts to the seven-value set; confirm integer values with the .NET service. Without this fix the UI cannot render the Error SSE event when inference fails.

31. [RESOLVED 2026-05-10 — PARENT-DOC FIX APPLIED] AnnotationSource numeric serialization is canonical; UI is correct (AI=0, Manual=1). Parent ../../../../_docs/01_annotations.md §5 response example and AnnotationSource enum table updated to integer values; ../../../../_docs/09_dataset_explorer.md §1, §2, §4 examples likewise. Both files also got a "wire format is numeric" header note. No UI change needed.

32. [STEP 4 — UI FIX, user 2026-05-10] AnnotationsPage.handleSave must add Source and WaypointId to the request body and rename time → videoTime. Required body shape per parent ../../../../_docs/01_annotations.md §1 CreateAnnotationRequest:

    {
      "mediaId":    "<string>",
      "waypointId": "<guid | null>",
      "source":     1,
      "videoTime":  "HH:MM:SS.mmm",
      "detections": [ ... ]
    }
    

    Notes: (a) userId is supplied server-side from JWT — do not send. (b) image is omitted in the save flow; the server reuses the media's frame at videoTime. (c) source is Manual (= 1) for hand-edited save; AI inference posts use AI (= 0) and are sent server-to-server by the Detections service. (d) waypointId must come from Media.waypointId (already typed in src/types/index.ts:16); the UI currently does not thread waypoint association through to save. (e) Field name time must become videoTime (.NET PascalCase VideoTime → camelCase on the wire).

33. [RESOLVED 2026-05-10 — PARENT-DOC FIX APPLIED] DatasetItem.isSplit is correct on the UI side; parent spec response now includes it. ../../../../_docs/09_dataset_explorer.md §1 items[] shape updated with isSplit: boolean plus a description tying it to _docs/03_detections.md §4 Tile-Based Detection. No UI change needed.

34. [STEP 4 — UI FIX, user 2026-05-10] Detect endpoint needs conditional X-Refresh-Token header. Spec ../../../../_docs/03_detections.md §2 lists it as required for long-running video detection. Access tokens expire in 3600 s per ../../../../_docs/10_auth.md §1, so any video that takes >1 h to process loses auth on the server-to-server POST /annotations call. Action: when media.mediaType === MediaType.Video, attach X-Refresh-Token: <localStorage refreshToken> to the POST /api/detect/{mediaId} request. Caveat: /auth/refresh rotates the refresh token (per _docs/10_auth.md §2), so the value can go stale if the UI also refreshes its own token mid-flight. Step 4 must decide whether to (i) cache the refresh token at detect-start, (ii) use a long-lived service token, or (iii) accept the failure mode and surface it to the user. For images and short videos the header is unnecessary — but the UI cannot tell upfront, so default to "always send for video".

Tests

None.

Cross-doc references

• Parent suite Annotations API: ../../../../_docs/01_annotations.md
• Parent suite Detections (AI inference): ../../../../_docs/03_detections.md
• Parent suite Database schema: ../../../../_docs/00_database_schema.md
• Legacy WPF reference: ../../legacy/wpf-era.md §4 (Annotator window) and §10 (what survived).
• UI spec: ../../ui_design/README.md (Annotations Tab Layout + keyboard shortcuts + time-window + annotation row gradient + affiliation icons + combat readiness).
• Shared component used here: src/components/DetectionClasses.tsx (already documented).