Initial commit

Made-with: Cursor

_docs/02_plans/integration_tests/functional_tests.md

@@ -0,0 +1,323 @@
+# E2E Functional Tests
+
+## Positive Scenarios
+
+### FT-P-01: Tier 1 detects footpath from aerial image
+
+**Summary**: Submit a winter aerial image containing a visible footpath; verify Tier 1 (YOLOE) returns a detection with class "footpath" and a segmentation mask.
+**Traces to**: AC-YOLO-NEW-CLASSES, AC-SEMANTIC-PIPELINE
+**Category**: YOLO Object Detection — New Classes
+
+**Preconditions**:
+- Semantic detection service is running
+- Mock YOLO service returns pre-computed detections for semantic01.png including footpath class
+
+**Input data**: semantic01.png + mock-yolo-detections (footpath detected)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png to /api/v1/detect | 200 OK, processing started |
+| 2 | GET /api/v1/results after 200ms | Detection result array containing at least 1 detection with class="footpath", confidence > 0.5 |
+| 3 | Verify detection bbox covers the known footpath region in semantic01.png | bbox overlaps with annotated ground truth footpath region (IoU > 0.3) |
+
+**Expected outcome**: At least 1 footpath detection returned with confidence > 0.5
+**Max execution time**: 2s
+
+---
+
+### FT-P-02: Tier 2 traces footpath to endpoint and flags concealed position
+
+**Summary**: Given a frame with detected footpath, verify Tier 2 performs path tracing (skeletonization → endpoint detection) and identifies a dark mass at the endpoint as a potential concealed position.
+**Traces to**: AC-SEMANTIC-DETECTION, AC-SEMANTIC-PIPELINE
+**Category**: Semantic Detection Performance
+
+**Preconditions**:
+- Tier 1 has detected a footpath in the input frame
+- Mock YOLO provides footpath segmentation mask for semantic01.png
+
+**Input data**: semantic01.png + mock-yolo-detections (footpath with mask)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png to /api/v1/detect | Processing started |
+| 2 | Wait for Tier 2 processing (up to 500ms) | — |
+| 3 | GET /api/v1/results | Detection result includes tier2_result="concealed_position" with tier2_confidence > 0 |
+| 4 | Read detections.jsonl from output volume | Log entry exists with tier=2, class matches "concealed_position" or "branch_pile_endpoint" |
+
+**Expected outcome**: Tier 2 produces at least 1 endpoint detection flagged as potential concealed position
+**Max execution time**: 3s
+
+---
+
+### FT-P-03: Detection output format matches existing YOLO output schema
+
+**Summary**: Verify semantic detection output uses the same bounding box format as existing YOLO pipeline (centerX, centerY, width, height, classNum, label, confidence — all normalized).
+**Traces to**: AC-INTEGRATION
+**Category**: Integration
+
+**Preconditions**:
+- At least 1 detection produced from semantic pipeline
+
+**Input data**: semantic03.png + mock-yolo-detections
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic03.png to /api/v1/detect | Processing started |
+| 2 | GET /api/v1/results | Detection JSON array |
+| 3 | Validate each detection has fields: centerX (0-1), centerY (0-1), width (0-1), height (0-1), classNum (int), label (string), confidence (0-1) | All fields present, all values within valid ranges |
+
+**Expected outcome**: All output detections conform to existing YOLO output schema
+**Max execution time**: 2s
+
+---
+
+### FT-P-04: Tier 3 VLM analysis triggered for ambiguous Tier 2 result
+
+**Summary**: When Tier 2 confidence is below threshold (e.g., 0.3-0.6), verify Tier 3 VLM is invoked for deeper analysis and returns a structured response.
+**Traces to**: AC-LATENCY-TIER3, AC-SEMANTIC-PIPELINE
+**Category**: Semantic Analysis Pipeline
+
+**Preconditions**:
+- VLM stub is running and responds to IPC
+- Mock YOLO returns detections with ambiguous endpoint (moderate confidence)
+
+**Input data**: semantic02.png + mock-yolo-detections (footpath with ambiguous endpoint) + vlm-stub-responses
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic02.png to /api/v1/detect | Processing started |
+| 2 | Wait for Tier 3 processing (up to 6s) | — |
+| 3 | GET /api/v1/results | Detection result includes tier3_used=true |
+| 4 | Read detections.jsonl | Log entry with tier=3 and VLM analysis text present |
+
+**Expected outcome**: VLM was invoked, response is recorded in detection log, total latency ≤ 6s
+**Max execution time**: 8s
+
+---
+
+### FT-P-05: Frame quality gate rejects blurry frame
+
+**Summary**: Submit a blurred frame; verify the system rejects it via the frame quality gate and does not produce detections from it.
+**Traces to**: AC-SCAN-ALGORITHM
+**Category**: Scan Algorithm
+
+**Preconditions**:
+- Blurry test frames available in test data
+
+**Input data**: blurry-frames (Gaussian blur applied to semantic01.png)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST blurry_semantic01.png to /api/v1/detect | 200 OK |
+| 2 | GET /api/v1/results | Empty detection array or response indicating frame rejected (quality below threshold) |
+
+**Expected outcome**: No detections produced from blurry frame; frame quality metric logged
+**Max execution time**: 1s
+
+---
+
+### FT-P-06: Scan controller transitions from Level 1 to Level 2
+
+**Summary**: When Tier 1 detects a POI, verify the scan controller issues zoom-in gimbal commands and transitions to Level 2 state.
+**Traces to**: AC-SCAN-L1-TO-L2, AC-CAMERA-ZOOM
+**Category**: Scan Algorithm, Camera Control
+
+**Preconditions**:
+- Mock gimbal service is running and accepting commands
+- Scan controller starts in Level 1 mode
+
+**Input data**: synthetic-video-sequence (simulating Level 1 sweep) + mock-yolo-detections (POI detected mid-sequence)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST first 10 frames (Level 1 sweep, no POI) | Gimbal commands show pan sweep pattern |
+| 2 | POST frame 11 with mock YOLO returning a footpath detection | Scan controller queues POI |
+| 3 | POST frame 12-15 | Gimbal command log shows zoom-in command issued |
+| 4 | Read gimbal command log | Transition from sweep commands to zoom + hold commands within 2s of POI detection |
+
+**Expected outcome**: Gimbal transitions from Level 1 sweep to Level 2 zoom within 2 seconds
+**Max execution time**: 5s
+
+---
+
+### FT-P-07: Detection logging writes complete JSON-lines entries
+
+**Summary**: After processing multiple frames, verify the detection log contains properly formatted JSON-lines entries with all required fields.
+**Traces to**: AC-INTEGRATION
+**Category**: Recording, Logging & Telemetry
+
+**Preconditions**:
+- Multiple frames processed with detections
+
+**Input data**: semantic01.png, semantic02.png + mock-yolo-detections
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png, then semantic02.png | Detections produced |
+| 2 | Read /data/output/detections.jsonl | File exists, contains ≥1 JSON line |
+| 3 | Parse each line as JSON | Valid JSON with fields: ts, frame_id, tier, class, confidence, bbox |
+| 4 | Verify timestamps are ISO 8601, bbox values 0-1, confidence 0-1 | All values within valid ranges |
+
+**Expected outcome**: All detection log entries are valid JSON with all required fields
+**Max execution time**: 3s
+
+---
+
+### FT-P-08: Freshness metadata attached to footpath detections
+
+**Summary**: Verify that footpath detections include freshness metadata (contrast ratio) as "high_contrast" or "low_contrast" tag.
+**Traces to**: AC-SEMANTIC-PIPELINE
+**Category**: Semantic Analysis Pipeline
+
+**Preconditions**:
+- Footpath detected in Tier 1
+
+**Input data**: semantic01.png + mock-yolo-detections (footpath)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png | Detections produced |
+| 2 | GET /api/v1/results | Footpath detection includes freshness field |
+| 3 | Verify freshness is one of: "high_contrast", "low_contrast" | Valid freshness tag present |
+
+**Expected outcome**: Freshness metadata present on all footpath detections
+**Max execution time**: 2s
+
+---
+
+## Negative Scenarios
+
+### FT-N-01: No detections from empty scene
+
+**Summary**: Submit a frame where YOLO returns zero detections; verify semantic pipeline returns empty results without errors.
+**Traces to**: AC-SEMANTIC-PIPELINE (negative case)
+**Category**: Semantic Analysis Pipeline
+
+**Preconditions**:
+- Mock YOLO returns empty detection array
+
+**Input data**: semantic01.png + mock-yolo-empty
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png with mock YOLO returning zero detections | 200 OK |
+| 2 | GET /api/v1/results | Empty detection array, no errors |
+
+**Expected outcome**: System returns empty results gracefully
+**Max execution time**: 1s
+
+---
+
+### FT-N-02: System handles high-volume false positive YOLO input
+
+**Summary**: Submit a frame where YOLO returns 50+ random false positive bounding boxes; verify system processes without crash and Tier 2 filters most.
+**Traces to**: AC-SEMANTIC-DETECTION, RESTRICT-RESOURCE
+**Category**: Semantic Detection Performance
+
+**Preconditions**:
+- Mock YOLO returns 50 random detections
+
+**Input data**: semantic01.png + mock-yolo-noise (50 random bboxes)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST semantic01.png with noisy YOLO output | 200 OK, processing started |
+| 2 | Wait 2s, GET /api/v1/results | Results returned without crash |
+| 3 | Verify result count ≤ 50 | Tier 2 filtering reduces candidate count |
+
+**Expected outcome**: System handles noisy input without crash; processes within time budget
+**Max execution time**: 5s
+
+---
+
+### FT-N-03: Invalid image format rejected
+
+**Summary**: Submit a 0-byte file and a truncated JPEG; verify system rejects with appropriate error.
+**Traces to**: RESTRICT-SOFTWARE
+**Category**: Software
+
+**Preconditions**:
+- Service is running
+
+**Input data**: 0-byte file, truncated JPEG (first 100 bytes of semantic01.png)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST 0-byte file to /api/v1/detect | 400 Bad Request or skip with warning |
+| 2 | POST truncated JPEG | 400 Bad Request or skip with warning |
+
+**Expected outcome**: System rejects invalid input without crash
+**Max execution time**: 1s
+
+---
+
+### FT-N-04: Gimbal communication failure triggers graceful degradation
+
+**Summary**: When mock gimbal stops responding, verify system degrades to Level 3 (no gimbal) and continues YOLO-only detection.
+**Traces to**: AC-SCAN-ALGORITHM, RESTRICT-HARDWARE
+**Category**: Scan Algorithm, Resilience
+
+**Preconditions**:
+- Mock gimbal is initially running, then stopped mid-test
+
+**Input data**: semantic01.png + mock-yolo-detections
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | POST frame, verify gimbal commands are sent | Gimbal commands in log |
+| 2 | Stop mock-gimbal service | — |
+| 3 | POST next frame | System detects gimbal timeout |
+| 4 | POST 3 more frames | System enters degradation Level 3 (no gimbal), continues producing YOLO-only detections |
+| 5 | GET /api/v1/results | Detections still returned (from existing YOLO pipeline) |
+
+**Expected outcome**: System degrades gracefully to Level 3, continues detecting without gimbal
+**Max execution time**: 15s
+
+---
+
+### FT-N-05: VLM process crash triggers Tier 3 unavailability
+
+**Summary**: When VLM stub crashes, verify Tier 3 is marked unavailable and Tier 1+2 continue operating.
+**Traces to**: AC-SEMANTIC-PIPELINE, RESTRICT-SOFTWARE
+**Category**: Resilience
+
+**Preconditions**:
+- VLM stub initially running, then killed
+
+**Input data**: semantic02.png + mock-yolo-detections (ambiguous endpoint that would trigger VLM)
+
+**Steps**:
+
+| Step | Consumer Action | Expected System Response |
+|------|----------------|------------------------|
+| 1 | Kill vlm-stub process | — |
+| 2 | POST semantic02.png with ambiguous detection | Processing starts |
+| 3 | GET /api/v1/results after 3s | Detection result with tier3_used=false (VLM unavailable), Tier 1+2 results still present |
+| 4 | Read detection log | Log entry shows tier3 skipped with reason "vlm_unavailable" |
+
+**Expected outcome**: Tier 1+2 results are returned; Tier 3 is gracefully skipped
+**Max execution time**: 5s