Oleksandr Bezdieniezhnykh
12 KiB
E2E Functional Tests
Positive Scenarios
FT-P-01: Tier 1 detects footpath from aerial image
Summary: Submit a winter aerial image containing a visible footpath; verify Tier 1 (YOLOE) returns a detection with class "footpath" and a segmentation mask. Traces to: AC-YOLO-NEW-CLASSES, AC-SEMANTIC-PIPELINE Category: YOLO Object Detection — New Classes
Preconditions:
- Semantic detection service is running
- Mock YOLO service returns pre-computed detections for semantic01.png including footpath class
Input data: semantic01.png + mock-yolo-detections (footpath detected)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png to /api/v1/detect | 200 OK, processing started |
| 2 | GET /api/v1/results after 200ms | Detection result array containing at least 1 detection with class="footpath", confidence > 0.5 |
| 3 | Verify detection bbox covers the known footpath region in semantic01.png | bbox overlaps with annotated ground truth footpath region (IoU > 0.3) |
Expected outcome: At least 1 footpath detection returned with confidence > 0.5 Max execution time: 2s
FT-P-02: Tier 2 traces footpath to endpoint and flags concealed position
Summary: Given a frame with detected footpath, verify Tier 2 performs path tracing (skeletonization → endpoint detection) and identifies a dark mass at the endpoint as a potential concealed position. Traces to: AC-SEMANTIC-DETECTION, AC-SEMANTIC-PIPELINE Category: Semantic Detection Performance
Preconditions:
- Tier 1 has detected a footpath in the input frame
- Mock YOLO provides footpath segmentation mask for semantic01.png
Input data: semantic01.png + mock-yolo-detections (footpath with mask)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png to /api/v1/detect | Processing started |
| 2 | Wait for Tier 2 processing (up to 500ms) | — |
| 3 | GET /api/v1/results | Detection result includes tier2_result="concealed_position" with tier2_confidence > 0 |
| 4 | Read detections.jsonl from output volume | Log entry exists with tier=2, class matches "concealed_position" or "branch_pile_endpoint" |
Expected outcome: Tier 2 produces at least 1 endpoint detection flagged as potential concealed position Max execution time: 3s
FT-P-03: Detection output format matches existing YOLO output schema
Summary: Verify semantic detection output uses the same bounding box format as existing YOLO pipeline (centerX, centerY, width, height, classNum, label, confidence — all normalized). Traces to: AC-INTEGRATION Category: Integration
Preconditions:
- At least 1 detection produced from semantic pipeline
Input data: semantic03.png + mock-yolo-detections
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic03.png to /api/v1/detect | Processing started |
| 2 | GET /api/v1/results | Detection JSON array |
| 3 | Validate each detection has fields: centerX (0-1), centerY (0-1), width (0-1), height (0-1), classNum (int), label (string), confidence (0-1) | All fields present, all values within valid ranges |
Expected outcome: All output detections conform to existing YOLO output schema Max execution time: 2s
FT-P-04: Tier 3 VLM analysis triggered for ambiguous Tier 2 result
Summary: When Tier 2 confidence is below threshold (e.g., 0.3-0.6), verify Tier 3 VLM is invoked for deeper analysis and returns a structured response. Traces to: AC-LATENCY-TIER3, AC-SEMANTIC-PIPELINE Category: Semantic Analysis Pipeline
Preconditions:
- VLM stub is running and responds to IPC
- Mock YOLO returns detections with ambiguous endpoint (moderate confidence)
Input data: semantic02.png + mock-yolo-detections (footpath with ambiguous endpoint) + vlm-stub-responses
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic02.png to /api/v1/detect | Processing started |
| 2 | Wait for Tier 3 processing (up to 6s) | — |
| 3 | GET /api/v1/results | Detection result includes tier3_used=true |
| 4 | Read detections.jsonl | Log entry with tier=3 and VLM analysis text present |
Expected outcome: VLM was invoked, response is recorded in detection log, total latency ≤ 6s Max execution time: 8s
FT-P-05: Frame quality gate rejects blurry frame
Summary: Submit a blurred frame; verify the system rejects it via the frame quality gate and does not produce detections from it. Traces to: AC-SCAN-ALGORITHM Category: Scan Algorithm
Preconditions:
- Blurry test frames available in test data
Input data: blurry-frames (Gaussian blur applied to semantic01.png)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST blurry_semantic01.png to /api/v1/detect | 200 OK |
| 2 | GET /api/v1/results | Empty detection array or response indicating frame rejected (quality below threshold) |
Expected outcome: No detections produced from blurry frame; frame quality metric logged Max execution time: 1s
FT-P-06: Scan controller transitions from Level 1 to Level 2
Summary: When Tier 1 detects a POI, verify the scan controller issues zoom-in gimbal commands and transitions to Level 2 state. Traces to: AC-SCAN-L1-TO-L2, AC-CAMERA-ZOOM Category: Scan Algorithm, Camera Control
Preconditions:
- Mock gimbal service is running and accepting commands
- Scan controller starts in Level 1 mode
Input data: synthetic-video-sequence (simulating Level 1 sweep) + mock-yolo-detections (POI detected mid-sequence)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST first 10 frames (Level 1 sweep, no POI) | Gimbal commands show pan sweep pattern |
| 2 | POST frame 11 with mock YOLO returning a footpath detection | Scan controller queues POI |
| 3 | POST frame 12-15 | Gimbal command log shows zoom-in command issued |
| 4 | Read gimbal command log | Transition from sweep commands to zoom + hold commands within 2s of POI detection |
Expected outcome: Gimbal transitions from Level 1 sweep to Level 2 zoom within 2 seconds Max execution time: 5s
FT-P-07: Detection logging writes complete JSON-lines entries
Summary: After processing multiple frames, verify the detection log contains properly formatted JSON-lines entries with all required fields. Traces to: AC-INTEGRATION Category: Recording, Logging & Telemetry
Preconditions:
- Multiple frames processed with detections
Input data: semantic01.png, semantic02.png + mock-yolo-detections
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png, then semantic02.png | Detections produced |
| 2 | Read /data/output/detections.jsonl | File exists, contains ≥1 JSON line |
| 3 | Parse each line as JSON | Valid JSON with fields: ts, frame_id, tier, class, confidence, bbox |
| 4 | Verify timestamps are ISO 8601, bbox values 0-1, confidence 0-1 | All values within valid ranges |
Expected outcome: All detection log entries are valid JSON with all required fields Max execution time: 3s
FT-P-08: Freshness metadata attached to footpath detections
Summary: Verify that footpath detections include freshness metadata (contrast ratio) as "high_contrast" or "low_contrast" tag. Traces to: AC-SEMANTIC-PIPELINE Category: Semantic Analysis Pipeline
Preconditions:
- Footpath detected in Tier 1
Input data: semantic01.png + mock-yolo-detections (footpath)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png | Detections produced |
| 2 | GET /api/v1/results | Footpath detection includes freshness field |
| 3 | Verify freshness is one of: "high_contrast", "low_contrast" | Valid freshness tag present |
Expected outcome: Freshness metadata present on all footpath detections Max execution time: 2s
Negative Scenarios
FT-N-01: No detections from empty scene
Summary: Submit a frame where YOLO returns zero detections; verify semantic pipeline returns empty results without errors. Traces to: AC-SEMANTIC-PIPELINE (negative case) Category: Semantic Analysis Pipeline
Preconditions:
- Mock YOLO returns empty detection array
Input data: semantic01.png + mock-yolo-empty
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png with mock YOLO returning zero detections | 200 OK |
| 2 | GET /api/v1/results | Empty detection array, no errors |
Expected outcome: System returns empty results gracefully Max execution time: 1s
FT-N-02: System handles high-volume false positive YOLO input
Summary: Submit a frame where YOLO returns 50+ random false positive bounding boxes; verify system processes without crash and Tier 2 filters most. Traces to: AC-SEMANTIC-DETECTION, RESTRICT-RESOURCE Category: Semantic Detection Performance
Preconditions:
- Mock YOLO returns 50 random detections
Input data: semantic01.png + mock-yolo-noise (50 random bboxes)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST semantic01.png with noisy YOLO output | 200 OK, processing started |
| 2 | Wait 2s, GET /api/v1/results | Results returned without crash |
| 3 | Verify result count ≤ 50 | Tier 2 filtering reduces candidate count |
Expected outcome: System handles noisy input without crash; processes within time budget Max execution time: 5s
FT-N-03: Invalid image format rejected
Summary: Submit a 0-byte file and a truncated JPEG; verify system rejects with appropriate error. Traces to: RESTRICT-SOFTWARE Category: Software
Preconditions:
- Service is running
Input data: 0-byte file, truncated JPEG (first 100 bytes of semantic01.png)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST 0-byte file to /api/v1/detect | 400 Bad Request or skip with warning |
| 2 | POST truncated JPEG | 400 Bad Request or skip with warning |
Expected outcome: System rejects invalid input without crash Max execution time: 1s
FT-N-04: Gimbal communication failure triggers graceful degradation
Summary: When mock gimbal stops responding, verify system degrades to Level 3 (no gimbal) and continues YOLO-only detection. Traces to: AC-SCAN-ALGORITHM, RESTRICT-HARDWARE Category: Scan Algorithm, Resilience
Preconditions:
- Mock gimbal is initially running, then stopped mid-test
Input data: semantic01.png + mock-yolo-detections
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | POST frame, verify gimbal commands are sent | Gimbal commands in log |
| 2 | Stop mock-gimbal service | — |
| 3 | POST next frame | System detects gimbal timeout |
| 4 | POST 3 more frames | System enters degradation Level 3 (no gimbal), continues producing YOLO-only detections |
| 5 | GET /api/v1/results | Detections still returned (from existing YOLO pipeline) |
Expected outcome: System degrades gracefully to Level 3, continues detecting without gimbal Max execution time: 15s
FT-N-05: VLM process crash triggers Tier 3 unavailability
Summary: When VLM stub crashes, verify Tier 3 is marked unavailable and Tier 1+2 continue operating. Traces to: AC-SEMANTIC-PIPELINE, RESTRICT-SOFTWARE Category: Resilience
Preconditions:
- VLM stub initially running, then killed
Input data: semantic02.png + mock-yolo-detections (ambiguous endpoint that would trigger VLM)
Steps:
| Step | Consumer Action | Expected System Response |
|---|---|---|
| 1 | Kill vlm-stub process | — |
| 2 | POST semantic02.png with ambiguous detection | Processing starts |
| 3 | GET /api/v1/results after 3s | Detection result with tier3_used=false (VLM unavailable), Tier 1+2 results still present |
| 4 | Read detection log | Log entry shows tier3 skipped with reason "vlm_unavailable" |
Expected outcome: Tier 1+2 results are returned; Tier 3 is gracefully skipped Max execution time: 5s