detections/_docs/02_document/tests/blackbox-tests.md

# Blackbox Tests

## Positive Scenarios

### FT-P-01: Health check returns status before engine initialization

**Summary**: Verify the health endpoint responds correctly when the inference engine has not yet been initialized.
**Traces to**: AC-API-1, AC-EL-1
**Category**: API, Engine Lifecycle

**Preconditions**:
- Detections service is running
- No detection requests have been made (engine is not initialized)

**Input data**: None

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `GET /health` | 200 OK with `{"status": "healthy", "aiAvailability": "None"}` |

**Expected outcome**: Health endpoint returns `status: "healthy"` and `aiAvailability: "None"` (engine not yet loaded).
**Max execution time**: 2s

---

### FT-P-02: Health check reflects engine availability after initialization

**Summary**: Verify the health endpoint reports the correct engine state after the engine has been initialized by a detection request.
**Traces to**: AC-API-1, AC-EL-2
**Category**: API, Engine Lifecycle

**Preconditions**:
- Detections service is running
- Mock-loader serves the ONNX model file
- At least one successful detection has been performed (engine initialized)

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image (trigger engine init) | 200 OK with detection results |
| 2 | `GET /health` | 200 OK with `aiAvailability` set to `"Enabled"` or `"Warning"` |

**Expected outcome**: `aiAvailability` reflects an initialized engine state (not `"None"` or `"Downloading"`).
**Max execution time**: 30s (includes engine init on first call)

---

### FT-P-03: Single image detection returns detections

**Summary**: Verify that a valid small image submitted via POST /detect returns structured detection results.
**Traces to**: AC-DA-1, AC-API-2
**Category**: Detection Accuracy, API

**Preconditions**:
- Engine is initialized (or will be on this call)
- Mock-loader serves the model

**Input data**: small-image (640×480, contains detectable objects)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image as multipart file | 200 OK |
| 2 | Parse response JSON | Array of detection objects, each with `x`, `y`, `width`, `height`, `label`, `confidence` |
| 3 | Verify all confidence values | Every detection has `confidence >= 0.25` (default probability_threshold) |

**Expected outcome**: Non-empty array of DetectionDto objects. All confidences meet threshold. Each detection has valid bounding box coordinates (0.0–1.0 range).
**Max execution time**: 30s

---

### FT-P-04: Large image triggers GSD-based tiling

**Summary**: Verify that an image exceeding 1.5× model dimensions is tiled and processed with tile-level detection results merged.
**Traces to**: AC-IP-1, AC-IP-2
**Category**: Image Processing

**Preconditions**:
- Engine is initialized
- Config includes altitude, focal_length, sensor_width for GSD calculation

**Input data**: large-image (4000×3000)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with large-image and config `{"altitude": 400, "focal_length": 24, "sensor_width": 23.5}` | 200 OK |
| 2 | Parse response JSON | Array of detections |
| 3 | Verify detection coordinates | Bounding box coordinates are in 0.0–1.0 range relative to the full original image |

**Expected outcome**: Detections returned for the full image. Coordinates are normalized to original image dimensions (not tile dimensions). Processing time is longer than small-image due to tiling.
**Max execution time**: 60s

---

### FT-P-05: Detection confidence filtering respects threshold

**Summary**: Verify that detections below the configured probability_threshold are filtered out.
**Traces to**: AC-DA-1
**Category**: Detection Accuracy

**Preconditions**:
- Engine is initialized

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image and config `{"probability_threshold": 0.8}` | 200 OK |
| 2 | Parse response JSON | All returned detections have `confidence >= 0.8` |
| 3 | `POST /detect` with same image and config `{"probability_threshold": 0.1}` | 200 OK |
| 4 | Compare result counts | Step 3 returns >= number of detections from Step 1 |

**Expected outcome**: Higher threshold produces fewer or equal detections. No detection below threshold appears in results.
**Max execution time**: 30s

---

### FT-P-06: Overlapping detections are deduplicated

**Summary**: Verify that overlapping detections with containment ratio above threshold are deduplicated, keeping the higher-confidence one.
**Traces to**: AC-DA-2
**Category**: Detection Accuracy

**Preconditions**:
- Engine is initialized
- Image produces overlapping detections (dense scene)

**Input data**: small-image (scene with clustered objects)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image and config `{"tracking_intersection_threshold": 0.6}` | 200 OK |
| 2 | Collect detections | No two detections of the same class overlap by more than 60% containment ratio |
| 3 | `POST /detect` with same image and config `{"tracking_intersection_threshold": 0.01}` | 200 OK |
| 4 | Compare result counts | Step 3 returns fewer or equal detections (more aggressive dedup) |

**Expected outcome**: No pair of returned detections exceeds the configured overlap threshold.
**Max execution time**: 30s

---

### FT-P-07: Physical size filtering removes oversized detections

**Summary**: Verify that detections exceeding the MaxSizeM for their class (given GSD) are removed.
**Traces to**: AC-DA-4
**Category**: Detection Accuracy

**Preconditions**:
- Engine is initialized
- classes.json loaded with MaxSizeM values

**Input data**: small-image, config with known GSD parameters

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image and config `{"altitude": 400, "focal_length": 24, "sensor_width": 23.5}` | 200 OK |
| 2 | For each detection, compute physical size from bounding box + GSD | No detection's physical size exceeds the MaxSizeM defined for its class in classes.json |

**Expected outcome**: All returned detections have plausible physical dimensions for their class.
**Max execution time**: 30s

---

### FT-P-08: Async media detection returns "started" immediately

**Summary**: Verify that POST /detect/{media_id} returns immediately with status "started" while processing continues in background.
**Traces to**: AC-API-3
**Category**: API

**Preconditions**:
- Engine is initialized
- Media file paths are available via config

**Input data**: jwt-token, test-video path in config

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect/test-media-001` with config paths and auth headers | 200 OK, `{"status": "started"}` |
| 2 | Measure response time | Response arrives within 1s (before video processing completes) |

**Expected outcome**: Immediate response with `{"status": "started"}`. Processing continues asynchronously.
**Max execution time**: 2s (response only; processing continues in background)

---

### FT-P-09: SSE streaming delivers detection events during async processing

**Summary**: Verify that SSE clients receive real-time detection events during async media detection.
**Traces to**: AC-API-4, AC-API-3
**Category**: API

**Preconditions**:
- Engine is initialized
- SSE client connected before triggering detection

**Input data**: jwt-token, test-video path in config

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Open SSE connection: `GET /detect/stream` | Connection established |
| 2 | `POST /detect/test-media-002` with config and auth headers | `{"status": "started"}` |
| 3 | Listen on SSE connection | Receive events with `mediaStatus: "AIProcessing"` as frames are processed |
| 4 | Wait for completion | Final event with `mediaStatus: "AIProcessed"` and `percent: 100` |

**Expected outcome**: Multiple SSE events received. Events include detection data. Final event signals completion.
**Max execution time**: 120s

---

### FT-P-10: Video frame sampling processes every Nth frame

**Summary**: Verify that video processing respects the `frame_period_recognition` setting.
**Traces to**: AC-VP-1
**Category**: Video Processing

**Preconditions**:
- Engine is initialized
- SSE client connected

**Input data**: test-video (10s, 30fps = 300 frames), config `{"frame_period_recognition": 4}`

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Open SSE connection | Connection established |
| 2 | `POST /detect/test-media-003` with config `{"frame_period_recognition": 4, "paths": ["/media/test-video.mp4"]}` | `{"status": "started"}` |
| 3 | Count distinct SSE events with detection data | Number of processed frames ≈ 300/4 = 75 (±10% tolerance for start/end frames) |

**Expected outcome**: Approximately 75 frames processed (not all 300). The count scales proportionally with frame_period_recognition.
**Max execution time**: 120s

---

### FT-P-11: Video annotation interval enforcement

**Summary**: Verify that annotations are not reported more frequently than `frame_recognition_seconds`.
**Traces to**: AC-VP-2
**Category**: Video Processing

**Preconditions**:
- Engine is initialized
- SSE client connected

**Input data**: test-video, config `{"frame_recognition_seconds": 2}`

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Open SSE connection | Connection established |
| 2 | `POST /detect/test-media-004` with config `{"frame_recognition_seconds": 2, "paths": ["/media/test-video.mp4"]}` | `{"status": "started"}` |
| 3 | Record timestamps of consecutive SSE detection events | Minimum gap between consecutive annotation events ≥ 2 seconds |

**Expected outcome**: No two annotation events are closer than 2 seconds apart.
**Max execution time**: 120s

---

### FT-P-12: Video tracking accepts new annotations on movement

**Summary**: Verify that new annotations are accepted when detections move beyond the tracking threshold.
**Traces to**: AC-VP-3
**Category**: Video Processing

**Preconditions**:
- Engine is initialized
- SSE client connected
- Video contains moving objects

**Input data**: test-video, config with `tracking_distance_confidence > 0`

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Open SSE connection | Connection established |
| 2 | `POST /detect/test-media-005` with config `{"tracking_distance_confidence": 0.05, "paths": ["/media/test-video.mp4"]}` | `{"status": "started"}` |
| 3 | Collect SSE events | Annotations are emitted when object positions change between frames |

**Expected outcome**: Annotations contain updated positions reflecting object movement. Static objects do not generate redundant annotations.
**Max execution time**: 120s

---

### FT-P-13: Weather mode class variants

**Summary**: Verify that the system supports detection across different weather mode class variants (Norm, Wint, Night).
**Traces to**: AC-OC-1
**Category**: Object Classes

**Preconditions**:
- Engine is initialized
- classes.json includes weather-mode variants

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image | 200 OK |
| 2 | Inspect returned detection labels | Labels correspond to valid class names from classes.json (base or weather-variant) |

**Expected outcome**: All returned labels are valid entries from the 19-class × 3-mode registry.
**Max execution time**: 30s

---

### FT-P-14: Engine lazy initialization on first detection request

**Summary**: Verify that the engine is not initialized at startup but is initialized on the first detection request.
**Traces to**: AC-EL-1, AC-EL-2
**Category**: Engine Lifecycle

**Preconditions**:
- Fresh service start, no prior requests

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `GET /health` immediately after service starts | `aiAvailability: "None"` — engine not loaded |
| 2 | `POST /detect` with small-image | 200 OK (may take longer — engine initializing) |
| 3 | `GET /health` | `aiAvailability` changed to `"Enabled"` or status indicating engine is active |

**Expected outcome**: Engine transitions from "None" to an active state only after a detection request.
**Max execution time**: 60s

---

### FT-P-15: ONNX fallback when GPU unavailable

**Summary**: Verify that the system falls back to ONNX Runtime when no compatible GPU is available.
**Traces to**: AC-EL-2, RESTRICT-HW-1
**Category**: Engine Lifecycle

**Preconditions**:
- Detections service running WITHOUT GPU runtime (CPU-only Docker profile)
- Mock-loader serves ONNX model

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with small-image | 200 OK with detection results |
| 2 | `GET /health` | `aiAvailability` indicates engine is active (ONNX fallback) |

**Expected outcome**: Detection succeeds via ONNX Runtime. No TensorRT-related errors.
**Max execution time**: 60s

---

### FT-P-16: Tile deduplication removes duplicate detections at tile boundaries

**Summary**: Verify that detections appearing in overlapping tile regions are deduplicated.
**Traces to**: AC-DA-3
**Category**: Detection Accuracy

**Preconditions**:
- Engine is initialized
- Large image that triggers tiling

**Input data**: large-image with config including GSD parameters and `big_image_tile_overlap_percent: 20`

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with large-image and tiling config | 200 OK |
| 2 | Inspect detections near tile boundaries | No two detections of the same class are within 0.01 coordinate difference of each other (TILE_DUPLICATE_CONFIDENCE_THRESHOLD) |

**Expected outcome**: Tile boundary detections are merged. No duplicates with near-identical coordinates remain.
**Max execution time**: 60s

---

## Negative Scenarios

### FT-N-01: Empty image returns 400

**Summary**: Verify that submitting an empty file to POST /detect returns a 400 error.
**Traces to**: AC-API-2 (negative case)
**Category**: API

**Preconditions**:
- Detections service is running

**Input data**: empty-image (zero-byte file)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with empty-image as multipart file | 400 Bad Request |

**Expected outcome**: HTTP 400 with error message indicating empty or invalid image.
**Max execution time**: 5s

---

### FT-N-02: Invalid image data returns 400

**Summary**: Verify that submitting a corrupt/non-image file returns a 400 error.
**Traces to**: AC-API-2 (negative case)
**Category**: API

**Preconditions**:
- Detections service is running

**Input data**: corrupt-image (random binary data)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect` with corrupt-image as multipart file | 400 Bad Request |

**Expected outcome**: HTTP 400. Image decoding fails gracefully with an error response (not a 500).
**Max execution time**: 5s

---

### FT-N-03: Detection when engine unavailable returns 503

**Summary**: Verify that a detection request returns 503 when the engine cannot be initialized.
**Traces to**: AC-API-2 (negative case), AC-EL-2
**Category**: API, Engine Lifecycle

**Preconditions**:
- Mock-loader configured to return errors (model download fails)
- Engine has not been previously initialized

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Configure mock-loader to return 503 on model requests | — |
| 2 | `POST /detect` with small-image | 503 Service Unavailable or 422 |

**Expected outcome**: HTTP 503 or 422 error indicating engine is not available. No crash or unhandled exception.
**Max execution time**: 30s

---

### FT-N-04: Duplicate media_id returns 409

**Summary**: Verify that submitting a second async detection request with an already-active media_id returns 409.
**Traces to**: AC-API-3 (negative case)
**Category**: API

**Preconditions**:
- Engine is initialized
- An async detection is already in progress for media_id "dup-test"

**Input data**: jwt-token, test-video

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | `POST /detect/dup-test` with config and auth headers | `{"status": "started"}` |
| 2 | Immediately `POST /detect/dup-test` again (same media_id) | 409 Conflict |

**Expected outcome**: Second request is rejected with 409. First detection continues normally.
**Max execution time**: 5s

---

### FT-N-05: Missing classes.json prevents startup

**Summary**: Verify that the service fails or returns no detections when classes.json is not present.
**Traces to**: RESTRICT-SW-4
**Category**: Restrictions

**Preconditions**:
- Detections service started WITHOUT classes.json volume mount

**Input data**: None

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Attempt to start detections service without classes.json | Service fails to start OR starts with empty class registry |
| 2 | If started: `POST /detect` with small-image | Empty detections or error response |

**Expected outcome**: Service either fails to start or returns no detections. No unhandled crash.
**Max execution time**: 30s

---

### FT-N-06: Loader service unreachable during model download

**Summary**: Verify that the system handles Loader service being unreachable during engine initialization.
**Traces to**: RESTRICT-ENV-1, AC-EL-2
**Category**: Resilience, Engine Lifecycle

**Preconditions**:
- Mock-loader is stopped or unreachable
- Engine not yet initialized

**Input data**: small-image

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Stop mock-loader service | — |
| 2 | `POST /detect` with small-image | Error response (503 or 422) |
| 3 | `GET /health` | `aiAvailability` reflects error state |

**Expected outcome**: Detection fails gracefully. Health endpoint reflects the engine error state.
**Max execution time**: 30s

---

### FT-N-07: Annotations service unreachable — detection continues

**Summary**: Verify that async detection continues even when the Annotations service is unreachable.
**Traces to**: RESTRICT-ENV-2
**Category**: Resilience

**Preconditions**:
- Engine is initialized
- Mock-annotations is stopped or returns errors
- SSE client connected

**Input data**: jwt-token, test-video

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Stop mock-annotations service | — |
| 2 | `POST /detect/test-media-006` with config and auth | `{"status": "started"}` |
| 3 | Listen on SSE | Detection events still arrive (annotations POST failure is silently caught) |
| 4 | Wait for completion | Final `AIProcessed` event received |

**Expected outcome**: Detection processing completes. SSE events are delivered. Annotations POST failure does not stop the detection pipeline.
**Max execution time**: 120s

---

### FT-N-08: SSE queue overflow is silently dropped

**Summary**: Verify that when an SSE client's queue reaches 100 events, additional events are dropped without error.
**Traces to**: AC-API-4
**Category**: API

**Preconditions**:
- Engine is initialized
- SSE client connected but NOT consuming events (stalled reader)

**Input data**: test-video (generates many events)

**Steps**:

| Step | Consumer Action | Expected System Response |
|------|----------------|------------------------|
| 1 | Open SSE connection but pause reading | Connection established |
| 2 | `POST /detect/test-media-007` with config that generates > 100 events | `{"status": "started"}` |
| 3 | Wait for processing to complete | No error on the detection side |
| 4 | Resume reading SSE | Receive ≤ 100 events (queue max depth) |

**Expected outcome**: No crash or error. Overflow events are silently dropped. Detection completes normally.
**Max execution time**: 120s