detections/_docs/02_document/tests/blackbox-tests.md

Blackbox Tests

Positive Scenarios

FT-P-01: Health check returns status before engine initialization

Summary: Verify the health endpoint responds correctly when the inference engine has not yet been initialized. Traces to: AC-API-1, AC-EL-1 Category: API, Engine Lifecycle

Preconditions:

• Detections service is running
• No detection requests have been made (engine is not initialized)

Input data: None

Steps:

Step	Consumer Action	Expected System Response
1	GET /health	200 OK with {"status": "healthy", "aiAvailability": "None"}

Expected outcome: Health endpoint returns status: "healthy" and aiAvailability: "None" (engine not yet loaded). Max execution time: 2s

---

FT-P-02: Health check reflects engine availability after initialization

Summary: Verify the health endpoint reports the correct engine state after the engine has been initialized by a detection request. Traces to: AC-API-1, AC-EL-2 Category: API, Engine Lifecycle

Preconditions:

• Detections service is running
• Mock-loader serves the ONNX model file
• At least one successful detection has been performed (engine initialized)

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image (trigger engine init)	200 OK with detection results
2	GET /health	200 OK with aiAvailability set to "Enabled" or "Warning"

Expected outcome: aiAvailability reflects an initialized engine state (not "None" or "Downloading"). Max execution time: 30s (includes engine init on first call)

---

FT-P-03: Single image detection returns detections

Summary: Verify that a valid small image submitted via POST /detect returns structured detection results. Traces to: AC-DA-1, AC-API-2 Category: Detection Accuracy, API

Preconditions:

• Engine is initialized (or will be on this call)
• Mock-loader serves the model

Input data: small-image (640×480, contains detectable objects)

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image as multipart file	200 OK
2	Parse response JSON	Array of detection objects, each with x, y, width, height, label, confidence
3	Verify all confidence values	Every detection has confidence >= 0.25 (default probability_threshold)

Expected outcome: Non-empty array of DetectionDto objects. All confidences meet threshold. Each detection has valid bounding box coordinates (0.0–1.0 range). Max execution time: 30s

---

FT-P-04: Large image triggers GSD-based tiling

Summary: Verify that an image exceeding 1.5× model dimensions is tiled and processed with tile-level detection results merged. Traces to: AC-IP-1, AC-IP-2 Category: Image Processing

Preconditions:

• Engine is initialized
• Config includes altitude, focal_length, sensor_width for GSD calculation

Input data: large-image (4000×3000)

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with large-image and config {"altitude": 400, "focal_length": 24, "sensor_width": 23.5}	200 OK
2	Parse response JSON	Array of detections
3	Verify detection coordinates	Bounding box coordinates are in 0.0–1.0 range relative to the full original image

Expected outcome: Detections returned for the full image. Coordinates are normalized to original image dimensions (not tile dimensions). Processing time is longer than small-image due to tiling. Max execution time: 60s

---

FT-P-05: Detection confidence filtering respects threshold

Summary: Verify that detections below the configured probability_threshold are filtered out. Traces to: AC-DA-1 Category: Detection Accuracy

Preconditions:

• Engine is initialized

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image and config {"probability_threshold": 0.8}	200 OK
2	Parse response JSON	All returned detections have confidence >= 0.8
3	POST /detect with same image and config {"probability_threshold": 0.1}	200 OK
4	Compare result counts	Step 3 returns >= number of detections from Step 1

Expected outcome: Higher threshold produces fewer or equal detections. No detection below threshold appears in results. Max execution time: 30s

---

FT-P-06: Overlapping detections are deduplicated

Summary: Verify that overlapping detections with containment ratio above threshold are deduplicated, keeping the higher-confidence one. Traces to: AC-DA-2 Category: Detection Accuracy

Preconditions:

• Engine is initialized
• Image produces overlapping detections (dense scene)

Input data: small-image (scene with clustered objects)

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image and config {"tracking_intersection_threshold": 0.6}	200 OK
2	Collect detections	No two detections of the same class overlap by more than 60% containment ratio
3	POST /detect with same image and config {"tracking_intersection_threshold": 0.01}	200 OK
4	Compare result counts	Step 3 returns fewer or equal detections (more aggressive dedup)

Expected outcome: No pair of returned detections exceeds the configured overlap threshold. Max execution time: 30s

---

FT-P-07: Physical size filtering removes oversized detections

Summary: Verify that detections exceeding the MaxSizeM for their class (given GSD) are removed. Traces to: AC-DA-4 Category: Detection Accuracy

Preconditions:

• Engine is initialized
• classes.json loaded with MaxSizeM values

Input data: small-image, config with known GSD parameters

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image and config {"altitude": 400, "focal_length": 24, "sensor_width": 23.5}	200 OK
2	For each detection, compute physical size from bounding box + GSD	No detection's physical size exceeds the MaxSizeM defined for its class in classes.json

Expected outcome: All returned detections have plausible physical dimensions for their class. Max execution time: 30s

---

FT-P-08: Async media detection returns "started" immediately

Summary: Verify that POST /detect/{media_id} returns immediately with status "started" while processing continues in background. Traces to: AC-API-3 Category: API

Preconditions:

• Engine is initialized
• Media file paths are available via config

Input data: jwt-token, test-video path in config

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect/test-media-001 with config paths and auth headers	200 OK, {"status": "started"}
2	Measure response time	Response arrives within 1s (before video processing completes)

Expected outcome: Immediate response with {"status": "started"}. Processing continues asynchronously. Max execution time: 2s (response only; processing continues in background)

---

FT-P-09: SSE streaming delivers detection events during async processing

Summary: Verify that SSE clients receive real-time detection events during async media detection. Traces to: AC-API-4, AC-API-3 Category: API

Preconditions:

• Engine is initialized
• SSE client connected before triggering detection

Input data: jwt-token, test-video path in config

Steps:

Step	Consumer Action	Expected System Response
1	Open SSE connection: GET /detect/stream	Connection established
2	POST /detect/test-media-002 with config and auth headers	{"status": "started"}
3	Listen on SSE connection	Receive events with mediaStatus: "AIProcessing" as frames are processed
4	Wait for completion	Final event with mediaStatus: "AIProcessed" and percent: 100

Expected outcome: Multiple SSE events received. Events include detection data. Final event signals completion. Max execution time: 120s

---

FT-P-10: Video frame sampling processes every Nth frame

Summary: Verify that video processing respects the frame_period_recognition setting. Traces to: AC-VP-1 Category: Video Processing

Preconditions:

• Engine is initialized
• SSE client connected

Input data: test-video (10s, 30fps = 300 frames), config {"frame_period_recognition": 4}

Steps:

Step	Consumer Action	Expected System Response
1	Open SSE connection	Connection established
2	POST /detect/test-media-003 with config {"frame_period_recognition": 4, "paths": ["/media/test-video.mp4"]}	{"status": "started"}
3	Count distinct SSE events with detection data	Number of processed frames ≈ 300/4 = 75 (±10% tolerance for start/end frames)

Expected outcome: Approximately 75 frames processed (not all 300). The count scales proportionally with frame_period_recognition. Max execution time: 120s

---

FT-P-11: Video annotation interval enforcement

Summary: Verify that annotations are not reported more frequently than frame_recognition_seconds. Traces to: AC-VP-2 Category: Video Processing

Preconditions:

• Engine is initialized
• SSE client connected

Input data: test-video, config {"frame_recognition_seconds": 2}

Steps:

Step	Consumer Action	Expected System Response
1	Open SSE connection	Connection established
2	POST /detect/test-media-004 with config {"frame_recognition_seconds": 2, "paths": ["/media/test-video.mp4"]}	{"status": "started"}
3	Record timestamps of consecutive SSE detection events	Minimum gap between consecutive annotation events ≥ 2 seconds

Expected outcome: No two annotation events are closer than 2 seconds apart. Max execution time: 120s

---

FT-P-12: Video tracking accepts new annotations on movement

Summary: Verify that new annotations are accepted when detections move beyond the tracking threshold. Traces to: AC-VP-3 Category: Video Processing

Preconditions:

• Engine is initialized
• SSE client connected
• Video contains moving objects

Input data: test-video, config with tracking_distance_confidence > 0

Steps:

Step	Consumer Action	Expected System Response
1	Open SSE connection	Connection established
2	POST /detect/test-media-005 with config {"tracking_distance_confidence": 0.05, "paths": ["/media/test-video.mp4"]}	{"status": "started"}
3	Collect SSE events	Annotations are emitted when object positions change between frames

Expected outcome: Annotations contain updated positions reflecting object movement. Static objects do not generate redundant annotations. Max execution time: 120s

---

FT-P-13: Weather mode class variants

Summary: Verify that the system supports detection across different weather mode class variants (Norm, Wint, Night). Traces to: AC-OC-1 Category: Object Classes

Preconditions:

• Engine is initialized
• classes.json includes weather-mode variants

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image	200 OK
2	Inspect returned detection labels	Labels correspond to valid class names from classes.json (base or weather-variant)

Expected outcome: All returned labels are valid entries from the 19-class × 3-mode registry. Max execution time: 30s

---

FT-P-14: Engine lazy initialization on first detection request

Summary: Verify that the engine is not initialized at startup but is initialized on the first detection request. Traces to: AC-EL-1, AC-EL-2 Category: Engine Lifecycle

Preconditions:

• Fresh service start, no prior requests

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	GET /health immediately after service starts	aiAvailability: "None" — engine not loaded
2	POST /detect with small-image	200 OK (may take longer — engine initializing)
3	GET /health	aiAvailability changed to "Enabled" or status indicating engine is active

Expected outcome: Engine transitions from "None" to an active state only after a detection request. Max execution time: 60s

---

FT-P-15: ONNX fallback when GPU unavailable

Summary: Verify that the system falls back to ONNX Runtime when no compatible GPU is available. Traces to: AC-EL-2, RESTRICT-HW-1 Category: Engine Lifecycle

Preconditions:

• Detections service running WITHOUT GPU runtime (CPU-only Docker profile)
• Mock-loader serves ONNX model

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with small-image	200 OK with detection results
2	GET /health	aiAvailability indicates engine is active (ONNX fallback)

Expected outcome: Detection succeeds via ONNX Runtime. No TensorRT-related errors. Max execution time: 60s

---

FT-P-16: Tile deduplication removes duplicate detections at tile boundaries

Summary: Verify that detections appearing in overlapping tile regions are deduplicated. Traces to: AC-DA-3 Category: Detection Accuracy

Preconditions:

• Engine is initialized
• Large image that triggers tiling

Input data: large-image with config including GSD parameters and big_image_tile_overlap_percent: 20

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with large-image and tiling config	200 OK
2	Inspect detections near tile boundaries	No two detections of the same class are within 0.01 coordinate difference of each other (TILE_DUPLICATE_CONFIDENCE_THRESHOLD)

Expected outcome: Tile boundary detections are merged. No duplicates with near-identical coordinates remain. Max execution time: 60s

---

Negative Scenarios

FT-N-01: Empty image returns 400

Summary: Verify that submitting an empty file to POST /detect returns a 400 error. Traces to: AC-API-2 (negative case) Category: API

Preconditions:

• Detections service is running

Input data: empty-image (zero-byte file)

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with empty-image as multipart file	400 Bad Request

Expected outcome: HTTP 400 with error message indicating empty or invalid image. Max execution time: 5s

---

FT-N-02: Invalid image data returns 400

Summary: Verify that submitting a corrupt/non-image file returns a 400 error. Traces to: AC-API-2 (negative case) Category: API

Preconditions:

• Detections service is running

Input data: corrupt-image (random binary data)

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect with corrupt-image as multipart file	400 Bad Request

Expected outcome: HTTP 400. Image decoding fails gracefully with an error response (not a 500). Max execution time: 5s

---

FT-N-03: Detection when engine unavailable returns 503

Summary: Verify that a detection request returns 503 when the engine cannot be initialized. Traces to: AC-API-2 (negative case), AC-EL-2 Category: API, Engine Lifecycle

Preconditions:

• Mock-loader configured to return errors (model download fails)
• Engine has not been previously initialized

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	Configure mock-loader to return 503 on model requests	—
2	POST /detect with small-image	503 Service Unavailable or 422

Expected outcome: HTTP 503 or 422 error indicating engine is not available. No crash or unhandled exception. Max execution time: 30s

---

FT-N-04: Duplicate media_id returns 409

Summary: Verify that submitting a second async detection request with an already-active media_id returns 409. Traces to: AC-API-3 (negative case) Category: API

Preconditions:

• Engine is initialized
• An async detection is already in progress for media_id "dup-test"

Input data: jwt-token, test-video

Steps:

Step	Consumer Action	Expected System Response
1	POST /detect/dup-test with config and auth headers	{"status": "started"}
2	Immediately POST /detect/dup-test again (same media_id)	409 Conflict

Expected outcome: Second request is rejected with 409. First detection continues normally. Max execution time: 5s

---

FT-N-05: Missing classes.json prevents startup

Summary: Verify that the service fails or returns no detections when classes.json is not present. Traces to: RESTRICT-SW-4 Category: Restrictions

Preconditions:

• Detections service started WITHOUT classes.json volume mount

Input data: None

Steps:

Step	Consumer Action	Expected System Response
1	Attempt to start detections service without classes.json	Service fails to start OR starts with empty class registry
2	If started: POST /detect with small-image	Empty detections or error response

Expected outcome: Service either fails to start or returns no detections. No unhandled crash. Max execution time: 30s

---

FT-N-06: Loader service unreachable during model download

Summary: Verify that the system handles Loader service being unreachable during engine initialization. Traces to: RESTRICT-ENV-1, AC-EL-2 Category: Resilience, Engine Lifecycle

Preconditions:

• Mock-loader is stopped or unreachable
• Engine not yet initialized

Input data: small-image

Steps:

Step	Consumer Action	Expected System Response
1	Stop mock-loader service	—
2	POST /detect with small-image	Error response (503 or 422)
3	GET /health	aiAvailability reflects error state

Expected outcome: Detection fails gracefully. Health endpoint reflects the engine error state. Max execution time: 30s

---

FT-N-07: Annotations service unreachable — detection continues

Summary: Verify that async detection continues even when the Annotations service is unreachable. Traces to: RESTRICT-ENV-2 Category: Resilience

Preconditions:

• Engine is initialized
• Mock-annotations is stopped or returns errors
• SSE client connected

Input data: jwt-token, test-video

Steps:

Step	Consumer Action	Expected System Response
1	Stop mock-annotations service	—
2	POST /detect/test-media-006 with config and auth	{"status": "started"}
3	Listen on SSE	Detection events still arrive (annotations POST failure is silently caught)
4	Wait for completion	Final AIProcessed event received

Expected outcome: Detection processing completes. SSE events are delivered. Annotations POST failure does not stop the detection pipeline. Max execution time: 120s

---

FT-N-08: SSE queue overflow is silently dropped

Summary: Verify that when an SSE client's queue reaches 100 events, additional events are dropped without error. Traces to: AC-API-4 Category: API

Preconditions:

• Engine is initialized
• SSE client connected but NOT consuming events (stalled reader)

Input data: test-video (generates many events)

Steps:

Step	Consumer Action	Expected System Response
1	Open SSE connection but pause reading	Connection established
2	POST /detect/test-media-007 with config that generates > 100 events	{"status": "started"}
3	Wait for processing to complete	No error on the detection side
4	Resume reading SSE	Receive ≤ 100 events (queue max depth)

Expected outcome: No crash or error. Overflow events are silently dropped. Detection completes normally. Max execution time: 120s