mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 15:16:36 +00:00
Refactor constants management to use Pydantic BaseModel for configuration
- Replaced module-level path variables in constants.py with a structured Pydantic Config class. - Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure. - Fixed bugs related to image processing and model saving. - Enhanced test infrastructure to accommodate the new configuration approach. This refactor improves code maintainability and clarity by centralizing configuration management.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,285 @@
|
||||
# Blackbox Test Scenarios
|
||||
|
||||
## BT-AUG: Augmentation Pipeline
|
||||
|
||||
### BT-AUG-01: Single image produces 8 outputs
|
||||
- **Input**: 1 image + 1 valid label from fixture dataset
|
||||
- **Action**: Run `Augmentator.augment_inner()` on the image
|
||||
- **Expected**: Returns list of exactly 8 ImageLabel objects
|
||||
- **Traces**: AC: 8× augmentation ratio
|
||||
|
||||
### BT-AUG-02: Augmented filenames follow naming convention
|
||||
- **Input**: Image with stem "test_image"
|
||||
- **Action**: Run `augment_inner()`
|
||||
- **Expected**: Output filenames: `test_image.jpg`, `test_image_1.jpg` through `test_image_7.jpg`; matching `.txt` labels
|
||||
- **Traces**: AC: Augmentation output format
|
||||
|
||||
### BT-AUG-03: All output bounding boxes in valid range
|
||||
- **Input**: 1 image + label with multiple bboxes
|
||||
- **Action**: Run `augment_inner()`
|
||||
- **Expected**: Every bbox coordinate in every output label is in [0.0, 1.0]
|
||||
- **Traces**: AC: Bounding boxes clipped to [0, 1]
|
||||
|
||||
### BT-AUG-04: Bounding box correction clips edge bboxes
|
||||
- **Input**: Label with bbox near edge: `0 0.99 0.5 0.2 0.1`
|
||||
- **Action**: Run `correct_bboxes()`
|
||||
- **Expected**: Width reduced so bbox fits within [margin, 1-margin]; no coordinate exceeds bounds
|
||||
- **Traces**: AC: Bounding boxes clipped to [0, 1]
|
||||
|
||||
### BT-AUG-05: Tiny bounding boxes removed after correction
|
||||
- **Input**: Label with tiny bbox that becomes < 0.01 after clipping
|
||||
- **Action**: Run `correct_bboxes()`
|
||||
- **Expected**: Bbox removed from output (area < correct_min_bbox_size)
|
||||
- **Traces**: AC: Bounding boxes with area < 0.01% discarded
|
||||
|
||||
### BT-AUG-06: Empty label produces 8 outputs with empty labels
|
||||
- **Input**: 1 image + empty label file
|
||||
- **Action**: Run `augment_inner()`
|
||||
- **Expected**: 8 ImageLabel objects returned; all have empty labels lists
|
||||
- **Traces**: AC: Augmentation handles empty annotations
|
||||
|
||||
### BT-AUG-07: Full augmentation pipeline (filesystem integration)
|
||||
- **Input**: 5 images + labels copied to data/ directory in tmp_path
|
||||
- **Action**: Run `augment_annotations()` with patched paths
|
||||
- **Expected**: 40 images (5 × 8) in processed images dir; 40 matching labels in processed labels dir
|
||||
- **Traces**: AC: 8× augmentation, filesystem output
|
||||
|
||||
### BT-AUG-08: Augmentation skips already-processed images
|
||||
- **Input**: 5 images in data/; 3 already present in processed/ dir
|
||||
- **Action**: Run `augment_annotations()`
|
||||
- **Expected**: Only 2 new images processed (16 new outputs); existing 3 untouched
|
||||
- **Traces**: AC: Augmentation processes only unprocessed images
|
||||
|
||||
---
|
||||
|
||||
## BT-DSF: Dataset Formation
|
||||
|
||||
### BT-DSF-01: 70/20/10 split ratio
|
||||
- **Input**: 100 images + labels in processed/ dir
|
||||
- **Action**: Run `form_dataset()` with patched paths
|
||||
- **Expected**: train: 70 images+labels, valid: 20, test: 10
|
||||
- **Traces**: AC: Dataset split 70/20/10
|
||||
|
||||
### BT-DSF-02: Split directories structure
|
||||
- **Input**: 100 images + labels
|
||||
- **Action**: Run `form_dataset()`
|
||||
- **Expected**: Created: `train/images/`, `train/labels/`, `valid/images/`, `valid/labels/`, `test/images/`, `test/labels/`
|
||||
- **Traces**: AC: YOLO dataset directory structure
|
||||
|
||||
### BT-DSF-03: Total files preserved across splits
|
||||
- **Input**: 100 valid images + labels
|
||||
- **Action**: Run `form_dataset()`
|
||||
- **Expected**: `count(train) + count(valid) + count(test) == 100` (no data loss)
|
||||
- **Traces**: AC: Dataset integrity
|
||||
|
||||
### BT-DSF-04: Corrupted labels moved to corrupted directory
|
||||
- **Input**: 95 valid + 5 corrupted labels (coords > 1.0)
|
||||
- **Action**: Run `form_dataset()` with patched paths
|
||||
- **Expected**: 5 images+labels in `data-corrupted/`; 95 across train/valid/test splits
|
||||
- **Traces**: AC: Corrupted labels filtered
|
||||
|
||||
---
|
||||
|
||||
## BT-LBL: Label Validation
|
||||
|
||||
### BT-LBL-01: Valid label accepted
|
||||
- **Input**: Label file: `0 0.5 0.5 0.1 0.1`
|
||||
- **Action**: Call `check_label(path)`
|
||||
- **Expected**: Returns `True`
|
||||
- **Traces**: AC: Valid YOLO label format
|
||||
|
||||
### BT-LBL-02: Label with x > 1.0 rejected
|
||||
- **Input**: Label file: `0 1.5 0.5 0.1 0.1`
|
||||
- **Action**: Call `check_label(path)`
|
||||
- **Expected**: Returns `False`
|
||||
- **Traces**: AC: Corrupted labels detected
|
||||
|
||||
### BT-LBL-03: Label with height > 1.0 rejected
|
||||
- **Input**: Label file: `0 0.5 0.5 0.1 1.2`
|
||||
- **Action**: Call `check_label(path)`
|
||||
- **Expected**: Returns `False`
|
||||
- **Traces**: AC: Corrupted labels detected
|
||||
|
||||
### BT-LBL-04: Missing label file rejected
|
||||
- **Input**: Non-existent file path
|
||||
- **Action**: Call `check_label(path)`
|
||||
- **Expected**: Returns `False`
|
||||
- **Traces**: AC: Missing labels handled
|
||||
|
||||
### BT-LBL-05: Multi-line label with one corrupted line
|
||||
- **Input**: Label file: `0 0.5 0.5 0.1 0.1\n3 0.5 0.5 0.1 1.5`
|
||||
- **Action**: Call `check_label(path)`
|
||||
- **Expected**: Returns `False` (any corrupted line fails the whole file)
|
||||
- **Traces**: AC: Corrupted labels detected
|
||||
|
||||
---
|
||||
|
||||
## BT-ENC: Encryption
|
||||
|
||||
### BT-ENC-01: Encrypt-decrypt roundtrip (arbitrary data)
|
||||
- **Input**: 1024 random bytes, key "test-key"
|
||||
- **Action**: `decrypt_to(encrypt_to(data, key), key)`
|
||||
- **Expected**: Output equals input bytes exactly
|
||||
- **Traces**: AC: AES-256-CBC encryption
|
||||
|
||||
### BT-ENC-02: Encrypt-decrypt roundtrip (ONNX model)
|
||||
- **Input**: `azaion.onnx` bytes, model encryption key
|
||||
- **Action**: `decrypt_to(encrypt_to(model_bytes, key), key)`
|
||||
- **Expected**: Output equals input bytes exactly
|
||||
- **Traces**: AC: Model encryption
|
||||
|
||||
### BT-ENC-03: Empty input roundtrip
|
||||
- **Input**: `b""`, key "test-key"
|
||||
- **Action**: `decrypt_to(encrypt_to(b"", key), key)`
|
||||
- **Expected**: Output equals `b""`
|
||||
- **Traces**: AC: Edge case handling
|
||||
|
||||
### BT-ENC-04: Single byte roundtrip
|
||||
- **Input**: `b"\x00"`, key "test-key"
|
||||
- **Action**: `decrypt_to(encrypt_to(b"\x00", key), key)`
|
||||
- **Expected**: Output equals `b"\x00"`
|
||||
- **Traces**: AC: Edge case handling
|
||||
|
||||
### BT-ENC-05: Different keys produce different ciphertext
|
||||
- **Input**: Same 1024 bytes, keys "key-a" and "key-b"
|
||||
- **Action**: `encrypt_to(data, "key-a")` vs `encrypt_to(data, "key-b")`
|
||||
- **Expected**: Ciphertexts differ
|
||||
- **Traces**: AC: Key-dependent encryption
|
||||
|
||||
### BT-ENC-06: Wrong key fails decryption
|
||||
- **Input**: Encrypted with "key-a", decrypt with "key-b"
|
||||
- **Action**: `decrypt_to(encrypted, "key-b")`
|
||||
- **Expected**: Output does NOT equal original input
|
||||
- **Traces**: AC: Key-dependent encryption
|
||||
|
||||
---
|
||||
|
||||
## BT-SPL: Model Split Storage
|
||||
|
||||
### BT-SPL-01: Split respects size constraint
|
||||
- **Input**: 10000 encrypted bytes
|
||||
- **Action**: Split into small + big per `SMALL_SIZE_KB = 3` logic
|
||||
- **Expected**: small ≤ max(3072 bytes, 20% of total); big = remainder
|
||||
- **Traces**: AC: Model split ≤3KB or 20%
|
||||
|
||||
### BT-SPL-02: Reassembly produces original
|
||||
- **Input**: 10000 encrypted bytes → split → reassemble
|
||||
- **Action**: `small + big`
|
||||
- **Expected**: Equals original encrypted bytes
|
||||
- **Traces**: AC: Split model integrity
|
||||
|
||||
---
|
||||
|
||||
## BT-CLS: Annotation Class Loading
|
||||
|
||||
### BT-CLS-01: Load 17 base classes
|
||||
- **Input**: `classes.json`
|
||||
- **Action**: `AnnotationClass.read_json()`
|
||||
- **Expected**: Dict with 17 unique class entries (base IDs)
|
||||
- **Traces**: AC: 17 base classes
|
||||
|
||||
### BT-CLS-02: Weather mode expansion
|
||||
- **Input**: `classes.json`
|
||||
- **Action**: `AnnotationClass.read_json()`
|
||||
- **Expected**: Same class at offset 0 (Norm), 20 (Wint), 40 (Night); e.g., ID 0, 20, 40 all represent ArmorVehicle
|
||||
- **Traces**: AC: 3 weather modes
|
||||
|
||||
### BT-CLS-03: YAML generation produces 80 class names
|
||||
- **Input**: `classes.json` + dataset path
|
||||
- **Action**: `create_yaml()` with patched paths
|
||||
- **Expected**: data.yaml contains `nc: 80`, 17 named classes + 63 `Class-N` placeholders
|
||||
- **Traces**: AC: 80 total class slots
|
||||
|
||||
---
|
||||
|
||||
## BT-HSH: Hardware Hash
|
||||
|
||||
### BT-HSH-01: Deterministic output
|
||||
- **Input**: "test-hardware-info"
|
||||
- **Action**: `Security.get_hw_hash()` called twice
|
||||
- **Expected**: Both calls return identical string
|
||||
- **Traces**: AC: Hardware fingerprinting determinism
|
||||
|
||||
### BT-HSH-02: Different inputs produce different hashes
|
||||
- **Input**: "hw-a" and "hw-b"
|
||||
- **Action**: `Security.get_hw_hash()` on each
|
||||
- **Expected**: Results differ
|
||||
- **Traces**: AC: Hardware-bound uniqueness
|
||||
|
||||
### BT-HSH-03: Output is valid base64
|
||||
- **Input**: "test-hardware-info"
|
||||
- **Action**: `Security.get_hw_hash()`
|
||||
- **Expected**: Matches regex `^[A-Za-z0-9+/]+=*$`
|
||||
- **Traces**: AC: Hash format
|
||||
|
||||
---
|
||||
|
||||
## BT-INF: ONNX Inference
|
||||
|
||||
### BT-INF-01: Model loads successfully
|
||||
- **Input**: `azaion.onnx` bytes
|
||||
- **Action**: `OnnxEngine(model_bytes)`
|
||||
- **Expected**: No exception; engine object created with valid input_shape and batch_size
|
||||
- **Traces**: AC: ONNX inference capability
|
||||
|
||||
### BT-INF-02: Inference returns output
|
||||
- **Input**: ONNX engine + 1 preprocessed image
|
||||
- **Action**: `engine.run(input_blob)`
|
||||
- **Expected**: Returns list of numpy arrays; first array has shape [batch, N, 6+]
|
||||
- **Traces**: AC: ONNX inference produces results
|
||||
|
||||
### BT-INF-03: Postprocessing returns valid detections
|
||||
- **Input**: ONNX engine output from real image
|
||||
- **Action**: `Inference.postprocess()`
|
||||
- **Expected**: Returns list of Annotation objects; each Detection has x,y,w,h ∈ [0,1], cls ∈ [0,79], confidence ∈ [0,1]
|
||||
- **Traces**: AC: Detection format validity
|
||||
|
||||
---
|
||||
|
||||
## BT-NMS: Overlap Removal
|
||||
|
||||
### BT-NMS-01: Overlapping detections — keep higher confidence
|
||||
- **Input**: 2 Detection objects at same position, confidence 0.9 and 0.5, IoU > 0.3
|
||||
- **Action**: `remove_overlapping_detections()`
|
||||
- **Expected**: 1 detection returned (confidence 0.9)
|
||||
- **Traces**: AC: NMS IoU threshold 0.3
|
||||
|
||||
### BT-NMS-02: Non-overlapping detections — keep both
|
||||
- **Input**: 2 Detection objects at distant positions, IoU < 0.3
|
||||
- **Action**: `remove_overlapping_detections()`
|
||||
- **Expected**: 2 detections returned
|
||||
- **Traces**: AC: NMS preserves non-overlapping
|
||||
|
||||
### BT-NMS-03: Chain overlap resolution
|
||||
- **Input**: 3 Detection objects: A overlaps B (IoU > 0.3), B overlaps C (IoU > 0.3), A doesn't overlap C
|
||||
- **Action**: `remove_overlapping_detections()`
|
||||
- **Expected**: ≤ 2 detections; highest confidence per overlapping pair kept
|
||||
- **Traces**: AC: NMS handles chains
|
||||
|
||||
---
|
||||
|
||||
## BT-AQM: Annotation Queue Message Parsing
|
||||
|
||||
### BT-AQM-01: Parse Created annotation message
|
||||
- **Input**: Msgpack bytes matching AnnotationMessage schema (status=Created, role=Validator)
|
||||
- **Action**: Decode and construct AnnotationMessage
|
||||
- **Expected**: All fields populated: name, detections, image bytes, status == "Created", role == "Validator"
|
||||
- **Traces**: AC: Annotation message parsing
|
||||
|
||||
### BT-AQM-02: Parse Validated bulk message
|
||||
- **Input**: Msgpack bytes with status=Validated, list of names
|
||||
- **Action**: Decode and construct AnnotationBulkMessage
|
||||
- **Expected**: Status == "Validated", names list matches input
|
||||
- **Traces**: AC: Bulk validation parsing
|
||||
|
||||
### BT-AQM-03: Parse Deleted bulk message
|
||||
- **Input**: Msgpack bytes with status=Deleted, list of names
|
||||
- **Action**: Decode and construct AnnotationBulkMessage
|
||||
- **Expected**: Status == "Deleted", names list matches input
|
||||
- **Traces**: AC: Bulk deletion parsing
|
||||
|
||||
### BT-AQM-04: Malformed message raises exception
|
||||
- **Input**: Invalid msgpack bytes
|
||||
- **Action**: Attempt to decode
|
||||
- **Expected**: Exception raised
|
||||
- **Traces**: AC: Error handling for malformed messages
|
||||
@@ -0,0 +1,71 @@
|
||||
# Test Environment
|
||||
|
||||
## Runtime Requirements
|
||||
|
||||
| Requirement | Specification |
|
||||
|-------------|--------------|
|
||||
| Python | 3.10+ |
|
||||
| OS | Linux or macOS (POSIX filesystem paths) |
|
||||
| GPU | Optional — ONNX inference falls back to CPUExecutionProvider |
|
||||
| Disk | Temp directory for fixture data (~500MB for augmentation output) |
|
||||
| Network | Not required (all tests are offline) |
|
||||
|
||||
## Execution Modes
|
||||
|
||||
Tests MUST be runnable in two ways:
|
||||
|
||||
### 1. Local (no Docker) — primary mode
|
||||
Run directly on the host machine. Required for macOS development where Docker has GPU/performance limitations.
|
||||
|
||||
```bash
|
||||
scripts/run-tests-local.sh
|
||||
```
|
||||
|
||||
### 2. Docker — CI/portable mode
|
||||
Run inside a container for reproducible CI environments (Linux-based CI runners).
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.test.yml up --build --abort-on-container-exit
|
||||
```
|
||||
|
||||
Both modes run the same pytest suite; the only difference is the runtime environment.
|
||||
|
||||
## Dependencies
|
||||
|
||||
All test dependencies are a subset of the production `requirements.txt` plus pytest:
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| pytest | Test runner |
|
||||
| albumentations | Augmentation tests |
|
||||
| opencv-python-headless | Image I/O (headless — no GUI) |
|
||||
| numpy | Array operations |
|
||||
| onnxruntime | ONNX inference (CPU fallback) |
|
||||
| cryptography | Encryption tests |
|
||||
| msgpack | Annotation queue message tests |
|
||||
| PyYAML | Config/YAML generation tests |
|
||||
|
||||
## Fixture Data
|
||||
|
||||
| Fixture | Location | Size |
|
||||
|---------|----------|------|
|
||||
| 100 annotated images | `_docs/00_problem/input_data/dataset/images/` | ~50MB |
|
||||
| 100 YOLO labels | `_docs/00_problem/input_data/dataset/labels/` | ~10KB |
|
||||
| ONNX model | `_docs/00_problem/input_data/azaion.onnx` | 81MB |
|
||||
| Class definitions | `classes.json` (project root) | 2KB |
|
||||
|
||||
## Test Isolation
|
||||
|
||||
- Each test creates a temporary directory (via `tmp_path` pytest fixture) for filesystem operations
|
||||
- No tests modify the actual `/azaion/` directory structure
|
||||
- No tests require running external services (RabbitMQ, Azaion API, S3 CDN)
|
||||
- Constants paths are patched/overridden to point to temp directories during tests
|
||||
|
||||
## Excluded (Require External Services)
|
||||
|
||||
| Component | Service Required | Reason for Exclusion |
|
||||
|-----------|-----------------|---------------------|
|
||||
| API upload/download | Azaion REST API | No mock server; real API has auth |
|
||||
| CDN upload/download | S3-compatible CDN | No mock S3; real CDN has credentials |
|
||||
| Queue consumption | RabbitMQ Streams | No mock broker; rstream requires live connection |
|
||||
| TensorRT inference | NVIDIA GPU + TensorRT | Hardware-specific; cannot run in CI without GPU |
|
||||
@@ -0,0 +1,33 @@
|
||||
# Performance Test Scenarios
|
||||
|
||||
## PT-AUG-01: Augmentation throughput
|
||||
- **Input**: 10 images from fixture dataset
|
||||
- **Action**: Run `augment_annotations()`, measure wall time
|
||||
- **Expected**: Completes within 60 seconds (10 images × 8 outputs = 80 files)
|
||||
- **Traces**: Restriction: Augmentation runs continuously
|
||||
- **Note**: Threshold is generous; actual performance depends on CPU
|
||||
|
||||
## PT-AUG-02: Parallel augmentation speedup
|
||||
- **Input**: 10 images from fixture dataset
|
||||
- **Action**: Run with ThreadPoolExecutor vs sequential, compare times
|
||||
- **Expected**: Parallel is ≥ 1.5× faster than sequential
|
||||
- **Traces**: AC: Parallelized per-image processing
|
||||
|
||||
## PT-DSF-01: Dataset formation throughput
|
||||
- **Input**: 100 images + labels
|
||||
- **Action**: Run `form_dataset()`, measure wall time
|
||||
- **Expected**: Completes within 30 seconds
|
||||
- **Traces**: Restriction: Dataset formation before training
|
||||
|
||||
## PT-ENC-01: Encryption throughput
|
||||
- **Input**: 10MB random bytes
|
||||
- **Action**: Encrypt + decrypt roundtrip, measure wall time
|
||||
- **Expected**: Completes within 5 seconds
|
||||
- **Traces**: AC: Model encryption feasible for large models
|
||||
|
||||
## PT-INF-01: ONNX inference latency (single image)
|
||||
- **Input**: 1 preprocessed image + ONNX model
|
||||
- **Action**: Run single inference, measure wall time
|
||||
- **Expected**: Completes within 10 seconds on CPU (no GPU requirement for test)
|
||||
- **Traces**: AC: Inference capability
|
||||
- **Note**: Production uses GPU; CPU is slower but validates correctness
|
||||
@@ -0,0 +1,37 @@
|
||||
# Resilience Test Scenarios
|
||||
|
||||
## RT-AUG-01: Augmentation handles corrupted image gracefully
|
||||
- **Input**: 1 valid image + 1 corrupted image file (truncated JPEG) in data/ dir
|
||||
- **Action**: Run `augment_annotations()`
|
||||
- **Expected**: Valid image produces 8 outputs; corrupted image skipped without crashing pipeline; total output: 8 files
|
||||
- **Traces**: Restriction: Augmentation exception handling per-image
|
||||
|
||||
## RT-AUG-02: Augmentation handles missing label file
|
||||
- **Input**: 1 image with no matching label file
|
||||
- **Action**: Run `augment_annotation()` on the image
|
||||
- **Expected**: Exception caught per-thread; does not crash pipeline
|
||||
- **Traces**: Restriction: Augmentation exception handling
|
||||
|
||||
## RT-AUG-03: Augmentation transform failure produces fewer variants
|
||||
- **Input**: 1 image + label that causes some transforms to fail (extremely narrow bbox)
|
||||
- **Action**: Run `augment_inner()`
|
||||
- **Expected**: Returns 1-8 ImageLabel objects (original always present; failed variants skipped); no crash
|
||||
- **Traces**: Restriction: Transform failure handling
|
||||
|
||||
## RT-DSF-01: Dataset formation with empty processed directory
|
||||
- **Input**: Empty processed images dir
|
||||
- **Action**: Run `form_dataset()`
|
||||
- **Expected**: Creates empty train/valid/test directories; no crash
|
||||
- **Traces**: Restriction: Edge case handling
|
||||
|
||||
## RT-ENC-01: Decrypt with corrupted ciphertext
|
||||
- **Input**: Randomly modified ciphertext bytes
|
||||
- **Action**: `Security.decrypt_to(corrupted_bytes, key)`
|
||||
- **Expected**: Either raises exception or returns garbage bytes (not original)
|
||||
- **Traces**: AC: Encryption integrity
|
||||
|
||||
## RT-AQM-01: Malformed msgpack message
|
||||
- **Input**: Random bytes that aren't valid msgpack
|
||||
- **Action**: Pass to message handler
|
||||
- **Expected**: Exception caught; handler doesn't crash
|
||||
- **Traces**: AC: Error handling for malformed messages
|
||||
@@ -0,0 +1,31 @@
|
||||
# Resource Limit Test Scenarios
|
||||
|
||||
## RL-AUG-01: Augmentation output count bounded
|
||||
- **Input**: 1 image
|
||||
- **Action**: Run `augment_inner()`
|
||||
- **Expected**: Returns exactly 8 outputs (never more, even with retries)
|
||||
- **Traces**: AC: 8× augmentation ratio (1 original + 7 augmented)
|
||||
|
||||
## RL-DSF-01: Dataset split ratios sum to 100%
|
||||
- **Input**: Any number of images
|
||||
- **Action**: Check `train_set + valid_set + test_set`
|
||||
- **Expected**: Equals 100
|
||||
- **Traces**: AC: 70/20/10 split
|
||||
|
||||
## RL-DSF-02: No data duplication across splits
|
||||
- **Input**: 100 images
|
||||
- **Action**: Run `form_dataset()`, collect all filenames across train/valid/test
|
||||
- **Expected**: No filename appears in more than one split
|
||||
- **Traces**: AC: Dataset integrity
|
||||
|
||||
## RL-ENC-01: Encrypted output size bounded
|
||||
- **Input**: N bytes plaintext
|
||||
- **Action**: Encrypt
|
||||
- **Expected**: Ciphertext size ≤ N + 32 bytes (16 IV + up to 16 padding)
|
||||
- **Traces**: Restriction: AES-256-CBC overhead
|
||||
|
||||
## RL-CLS-01: Total class count is exactly 80
|
||||
- **Input**: `classes.json`
|
||||
- **Action**: Generate class list for YAML
|
||||
- **Expected**: Exactly 80 entries (17 named × 3 weather + 29 placeholders = 80)
|
||||
- **Traces**: AC: 80 total class slots
|
||||
@@ -0,0 +1,43 @@
|
||||
# Security Test Scenarios
|
||||
|
||||
## ST-ENC-01: Encryption produces different ciphertext each time (random IV)
|
||||
- **Input**: Same 1024 bytes, same key, encrypt twice
|
||||
- **Action**: Compare two ciphertexts
|
||||
- **Expected**: Ciphertexts differ (random IV ensures non-deterministic output)
|
||||
- **Traces**: AC: AES-256-CBC with random IV
|
||||
|
||||
## ST-ENC-02: Wrong key cannot recover plaintext
|
||||
- **Input**: Encrypt with "key-a", attempt decrypt with "key-b"
|
||||
- **Action**: `Security.decrypt_to(encrypted, "key-b")`
|
||||
- **Expected**: Output != original plaintext
|
||||
- **Traces**: AC: Key-dependent encryption
|
||||
|
||||
## ST-ENC-03: Model encryption key is deterministic
|
||||
- **Input**: Call `Security.get_model_encryption_key()` twice
|
||||
- **Action**: Compare results
|
||||
- **Expected**: Identical strings
|
||||
- **Traces**: AC: Static model encryption key
|
||||
|
||||
## ST-HSH-01: Hardware hash is deterministic for same input
|
||||
- **Input**: Same hardware info string
|
||||
- **Action**: `Security.get_hw_hash()` called twice
|
||||
- **Expected**: Identical output
|
||||
- **Traces**: AC: Hardware fingerprinting determinism
|
||||
|
||||
## ST-HSH-02: Different hardware produces different hash
|
||||
- **Input**: Two different hardware info strings
|
||||
- **Action**: `Security.get_hw_hash()` on each
|
||||
- **Expected**: Different outputs
|
||||
- **Traces**: AC: Hardware-bound uniqueness
|
||||
|
||||
## ST-HSH-03: API encryption key depends on credentials + hardware
|
||||
- **Input**: Same credentials with different hardware hashes
|
||||
- **Action**: `Security.get_api_encryption_key()` for each
|
||||
- **Expected**: Different keys
|
||||
- **Traces**: AC: Hardware-bound API encryption
|
||||
|
||||
## ST-HSH-04: API encryption key depends on credentials
|
||||
- **Input**: Different credentials with same hardware hash
|
||||
- **Action**: `Security.get_api_encryption_key()` for each
|
||||
- **Expected**: Different keys
|
||||
- **Traces**: AC: Credential-dependent API encryption
|
||||
@@ -0,0 +1,26 @@
|
||||
# Test Data Management
|
||||
|
||||
## Fixture Sources
|
||||
|
||||
| ID | Data Item | Source | Format | Preparation |
|
||||
|----|-----------|--------|--------|-------------|
|
||||
| FD-01 | Annotated images (100) | `_docs/00_problem/input_data/dataset/images/` | JPEG | Copy subset to tmp_path at test start |
|
||||
| FD-02 | YOLO labels (100) | `_docs/00_problem/input_data/dataset/labels/` | TXT | Copy subset to tmp_path at test start |
|
||||
| FD-03 | ONNX model | `_docs/00_problem/input_data/azaion.onnx` | ONNX | Read bytes at test start |
|
||||
| FD-04 | Class definitions | `classes.json` (project root) | JSON | Copy to tmp_path at test start |
|
||||
| FD-05 | Corrupted labels | Generated at test time | TXT | Create labels with coords > 1.0 |
|
||||
| FD-06 | Edge-case bboxes | Generated at test time | In-memory | Construct bboxes near image boundaries |
|
||||
| FD-07 | Detection objects | Generated at test time | In-memory | Construct Detection instances for NMS tests |
|
||||
| FD-08 | Msgpack messages | Generated at test time | bytes | Construct AnnotationMessage-compatible msgpack |
|
||||
| FD-09 | Random binary data | Generated at test time | bytes | `os.urandom(N)` for encryption tests |
|
||||
| FD-10 | Empty label file | Generated at test time | TXT | Empty file for augmentation edge case |
|
||||
|
||||
## Data Lifecycle
|
||||
|
||||
1. **Setup**: pytest `conftest.py` copies fixture files to `tmp_path`
|
||||
2. **Execution**: Tests operate on copied data in isolation
|
||||
3. **Teardown**: `tmp_path` is automatically cleaned by pytest
|
||||
|
||||
## Expected Results Location
|
||||
|
||||
All expected results are defined in `_docs/00_problem/input_data/expected_results/results_report.md` (37 test scenarios mapped).
|
||||
@@ -0,0 +1,67 @@
|
||||
# Traceability Matrix
|
||||
|
||||
## Acceptance Criteria Coverage
|
||||
|
||||
| AC / Restriction | Test IDs | Coverage |
|
||||
|------------------|----------|----------|
|
||||
| 8× augmentation ratio | BT-AUG-01, BT-AUG-06, BT-AUG-07, RL-AUG-01 | Full |
|
||||
| Augmentation naming convention | BT-AUG-02 | Full |
|
||||
| Bounding boxes clipped to [0,1] | BT-AUG-03, BT-AUG-04 | Full |
|
||||
| Tiny bboxes (< 0.01) discarded | BT-AUG-05 | Full |
|
||||
| Augmentation skips already-processed | BT-AUG-08 | Full |
|
||||
| Augmentation parallelized | PT-AUG-02 | Full |
|
||||
| Augmentation handles corrupted images | RT-AUG-01 | Full |
|
||||
| Augmentation handles missing labels | RT-AUG-02 | Full |
|
||||
| Transform failure graceful | RT-AUG-03 | Full |
|
||||
| Dataset split 70/20/10 | BT-DSF-01, RL-DSF-01 | Full |
|
||||
| Dataset directory structure | BT-DSF-02 | Full |
|
||||
| Dataset integrity (no data loss) | BT-DSF-03, RL-DSF-02 | Full |
|
||||
| Corrupted label filtering | BT-DSF-04, BT-LBL-01 to BT-LBL-05 | Full |
|
||||
| AES-256-CBC encryption | BT-ENC-01 to BT-ENC-06, ST-ENC-01, ST-ENC-02 | Full |
|
||||
| Model encryption roundtrip | BT-ENC-02 | Full |
|
||||
| Model split ≤3KB or 20% | BT-SPL-01, BT-SPL-02 | Full |
|
||||
| 17 base classes | BT-CLS-01 | Full |
|
||||
| 3 weather modes (Norm/Wint/Night) | BT-CLS-02 | Full |
|
||||
| 80 total class slots | BT-CLS-03, RL-CLS-01 | Full |
|
||||
| YAML generation (nc: 80) | BT-CLS-03 | Full |
|
||||
| Hardware hash determinism | BT-HSH-01 to BT-HSH-03, ST-HSH-01, ST-HSH-02 | Full |
|
||||
| Hardware-bound API encryption | ST-HSH-03, ST-HSH-04 | Full |
|
||||
| ONNX inference loads model | BT-INF-01 | Full |
|
||||
| ONNX inference returns detections | BT-INF-02, BT-INF-03 | Full |
|
||||
| NMS overlap removal (IoU 0.3) | BT-NMS-01, BT-NMS-02, BT-NMS-03 | Full |
|
||||
| Annotation message parsing | BT-AQM-01 to BT-AQM-04, RT-AQM-01 | Full |
|
||||
| Encryption size overhead bounded | RL-ENC-01 | Full |
|
||||
| Static model encryption key | ST-ENC-03 | Full |
|
||||
| Random IV per encryption | ST-ENC-01 | Full |
|
||||
|
||||
## Uncovered (Require External Services)
|
||||
|
||||
| AC / Restriction | Reason |
|
||||
|------------------|--------|
|
||||
| TensorRT inference (54s for 200s video) | Requires NVIDIA GPU + TensorRT runtime |
|
||||
| API upload/download with JWT auth | Requires live Azaion API |
|
||||
| CDN upload/download (S3) | Requires live S3-compatible CDN |
|
||||
| Queue offset persistence | Requires live RabbitMQ Streams |
|
||||
| Auto-relogin on 401/403 | Requires live Azaion API |
|
||||
| Frame sampling every 4th frame | Requires video file (fixture not provided) |
|
||||
| Confidence threshold 0.3 filtering | Partially covered by BT-INF-03 (validates range, not exact threshold) |
|
||||
|
||||
## Summary
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total AC + Restrictions | 36 |
|
||||
| Covered by tests | 29 |
|
||||
| Uncovered (external deps) | 7 |
|
||||
| **Coverage** | **80.6%** |
|
||||
|
||||
## Test Count Summary
|
||||
|
||||
| Category | Count |
|
||||
|----------|-------|
|
||||
| Blackbox tests | 32 |
|
||||
| Performance tests | 5 |
|
||||
| Resilience tests | 6 |
|
||||
| Security tests | 7 |
|
||||
| Resource limit tests | 5 |
|
||||
| **Total** | **55** |
|
||||
Reference in New Issue
Block a user