mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 08:56:35 +00:00
Oleksandr Bezdieniezhnykh
142c6c4de8
- Replaced module-level path variables in constants.py with a structured Pydantic Config class. - Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure. - Fixed bugs related to image processing and model saving. - Enhanced test infrastructure to accommodate the new configuration approach. This refactor improves code maintainability and clarity by centralizing configuration management.
286 lines
11 KiB
Markdown
286 lines
11 KiB
Markdown
# Blackbox Test Scenarios
|
||
|
||
## BT-AUG: Augmentation Pipeline
|
||
|
||
### BT-AUG-01: Single image produces 8 outputs
|
||
- **Input**: 1 image + 1 valid label from fixture dataset
|
||
- **Action**: Run `Augmentator.augment_inner()` on the image
|
||
- **Expected**: Returns list of exactly 8 ImageLabel objects
|
||
- **Traces**: AC: 8× augmentation ratio
|
||
|
||
### BT-AUG-02: Augmented filenames follow naming convention
|
||
- **Input**: Image with stem "test_image"
|
||
- **Action**: Run `augment_inner()`
|
||
- **Expected**: Output filenames: `test_image.jpg`, `test_image_1.jpg` through `test_image_7.jpg`; matching `.txt` labels
|
||
- **Traces**: AC: Augmentation output format
|
||
|
||
### BT-AUG-03: All output bounding boxes in valid range
|
||
- **Input**: 1 image + label with multiple bboxes
|
||
- **Action**: Run `augment_inner()`
|
||
- **Expected**: Every bbox coordinate in every output label is in [0.0, 1.0]
|
||
- **Traces**: AC: Bounding boxes clipped to [0, 1]
|
||
|
||
### BT-AUG-04: Bounding box correction clips edge bboxes
|
||
- **Input**: Label with bbox near edge: `0 0.99 0.5 0.2 0.1`
|
||
- **Action**: Run `correct_bboxes()`
|
||
- **Expected**: Width reduced so bbox fits within [margin, 1-margin]; no coordinate exceeds bounds
|
||
- **Traces**: AC: Bounding boxes clipped to [0, 1]
|
||
|
||
### BT-AUG-05: Tiny bounding boxes removed after correction
|
||
- **Input**: Label with tiny bbox that becomes < 0.01 after clipping
|
||
- **Action**: Run `correct_bboxes()`
|
||
- **Expected**: Bbox removed from output (area < correct_min_bbox_size)
|
||
- **Traces**: AC: Bounding boxes with area < 0.01% discarded
|
||
|
||
### BT-AUG-06: Empty label produces 8 outputs with empty labels
|
||
- **Input**: 1 image + empty label file
|
||
- **Action**: Run `augment_inner()`
|
||
- **Expected**: 8 ImageLabel objects returned; all have empty labels lists
|
||
- **Traces**: AC: Augmentation handles empty annotations
|
||
|
||
### BT-AUG-07: Full augmentation pipeline (filesystem integration)
|
||
- **Input**: 5 images + labels copied to data/ directory in tmp_path
|
||
- **Action**: Run `augment_annotations()` with patched paths
|
||
- **Expected**: 40 images (5 × 8) in processed images dir; 40 matching labels in processed labels dir
|
||
- **Traces**: AC: 8× augmentation, filesystem output
|
||
|
||
### BT-AUG-08: Augmentation skips already-processed images
|
||
- **Input**: 5 images in data/; 3 already present in processed/ dir
|
||
- **Action**: Run `augment_annotations()`
|
||
- **Expected**: Only 2 new images processed (16 new outputs); existing 3 untouched
|
||
- **Traces**: AC: Augmentation processes only unprocessed images
|
||
|
||
---
|
||
|
||
## BT-DSF: Dataset Formation
|
||
|
||
### BT-DSF-01: 70/20/10 split ratio
|
||
- **Input**: 100 images + labels in processed/ dir
|
||
- **Action**: Run `form_dataset()` with patched paths
|
||
- **Expected**: train: 70 images+labels, valid: 20, test: 10
|
||
- **Traces**: AC: Dataset split 70/20/10
|
||
|
||
### BT-DSF-02: Split directories structure
|
||
- **Input**: 100 images + labels
|
||
- **Action**: Run `form_dataset()`
|
||
- **Expected**: Created: `train/images/`, `train/labels/`, `valid/images/`, `valid/labels/`, `test/images/`, `test/labels/`
|
||
- **Traces**: AC: YOLO dataset directory structure
|
||
|
||
### BT-DSF-03: Total files preserved across splits
|
||
- **Input**: 100 valid images + labels
|
||
- **Action**: Run `form_dataset()`
|
||
- **Expected**: `count(train) + count(valid) + count(test) == 100` (no data loss)
|
||
- **Traces**: AC: Dataset integrity
|
||
|
||
### BT-DSF-04: Corrupted labels moved to corrupted directory
|
||
- **Input**: 95 valid + 5 corrupted labels (coords > 1.0)
|
||
- **Action**: Run `form_dataset()` with patched paths
|
||
- **Expected**: 5 images+labels in `data-corrupted/`; 95 across train/valid/test splits
|
||
- **Traces**: AC: Corrupted labels filtered
|
||
|
||
---
|
||
|
||
## BT-LBL: Label Validation
|
||
|
||
### BT-LBL-01: Valid label accepted
|
||
- **Input**: Label file: `0 0.5 0.5 0.1 0.1`
|
||
- **Action**: Call `check_label(path)`
|
||
- **Expected**: Returns `True`
|
||
- **Traces**: AC: Valid YOLO label format
|
||
|
||
### BT-LBL-02: Label with x > 1.0 rejected
|
||
- **Input**: Label file: `0 1.5 0.5 0.1 0.1`
|
||
- **Action**: Call `check_label(path)`
|
||
- **Expected**: Returns `False`
|
||
- **Traces**: AC: Corrupted labels detected
|
||
|
||
### BT-LBL-03: Label with height > 1.0 rejected
|
||
- **Input**: Label file: `0 0.5 0.5 0.1 1.2`
|
||
- **Action**: Call `check_label(path)`
|
||
- **Expected**: Returns `False`
|
||
- **Traces**: AC: Corrupted labels detected
|
||
|
||
### BT-LBL-04: Missing label file rejected
|
||
- **Input**: Non-existent file path
|
||
- **Action**: Call `check_label(path)`
|
||
- **Expected**: Returns `False`
|
||
- **Traces**: AC: Missing labels handled
|
||
|
||
### BT-LBL-05: Multi-line label with one corrupted line
|
||
- **Input**: Label file: `0 0.5 0.5 0.1 0.1\n3 0.5 0.5 0.1 1.5`
|
||
- **Action**: Call `check_label(path)`
|
||
- **Expected**: Returns `False` (any corrupted line fails the whole file)
|
||
- **Traces**: AC: Corrupted labels detected
|
||
|
||
---
|
||
|
||
## BT-ENC: Encryption
|
||
|
||
### BT-ENC-01: Encrypt-decrypt roundtrip (arbitrary data)
|
||
- **Input**: 1024 random bytes, key "test-key"
|
||
- **Action**: `decrypt_to(encrypt_to(data, key), key)`
|
||
- **Expected**: Output equals input bytes exactly
|
||
- **Traces**: AC: AES-256-CBC encryption
|
||
|
||
### BT-ENC-02: Encrypt-decrypt roundtrip (ONNX model)
|
||
- **Input**: `azaion.onnx` bytes, model encryption key
|
||
- **Action**: `decrypt_to(encrypt_to(model_bytes, key), key)`
|
||
- **Expected**: Output equals input bytes exactly
|
||
- **Traces**: AC: Model encryption
|
||
|
||
### BT-ENC-03: Empty input roundtrip
|
||
- **Input**: `b""`, key "test-key"
|
||
- **Action**: `decrypt_to(encrypt_to(b"", key), key)`
|
||
- **Expected**: Output equals `b""`
|
||
- **Traces**: AC: Edge case handling
|
||
|
||
### BT-ENC-04: Single byte roundtrip
|
||
- **Input**: `b"\x00"`, key "test-key"
|
||
- **Action**: `decrypt_to(encrypt_to(b"\x00", key), key)`
|
||
- **Expected**: Output equals `b"\x00"`
|
||
- **Traces**: AC: Edge case handling
|
||
|
||
### BT-ENC-05: Different keys produce different ciphertext
|
||
- **Input**: Same 1024 bytes, keys "key-a" and "key-b"
|
||
- **Action**: `encrypt_to(data, "key-a")` vs `encrypt_to(data, "key-b")`
|
||
- **Expected**: Ciphertexts differ
|
||
- **Traces**: AC: Key-dependent encryption
|
||
|
||
### BT-ENC-06: Wrong key fails decryption
|
||
- **Input**: Encrypted with "key-a", decrypt with "key-b"
|
||
- **Action**: `decrypt_to(encrypted, "key-b")`
|
||
- **Expected**: Output does NOT equal original input
|
||
- **Traces**: AC: Key-dependent encryption
|
||
|
||
---
|
||
|
||
## BT-SPL: Model Split Storage
|
||
|
||
### BT-SPL-01: Split respects size constraint
|
||
- **Input**: 10000 encrypted bytes
|
||
- **Action**: Split into small + big per `SMALL_SIZE_KB = 3` logic
|
||
- **Expected**: small ≤ max(3072 bytes, 20% of total); big = remainder
|
||
- **Traces**: AC: Model split ≤3KB or 20%
|
||
|
||
### BT-SPL-02: Reassembly produces original
|
||
- **Input**: 10000 encrypted bytes → split → reassemble
|
||
- **Action**: `small + big`
|
||
- **Expected**: Equals original encrypted bytes
|
||
- **Traces**: AC: Split model integrity
|
||
|
||
---
|
||
|
||
## BT-CLS: Annotation Class Loading
|
||
|
||
### BT-CLS-01: Load 17 base classes
|
||
- **Input**: `classes.json`
|
||
- **Action**: `AnnotationClass.read_json()`
|
||
- **Expected**: Dict with 17 unique class entries (base IDs)
|
||
- **Traces**: AC: 17 base classes
|
||
|
||
### BT-CLS-02: Weather mode expansion
|
||
- **Input**: `classes.json`
|
||
- **Action**: `AnnotationClass.read_json()`
|
||
- **Expected**: Same class at offset 0 (Norm), 20 (Wint), 40 (Night); e.g., ID 0, 20, 40 all represent ArmorVehicle
|
||
- **Traces**: AC: 3 weather modes
|
||
|
||
### BT-CLS-03: YAML generation produces 80 class names
|
||
- **Input**: `classes.json` + dataset path
|
||
- **Action**: `create_yaml()` with patched paths
|
||
- **Expected**: data.yaml contains `nc: 80`, 17 named classes + 63 `Class-N` placeholders
|
||
- **Traces**: AC: 80 total class slots
|
||
|
||
---
|
||
|
||
## BT-HSH: Hardware Hash
|
||
|
||
### BT-HSH-01: Deterministic output
|
||
- **Input**: "test-hardware-info"
|
||
- **Action**: `Security.get_hw_hash()` called twice
|
||
- **Expected**: Both calls return identical string
|
||
- **Traces**: AC: Hardware fingerprinting determinism
|
||
|
||
### BT-HSH-02: Different inputs produce different hashes
|
||
- **Input**: "hw-a" and "hw-b"
|
||
- **Action**: `Security.get_hw_hash()` on each
|
||
- **Expected**: Results differ
|
||
- **Traces**: AC: Hardware-bound uniqueness
|
||
|
||
### BT-HSH-03: Output is valid base64
|
||
- **Input**: "test-hardware-info"
|
||
- **Action**: `Security.get_hw_hash()`
|
||
- **Expected**: Matches regex `^[A-Za-z0-9+/]+=*$`
|
||
- **Traces**: AC: Hash format
|
||
|
||
---
|
||
|
||
## BT-INF: ONNX Inference
|
||
|
||
### BT-INF-01: Model loads successfully
|
||
- **Input**: `azaion.onnx` bytes
|
||
- **Action**: `OnnxEngine(model_bytes)`
|
||
- **Expected**: No exception; engine object created with valid input_shape and batch_size
|
||
- **Traces**: AC: ONNX inference capability
|
||
|
||
### BT-INF-02: Inference returns output
|
||
- **Input**: ONNX engine + 1 preprocessed image
|
||
- **Action**: `engine.run(input_blob)`
|
||
- **Expected**: Returns list of numpy arrays; first array has shape [batch, N, 6+]
|
||
- **Traces**: AC: ONNX inference produces results
|
||
|
||
### BT-INF-03: Postprocessing returns valid detections
|
||
- **Input**: ONNX engine output from real image
|
||
- **Action**: `Inference.postprocess()`
|
||
- **Expected**: Returns list of Annotation objects; each Detection has x,y,w,h ∈ [0,1], cls ∈ [0,79], confidence ∈ [0,1]
|
||
- **Traces**: AC: Detection format validity
|
||
|
||
---
|
||
|
||
## BT-NMS: Overlap Removal
|
||
|
||
### BT-NMS-01: Overlapping detections — keep higher confidence
|
||
- **Input**: 2 Detection objects at same position, confidence 0.9 and 0.5, IoU > 0.3
|
||
- **Action**: `remove_overlapping_detections()`
|
||
- **Expected**: 1 detection returned (confidence 0.9)
|
||
- **Traces**: AC: NMS IoU threshold 0.3
|
||
|
||
### BT-NMS-02: Non-overlapping detections — keep both
|
||
- **Input**: 2 Detection objects at distant positions, IoU < 0.3
|
||
- **Action**: `remove_overlapping_detections()`
|
||
- **Expected**: 2 detections returned
|
||
- **Traces**: AC: NMS preserves non-overlapping
|
||
|
||
### BT-NMS-03: Chain overlap resolution
|
||
- **Input**: 3 Detection objects: A overlaps B (IoU > 0.3), B overlaps C (IoU > 0.3), A doesn't overlap C
|
||
- **Action**: `remove_overlapping_detections()`
|
||
- **Expected**: ≤ 2 detections; highest confidence per overlapping pair kept
|
||
- **Traces**: AC: NMS handles chains
|
||
|
||
---
|
||
|
||
## BT-AQM: Annotation Queue Message Parsing
|
||
|
||
### BT-AQM-01: Parse Created annotation message
|
||
- **Input**: Msgpack bytes matching AnnotationMessage schema (status=Created, role=Validator)
|
||
- **Action**: Decode and construct AnnotationMessage
|
||
- **Expected**: All fields populated: name, detections, image bytes, status == "Created", role == "Validator"
|
||
- **Traces**: AC: Annotation message parsing
|
||
|
||
### BT-AQM-02: Parse Validated bulk message
|
||
- **Input**: Msgpack bytes with status=Validated, list of names
|
||
- **Action**: Decode and construct AnnotationBulkMessage
|
||
- **Expected**: Status == "Validated", names list matches input
|
||
- **Traces**: AC: Bulk validation parsing
|
||
|
||
### BT-AQM-03: Parse Deleted bulk message
|
||
- **Input**: Msgpack bytes with status=Deleted, list of names
|
||
- **Action**: Decode and construct AnnotationBulkMessage
|
||
- **Expected**: Status == "Deleted", names list matches input
|
||
- **Traces**: AC: Bulk deletion parsing
|
||
|
||
### BT-AQM-04: Malformed message raises exception
|
||
- **Input**: Invalid msgpack bytes
|
||
- **Action**: Attempt to decode
|
||
- **Expected**: Exception raised
|
||
- **Traces**: AC: Error handling for malformed messages
|