mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 22:16:35 +00:00
Oleksandr Bezdieniezhnykh
a47fa135de
- Modified `.gitignore` to include test fixture data while excluding test results. - Updated `config.yaml` to change the model from 'yolo11m.yaml' to 'yolo26m.pt'. - Enhanced `.cursor/rules/coderule.mdc` with additional guidelines for test environment consistency and infrastructure handling. - Revised autopilot state management in `_docs/_autopilot_state.md` to reflect current progress and tasks. - Removed outdated augmentation tests and adjusted dataset formation tests to align with the new structure. These changes streamline the configuration and testing processes, ensuring better organization and clarity in the project.
234 lines
8.4 KiB
Markdown
234 lines
8.4 KiB
Markdown
# Blackbox Test Scenarios
|
|
|
|
## BT-DSF: Dataset Formation
|
|
|
|
### BT-DSF-01: 70/20/10 split ratio
|
|
- **Input**: 100 images + labels in data/ dir
|
|
- **Action**: Run `form_dataset()` with patched paths
|
|
- **Expected**: train: 70 images+labels, valid: 20, test: 10
|
|
- **Traces**: AC: Dataset split 70/20/10
|
|
|
|
### BT-DSF-02: Split directories structure
|
|
- **Input**: 100 images + labels
|
|
- **Action**: Run `form_dataset()`
|
|
- **Expected**: Created: `train/images/`, `train/labels/`, `valid/images/`, `valid/labels/`, `test/images/`, `test/labels/`
|
|
- **Traces**: AC: YOLO dataset directory structure
|
|
|
|
### BT-DSF-03: Total files preserved across splits
|
|
- **Input**: 100 valid images + labels
|
|
- **Action**: Run `form_dataset()`
|
|
- **Expected**: `count(train) + count(valid) + count(test) == 100` (no data loss)
|
|
- **Traces**: AC: Dataset integrity
|
|
|
|
### BT-DSF-04: Corrupted labels moved to corrupted directory
|
|
- **Input**: 95 valid + 5 corrupted labels (coords > 1.0)
|
|
- **Action**: Run `form_dataset()` with patched paths
|
|
- **Expected**: 5 images+labels in `data-corrupted/`; 95 across train/valid/test splits
|
|
- **Traces**: AC: Corrupted labels filtered
|
|
|
|
---
|
|
|
|
## BT-LBL: Label Validation
|
|
|
|
### BT-LBL-01: Valid label accepted
|
|
- **Input**: Label file: `0 0.5 0.5 0.1 0.1`
|
|
- **Action**: Call `check_label(path)`
|
|
- **Expected**: Returns `True`
|
|
- **Traces**: AC: Valid YOLO label format
|
|
|
|
### BT-LBL-02: Label with x > 1.0 rejected
|
|
- **Input**: Label file: `0 1.5 0.5 0.1 0.1`
|
|
- **Action**: Call `check_label(path)`
|
|
- **Expected**: Returns `False`
|
|
- **Traces**: AC: Corrupted labels detected
|
|
|
|
### BT-LBL-03: Label with height > 1.0 rejected
|
|
- **Input**: Label file: `0 0.5 0.5 0.1 1.2`
|
|
- **Action**: Call `check_label(path)`
|
|
- **Expected**: Returns `False`
|
|
- **Traces**: AC: Corrupted labels detected
|
|
|
|
### BT-LBL-04: Missing label file rejected
|
|
- **Input**: Non-existent file path
|
|
- **Action**: Call `check_label(path)`
|
|
- **Expected**: Returns `False`
|
|
- **Traces**: AC: Missing labels handled
|
|
|
|
### BT-LBL-05: Multi-line label with one corrupted line
|
|
- **Input**: Label file: `0 0.5 0.5 0.1 0.1\n3 0.5 0.5 0.1 1.5`
|
|
- **Action**: Call `check_label(path)`
|
|
- **Expected**: Returns `False` (any corrupted line fails the whole file)
|
|
- **Traces**: AC: Corrupted labels detected
|
|
|
|
---
|
|
|
|
## BT-ENC: Encryption
|
|
|
|
### BT-ENC-01: Encrypt-decrypt roundtrip (arbitrary data)
|
|
- **Input**: 1024 random bytes, key "test-key"
|
|
- **Action**: `decrypt_to(encrypt_to(data, key), key)`
|
|
- **Expected**: Output equals input bytes exactly
|
|
- **Traces**: AC: AES-256-CBC encryption
|
|
|
|
### BT-ENC-02: Encrypt-decrypt roundtrip (ONNX model)
|
|
- **Input**: `azaion.onnx` bytes, model encryption key
|
|
- **Action**: `decrypt_to(encrypt_to(model_bytes, key), key)`
|
|
- **Expected**: Output equals input bytes exactly
|
|
- **Traces**: AC: Model encryption
|
|
|
|
### BT-ENC-03: Empty input roundtrip
|
|
- **Input**: `b""`, key "test-key"
|
|
- **Action**: `decrypt_to(encrypt_to(b"", key), key)`
|
|
- **Expected**: Output equals `b""`
|
|
- **Traces**: AC: Edge case handling
|
|
|
|
### BT-ENC-04: Single byte roundtrip
|
|
- **Input**: `b"\x00"`, key "test-key"
|
|
- **Action**: `decrypt_to(encrypt_to(b"\x00", key), key)`
|
|
- **Expected**: Output equals `b"\x00"`
|
|
- **Traces**: AC: Edge case handling
|
|
|
|
### BT-ENC-05: Different keys produce different ciphertext
|
|
- **Input**: Same 1024 bytes, keys "key-a" and "key-b"
|
|
- **Action**: `encrypt_to(data, "key-a")` vs `encrypt_to(data, "key-b")`
|
|
- **Expected**: Ciphertexts differ
|
|
- **Traces**: AC: Key-dependent encryption
|
|
|
|
### BT-ENC-06: Wrong key fails decryption
|
|
- **Input**: Encrypted with "key-a", decrypt with "key-b"
|
|
- **Action**: `decrypt_to(encrypted, "key-b")`
|
|
- **Expected**: Output does NOT equal original input
|
|
- **Traces**: AC: Key-dependent encryption
|
|
|
|
---
|
|
|
|
## BT-SPL: Model Split Storage
|
|
|
|
### BT-SPL-01: Split respects size constraint
|
|
- **Input**: 10000 encrypted bytes
|
|
- **Action**: Split into small + big per `SMALL_SIZE_KB = 3` logic
|
|
- **Expected**: small ≤ max(3072 bytes, 20% of total); big = remainder
|
|
- **Traces**: AC: Model split ≤3KB or 20%
|
|
|
|
### BT-SPL-02: Reassembly produces original
|
|
- **Input**: 10000 encrypted bytes → split → reassemble
|
|
- **Action**: `small + big`
|
|
- **Expected**: Equals original encrypted bytes
|
|
- **Traces**: AC: Split model integrity
|
|
|
|
---
|
|
|
|
## BT-CLS: Annotation Class Loading
|
|
|
|
### BT-CLS-01: Load 17 base classes
|
|
- **Input**: `classes.json`
|
|
- **Action**: `AnnotationClass.read_json()`
|
|
- **Expected**: Dict with 17 unique class entries (base IDs)
|
|
- **Traces**: AC: 17 base classes
|
|
|
|
### BT-CLS-02: Weather mode expansion
|
|
- **Input**: `classes.json`
|
|
- **Action**: `AnnotationClass.read_json()`
|
|
- **Expected**: Same class at offset 0 (Norm), 20 (Wint), 40 (Night); e.g., ID 0, 20, 40 all represent ArmorVehicle
|
|
- **Traces**: AC: 3 weather modes
|
|
|
|
### BT-CLS-03: YAML generation produces 80 class names
|
|
- **Input**: `classes.json` + dataset path
|
|
- **Action**: `create_yaml()` with patched paths
|
|
- **Expected**: data.yaml contains `nc: 80`, 17 named classes + 63 `Class-N` placeholders
|
|
- **Traces**: AC: 80 total class slots
|
|
|
|
---
|
|
|
|
## BT-HSH: Hardware Hash
|
|
|
|
### BT-HSH-01: Deterministic output
|
|
- **Input**: "test-hardware-info"
|
|
- **Action**: `Security.get_hw_hash()` called twice
|
|
- **Expected**: Both calls return identical string
|
|
- **Traces**: AC: Hardware fingerprinting determinism
|
|
|
|
### BT-HSH-02: Different inputs produce different hashes
|
|
- **Input**: "hw-a" and "hw-b"
|
|
- **Action**: `Security.get_hw_hash()` on each
|
|
- **Expected**: Results differ
|
|
- **Traces**: AC: Hardware-bound uniqueness
|
|
|
|
### BT-HSH-03: Output is valid base64
|
|
- **Input**: "test-hardware-info"
|
|
- **Action**: `Security.get_hw_hash()`
|
|
- **Expected**: Matches regex `^[A-Za-z0-9+/]+=*$`
|
|
- **Traces**: AC: Hash format
|
|
|
|
---
|
|
|
|
## BT-INF: ONNX Inference
|
|
|
|
### BT-INF-01: Model loads successfully
|
|
- **Input**: `azaion.onnx` bytes
|
|
- **Action**: `OnnxEngine(model_bytes)`
|
|
- **Expected**: No exception; engine object created with valid input_shape and batch_size
|
|
- **Traces**: AC: ONNX inference capability
|
|
|
|
### BT-INF-02: Inference returns output
|
|
- **Input**: ONNX engine + 1 preprocessed image
|
|
- **Action**: `engine.run(input_blob)`
|
|
- **Expected**: Returns list of numpy arrays; first array has shape [batch, N, 6+]
|
|
- **Traces**: AC: ONNX inference produces results
|
|
|
|
### BT-INF-03: Postprocessing returns valid detections
|
|
- **Input**: ONNX engine output from real image
|
|
- **Action**: `Inference.postprocess()`
|
|
- **Expected**: Returns list of Annotation objects; each Detection has x,y,w,h ∈ [0,1], cls ∈ [0,79], confidence ∈ [0,1]
|
|
- **Traces**: AC: Detection format validity
|
|
|
|
---
|
|
|
|
## BT-NMS: Overlap Removal
|
|
|
|
### BT-NMS-01: Overlapping detections — keep higher confidence
|
|
- **Input**: 2 Detection objects at same position, confidence 0.9 and 0.5, IoU > 0.3
|
|
- **Action**: `remove_overlapping_detections()`
|
|
- **Expected**: 1 detection returned (confidence 0.9)
|
|
- **Traces**: AC: NMS IoU threshold 0.3
|
|
|
|
### BT-NMS-02: Non-overlapping detections — keep both
|
|
- **Input**: 2 Detection objects at distant positions, IoU < 0.3
|
|
- **Action**: `remove_overlapping_detections()`
|
|
- **Expected**: 2 detections returned
|
|
- **Traces**: AC: NMS preserves non-overlapping
|
|
|
|
### BT-NMS-03: Chain overlap resolution
|
|
- **Input**: 3 Detection objects: A overlaps B (IoU > 0.3), B overlaps C (IoU > 0.3), A doesn't overlap C
|
|
- **Action**: `remove_overlapping_detections()`
|
|
- **Expected**: ≤ 2 detections; highest confidence per overlapping pair kept
|
|
- **Traces**: AC: NMS handles chains
|
|
|
|
---
|
|
|
|
## BT-AQM: Annotation Queue Message Parsing
|
|
|
|
### BT-AQM-01: Parse Created annotation message
|
|
- **Input**: Msgpack bytes matching AnnotationMessage schema (status=Created, role=Validator)
|
|
- **Action**: Decode and construct AnnotationMessage
|
|
- **Expected**: All fields populated: name, detections, image bytes, status == "Created", role == "Validator"
|
|
- **Traces**: AC: Annotation message parsing
|
|
|
|
### BT-AQM-02: Parse Validated bulk message
|
|
- **Input**: Msgpack bytes with status=Validated, list of names
|
|
- **Action**: Decode and construct AnnotationBulkMessage
|
|
- **Expected**: Status == "Validated", names list matches input
|
|
- **Traces**: AC: Bulk validation parsing
|
|
|
|
### BT-AQM-03: Parse Deleted bulk message
|
|
- **Input**: Msgpack bytes with status=Deleted, list of names
|
|
- **Action**: Decode and construct AnnotationBulkMessage
|
|
- **Expected**: Status == "Deleted", names list matches input
|
|
- **Traces**: AC: Bulk deletion parsing
|
|
|
|
### BT-AQM-04: Malformed message raises exception
|
|
- **Input**: Invalid msgpack bytes
|
|
- **Action**: Attempt to decode
|
|
- **Expected**: Exception raised
|
|
- **Traces**: AC: Error handling for malformed messages
|