Refactor constants management to use Pydantic BaseModel for configuration

- Replaced module-level path variables in constants.py with a structured Pydantic Config class.
- Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure.
- Fixed bugs related to image processing and model saving.
- Enhanced test infrastructure to accommodate the new configuration approach.

This refactor improves code maintainability and clarity by centralizing configuration management.
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-03-27 18:18:30 +02:00
parent b68c07b540
commit 142c6c4de8
106 changed files with 5706 additions and 654 deletions
@@ -0,0 +1,74 @@
# Component: Data Pipeline
## Overview
Tools for preparing and managing annotation data: augmentation of training images, format conversion from external annotation systems, and visual inspection of annotated datasets.
**Pattern**: Batch processing tools (standalone scripts + library)
**Upstream**: Core (constants), Data Models (ImageLabel, AnnotationClass)
**Downstream**: Training (augmented images feed into dataset formation)
## Modules
- `augmentation` — image augmentation pipeline (albumentations)
- `convert-annotations` — Pascal VOC / oriented bbox → YOLO format converter
- `dataset-visualiser` — interactive annotation visualization tool
## Internal Interfaces
### Augmentator
```python
Augmentator()
Augmentator.augment_annotations(from_scratch: bool = False) -> None
Augmentator.augment_inner(img_ann: ImageLabel) -> list[ImageLabel]
Augmentator.correct_bboxes(labels) -> list
Augmentator.read_labels(labels_path) -> list[list]
```
### convert-annotations (functions)
```python
convert(folder, dest_folder, read_annotations, ann_format) -> None
minmax2yolo(width, height, xmin, xmax, ymin, ymax) -> tuple
read_pascal_voc(width, height, s: str) -> list[str]
read_bbox_oriented(width, height, s: str) -> list[str]
```
### dataset-visualiser (functions)
```python
visualise_dataset() -> None
visualise_processed_folder() -> None
```
## Data Access Patterns
- **Augmentation**: Reads from `/azaion/data/images/` + `/azaion/data/labels/`, writes to `/azaion/data-processed/images/` + `/azaion/data-processed/labels/`
- **Conversion**: Reads from user-specified source folder, writes to destination folder
- **Visualiser**: Reads from datasets or processed folder, renders to matplotlib window
## Implementation Details
- **Augmentation pipeline**: Per image → 1 original copy + 7 augmented variants (8× data expansion)
- HorizontalFlip (60%), BrightnessContrast (40%), Affine (80%), MotionBlur (10%), HueSaturation (40%)
- Bbox correction clips outside-boundary boxes, removes boxes < 1% of image
- Incremental: skips already-processed images
- Continuous mode: infinite loop with 5-minute sleep between rounds
- Concurrent: ThreadPoolExecutor for parallel image processing
- **Format conversion**: Pluggable reader pattern — `convert()` accepts any reader function that maps (width, height, text) → YOLO lines
- **Visualiser**: Interactive (waits for keypress) — developer debugging tool
## Caveats
- `dataset-visualiser` imports from `preprocessing` module which does not exist — broken import
- `dataset-visualiser` has hardcoded dataset date (`2024-06-18`) and start index (35247)
- `convert-annotations` hardcodes class mappings (Truck=1, Car/Taxi=2) — not configurable
- Augmentation parameters are hardcoded, not configurable via config file
- Augmentation `total_to_process` attribute referenced in `augment_annotation` but never set (uses `total_images_to_process`)
## Dependency Graph
```mermaid
graph TD
constants --> augmentation
dto_imageLabel[dto/imageLabel] --> augmentation
constants --> dataset-visualiser
dto_annotationClass[dto/annotationClass] --> dataset-visualiser
dto_imageLabel --> dataset-visualiser
augmentation --> manual_run
```
## Logging Strategy
Print statements for progress tracking (processed count, errors). No structured logging.