mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 17:46:41 +00:00
Refactor constants management to use Pydantic BaseModel for configuration
- Replaced module-level path variables in constants.py with a structured Pydantic Config class. - Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure. - Fixed bugs related to image processing and model saving. - Enhanced test infrastructure to accommodate the new configuration approach. This refactor improves code maintainability and clarity by centralizing configuration management.
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Component: Data Pipeline
|
||||
|
||||
## Overview
|
||||
Tools for preparing and managing annotation data: augmentation of training images, format conversion from external annotation systems, and visual inspection of annotated datasets.
|
||||
|
||||
**Pattern**: Batch processing tools (standalone scripts + library)
|
||||
**Upstream**: Core (constants), Data Models (ImageLabel, AnnotationClass)
|
||||
**Downstream**: Training (augmented images feed into dataset formation)
|
||||
|
||||
## Modules
|
||||
- `augmentation` — image augmentation pipeline (albumentations)
|
||||
- `convert-annotations` — Pascal VOC / oriented bbox → YOLO format converter
|
||||
- `dataset-visualiser` — interactive annotation visualization tool
|
||||
|
||||
## Internal Interfaces
|
||||
|
||||
### Augmentator
|
||||
```python
|
||||
Augmentator()
|
||||
Augmentator.augment_annotations(from_scratch: bool = False) -> None
|
||||
Augmentator.augment_inner(img_ann: ImageLabel) -> list[ImageLabel]
|
||||
Augmentator.correct_bboxes(labels) -> list
|
||||
Augmentator.read_labels(labels_path) -> list[list]
|
||||
```
|
||||
|
||||
### convert-annotations (functions)
|
||||
```python
|
||||
convert(folder, dest_folder, read_annotations, ann_format) -> None
|
||||
minmax2yolo(width, height, xmin, xmax, ymin, ymax) -> tuple
|
||||
read_pascal_voc(width, height, s: str) -> list[str]
|
||||
read_bbox_oriented(width, height, s: str) -> list[str]
|
||||
```
|
||||
|
||||
### dataset-visualiser (functions)
|
||||
```python
|
||||
visualise_dataset() -> None
|
||||
visualise_processed_folder() -> None
|
||||
```
|
||||
|
||||
## Data Access Patterns
|
||||
- **Augmentation**: Reads from `/azaion/data/images/` + `/azaion/data/labels/`, writes to `/azaion/data-processed/images/` + `/azaion/data-processed/labels/`
|
||||
- **Conversion**: Reads from user-specified source folder, writes to destination folder
|
||||
- **Visualiser**: Reads from datasets or processed folder, renders to matplotlib window
|
||||
|
||||
## Implementation Details
|
||||
- **Augmentation pipeline**: Per image → 1 original copy + 7 augmented variants (8× data expansion)
|
||||
- HorizontalFlip (60%), BrightnessContrast (40%), Affine (80%), MotionBlur (10%), HueSaturation (40%)
|
||||
- Bbox correction clips outside-boundary boxes, removes boxes < 1% of image
|
||||
- Incremental: skips already-processed images
|
||||
- Continuous mode: infinite loop with 5-minute sleep between rounds
|
||||
- Concurrent: ThreadPoolExecutor for parallel image processing
|
||||
- **Format conversion**: Pluggable reader pattern — `convert()` accepts any reader function that maps (width, height, text) → YOLO lines
|
||||
- **Visualiser**: Interactive (waits for keypress) — developer debugging tool
|
||||
|
||||
## Caveats
|
||||
- `dataset-visualiser` imports from `preprocessing` module which does not exist — broken import
|
||||
- `dataset-visualiser` has hardcoded dataset date (`2024-06-18`) and start index (35247)
|
||||
- `convert-annotations` hardcodes class mappings (Truck=1, Car/Taxi=2) — not configurable
|
||||
- Augmentation parameters are hardcoded, not configurable via config file
|
||||
- Augmentation `total_to_process` attribute referenced in `augment_annotation` but never set (uses `total_images_to_process`)
|
||||
|
||||
## Dependency Graph
|
||||
```mermaid
|
||||
graph TD
|
||||
constants --> augmentation
|
||||
dto_imageLabel[dto/imageLabel] --> augmentation
|
||||
constants --> dataset-visualiser
|
||||
dto_annotationClass[dto/annotationClass] --> dataset-visualiser
|
||||
dto_imageLabel --> dataset-visualiser
|
||||
augmentation --> manual_run
|
||||
```
|
||||
|
||||
## Logging Strategy
|
||||
Print statements for progress tracking (processed count, errors). No structured logging.
|
||||
Reference in New Issue
Block a user