mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 22:06:36 +00:00
Oleksandr Bezdieniezhnykh
142c6c4de8
- Replaced module-level path variables in constants.py with a structured Pydantic Config class. - Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure. - Fixed bugs related to image processing and model saving. - Enhanced test infrastructure to accommodate the new configuration approach. This refactor improves code maintainability and clarity by centralizing configuration management.
52 lines
1.9 KiB
Markdown
52 lines
1.9 KiB
Markdown
# Module: cdn_manager
|
|
|
|
## Purpose
|
|
Manages file upload and download to/from an S3-compatible CDN (MinIO/similar) using separate credentials for upload and download operations.
|
|
|
|
## Public Interface
|
|
|
|
### CDNCredentials
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `host` | str | CDN endpoint URL |
|
|
| `downloader_access_key` | str | S3 access key for downloads |
|
|
| `downloader_access_secret` | str | S3 secret for downloads |
|
|
| `uploader_access_key` | str | S3 access key for uploads |
|
|
| `uploader_access_secret` | str | S3 secret for uploads |
|
|
|
|
### CDNManager
|
|
| Method | Signature | Returns | Description |
|
|
|--------|-----------|---------|-------------|
|
|
| `__init__` | `(credentials: CDNCredentials)` | — | Creates two boto3 S3 clients (download + upload) |
|
|
| `upload` | `(bucket: str, filename: str, file_bytes: bytearray) -> bool` | True on success | Uploads bytes to S3 bucket |
|
|
| `download` | `(bucket: str, filename: str) -> bool` | True on success | Downloads file from S3 to current directory |
|
|
|
|
## Internal Logic
|
|
- Maintains two separate boto3 S3 clients with different credentials (read vs write separation)
|
|
- Upload uses `upload_fileobj` with in-memory BytesIO wrapper
|
|
- Download uses `download_file` (saves directly to disk with same filename)
|
|
- Both methods catch all exceptions, print error, return bool
|
|
|
|
## Dependencies
|
|
- `boto3` (external) — S3 client
|
|
- `io`, `sys`, `yaml`, `os` (stdlib) — Note: `sys`, `yaml`, `os` are imported but unused
|
|
|
|
## Consumers
|
|
api_client, exports, train, start_inference
|
|
|
|
## Data Models
|
|
`CDNCredentials` — plain data class holding S3 access credentials.
|
|
|
|
## Configuration
|
|
Credentials loaded from `cdn.yaml` by callers (not by this module directly).
|
|
|
|
## External Integrations
|
|
- S3-compatible object storage (configured via `CDNCredentials.host`)
|
|
|
|
## Security
|
|
- Separate read/write credentials enforce least-privilege access
|
|
- Credentials passed in at construction time, not hardcoded here
|
|
|
|
## Tests
|
|
None.
|