mirror of
https://github.com/azaion/ai-training.git
synced 2026-04-22 12:56:35 +00:00
Refactor constants management to use Pydantic BaseModel for configuration
- Replaced module-level path variables in constants.py with a structured Pydantic Config class. - Updated all relevant modules (train.py, augmentation.py, exports.py, dataset-visualiser.py, manual_run.py) to access paths through the new config structure. - Fixed bugs related to image processing and model saving. - Enhanced test infrastructure to accommodate the new configuration approach. This refactor improves code maintainability and clarity by centralizing configuration management.
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# Module: cdn_manager
|
||||
|
||||
## Purpose
|
||||
Manages file upload and download to/from an S3-compatible CDN (MinIO/similar) using separate credentials for upload and download operations.
|
||||
|
||||
## Public Interface
|
||||
|
||||
### CDNCredentials
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `host` | str | CDN endpoint URL |
|
||||
| `downloader_access_key` | str | S3 access key for downloads |
|
||||
| `downloader_access_secret` | str | S3 secret for downloads |
|
||||
| `uploader_access_key` | str | S3 access key for uploads |
|
||||
| `uploader_access_secret` | str | S3 secret for uploads |
|
||||
|
||||
### CDNManager
|
||||
| Method | Signature | Returns | Description |
|
||||
|--------|-----------|---------|-------------|
|
||||
| `__init__` | `(credentials: CDNCredentials)` | — | Creates two boto3 S3 clients (download + upload) |
|
||||
| `upload` | `(bucket: str, filename: str, file_bytes: bytearray) -> bool` | True on success | Uploads bytes to S3 bucket |
|
||||
| `download` | `(bucket: str, filename: str) -> bool` | True on success | Downloads file from S3 to current directory |
|
||||
|
||||
## Internal Logic
|
||||
- Maintains two separate boto3 S3 clients with different credentials (read vs write separation)
|
||||
- Upload uses `upload_fileobj` with in-memory BytesIO wrapper
|
||||
- Download uses `download_file` (saves directly to disk with same filename)
|
||||
- Both methods catch all exceptions, print error, return bool
|
||||
|
||||
## Dependencies
|
||||
- `boto3` (external) — S3 client
|
||||
- `io`, `sys`, `yaml`, `os` (stdlib) — Note: `sys`, `yaml`, `os` are imported but unused
|
||||
|
||||
## Consumers
|
||||
api_client, exports, train, start_inference
|
||||
|
||||
## Data Models
|
||||
`CDNCredentials` — plain data class holding S3 access credentials.
|
||||
|
||||
## Configuration
|
||||
Credentials loaded from `cdn.yaml` by callers (not by this module directly).
|
||||
|
||||
## External Integrations
|
||||
- S3-compatible object storage (configured via `CDNCredentials.host`)
|
||||
|
||||
## Security
|
||||
- Separate read/write credentials enforce least-privilege access
|
||||
- Credentials passed in at construction time, not hardcoded here
|
||||
|
||||
## Tests
|
||||
None.
|
||||
Reference in New Issue
Block a user