[AZ-185][AZ-186] Batch 2

Made-with: Cursor
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-04-15 07:32:37 +03:00
parent d244799f02
commit 9a0248af72
18 changed files with 1857 additions and 26 deletions
@@ -0,0 +1,53 @@
# Publish artifact script (AZ-186)
Training services and CI/CD call `scripts/publish_artifact.py` after producing an artifact (for example a `.trt` model or a Docker image tarball). The script gzip-compresses the file, encrypts it with a random 32-byte AES-256 key (AES-CBC with PKCS7, IV prefixed), uploads the ciphertext to S3, and registers metadata with the admin API.
## CLI
```text
python scripts/publish_artifact.py \
--file /path/to/artifact \
--resource-name my_model \
--dev-stage dev \
--architecture arm64 \
--version 2026-04-15
```
Object key: `{dev_stage}/{resource_name}-{architecture}-{version}.enc`
## Environment variables
| Variable | Required | Purpose |
|----------|----------|---------|
| `S3_ENDPOINT` | yes | S3-compatible endpoint URL |
| `S3_ACCESS_KEY` | yes | Upload credentials |
| `S3_SECRET_KEY` | yes | Upload credentials |
| `S3_BUCKET` | yes | Target bucket |
| `ADMIN_API_URL` | yes | Admin API base URL (no trailing path for publish) |
| `ADMIN_API_TOKEN` | yes | Bearer token for the publish request |
| `CDN_PUBLIC_BASE_URL` | no | If set, `cdn_url` in the registration payload is `{CDN_PUBLIC_BASE_URL}/{object_key}`; otherwise it defaults to `{S3_ENDPOINT}/{S3_BUCKET}/{object_key}` |
| `ADMIN_API_PUBLISH_PATH` | no | Defaults to `internal/resources/publish`; POST is sent to `{ADMIN_API_URL}/{ADMIN_API_PUBLISH_PATH}` |
## Admin API contract
`POST {ADMIN_API_URL}/internal/resources/publish` (unless overridden) with JSON body:
- `resource_name`, `dev_stage`, `architecture`, `version` (strings)
- `cdn_url` (string)
- `sha256` (lowercase hex of the uploaded ciphertext file, including the 16-byte IV)
- `encryption_key` (64-character hex encoding of the raw 32-byte AES key)
- `size_bytes` (integer size of the uploaded ciphertext file)
The loader expects the same `encryption_key` and `sha256` semantics as returned by fleet `POST /get-update` (hex key, hash of the ciphertext object).
## Dependencies
Use the same major versions as the loader: `boto3`, `cryptography`, `requests` (see `requirements.txt`). A minimal install for a training host is:
```text
pip install boto3==1.40.9 cryptography==44.0.2 requests==2.32.4
```
## Woodpecker
Pipeline `.woodpecker/build-arm.yml` saves the built image to `loader-image.tar` and runs this script in a follow-up step. Configure the environment variables above as Woodpecker secrets for that step.