mirror of
https://github.com/azaion/loader.git
synced 2026-04-22 09:46:32 +00:00
Add E2E tests, fix bugs
Made-with: Cursor
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,117 @@
|
||||
# Test Infrastructure
|
||||
|
||||
**Task**: 01_test_infrastructure
|
||||
**Name**: Test Infrastructure
|
||||
**Description**: Scaffold the blackbox test project — pytest runner, mock API server, mock CDN (MinIO), Docker test environment, test data fixtures, CSV reporting
|
||||
**Complexity**: 5 points
|
||||
**Dependencies**: None
|
||||
**Component**: Blackbox Tests
|
||||
**Tracker**: pending
|
||||
**Epic**: pending
|
||||
|
||||
## Test Project Folder Layout
|
||||
|
||||
```
|
||||
e2e/
|
||||
├── conftest.py
|
||||
├── requirements.txt
|
||||
├── mocks/
|
||||
│ └── mock_api/
|
||||
│ ├── Dockerfile
|
||||
│ └── app.py
|
||||
├── fixtures/
|
||||
│ ├── test_resource.bin
|
||||
│ └── test_archive.enc
|
||||
├── tests/
|
||||
│ ├── test_health.py
|
||||
│ ├── test_auth.py
|
||||
│ ├── test_resources.py
|
||||
│ ├── test_unlock.py
|
||||
│ ├── test_security.py
|
||||
│ ├── test_performance.py
|
||||
│ └── test_resilience.py
|
||||
└── docker-compose.test.yml
|
||||
```
|
||||
|
||||
## Mock Services
|
||||
|
||||
| Mock Service | Replaces | Endpoints | Behavior |
|
||||
|-------------|----------|-----------|----------|
|
||||
| mock-api | Azaion Resource API | POST /login, POST /resources/get/{folder}, POST /resources/{folder}, GET /resources/list/{folder}, GET /binary-split/key-fragment | Returns canned JWT, encrypted test resources, key fragment |
|
||||
| mock-cdn (MinIO) | S3 CDN | S3 API (standard) | S3-compatible storage with pre-seeded test .big files |
|
||||
|
||||
## Docker Test Environment
|
||||
|
||||
### docker-compose.test.yml Structure
|
||||
|
||||
| Service | Image / Build | Purpose | Depends On |
|
||||
|---------|--------------|---------|------------|
|
||||
| system-under-test | Build from Dockerfile | Azaion.Loader | mock-api, mock-cdn |
|
||||
| mock-api | Build from e2e/mocks/mock_api/ | Mock Azaion Resource API | — |
|
||||
| mock-cdn | minio/minio | Mock S3 CDN | — |
|
||||
| e2e-consumer | python:3.11-slim + e2e/ | Pytest test runner | system-under-test |
|
||||
|
||||
### Networks and Volumes
|
||||
|
||||
- `e2e-net`: isolated test network connecting all services
|
||||
- `test-data` volume: mounted to e2e-consumer for test fixtures
|
||||
- Docker socket: mounted to system-under-test for unlock flow
|
||||
|
||||
## Test Runner Configuration
|
||||
|
||||
**Framework**: pytest
|
||||
**Plugins**: pytest-csv (reporting), requests (HTTP client)
|
||||
**Entry point**: `pytest tests/ --csv=/results/report.csv -v`
|
||||
|
||||
### Fixture Strategy
|
||||
|
||||
| Fixture | Scope | Purpose |
|
||||
|---------|-------|---------|
|
||||
| base_url | session | URL of the system-under-test |
|
||||
| logged_in_client | function | requests.Session with /login called |
|
||||
| mock_api_url | session | URL of the mock API |
|
||||
|
||||
## Test Data Fixtures
|
||||
|
||||
| Data Set | Source | Format | Used By |
|
||||
|----------|--------|--------|---------|
|
||||
| test_resource.bin | Generated (small binary) | Binary | test_resources.py |
|
||||
| test_archive.enc | Generated (AES-encrypted tar) | Binary | test_unlock.py |
|
||||
| cdn.yaml | Generated (mock CDN config) | YAML | conftest.py (served by mock-api) |
|
||||
|
||||
### Data Isolation
|
||||
|
||||
Fresh container restart per test run. Mock API state is stateless (canned responses). MinIO bucket re-created on startup.
|
||||
|
||||
## Test Reporting
|
||||
|
||||
**Format**: CSV
|
||||
**Columns**: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message
|
||||
**Output path**: `/results/report.csv` → mounted to `./e2e-results/report.csv` on host
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: Test environment starts**
|
||||
Given the docker-compose.test.yml
|
||||
When `docker compose -f e2e/docker-compose.test.yml up` is executed
|
||||
Then all services start and the system-under-test health endpoint responds
|
||||
|
||||
**AC-2: Mock API responds**
|
||||
Given the test environment is running
|
||||
When the e2e-consumer sends POST /login to the mock API
|
||||
Then the mock API returns a valid JWT response
|
||||
|
||||
**AC-3: Mock CDN operational**
|
||||
Given the test environment is running
|
||||
When the e2e-consumer uploads/downloads a file to MinIO
|
||||
Then S3 operations succeed
|
||||
|
||||
**AC-4: Test runner discovers tests**
|
||||
Given the test environment is running
|
||||
When the e2e-consumer starts
|
||||
Then pytest discovers all test files in e2e/tests/
|
||||
|
||||
**AC-5: Test report generated**
|
||||
Given tests have completed
|
||||
When the test run finishes
|
||||
Then a CSV report exists at /results/report.csv with correct columns
|
||||
@@ -0,0 +1,71 @@
|
||||
# Health & Authentication Tests
|
||||
|
||||
**Task**: 02_test_health_auth
|
||||
**Name**: Health & Authentication Tests
|
||||
**Description**: Implement blackbox tests for health, status, and login endpoints (positive and negative scenarios)
|
||||
**Complexity**: 3 points
|
||||
**Dependencies**: 01_test_infrastructure
|
||||
**Component**: Blackbox Tests
|
||||
**Tracker**: pending
|
||||
**Epic**: pending
|
||||
|
||||
## Problem
|
||||
|
||||
The loader has no test coverage for its health and authentication endpoints. These are the most basic verification points for service liveness and user access.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Health endpoint test passes (FT-P-01)
|
||||
- Status endpoint tests pass — unauthenticated and authenticated (FT-P-02, FT-P-03 step 2)
|
||||
- Login positive test passes (FT-P-03)
|
||||
- Login negative tests pass — invalid credentials and missing fields (FT-N-01, FT-N-02)
|
||||
|
||||
## Scope
|
||||
|
||||
### Included
|
||||
- FT-P-01: Health endpoint returns healthy
|
||||
- FT-P-02: Status reports unauthenticated state
|
||||
- FT-P-03: Login with valid credentials (including authenticated status check)
|
||||
- FT-N-01: Login with invalid credentials
|
||||
- FT-N-02: Login with missing fields
|
||||
|
||||
### Excluded
|
||||
- Resource download/upload tests
|
||||
- Unlock workflow tests
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: Health returns 200**
|
||||
Given the loader is running
|
||||
When GET /health is called
|
||||
Then HTTP 200 with body `{"status": "healthy"}`
|
||||
|
||||
**AC-2: Status shows unauthenticated before login**
|
||||
Given the loader is running with no prior login
|
||||
When GET /status is called
|
||||
Then HTTP 200 with `authenticated: false`
|
||||
|
||||
**AC-3: Login succeeds with valid credentials**
|
||||
Given the mock API accepts test credentials
|
||||
When POST /login with valid email/password
|
||||
Then HTTP 200 with `{"status": "ok"}`
|
||||
|
||||
**AC-4: Login fails with invalid credentials**
|
||||
Given the mock API rejects test credentials
|
||||
When POST /login with wrong email/password
|
||||
Then HTTP 401
|
||||
|
||||
**AC-5: Login rejects empty body**
|
||||
Given the loader is running
|
||||
When POST /login with empty JSON
|
||||
Then HTTP 422
|
||||
|
||||
## Blackbox Tests
|
||||
|
||||
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||
|--------|------------------------|-------------|-------------------|----------------|
|
||||
| AC-1 | Loader running | GET /health | 200, {"status": "healthy"} | NFT-PERF-01 |
|
||||
| AC-2 | No prior login | GET /status | 200, authenticated=false | — |
|
||||
| AC-3 | Mock API accepts creds | POST /login valid | 200, status ok | NFT-PERF-02 |
|
||||
| AC-4 | Mock API rejects creds | POST /login invalid | 401 | — |
|
||||
| AC-5 | — | POST /login empty | 422 | — |
|
||||
@@ -0,0 +1,86 @@
|
||||
# Resource Download & Upload Tests
|
||||
|
||||
**Task**: 03_test_resources
|
||||
**Name**: Resource Download & Upload Tests
|
||||
**Description**: Implement blackbox tests for resource download (binary-split) and upload endpoints
|
||||
**Complexity**: 5 points
|
||||
**Dependencies**: 01_test_infrastructure, 02_test_health_auth
|
||||
**Component**: Blackbox Tests
|
||||
**Tracker**: pending
|
||||
**Epic**: pending
|
||||
|
||||
## Problem
|
||||
|
||||
The resource download/upload flow involves complex encryption, binary splitting, and CDN coordination. No test coverage exists to verify this critical path.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Resource download test passes (FT-P-04)
|
||||
- Resource upload test passes (FT-P-05)
|
||||
- Non-existent resource download returns error (FT-N-04)
|
||||
- Upload without file attachment returns error (FT-N-03)
|
||||
- Encryption round-trip integrity verified (NFT-SEC-02)
|
||||
|
||||
## Scope
|
||||
|
||||
### Included
|
||||
- FT-P-04: Download resource via binary-split
|
||||
- FT-P-05: Upload resource via binary-split
|
||||
- FT-N-03: Upload without file attachment
|
||||
- FT-N-04: Download non-existent resource
|
||||
- NFT-SEC-01: Unauthenticated resource access
|
||||
- NFT-SEC-02: Encryption round-trip integrity
|
||||
- NFT-RES-LIM-01: Large file upload
|
||||
|
||||
### Excluded
|
||||
- Unlock workflow tests
|
||||
- Performance benchmarking (separate task)
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: Download returns decrypted resource**
|
||||
Given valid credentials are set and mock API+CDN serve test data
|
||||
When POST /load/testmodel is called
|
||||
Then HTTP 200 with binary content matching the original test resource
|
||||
|
||||
**AC-2: Upload succeeds**
|
||||
Given valid credentials are set
|
||||
When POST /upload/testmodel with file attachment
|
||||
Then HTTP 200 with `{"status": "ok"}`
|
||||
|
||||
**AC-3: Download non-existent resource fails**
|
||||
Given valid credentials are set but resource doesn't exist
|
||||
When POST /load/nonexistent
|
||||
Then HTTP 500 with error detail
|
||||
|
||||
**AC-4: Upload without file fails**
|
||||
Given valid credentials
|
||||
When POST /upload/testfile without file
|
||||
Then HTTP 422
|
||||
|
||||
**AC-5: Unauthenticated download fails**
|
||||
Given no prior login
|
||||
When POST /load/testfile
|
||||
Then HTTP 500
|
||||
|
||||
**AC-6: Encryption round-trip**
|
||||
Given valid credentials
|
||||
When upload a known file then download it back
|
||||
Then downloaded content matches uploaded content
|
||||
|
||||
## Blackbox Tests
|
||||
|
||||
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||
|--------|------------------------|-------------|-------------------|----------------|
|
||||
| AC-1 | Logged in, mock data | POST /load | 200, binary data | — |
|
||||
| AC-2 | Logged in | POST /upload multipart | 200, ok | NFT-RES-LIM-01 |
|
||||
| AC-3 | Logged in, no resource | POST /load | 500, error | — |
|
||||
| AC-4 | Logged in | POST /upload no file | 422 | — |
|
||||
| AC-5 | No login | POST /load | 500 | NFT-SEC-01 |
|
||||
| AC-6 | Logged in | Upload then download | Content matches | NFT-SEC-02 |
|
||||
|
||||
## Risks & Mitigation
|
||||
|
||||
**Risk 1: Mock API must correctly simulate encrypted responses**
|
||||
- *Risk*: Mock API needs to produce AES-256-CBC encrypted test data matching what the real API would return
|
||||
- *Mitigation*: Pre-generate encrypted test fixtures using a known key; mock serves these static files
|
||||
@@ -0,0 +1,82 @@
|
||||
# Unlock Workflow Tests
|
||||
|
||||
**Task**: 04_test_unlock
|
||||
**Name**: Unlock Workflow Tests
|
||||
**Description**: Implement blackbox tests for the Docker image unlock workflow including state machine transitions
|
||||
**Complexity**: 5 points
|
||||
**Dependencies**: 01_test_infrastructure, 02_test_health_auth
|
||||
**Component**: Blackbox Tests
|
||||
**Tracker**: pending
|
||||
**Epic**: pending
|
||||
|
||||
## Problem
|
||||
|
||||
The Docker unlock workflow is the most complex flow in the system — it involves authentication, key fragment download, archive decryption, and Docker image loading. No test coverage exists.
|
||||
|
||||
## Outcome
|
||||
|
||||
- Unlock starts and transitions through all states (FT-P-06)
|
||||
- Unlock detects already-loaded images (FT-P-07)
|
||||
- Unlock status polling works (FT-P-08)
|
||||
- Missing archive returns 404 (FT-N-05)
|
||||
- Concurrent unlock requests handled correctly (NFT-RES-LIM-02)
|
||||
|
||||
## Scope
|
||||
|
||||
### Included
|
||||
- FT-P-06: Unlock starts background workflow (full state cycle)
|
||||
- FT-P-07: Unlock detects already-loaded images
|
||||
- FT-P-08: Unlock status poll (idle state)
|
||||
- FT-N-05: Unlock without encrypted archive
|
||||
- NFT-RES-LIM-02: Concurrent unlock requests
|
||||
|
||||
### Excluded
|
||||
- Resource download/upload tests
|
||||
- Performance benchmarking
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: Unlock starts background workflow**
|
||||
Given encrypted test archive at IMAGES_PATH and mock API configured
|
||||
When POST /unlock with valid credentials
|
||||
Then response contains state field and status transitions to "ready"
|
||||
|
||||
**AC-2: Unlock detects loaded images**
|
||||
Given all API_SERVICES Docker images present with correct tags
|
||||
When POST /unlock
|
||||
Then immediate response with state="ready"
|
||||
|
||||
**AC-3: Unlock status returns current state**
|
||||
Given no unlock has been started
|
||||
When GET /unlock/status
|
||||
Then HTTP 200 with state="idle" and error=null
|
||||
|
||||
**AC-4: Missing archive returns 404**
|
||||
Given no file at IMAGES_PATH and images not loaded
|
||||
When POST /unlock
|
||||
Then HTTP 404 with "Encrypted archive not found"
|
||||
|
||||
**AC-5: Concurrent unlock handled**
|
||||
Given unlock is in progress
|
||||
When a second POST /unlock is sent
|
||||
Then second request returns current in-progress state without starting duplicate
|
||||
|
||||
## Blackbox Tests
|
||||
|
||||
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||
|--------|------------------------|-------------|-------------------|----------------|
|
||||
| AC-1 | Archive exists, mock API | POST /unlock + poll | States → ready | — |
|
||||
| AC-2 | Images loaded | POST /unlock | Immediate ready | — |
|
||||
| AC-3 | Idle state | GET /unlock/status | idle, null error | — |
|
||||
| AC-4 | No archive, no images | POST /unlock | 404 | — |
|
||||
| AC-5 | Unlock in progress | POST /unlock (2nd) | Returns current state | NFT-RES-LIM-02 |
|
||||
|
||||
## Risks & Mitigation
|
||||
|
||||
**Risk 1: Docker daemon required in test environment**
|
||||
- *Risk*: Unlock tests need a real Docker daemon for docker load/inspect
|
||||
- *Mitigation*: Mount Docker socket in test container; use small test images
|
||||
|
||||
**Risk 2: Test archive generation**
|
||||
- *Risk*: Need a valid encrypted archive + matching key fragment
|
||||
- *Mitigation*: Pre-generate a small test archive using the same AES-256-CBC scheme
|
||||
@@ -0,0 +1,66 @@
|
||||
# Resilience & Performance Tests
|
||||
|
||||
**Task**: 05_test_resilience_perf
|
||||
**Name**: Resilience & Performance Tests
|
||||
**Description**: Implement resilience tests (dependency failure) and performance latency tests
|
||||
**Complexity**: 3 points
|
||||
**Dependencies**: 01_test_infrastructure, 02_test_health_auth
|
||||
**Component**: Blackbox Tests
|
||||
**Tracker**: pending
|
||||
**Epic**: pending
|
||||
|
||||
## Problem
|
||||
|
||||
No tests verify system behavior when external dependencies fail, or baseline performance characteristics.
|
||||
|
||||
## Outcome
|
||||
|
||||
- API unavailable during login returns error (NFT-RES-01)
|
||||
- CDN unavailable during download returns error (NFT-RES-02)
|
||||
- Docker daemon unavailable during unlock reports error state (NFT-RES-03)
|
||||
- Health endpoint meets latency threshold (NFT-PERF-01)
|
||||
|
||||
## Scope
|
||||
|
||||
### Included
|
||||
- NFT-RES-01: API unavailable during login
|
||||
- NFT-RES-02: CDN unavailable during resource download
|
||||
- NFT-RES-03: Docker daemon unavailable during unlock
|
||||
- NFT-PERF-01: Health endpoint latency
|
||||
- NFT-PERF-02: Login latency
|
||||
- NFT-PERF-03: Resource download latency
|
||||
|
||||
### Excluded
|
||||
- Blackbox functional tests (covered in other tasks)
|
||||
- NFT-SEC-03 (hardware-bound key test — complex mock setup, tracked separately)
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
**AC-1: API failure handled gracefully**
|
||||
Given the mock API is stopped
|
||||
When POST /login is called
|
||||
Then HTTP 401 with error detail
|
||||
|
||||
**AC-2: CDN failure handled gracefully**
|
||||
Given logged in but mock CDN is stopped
|
||||
When POST /load/testmodel is called
|
||||
Then HTTP 500 with error detail
|
||||
|
||||
**AC-3: Docker failure reported in unlock state**
|
||||
Given Docker socket not mounted
|
||||
When POST /unlock and poll status
|
||||
Then state transitions to "error" with failure description
|
||||
|
||||
**AC-4: Health latency within threshold**
|
||||
Given the loader is running
|
||||
When 100 sequential GET /health requests are sent
|
||||
Then p95 latency ≤ 100ms
|
||||
|
||||
## Blackbox Tests
|
||||
|
||||
| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
|
||||
|--------|------------------------|-------------|-------------------|----------------|
|
||||
| AC-1 | Mock API stopped | POST /login | 401, error | NFT-RES-01 |
|
||||
| AC-2 | CDN stopped, no local cache | POST /load | 500, error | NFT-RES-02 |
|
||||
| AC-3 | No Docker socket | POST /unlock + poll | error state | NFT-RES-03 |
|
||||
| AC-4 | Normal operation | 100x GET /health | p95 ≤ 100ms | NFT-PERF-01 |
|
||||
Reference in New Issue
Block a user