Add E2E tests, fix bugs

Made-with: Cursor

_docs/02_tasks/_dependencies_table.md

@@ -0,0 +1,49 @@
+# Dependencies Table
+
+**Date**: 2026-04-13
+**Total Tasks**: 5
+**Total Complexity Points**: 21
+
+| Task | Name | Complexity | Dependencies | Epic |
+|------|------|-----------|-------------|------|
+| 01 | test_infrastructure | 5 | None | Blackbox Tests |
+| 02 | test_health_auth | 3 | 01 | Blackbox Tests |
+| 03 | test_resources | 5 | 01, 02 | Blackbox Tests |
+| 04 | test_unlock | 5 | 01, 02 | Blackbox Tests |
+| 05 | test_resilience_perf | 3 | 01, 02 | Blackbox Tests |
+
+## Execution Batches
+
+| Batch | Tasks | Parallel? | Total Points |
+|-------|-------|-----------|-------------|
+| 1 | 01_test_infrastructure | No | 5 |
+| 2 | 02_test_health_auth | No | 3 |
+| 3 | 03_test_resources, 04_test_unlock, 05_test_resilience_perf | Yes (parallel) | 13 |
+
+## Test Scenario Coverage
+
+| Test Scenario | Task |
+|--------------|------|
+| FT-P-01 Health | 02 |
+| FT-P-02 Status unauthenticated | 02 |
+| FT-P-03 Login valid | 02 |
+| FT-P-04 Download resource | 03 |
+| FT-P-05 Upload resource | 03 |
+| FT-P-06 Unlock workflow | 04 |
+| FT-P-07 Unlock detect loaded | 04 |
+| FT-P-08 Unlock status | 04 |
+| FT-N-01 Login invalid | 02 |
+| FT-N-02 Login missing fields | 02 |
+| FT-N-03 Upload no file | 03 |
+| FT-N-04 Download nonexistent | 03 |
+| FT-N-05 Unlock no archive | 04 |
+| NFT-PERF-01 Health latency | 05 |
+| NFT-PERF-02 Login latency | 05 |
+| NFT-PERF-03 Download latency | 05 |
+| NFT-RES-01 API unavailable | 05 |
+| NFT-RES-02 CDN unavailable | 05 |
+| NFT-RES-03 Docker unavailable | 05 |
+| NFT-RES-LIM-01 Large upload | 03 |
+| NFT-RES-LIM-02 Concurrent unlock | 04 |
+| NFT-SEC-01 Unauth access | 03 |
+| NFT-SEC-02 Encrypt round-trip | 03 |

_docs/02_tasks/done/01_test_infrastructure.md

@@ -0,0 +1,117 @@
+# Test Infrastructure
+
+**Task**: 01_test_infrastructure
+**Name**: Test Infrastructure
+**Description**: Scaffold the blackbox test project — pytest runner, mock API server, mock CDN (MinIO), Docker test environment, test data fixtures, CSV reporting
+**Complexity**: 5 points
+**Dependencies**: None
+**Component**: Blackbox Tests
+**Tracker**: pending
+**Epic**: pending
+
+## Test Project Folder Layout
+
+```
+e2e/
+├── conftest.py
+├── requirements.txt
+├── mocks/
+│   └── mock_api/
+│       ├── Dockerfile
+│       └── app.py
+├── fixtures/
+│   ├── test_resource.bin
+│   └── test_archive.enc
+├── tests/
+│   ├── test_health.py
+│   ├── test_auth.py
+│   ├── test_resources.py
+│   ├── test_unlock.py
+│   ├── test_security.py
+│   ├── test_performance.py
+│   └── test_resilience.py
+└── docker-compose.test.yml
+```
+
+## Mock Services
+
+| Mock Service | Replaces | Endpoints | Behavior |
+|-------------|----------|-----------|----------|
+| mock-api | Azaion Resource API | POST /login, POST /resources/get/{folder}, POST /resources/{folder}, GET /resources/list/{folder}, GET /binary-split/key-fragment | Returns canned JWT, encrypted test resources, key fragment |
+| mock-cdn (MinIO) | S3 CDN | S3 API (standard) | S3-compatible storage with pre-seeded test .big files |
+
+## Docker Test Environment
+
+### docker-compose.test.yml Structure
+
+| Service | Image / Build | Purpose | Depends On |
+|---------|--------------|---------|------------|
+| system-under-test | Build from Dockerfile | Azaion.Loader | mock-api, mock-cdn |
+| mock-api | Build from e2e/mocks/mock_api/ | Mock Azaion Resource API | — |
+| mock-cdn | minio/minio | Mock S3 CDN | — |
+| e2e-consumer | python:3.11-slim + e2e/ | Pytest test runner | system-under-test |
+
+### Networks and Volumes
+
+- `e2e-net`: isolated test network connecting all services
+- `test-data` volume: mounted to e2e-consumer for test fixtures
+- Docker socket: mounted to system-under-test for unlock flow
+
+## Test Runner Configuration
+
+**Framework**: pytest
+**Plugins**: pytest-csv (reporting), requests (HTTP client)
+**Entry point**: `pytest tests/ --csv=/results/report.csv -v`
+
+### Fixture Strategy
+
+| Fixture | Scope | Purpose |
+|---------|-------|---------|
+| base_url | session | URL of the system-under-test |
+| logged_in_client | function | requests.Session with /login called |
+| mock_api_url | session | URL of the mock API |
+
+## Test Data Fixtures
+
+| Data Set | Source | Format | Used By |
+|----------|--------|--------|---------|
+| test_resource.bin | Generated (small binary) | Binary | test_resources.py |
+| test_archive.enc | Generated (AES-encrypted tar) | Binary | test_unlock.py |
+| cdn.yaml | Generated (mock CDN config) | YAML | conftest.py (served by mock-api) |
+
+### Data Isolation
+
+Fresh container restart per test run. Mock API state is stateless (canned responses). MinIO bucket re-created on startup.
+
+## Test Reporting
+
+**Format**: CSV
+**Columns**: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message
+**Output path**: `/results/report.csv` → mounted to `./e2e-results/report.csv` on host
+
+## Acceptance Criteria
+
+**AC-1: Test environment starts**
+Given the docker-compose.test.yml
+When `docker compose -f e2e/docker-compose.test.yml up` is executed
+Then all services start and the system-under-test health endpoint responds
+
+**AC-2: Mock API responds**
+Given the test environment is running
+When the e2e-consumer sends POST /login to the mock API
+Then the mock API returns a valid JWT response
+
+**AC-3: Mock CDN operational**
+Given the test environment is running
+When the e2e-consumer uploads/downloads a file to MinIO
+Then S3 operations succeed
+
+**AC-4: Test runner discovers tests**
+Given the test environment is running
+When the e2e-consumer starts
+Then pytest discovers all test files in e2e/tests/
+
+**AC-5: Test report generated**
+Given tests have completed
+When the test run finishes
+Then a CSV report exists at /results/report.csv with correct columns

_docs/02_tasks/done/02_test_health_auth.md

@@ -0,0 +1,71 @@
+# Health & Authentication Tests
+
+**Task**: 02_test_health_auth
+**Name**: Health & Authentication Tests
+**Description**: Implement blackbox tests for health, status, and login endpoints (positive and negative scenarios)
+**Complexity**: 3 points
+**Dependencies**: 01_test_infrastructure
+**Component**: Blackbox Tests
+**Tracker**: pending
+**Epic**: pending
+
+## Problem
+
+The loader has no test coverage for its health and authentication endpoints. These are the most basic verification points for service liveness and user access.
+
+## Outcome
+
+- Health endpoint test passes (FT-P-01)
+- Status endpoint tests pass — unauthenticated and authenticated (FT-P-02, FT-P-03 step 2)
+- Login positive test passes (FT-P-03)
+- Login negative tests pass — invalid credentials and missing fields (FT-N-01, FT-N-02)
+
+## Scope
+
+### Included
+- FT-P-01: Health endpoint returns healthy
+- FT-P-02: Status reports unauthenticated state
+- FT-P-03: Login with valid credentials (including authenticated status check)
+- FT-N-01: Login with invalid credentials
+- FT-N-02: Login with missing fields
+
+### Excluded
+- Resource download/upload tests
+- Unlock workflow tests
+
+## Acceptance Criteria
+
+**AC-1: Health returns 200**
+Given the loader is running
+When GET /health is called
+Then HTTP 200 with body `{"status": "healthy"}`
+
+**AC-2: Status shows unauthenticated before login**
+Given the loader is running with no prior login
+When GET /status is called
+Then HTTP 200 with `authenticated: false`
+
+**AC-3: Login succeeds with valid credentials**
+Given the mock API accepts test credentials
+When POST /login with valid email/password
+Then HTTP 200 with `{"status": "ok"}`
+
+**AC-4: Login fails with invalid credentials**
+Given the mock API rejects test credentials
+When POST /login with wrong email/password
+Then HTTP 401
+
+**AC-5: Login rejects empty body**
+Given the loader is running
+When POST /login with empty JSON
+Then HTTP 422
+
+## Blackbox Tests
+
+| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
+|--------|------------------------|-------------|-------------------|----------------|
+| AC-1 | Loader running | GET /health | 200, {"status": "healthy"} | NFT-PERF-01 |
+| AC-2 | No prior login | GET /status | 200, authenticated=false | — |
+| AC-3 | Mock API accepts creds | POST /login valid | 200, status ok | NFT-PERF-02 |
+| AC-4 | Mock API rejects creds | POST /login invalid | 401 | — |
+| AC-5 | — | POST /login empty | 422 | — |

_docs/02_tasks/done/03_test_resources.md

@@ -0,0 +1,86 @@
+# Resource Download & Upload Tests
+
+**Task**: 03_test_resources
+**Name**: Resource Download & Upload Tests
+**Description**: Implement blackbox tests for resource download (binary-split) and upload endpoints
+**Complexity**: 5 points
+**Dependencies**: 01_test_infrastructure, 02_test_health_auth
+**Component**: Blackbox Tests
+**Tracker**: pending
+**Epic**: pending
+
+## Problem
+
+The resource download/upload flow involves complex encryption, binary splitting, and CDN coordination. No test coverage exists to verify this critical path.
+
+## Outcome
+
+- Resource download test passes (FT-P-04)
+- Resource upload test passes (FT-P-05)
+- Non-existent resource download returns error (FT-N-04)
+- Upload without file attachment returns error (FT-N-03)
+- Encryption round-trip integrity verified (NFT-SEC-02)
+
+## Scope
+
+### Included
+- FT-P-04: Download resource via binary-split
+- FT-P-05: Upload resource via binary-split
+- FT-N-03: Upload without file attachment
+- FT-N-04: Download non-existent resource
+- NFT-SEC-01: Unauthenticated resource access
+- NFT-SEC-02: Encryption round-trip integrity
+- NFT-RES-LIM-01: Large file upload
+
+### Excluded
+- Unlock workflow tests
+- Performance benchmarking (separate task)
+
+## Acceptance Criteria
+
+**AC-1: Download returns decrypted resource**
+Given valid credentials are set and mock API+CDN serve test data
+When POST /load/testmodel is called
+Then HTTP 200 with binary content matching the original test resource
+
+**AC-2: Upload succeeds**
+Given valid credentials are set
+When POST /upload/testmodel with file attachment
+Then HTTP 200 with `{"status": "ok"}`
+
+**AC-3: Download non-existent resource fails**
+Given valid credentials are set but resource doesn't exist
+When POST /load/nonexistent
+Then HTTP 500 with error detail
+
+**AC-4: Upload without file fails**
+Given valid credentials
+When POST /upload/testfile without file
+Then HTTP 422
+
+**AC-5: Unauthenticated download fails**
+Given no prior login
+When POST /load/testfile
+Then HTTP 500
+
+**AC-6: Encryption round-trip**
+Given valid credentials
+When upload a known file then download it back
+Then downloaded content matches uploaded content
+
+## Blackbox Tests
+
+| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
+|--------|------------------------|-------------|-------------------|----------------|
+| AC-1 | Logged in, mock data | POST /load | 200, binary data | — |
+| AC-2 | Logged in | POST /upload multipart | 200, ok | NFT-RES-LIM-01 |
+| AC-3 | Logged in, no resource | POST /load | 500, error | — |
+| AC-4 | Logged in | POST /upload no file | 422 | — |
+| AC-5 | No login | POST /load | 500 | NFT-SEC-01 |
+| AC-6 | Logged in | Upload then download | Content matches | NFT-SEC-02 |
+
+## Risks & Mitigation
+
+**Risk 1: Mock API must correctly simulate encrypted responses**
+- *Risk*: Mock API needs to produce AES-256-CBC encrypted test data matching what the real API would return
+- *Mitigation*: Pre-generate encrypted test fixtures using a known key; mock serves these static files

_docs/02_tasks/done/04_test_unlock.md

@@ -0,0 +1,82 @@
+# Unlock Workflow Tests
+
+**Task**: 04_test_unlock
+**Name**: Unlock Workflow Tests
+**Description**: Implement blackbox tests for the Docker image unlock workflow including state machine transitions
+**Complexity**: 5 points
+**Dependencies**: 01_test_infrastructure, 02_test_health_auth
+**Component**: Blackbox Tests
+**Tracker**: pending
+**Epic**: pending
+
+## Problem
+
+The Docker unlock workflow is the most complex flow in the system — it involves authentication, key fragment download, archive decryption, and Docker image loading. No test coverage exists.
+
+## Outcome
+
+- Unlock starts and transitions through all states (FT-P-06)
+- Unlock detects already-loaded images (FT-P-07)
+- Unlock status polling works (FT-P-08)
+- Missing archive returns 404 (FT-N-05)
+- Concurrent unlock requests handled correctly (NFT-RES-LIM-02)
+
+## Scope
+
+### Included
+- FT-P-06: Unlock starts background workflow (full state cycle)
+- FT-P-07: Unlock detects already-loaded images
+- FT-P-08: Unlock status poll (idle state)
+- FT-N-05: Unlock without encrypted archive
+- NFT-RES-LIM-02: Concurrent unlock requests
+
+### Excluded
+- Resource download/upload tests
+- Performance benchmarking
+
+## Acceptance Criteria
+
+**AC-1: Unlock starts background workflow**
+Given encrypted test archive at IMAGES_PATH and mock API configured
+When POST /unlock with valid credentials
+Then response contains state field and status transitions to "ready"
+
+**AC-2: Unlock detects loaded images**
+Given all API_SERVICES Docker images present with correct tags
+When POST /unlock
+Then immediate response with state="ready"
+
+**AC-3: Unlock status returns current state**
+Given no unlock has been started
+When GET /unlock/status
+Then HTTP 200 with state="idle" and error=null
+
+**AC-4: Missing archive returns 404**
+Given no file at IMAGES_PATH and images not loaded
+When POST /unlock
+Then HTTP 404 with "Encrypted archive not found"
+
+**AC-5: Concurrent unlock handled**
+Given unlock is in progress
+When a second POST /unlock is sent
+Then second request returns current in-progress state without starting duplicate
+
+## Blackbox Tests
+
+| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
+|--------|------------------------|-------------|-------------------|----------------|
+| AC-1 | Archive exists, mock API | POST /unlock + poll | States → ready | — |
+| AC-2 | Images loaded | POST /unlock | Immediate ready | — |
+| AC-3 | Idle state | GET /unlock/status | idle, null error | — |
+| AC-4 | No archive, no images | POST /unlock | 404 | — |
+| AC-5 | Unlock in progress | POST /unlock (2nd) | Returns current state | NFT-RES-LIM-02 |
+
+## Risks & Mitigation
+
+**Risk 1: Docker daemon required in test environment**
+- *Risk*: Unlock tests need a real Docker daemon for docker load/inspect
+- *Mitigation*: Mount Docker socket in test container; use small test images
+
+**Risk 2: Test archive generation**
+- *Risk*: Need a valid encrypted archive + matching key fragment
+- *Mitigation*: Pre-generate a small test archive using the same AES-256-CBC scheme

_docs/02_tasks/done/05_test_resilience_perf.md

@@ -0,0 +1,66 @@
+# Resilience & Performance Tests
+
+**Task**: 05_test_resilience_perf
+**Name**: Resilience & Performance Tests
+**Description**: Implement resilience tests (dependency failure) and performance latency tests
+**Complexity**: 3 points
+**Dependencies**: 01_test_infrastructure, 02_test_health_auth
+**Component**: Blackbox Tests
+**Tracker**: pending
+**Epic**: pending
+
+## Problem
+
+No tests verify system behavior when external dependencies fail, or baseline performance characteristics.
+
+## Outcome
+
+- API unavailable during login returns error (NFT-RES-01)
+- CDN unavailable during download returns error (NFT-RES-02)
+- Docker daemon unavailable during unlock reports error state (NFT-RES-03)
+- Health endpoint meets latency threshold (NFT-PERF-01)
+
+## Scope
+
+### Included
+- NFT-RES-01: API unavailable during login
+- NFT-RES-02: CDN unavailable during resource download
+- NFT-RES-03: Docker daemon unavailable during unlock
+- NFT-PERF-01: Health endpoint latency
+- NFT-PERF-02: Login latency
+- NFT-PERF-03: Resource download latency
+
+### Excluded
+- Blackbox functional tests (covered in other tasks)
+- NFT-SEC-03 (hardware-bound key test — complex mock setup, tracked separately)
+
+## Acceptance Criteria
+
+**AC-1: API failure handled gracefully**
+Given the mock API is stopped
+When POST /login is called
+Then HTTP 401 with error detail
+
+**AC-2: CDN failure handled gracefully**
+Given logged in but mock CDN is stopped
+When POST /load/testmodel is called
+Then HTTP 500 with error detail
+
+**AC-3: Docker failure reported in unlock state**
+Given Docker socket not mounted
+When POST /unlock and poll status
+Then state transitions to "error" with failure description
+
+**AC-4: Health latency within threshold**
+Given the loader is running
+When 100 sequential GET /health requests are sent
+Then p95 latency ≤ 100ms
+
+## Blackbox Tests
+
+| AC Ref | Initial Data/Conditions | What to Test | Expected Behavior | NFR References |
+|--------|------------------------|-------------|-------------------|----------------|
+| AC-1 | Mock API stopped | POST /login | 401, error | NFT-RES-01 |
+| AC-2 | CDN stopped, no local cache | POST /load | 500, error | NFT-RES-02 |
+| AC-3 | No Docker socket | POST /unlock + poll | error state | NFT-RES-03 |
+| AC-4 | Normal operation | 100x GET /health | p95 ≤ 100ms | NFT-PERF-01 |