mirror of
https://github.com/azaion/loader.git
synced 2026-04-22 12:16:32 +00:00
Add E2E tests, fix bugs
Made-with: Cursor
This commit is contained in:
@@ -0,0 +1,280 @@
|
||||
# Blackbox Tests
|
||||
|
||||
## Positive Scenarios
|
||||
|
||||
### FT-P-01: Health endpoint returns healthy
|
||||
|
||||
**Summary**: Verify the liveness probe returns a healthy status without authentication.
|
||||
**Traces to**: AC-1
|
||||
**Category**: Health Check
|
||||
|
||||
**Preconditions**: Loader service is running.
|
||||
|
||||
**Input data**: None
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | GET /health | HTTP 200, body: `{"status": "healthy"}` |
|
||||
|
||||
**Expected outcome**: HTTP 200 with exact body `{"status": "healthy"}`
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-02: Status reports unauthenticated state
|
||||
|
||||
**Summary**: Verify status endpoint reports no authentication before login.
|
||||
**Traces to**: AC-1
|
||||
**Category**: Health Check
|
||||
|
||||
**Preconditions**: Loader service is running, no prior login.
|
||||
|
||||
**Input data**: None
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | GET /status | HTTP 200, body contains `"authenticated": false` and `"modelCacheDir": "models"` |
|
||||
|
||||
**Expected outcome**: HTTP 200 with `authenticated=false`
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-03: Login with valid credentials
|
||||
|
||||
**Summary**: Verify login succeeds with valid email/password and sets credentials on the API client.
|
||||
**Traces to**: AC-2, AC-14
|
||||
**Category**: Authentication
|
||||
|
||||
**Preconditions**: Loader service is running, mock API configured to accept credentials.
|
||||
|
||||
**Input data**: `{"email": "test@azaion.com", "password": "validpass"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with valid credentials | HTTP 200, body: `{"status": "ok"}` |
|
||||
| 2 | GET /status | HTTP 200, body contains `"authenticated": true` |
|
||||
|
||||
**Expected outcome**: Login returns 200; subsequent status shows authenticated=true
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-04: Download resource via binary-split
|
||||
|
||||
**Summary**: Verify a resource can be downloaded and decrypted through the big/small split scheme.
|
||||
**Traces to**: AC-4, AC-11, AC-13
|
||||
**Category**: Resource Download
|
||||
|
||||
**Preconditions**: Logged in; mock API serves encrypted small part; mock CDN hosts big part.
|
||||
|
||||
**Input data**: `{"filename": "testmodel", "folder": "models"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with valid credentials | HTTP 200 |
|
||||
| 2 | POST /load/testmodel with body `{"filename": "testmodel", "folder": "models"}` | HTTP 200, Content-Type: application/octet-stream, non-empty body |
|
||||
|
||||
**Expected outcome**: HTTP 200 with binary content matching the original test resource
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-05: Upload resource via binary-split
|
||||
|
||||
**Summary**: Verify a resource can be uploaded, split, encrypted, and stored.
|
||||
**Traces to**: AC-5
|
||||
**Category**: Resource Upload
|
||||
|
||||
**Preconditions**: Logged in; mock API accepts uploads; mock CDN accepts writes.
|
||||
|
||||
**Input data**: Binary test file + folder="models"
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with valid credentials | HTTP 200 |
|
||||
| 2 | POST /upload/testmodel multipart (file=test_bytes, folder="models") | HTTP 200, body: `{"status": "ok"}` |
|
||||
|
||||
**Expected outcome**: Upload returns 200; big part present on CDN, small part on mock API
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-06: Unlock starts background workflow
|
||||
|
||||
**Summary**: Verify unlock endpoint starts the background decryption and Docker loading workflow.
|
||||
**Traces to**: AC-6, AC-9
|
||||
**Category**: Docker Unlock
|
||||
|
||||
**Preconditions**: Encrypted test archive at IMAGES_PATH; Docker daemon accessible; mock API configured.
|
||||
|
||||
**Input data**: `{"email": "test@azaion.com", "password": "validpass"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /unlock with valid credentials | HTTP 200, body contains `"state"` field |
|
||||
| 2 | Poll GET /unlock/status until state changes | States progress through: authenticating → downloading_key → decrypting → loading_images → ready |
|
||||
|
||||
**Expected outcome**: Final state is "ready"
|
||||
**Max execution time**: 60s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-07: Unlock detects already-loaded images
|
||||
|
||||
**Summary**: Verify unlock returns immediately when Docker images are already present.
|
||||
**Traces to**: AC-7
|
||||
**Category**: Docker Unlock
|
||||
|
||||
**Preconditions**: All 7 API_SERVICES Docker images already loaded with correct version tag.
|
||||
|
||||
**Input data**: `{"email": "test@azaion.com", "password": "validpass"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /unlock with valid credentials | HTTP 200, body: `{"state": "ready"}` |
|
||||
|
||||
**Expected outcome**: Immediate ready state, no background processing
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-P-08: Unlock status poll
|
||||
|
||||
**Summary**: Verify unlock status endpoint returns current state and error.
|
||||
**Traces to**: AC-8
|
||||
**Category**: Docker Unlock
|
||||
|
||||
**Preconditions**: No unlock started (idle state).
|
||||
|
||||
**Input data**: None
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | GET /unlock/status | HTTP 200, body: `{"state": "idle", "error": null}` |
|
||||
|
||||
**Expected outcome**: State is idle, error is null
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
## Negative Scenarios
|
||||
|
||||
### FT-N-01: Login with invalid credentials
|
||||
|
||||
**Summary**: Verify login rejects invalid credentials with HTTP 401.
|
||||
**Traces to**: AC-3
|
||||
**Category**: Authentication
|
||||
|
||||
**Preconditions**: Loader service is running; mock API rejects these credentials.
|
||||
|
||||
**Input data**: `{"email": "bad@test.com", "password": "wrongpass"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with invalid credentials | HTTP 401, body has `"detail"` field |
|
||||
|
||||
**Expected outcome**: HTTP 401 with error detail
|
||||
**Max execution time**: 5s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-02: Login with missing fields
|
||||
|
||||
**Summary**: Verify login rejects requests with missing email/password fields.
|
||||
**Traces to**: AC-3
|
||||
**Category**: Authentication
|
||||
|
||||
**Preconditions**: Loader service is running.
|
||||
|
||||
**Input data**: `{}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with empty JSON body | HTTP 422 (validation error) |
|
||||
|
||||
**Expected outcome**: HTTP 422 from Pydantic validation
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-03: Upload without file attachment
|
||||
|
||||
**Summary**: Verify upload rejects requests without a file.
|
||||
**Traces to**: AC-5 (negative)
|
||||
**Category**: Resource Upload
|
||||
|
||||
**Preconditions**: Logged in.
|
||||
|
||||
**Input data**: POST without multipart file
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /upload/testfile without file attachment | HTTP 422 |
|
||||
|
||||
**Expected outcome**: HTTP 422 validation error
|
||||
**Max execution time**: 2s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-04: Download non-existent resource
|
||||
|
||||
**Summary**: Verify download returns 500 when the requested resource does not exist.
|
||||
**Traces to**: AC-4 (negative)
|
||||
**Category**: Resource Download
|
||||
|
||||
**Preconditions**: Logged in; resource "nonexistent" does not exist on API or CDN.
|
||||
|
||||
**Input data**: `{"filename": "nonexistent", "folder": "models"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /load/nonexistent with body | HTTP 500, body has `"detail"` field |
|
||||
|
||||
**Expected outcome**: HTTP 500 with error detail
|
||||
**Max execution time**: 10s
|
||||
|
||||
---
|
||||
|
||||
### FT-N-05: Unlock without encrypted archive
|
||||
|
||||
**Summary**: Verify unlock returns 404 when no encrypted archive is present and images are not loaded.
|
||||
**Traces to**: AC-10
|
||||
**Category**: Docker Unlock
|
||||
|
||||
**Preconditions**: No file at IMAGES_PATH; Docker images not loaded.
|
||||
|
||||
**Input data**: `{"email": "test@azaion.com", "password": "validpass"}`
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /unlock with valid credentials | HTTP 404, body has `"detail"` containing "Encrypted archive not found" |
|
||||
|
||||
**Expected outcome**: HTTP 404 with archive-not-found message
|
||||
**Max execution time**: 5s
|
||||
@@ -0,0 +1,75 @@
|
||||
# Test Environment
|
||||
|
||||
## Overview
|
||||
|
||||
**System under test**: Azaion.Loader FastAPI service at `http://localhost:8080`
|
||||
**Consumer app purpose**: Python pytest suite exercising the loader through its HTTP API, validating black-box use cases without access to Cython internals.
|
||||
|
||||
## Test Execution
|
||||
|
||||
**Decision**: Local execution
|
||||
**Hardware dependencies found**:
|
||||
- `hardware_service.pyx`: uses `subprocess` with `lscpu`, `lspci`, `/sys/block/sda` (Linux) or PowerShell (Windows) — requires real OS hardware info
|
||||
- `binary_split.py`: calls `docker load` and `docker image inspect` — requires Docker daemon
|
||||
- Cython extensions: must be compiled natively for the target platform
|
||||
|
||||
**Execution instructions (local)**:
|
||||
1. Prerequisites: Python 3.11, GCC, Docker daemon running
|
||||
2. Install deps: `pip install -r requirements.txt && python setup.py build_ext --inplace`
|
||||
3. Start system: `uvicorn main:app --host 0.0.0.0 --port 8080`
|
||||
4. Run tests: `pytest tests/ -v --tb=short`
|
||||
5. Environment variables: `RESOURCE_API_URL`, `IMAGES_PATH`, `API_VERSION`
|
||||
|
||||
## Docker Environment
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Image / Build | Purpose | Ports |
|
||||
|---------|--------------|---------|-------|
|
||||
| system-under-test | Build from `Dockerfile` | Azaion.Loader | 8080 |
|
||||
| mock-api | Python (httpbin or custom) | Mock Azaion Resource API | 9090 |
|
||||
| mock-cdn | MinIO (S3-compatible) | Mock S3 CDN | 9000 |
|
||||
| e2e-consumer | `python:3.11-slim` + pytest | Black-box test runner | — |
|
||||
|
||||
### Networks
|
||||
|
||||
| Network | Services | Purpose |
|
||||
|---------|----------|---------|
|
||||
| e2e-net | all | Isolated test network |
|
||||
|
||||
### Volumes
|
||||
|
||||
| Volume | Mounted to | Purpose |
|
||||
|--------|-----------|---------|
|
||||
| test-data | e2e-consumer:/data | Test input files |
|
||||
| docker-sock | system-under-test:/var/run/docker.sock | Docker daemon access |
|
||||
|
||||
## Consumer Application
|
||||
|
||||
**Tech stack**: Python 3.11, pytest, requests
|
||||
**Entry point**: `pytest tests/ -v`
|
||||
|
||||
### Communication with system under test
|
||||
|
||||
| Interface | Protocol | Endpoint | Authentication |
|
||||
|-----------|----------|----------|----------------|
|
||||
| Loader API | HTTP | `http://system-under-test:8080` | POST /login first |
|
||||
|
||||
### What the consumer does NOT have access to
|
||||
|
||||
- No direct access to Cython `.so` modules
|
||||
- No shared filesystem with the main system (except Docker socket for verification)
|
||||
- No direct access to mock-api or mock-cdn internals
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
**When to run**: On push to dev/stage/main (extend `.woodpecker/build-arm.yml`)
|
||||
**Pipeline stage**: After build, before push
|
||||
**Gate behavior**: Block push on failure
|
||||
**Timeout**: 300 seconds (5 minutes)
|
||||
|
||||
## Reporting
|
||||
|
||||
**Format**: CSV
|
||||
**Columns**: Test ID, Test Name, Execution Time (ms), Result (PASS/FAIL/SKIP), Error Message
|
||||
**Output path**: `./test-results/report.csv`
|
||||
@@ -0,0 +1,50 @@
|
||||
# Performance Tests
|
||||
|
||||
### NFT-PERF-01: Health endpoint latency
|
||||
|
||||
**Summary**: Verify health endpoint responds within acceptable time under normal load.
|
||||
**Traces to**: AC-1
|
||||
**Category**: Latency
|
||||
|
||||
**Preconditions**: Loader service is running.
|
||||
|
||||
**Scenario**:
|
||||
- Send 100 sequential GET /health requests
|
||||
- Measure p95 response time
|
||||
|
||||
**Expected outcome**: p95 latency ≤ 100ms
|
||||
**Threshold**: `threshold_max: 100ms`
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-02: Login latency
|
||||
|
||||
**Summary**: Verify login completes within acceptable time.
|
||||
**Traces to**: AC-2
|
||||
**Category**: Latency
|
||||
|
||||
**Preconditions**: Loader service is running; mock API available.
|
||||
|
||||
**Scenario**:
|
||||
- Send 10 sequential POST /login requests
|
||||
- Measure p95 response time
|
||||
|
||||
**Expected outcome**: p95 latency ≤ 2000ms (includes mock API round-trip)
|
||||
**Threshold**: `threshold_max: 2000ms`
|
||||
|
||||
---
|
||||
|
||||
### NFT-PERF-03: Resource download latency (small resource)
|
||||
|
||||
**Summary**: Verify small resource download completes within acceptable time.
|
||||
**Traces to**: AC-4
|
||||
**Category**: Latency
|
||||
|
||||
**Preconditions**: Logged in; mock API and CDN serving a 10KB test resource.
|
||||
|
||||
**Scenario**:
|
||||
- Send 5 sequential POST /load/smallfile requests
|
||||
- Measure p95 response time
|
||||
|
||||
**Expected outcome**: p95 latency ≤ 5000ms
|
||||
**Threshold**: `threshold_max: 5000ms`
|
||||
@@ -0,0 +1,54 @@
|
||||
# Resilience Tests
|
||||
|
||||
### NFT-RES-01: API unavailable during login
|
||||
|
||||
**Summary**: Verify the system returns an error when the upstream API is unreachable.
|
||||
**Traces to**: AC-2 (negative), AC-3
|
||||
**Category**: External dependency failure
|
||||
|
||||
**Preconditions**: Loader service is running; mock API is stopped.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with valid credentials | HTTP 401, body has `"detail"` field with connection error |
|
||||
|
||||
**Expected outcome**: HTTP 401 with error message indicating API unreachable
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-02: CDN unavailable during resource download
|
||||
|
||||
**Summary**: Verify the system returns an error when CDN is unreachable and no local cache exists.
|
||||
**Traces to**: AC-4 (negative)
|
||||
**Category**: External dependency failure
|
||||
|
||||
**Preconditions**: Logged in; mock CDN is stopped; no local `.big` file cached.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /load/testmodel | HTTP 500, body has `"detail"` field |
|
||||
|
||||
**Expected outcome**: HTTP 500 indicating CDN download failure
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-03: Docker daemon unavailable during unlock
|
||||
|
||||
**Summary**: Verify unlock reports error when Docker daemon is not accessible.
|
||||
**Traces to**: AC-9 (negative)
|
||||
**Category**: External dependency failure
|
||||
|
||||
**Preconditions**: Docker socket not mounted / daemon stopped; encrypted archive exists.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /unlock with valid credentials | HTTP 200 (background task starts) |
|
||||
| 2 | Poll GET /unlock/status | State transitions to "error", error field describes Docker failure |
|
||||
|
||||
**Expected outcome**: unlock_state = "error" with CalledProcessError detail
|
||||
@@ -0,0 +1,37 @@
|
||||
# Resource Limit Tests
|
||||
|
||||
### NFT-RES-LIM-01: Large file upload
|
||||
|
||||
**Summary**: Verify the system handles uploading a large resource (>10MB) without crashing.
|
||||
**Traces to**: AC-5
|
||||
**Category**: File size limits
|
||||
|
||||
**Preconditions**: Logged in; mock API and CDN available.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /upload/largefile multipart (file=10MB random bytes) | HTTP 200, body: `{"status": "ok"}` |
|
||||
|
||||
**Expected outcome**: Upload succeeds; file is split into small (≤3KB or 30%) and big parts
|
||||
**Max execution time**: 30s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-LIM-02: Concurrent unlock requests
|
||||
|
||||
**Summary**: Verify the system correctly handles multiple simultaneous unlock requests (only one should proceed).
|
||||
**Traces to**: AC-6
|
||||
**Category**: Concurrency
|
||||
|
||||
**Preconditions**: Encrypted archive at IMAGES_PATH; Docker daemon accessible.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /unlock (request A) | HTTP 200, state starts processing |
|
||||
| 2 | POST /unlock (request B, concurrent) | HTTP 200, returns current in-progress state (does not start second unlock) |
|
||||
|
||||
**Expected outcome**: Only one unlock runs; second request returns current state without starting a duplicate
|
||||
@@ -0,0 +1,51 @@
|
||||
# Security Tests
|
||||
|
||||
### NFT-SEC-01: Unauthenticated resource access
|
||||
|
||||
**Summary**: Verify resource download fails when no credentials have been set.
|
||||
**Traces to**: AC-4 (negative), AC-14
|
||||
**Category**: Authentication enforcement
|
||||
|
||||
**Preconditions**: Loader service is running; no prior login.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /load/testfile without prior login | HTTP 500 (ApiClient has no credentials/token) |
|
||||
|
||||
**Expected outcome**: Resource access denied when not authenticated
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-02: Encryption round-trip integrity
|
||||
|
||||
**Summary**: Verify that encrypt→decrypt with the same key returns the original data (validates AES-256-CBC implementation).
|
||||
**Traces to**: AC-11
|
||||
**Category**: Data encryption
|
||||
|
||||
**Preconditions**: Upload a known resource, then download it back.
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Consumer Action | Expected System Response |
|
||||
|------|----------------|------------------------|
|
||||
| 1 | POST /login with valid credentials | HTTP 200 |
|
||||
| 2 | POST /upload/roundtrip multipart (file=known_bytes) | HTTP 200 |
|
||||
| 3 | POST /load/roundtrip with body `{"filename": "roundtrip", "folder": "models"}` | HTTP 200, body matches original known_bytes |
|
||||
|
||||
**Expected outcome**: Downloaded content is byte-identical to uploaded content
|
||||
|
||||
---
|
||||
|
||||
### NFT-SEC-03: Hardware-bound key produces different keys for different hardware strings
|
||||
|
||||
**Summary**: Verify that different hardware fingerprints produce different encryption keys (tested indirectly through behavior: a resource encrypted on one machine cannot be decrypted by another).
|
||||
**Traces to**: AC-12
|
||||
**Category**: Hardware binding
|
||||
|
||||
**Note**: This is a behavioral test — the consumer cannot directly call `get_hw_hash()` (Cython cdef). Instead, verify that a resource downloaded from the API cannot be decrypted with a different hardware context. This may require mocking the Resource API to return content encrypted with a known hardware-bound key.
|
||||
|
||||
**Preconditions**: Mock API configured with hardware-specific encrypted response.
|
||||
|
||||
**Expected outcome**: Decryption succeeds with matching hardware context; fails with mismatched context.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Test Data Management
|
||||
|
||||
## Seed Data Sets
|
||||
|
||||
| Data Set | Description | Used by Tests | How Loaded | Cleanup |
|
||||
|----------|-------------|---------------|-----------|---------|
|
||||
| mock-api-responses | Canned responses for mock Azaion Resource API (JWT, resources, key fragments) | All FT-P, FT-N tests | Mock server config | Container restart |
|
||||
| mock-cdn-data | Pre-uploaded `.big` files on MinIO | FT-P-04, FT-P-05, FT-N-04 | MinIO CLI seed script | Container restart |
|
||||
| test-resource | Small binary blob for encrypt/decrypt round-trip | FT-P-04, FT-P-05 | File on consumer volume | N/A (read-only) |
|
||||
| test-archive | Small encrypted `.enc` file + key fragment for unlock tests | FT-P-06, FT-P-07, FT-N-05 | File on SUT volume | Container restart |
|
||||
|
||||
## Data Isolation Strategy
|
||||
|
||||
Each test run starts with fresh container state. No shared mutable state between tests — mock API and CDN are reset per run.
|
||||
|
||||
## Input Data Mapping
|
||||
|
||||
| Input Data File | Source Location | Description | Covers Scenarios |
|
||||
|-----------------|----------------|-------------|-----------------|
|
||||
| data_parameters.md | `_docs/00_problem/input_data/data_parameters.md` | API request/response schemas | All tests (schema reference) |
|
||||
| results_report.md | `_docs/00_problem/input_data/expected_results/results_report.md` | Expected results mapping | All tests (expected outcomes) |
|
||||
|
||||
## Expected Results Mapping
|
||||
|
||||
| Test Scenario ID | Input Data | Expected Result | Comparison Method | Tolerance | Source |
|
||||
|-----------------|------------|-----------------|-------------------|-----------|--------|
|
||||
| FT-P-01 | GET /health | HTTP 200, `{"status": "healthy"}` | exact | N/A | inline |
|
||||
| FT-P-02 | GET /status (no login) | HTTP 200, authenticated=false | exact | N/A | inline |
|
||||
| FT-P-03 | POST /login valid creds | HTTP 200, `{"status": "ok"}` | exact | N/A | inline |
|
||||
| FT-P-04 | POST /load/testfile | HTTP 200, binary content | exact (status), threshold_min (length > 0) | N/A | inline |
|
||||
| FT-P-05 | POST /upload/testfile | HTTP 200, `{"status": "ok"}` | exact | N/A | inline |
|
||||
| FT-P-06 | POST /unlock valid creds | HTTP 200, state transition | exact | N/A | inline |
|
||||
| FT-P-07 | GET /unlock/status | HTTP 200, state + error fields | schema | N/A | inline |
|
||||
| FT-N-01 | POST /login invalid creds | HTTP 401 | exact (status) | N/A | inline |
|
||||
| FT-N-02 | POST /login empty body | HTTP 422 | exact (status) | N/A | inline |
|
||||
| FT-N-03 | POST /upload no file | HTTP 422 | exact (status) | N/A | inline |
|
||||
| FT-N-04 | POST /load nonexistent | HTTP 500 | exact (status) | N/A | inline |
|
||||
| FT-N-05 | POST /unlock no archive | HTTP 404 | exact (status) | N/A | inline |
|
||||
|
||||
## External Dependency Mocks
|
||||
|
||||
| External Service | Mock/Stub | How Provided | Behavior |
|
||||
|-----------------|-----------|-------------|----------|
|
||||
| Azaion Resource API | Custom Python HTTP server | Docker service (mock-api) | Returns canned JWT on /login; encrypted test data on /resources/get; key fragment on /binary-split/key-fragment |
|
||||
| S3 CDN | MinIO | Docker service (mock-cdn) | S3-compatible storage with pre-seeded test `.big` files |
|
||||
| Docker daemon | Real Docker (via socket) | Mounted volume | Required for unlock flow tests |
|
||||
|
||||
## Data Validation Rules
|
||||
|
||||
| Data Type | Validation | Invalid Examples | Expected System Behavior |
|
||||
|-----------|-----------|-----------------|------------------------|
|
||||
| email | String, non-empty | `""`, missing field | HTTP 422 |
|
||||
| password | String, non-empty | `""`, missing field | HTTP 422 |
|
||||
| filename | String, non-empty | `""` | HTTP 422 or 500 |
|
||||
| upload file | Binary, non-empty | Missing file | HTTP 422 |
|
||||
@@ -0,0 +1,55 @@
|
||||
# Traceability Matrix
|
||||
|
||||
## Acceptance Criteria Coverage
|
||||
|
||||
| AC ID | Acceptance Criterion | Test IDs | Coverage |
|
||||
|-------|---------------------|----------|----------|
|
||||
| AC-1 | Health endpoint responds | FT-P-01, FT-P-02, NFT-PERF-01 | Covered |
|
||||
| AC-2 | Login sets credentials | FT-P-03, NFT-PERF-02, NFT-RES-01 | Covered |
|
||||
| AC-3 | Login rejects invalid credentials | FT-N-01, FT-N-02 | Covered |
|
||||
| AC-4 | Resource download returns decrypted bytes | FT-P-04, FT-N-04, NFT-PERF-03, NFT-RES-02 | Covered |
|
||||
| AC-5 | Resource upload succeeds | FT-P-05, FT-N-03, NFT-RES-LIM-01 | Covered |
|
||||
| AC-6 | Unlock starts background workflow | FT-P-06, NFT-RES-LIM-02 | Covered |
|
||||
| AC-7 | Unlock detects already-loaded images | FT-P-07 | Covered |
|
||||
| AC-8 | Unlock status reports progress | FT-P-08 | Covered |
|
||||
| AC-9 | Unlock completes full cycle | FT-P-06, NFT-RES-03 | Covered |
|
||||
| AC-10 | Unlock handles missing archive | FT-N-05 | Covered |
|
||||
| AC-11 | Resources encrypted at rest | NFT-SEC-02 | Covered |
|
||||
| AC-12 | Hardware-bound key derivation | NFT-SEC-03 | Covered |
|
||||
| AC-13 | Binary split prevents single-source compromise | FT-P-04 (split download) | Covered |
|
||||
| AC-14 | JWT token from trusted API | FT-P-03, NFT-SEC-01 | Covered |
|
||||
| AC-15 | Auto-retry on expired token | — | NOT COVERED — requires mock API that returns 401 then 200 on retry; complex mock setup |
|
||||
| AC-16 | Docker images verified | FT-P-07 (checks via unlock) | Covered |
|
||||
| AC-17 | Logs rotate daily | — | NOT COVERED — operational config, not observable via HTTP API |
|
||||
| AC-18 | Container builds on ARM64 | — | NOT COVERED — CI pipeline concern, not black-box testable |
|
||||
|
||||
## Restrictions Coverage
|
||||
|
||||
| Restriction ID | Restriction | Test IDs | Coverage |
|
||||
|---------------|-------------|----------|----------|
|
||||
| R-HW-1 | ARM64 architecture | — | NOT COVERED — build/CI concern |
|
||||
| R-HW-2 | Docker daemon access | FT-P-06, FT-P-07, NFT-RES-03 | Covered |
|
||||
| R-HW-3 | Hardware fingerprint availability | NFT-SEC-03 | Covered |
|
||||
| R-SW-1 | Python 3.11 | — | Implicit (test environment uses Python 3.11) |
|
||||
| R-ENV-1 | RESOURCE_API_URL env var | FT-P-03 (uses configured URL) | Covered |
|
||||
| R-ENV-2 | IMAGES_PATH env var | FT-P-06, FT-N-05 | Covered |
|
||||
| R-ENV-3 | API_VERSION env var | FT-P-07 | Covered |
|
||||
| R-OP-1 | Single instance | NFT-RES-LIM-02 | Covered |
|
||||
|
||||
## Coverage Summary
|
||||
|
||||
| Category | Total Items | Covered | Not Covered | Coverage % |
|
||||
|----------|-----------|---------|-------------|-----------|
|
||||
| Acceptance Criteria | 18 | 15 | 3 | 83% |
|
||||
| Restrictions | 8 | 6 | 2 | 75% |
|
||||
| **Total** | **26** | **21** | **5** | **81%** |
|
||||
|
||||
## Uncovered Items Analysis
|
||||
|
||||
| Item | Reason Not Covered | Risk | Mitigation |
|
||||
|------|-------------------|------|-----------|
|
||||
| AC-15 (Auto-retry 401) | Requires complex mock that returns 401 on first call, 200 on retry | Medium — retry logic could silently break | Can be covered with a stateful mock API in integration tests |
|
||||
| AC-17 (Log rotation) | Operational config, not observable through HTTP API | Low — Loguru config is static | Manual verification of loguru configuration |
|
||||
| AC-18 (ARM64 build) | CI pipeline concern, not black-box testable | Low — CI pipeline runs on ARM64 runner | Covered by CI pipeline itself |
|
||||
| R-HW-1 (ARM64) | Build target, not runtime behavior | Low | CI pipeline |
|
||||
| R-SW-1 (Python 3.11) | Implicit in test environment | Low | Dockerfile specifies Python version |
|
||||
Reference in New Issue
Block a user