Oleksandr Bezdieniezhnykh
64542d32fc
Transitioned the autodev state to phase 21, reflecting the completion of Step 5 and the drafting of Step 6 epics. Revised the architecture documentation to clarify the roles of the Tile Manager and its components, ensuring accurate representation of the system's operational flow. Updated glossary entries for Flight State and Operator to incorporate recent changes and enhance clarity on component interactions and responsibilities.
6.2 KiB
Test Specification — C10 Pre-flight Cache Provisioning (engines + descriptors + manifest)
Component-scoped. Suite-level coverage in _docs/02_document/tests/*.md. C10 was narrowed in this Plan cycle: it builds model-derived artifacts (TensorRT engines, VPR descriptors, signed Manifest) from an already-populated C6 tile cache. Tile fetch is C11 TileDownloader's concern.
Acceptance Criteria Traceability
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|---|---|---|---|
| AC-8.3 | Imagery pre-loaded onto companion before flight (the manifest gate) | FT-P-15, FT-P-16, C10-IT-01 | Covered |
| AC-NEW-1 | Cold-start TTFF <30 s p95 (pre-built engines required) | NFT-PERF-03, C10-IT-02 | Covered |
| D-C10-1 | Manifest-hash idempotence on repeated build | C10-IT-03 | Covered |
| D-C10-3 | Takeoff content-hash gate refuses mismatch | covered at C7-IT-03; C10-IT-04 asserts the manifest signing path | Covered |
| D-C10-6 | Engine cache hardware-tied (SM 87 / JP 6.2 / TRT 10.3 / FP16) | C7-IT-04, C10-IT-05 | Covered |
| D-C10-7 | Engine filename schema enforcement | covered at C7-IT-04 | Covered |
Component-Internal Tests
C10-IT-01: end-to-end build from a pre-populated C6
Summary: given a C6 tile cache populated by C11 TileDownloader (10 GB Derkachi area), C10 produces (a) all required TensorRT engines, (b) the FAISS HNSW index over VPR descriptors, (c) a signed Manifest, in under the operator-tooling time budget.
Traces to: AC-8.3, AC-NEW-1
Description: stage a C6 with the Derkachi corpus already populated; run CacheProvisioner.build_artifacts; assert (a) the engine set under cache_artifacts/engines/ matches the configured model list, (b) descriptor_index.faiss is non-empty and queryable, (c) the Manifest is signed with the operator's signing key and content-hashes every artifact.
Input data: pre-populated C6 (tests/fixtures/c6_populated_derkachi/).
Expected result: all artifacts present + signed Manifest.
Max execution time: 12 min on Tier-1 (CPU TRT compile is slow; Tier-2 takes ~4 min and is the production path).
C10-IT-02: ManifestVerifier refuses unsigned / wrong-signature Manifest
Summary: ManifestVerifier.verify rejects a Manifest whose signature doesn't validate against the operator's public key.
Traces to: AC-NEW-1, D-C10-3
Description: build a valid Manifest; copy it; tamper one byte; call verify; assert ManifestSignatureError. Repeat: copy + replace signature with one signed by an unauthorized key; assert ManifestSignatureError.
Input data: valid Manifest + 2 tampered copies.
Expected result: both tampered Manifests rejected.
Max execution time: 5 s.
C10-IT-03: idempotence on repeated build
Summary: re-running build_artifacts against an unchanged C6 produces the same Manifest content-hash and skips already-built engines.
Traces to: D-C10-1
Description: run build once; record Manifest content-hash + engine compile timestamps. Re-run with no C6 changes; assert (a) Manifest content-hash unchanged, (b) engines reused (no recompile, asserted via timestamp comparison), (c) total wall-clock < 1 min on Tier-1.
Input data: as C10-IT-01.
Expected result: idempotent — same hash, no recompile.
Max execution time: 90 s (second-run only).
C10-IT-04: Manifest covers every shipped artifact
Summary: the Manifest's content-hash table includes every file under cache_artifacts/; an artifact present on disk but missing from the Manifest is a build failure.
Traces to: D-C10-3 (no smuggled artifacts can pass the takeoff gate)
Description: after a successful build, plant an extra file in cache_artifacts/; re-run build_artifacts (or call the build's post-step audit hook); assert build refuses to sign — output ManifestCoverageError listing the orphan file.
Input data: as C10-IT-01 plus an extra file.
Expected result: build fails with ManifestCoverageError.
Max execution time: 60 s.
C10-IT-05: Tier-2 hardware-tied engine compile produces SM-87 / JP-6.2 / TRT-10.3 binary
Summary: when run on the bench Jetson, C10 produces engines whose internal TRT metadata reports SM=87, JetPack=6.2, TRT=10.3, precision=FP16.
Traces to: D-C10-6
Description: run build_artifacts on the bench Jetson; for each engine, parse the internal TRT version footer; assert the quadruple matches.
Input data: bench Jetson + Derkachi C6 fixture.
Expected result: all engines tagged correctly.
Max execution time: 6 min on Tier-2.
Performance Tests
C10-PT-01: build wall-clock budget on Tier-1 (operator-tooling laptop)
Traces to: operator-tooling UX (no AC trace; an operator-tooling SLO)
Load scenario: full Derkachi corpus (10 GB, ~87 654 tiles).
Expected results:
| Metric | Target | Failure Threshold |
|---|---|---|
| Cold build wall-clock | ≤ 12 min on a developer laptop with NVIDIA GPU | 25 min |
| Warm idempotent re-run | ≤ 1 min | 3 min |
Security Tests
C10-ST-01: signing-key path uses operator-controlled key (not a baked-in dev key)
Summary: the build refuses to sign the Manifest if the configured signing-key path points to the baked-in dev key (caught via a hash-list check).
Traces to: defensive (production-key safety)
Test procedure:
- Configure C10 with the dev-key path that's hard-coded into the dev fixtures.
- Run
build_artifacts. - Assert refusal with
OperatorKeyRequiredError.
Pass criteria: refusal. Fail criteria: build succeeds with the dev key.
Acceptance Tests
Covered transitively via FT-P-15 / FT-P-16 (operator workflow tests).
Test Data Management
| Data Set | Source | Size |
|---|---|---|
tests/fixtures/c6_populated_derkachi/ |
C11 TileDownloader build artifact |
~10 GB on disk |
| Operator signing key (test-only) | generated per test run | <1 KB |
| Dev key (for the negative test) | curated, in-repo | <1 KB |
Setup: C11 TileDownloader integration test (under C11) populates C6 once; that artifact is reused.
Teardown: per-test temp dirs for cache_artifacts/ build outputs.
Data isolation: per-test temp cache_artifacts/.