Oleksandr Bezdieniezhnykh
64542d32fc
Transitioned the autodev state to phase 21, reflecting the completion of Step 5 and the drafting of Step 6 epics. Revised the architecture documentation to clarify the roles of the Tile Manager and its components, ensuring accurate representation of the system's operational flow. Updated glossary entries for Flight State and Operator to incorporate recent changes and enhance clarity on component interactions and responsibilities.
6.1 KiB
Test Specification — C7 On-Jetson Inference Runtime
Component-scoped. Suite-level coverage in _docs/02_document/tests/*.md.
Acceptance Criteria Traceability
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|---|---|---|---|
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01 (Tier-2), C7-PT-01 | Covered |
| AC-4.2 | Memory <8 GB on Jetson | NFT-LIM-01, C7-PT-02 | Covered |
| AC-NEW-1 | Cold-start TTFF <30 s p95 | NFT-PERF-03, C7-IT-01 | Covered |
| AC-NEW-5 | Operating envelope; thermal telemetry feed | NFT-LIM-04, C7-IT-02 | Covered (workstation portion) |
| D-C10-3 | Manifest content-hash takeoff gate | (gate is C10-owned, but the engine deserialise call is C7) | C7-IT-03 |
| D-C10-7 | Engine filename schema (SM/JP/TRT/precision) | Helper-doc cited; C7-IT-04 | Covered |
Component-Internal Tests
C7-IT-01: cold-start engine load + warm-up budget
Summary: from a cold (zero-resident-engines) Jetson process, every required engine deserialises and warms up in under the AC-NEW-1 30 s p95 budget.
Traces to: AC-NEW-1
Description: kill the companion process; restart; measure wall-clock from process start to "all engines warm" event in the FDR record stream. Repeat 10 times; assert p95 ≤ 30 s.
Input data: pre-built engine cache for the Derkachi fixture profile.
Expected result: p95 ≤ 30 s; no engine fails to warm.
Max execution time: 6 min (10 × ~30 s + overhead).
C7-IT-02: thermal telemetry feeds C4's hybrid
Summary: ThermalState from jetson-stats is published at ≥1 Hz and is observable to C4; under simulated throttle, throttle == true is reported within 1 s of the throttle event.
Traces to: AC-NEW-5 (workstation-baseline portion; chamber portion deferred per traceability matrix)
Description: simulate a thermal-throttle event by spoofing the jetson-stats sysfs reading; assert (a) ThermalState updates carry throttle == true within 1 s, (b) C4's current_covariance_mode flips to JACOBIAN within 1 frame after that.
Input data: scripted sysfs spoof.
Expected result: 1 s telemetry latency; 1-frame C4 reaction.
Max execution time: 30 s.
C7-IT-03: D-C10-3 takeoff gate refuses mismatched engine
Summary: when the manifest's content-hash for an engine does not match the on-disk engine's hash, C7 refuses to deserialise and the F2 takeoff aborts.
Traces to: D-C10-3
Description: corrupt one byte of a deployed engine after the manifest has been signed; trigger F2 takeoff load; assert (a) C7 raises EngineHashMismatchError, (b) the airborne process refuses to open the FC adapter, (c) the failure is logged at ERROR.
Input data: a deployed engine + its corrupted twin.
Expected result: takeoff aborts; ERROR logged.
Max execution time: 30 s.
C7-IT-04: SM / JetPack / TRT / precision filename schema enforcement
Summary: an engine file whose <sm>/<jp>/<trt>/<precision> quadruple in the filename does not match the running Jetson's actual quadruple is refused at deserialise time.
Traces to: D-C10-7
Description: copy a valid engine file but rename it with a mismatched SM (e.g., sm86 instead of sm87); call load_engine; assert EngineSchemaMismatchError and no GPU memory allocated.
Input data: a valid engine + a renamed copy.
Expected result: engine refused at filename-parse time.
Max execution time: 5 s.
C7-IT-05: ONNX-RT fallback when TRT engine unavailable
Summary: if the primary TRT engine is missing or unloadable, C7 falls back to ONNX-RT + TRT-EP and continues without dropping the request.
Traces to: defensive (engine-rule simple-baseline path)
Description: rename the TRT engine for one model away (so deserialise fails); call infer; assert the call succeeds via ONNX-RT path with a degraded-latency warning logged.
Input data: TRT engine + ONNX model side-by-side.
Expected result: successful inference; degraded-latency warning.
Max execution time: 30 s.
Performance Tests
C7-PT-01: per-call inference latency p95 by model
Traces to: AC-4.1
Load scenario: scripted call rate matching production — UltraVPR @ 3 Hz, LightGlue @ 9 Hz (3 cands × 3 Hz), AdHoP conditional (~25%).
Expected results:
| Model | Mode | p95 latency target | Failure threshold |
|---|---|---|---|
| UltraVPR | TRT FP16 | ≤ 60 ms | 100 ms |
| LightGlue | TRT FP16 | ≤ 30 ms | 60 ms |
| AdHoP | TRT FP16 | ≤ 90 ms | 150 ms |
| DISK | TRT FP16 | ≤ 50 ms | 90 ms |
C7-PT-02: aggregate GPU memory budget
Traces to: AC-4.2
Load scenario: all production-default engines resident concurrently.
Expected results:
| Metric | Target | Failure Threshold |
|---|---|---|
| GPU resident memory (all engines) | ≤ 4 GB | 5 GB |
| System RAM (process resident) | ≤ 1.5 GB | 2 GB |
(remaining 8 GB shared LPDDR5 budget partition belongs to OS + ROS-equivalents + scratch; tracked at the system level by NFT-LIM-01.)
Security Tests
C7-ST-01: engine deserialise refuses files with no SHA-256 sidecar
Summary: per Helper Sha256Sidecar, every engine has a sidecar .sha256 file; deserialising an engine without one is refused.
Traces to: D-C10-3 (defensive)
Test procedure:
- Delete the sidecar for one valid engine.
- Call
load_engineon it. - Assert refusal with
EngineSidecarMissingError.
Pass criteria: refusal + no GPU memory allocated. Fail criteria: load succeeds.
Acceptance Tests
C7 has no operator-facing behaviour; covered transitively via NFT-PERF-01 / NFT-PERF-03.
Test Data Management
| Data Set | Source | Size |
|---|---|---|
| Pre-built engine cache for Derkachi profile | C10 build artifact | ~600 MB |
Spoofed jetson-stats sysfs harness |
scripted | <1 MB |
| Corrupted-engine fixture | scripted | varies |
Setup: C10 must have built engines for SM 87 / JP 6.2 / TRT 10.3 / FP16 once before C7 tests can run on Tier-2. Teardown: read-only. Data isolation: per-test temp dirs.