mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-22 07:21:13 +00:00
Update autodev state, architecture documentation, and glossary terms
Transitioned the autodev state to phase 21, reflecting the completion of Step 5 and the drafting of Step 6 epics. Revised the architecture documentation to clarify the roles of the Tile Manager and its components, ensuring accurate representation of the system's operational flow. Updated glossary entries for Flight State and Operator to incorporate recent changes and enhance clarity on component interactions and responsibilities.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,170 @@
|
||||
# Test Specification — C7 On-Jetson Inference Runtime
|
||||
|
||||
Component-scoped. Suite-level coverage in `_docs/02_document/tests/*.md`.
|
||||
|
||||
## Acceptance Criteria Traceability
|
||||
|
||||
| AC ID | Acceptance Criterion (one-line) | Test IDs | Coverage |
|
||||
|-------|---------------------------------|----------|----------|
|
||||
| AC-4.1 | E2E latency <400 ms p95 | NFT-PERF-01 (Tier-2), **C7-PT-01** | Covered |
|
||||
| AC-4.2 | Memory <8 GB on Jetson | NFT-LIM-01, **C7-PT-02** | Covered |
|
||||
| AC-NEW-1 | Cold-start TTFF <30 s p95 | NFT-PERF-03, **C7-IT-01** | Covered |
|
||||
| AC-NEW-5 | Operating envelope; thermal telemetry feed | NFT-LIM-04, **C7-IT-02** | Covered (workstation portion) |
|
||||
| D-C10-3 | Manifest content-hash takeoff gate | (gate is C10-owned, but the engine deserialise call is C7) | **C7-IT-03** | Covered |
|
||||
| D-C10-7 | Engine filename schema (SM/JP/TRT/precision) | Helper-doc cited; **C7-IT-04** | Covered |
|
||||
|
||||
---
|
||||
|
||||
## Component-Internal Tests
|
||||
|
||||
### C7-IT-01: cold-start engine load + warm-up budget
|
||||
|
||||
**Summary**: from a cold (zero-resident-engines) Jetson process, every required engine deserialises and warms up in under the AC-NEW-1 30 s p95 budget.
|
||||
|
||||
**Traces to**: AC-NEW-1
|
||||
|
||||
**Description**: kill the companion process; restart; measure wall-clock from process start to "all engines warm" event in the FDR record stream. Repeat 10 times; assert p95 ≤ 30 s.
|
||||
|
||||
**Input data**: pre-built engine cache for the Derkachi fixture profile.
|
||||
|
||||
**Expected result**: p95 ≤ 30 s; no engine fails to warm.
|
||||
|
||||
**Max execution time**: 6 min (10 × ~30 s + overhead).
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-02: thermal telemetry feeds C4's hybrid
|
||||
|
||||
**Summary**: `ThermalState` from `jetson-stats` is published at ≥1 Hz and is observable to C4; under simulated throttle, `throttle == true` is reported within 1 s of the throttle event.
|
||||
|
||||
**Traces to**: AC-NEW-5 (workstation-baseline portion; chamber portion deferred per traceability matrix)
|
||||
|
||||
**Description**: simulate a thermal-throttle event by spoofing the `jetson-stats` sysfs reading; assert (a) `ThermalState` updates carry `throttle == true` within 1 s, (b) C4's `current_covariance_mode` flips to JACOBIAN within 1 frame after that.
|
||||
|
||||
**Input data**: scripted sysfs spoof.
|
||||
|
||||
**Expected result**: 1 s telemetry latency; 1-frame C4 reaction.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-03: D-C10-3 takeoff gate refuses mismatched engine
|
||||
|
||||
**Summary**: when the manifest's content-hash for an engine does not match the on-disk engine's hash, C7 refuses to deserialise and the F2 takeoff aborts.
|
||||
|
||||
**Traces to**: D-C10-3
|
||||
|
||||
**Description**: corrupt one byte of a deployed engine after the manifest has been signed; trigger F2 takeoff load; assert (a) C7 raises `EngineHashMismatchError`, (b) the airborne process refuses to open the FC adapter, (c) the failure is logged at ERROR.
|
||||
|
||||
**Input data**: a deployed engine + its corrupted twin.
|
||||
|
||||
**Expected result**: takeoff aborts; ERROR logged.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-04: SM / JetPack / TRT / precision filename schema enforcement
|
||||
|
||||
**Summary**: an engine file whose `<sm>/<jp>/<trt>/<precision>` quadruple in the filename does not match the running Jetson's actual quadruple is refused at deserialise time.
|
||||
|
||||
**Traces to**: D-C10-7
|
||||
|
||||
**Description**: copy a valid engine file but rename it with a mismatched SM (e.g., `sm86` instead of `sm87`); call `load_engine`; assert `EngineSchemaMismatchError` and no GPU memory allocated.
|
||||
|
||||
**Input data**: a valid engine + a renamed copy.
|
||||
|
||||
**Expected result**: engine refused at filename-parse time.
|
||||
|
||||
**Max execution time**: 5 s.
|
||||
|
||||
---
|
||||
|
||||
### C7-IT-05: ONNX-RT fallback when TRT engine unavailable
|
||||
|
||||
**Summary**: if the primary TRT engine is missing or unloadable, C7 falls back to ONNX-RT + TRT-EP and continues without dropping the request.
|
||||
|
||||
**Traces to**: defensive (engine-rule simple-baseline path)
|
||||
|
||||
**Description**: rename the TRT engine for one model away (so deserialise fails); call `infer`; assert the call succeeds via ONNX-RT path with a degraded-latency warning logged.
|
||||
|
||||
**Input data**: TRT engine + ONNX model side-by-side.
|
||||
|
||||
**Expected result**: successful inference; degraded-latency warning.
|
||||
|
||||
**Max execution time**: 30 s.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tests
|
||||
|
||||
### C7-PT-01: per-call inference latency p95 by model
|
||||
|
||||
**Traces to**: AC-4.1
|
||||
|
||||
**Load scenario**: scripted call rate matching production — UltraVPR @ 3 Hz, LightGlue @ 9 Hz (3 cands × 3 Hz), AdHoP conditional (~25%).
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Model | Mode | p95 latency target | Failure threshold |
|
||||
|-------|------|--------------------|-------------------|
|
||||
| UltraVPR | TRT FP16 | ≤ 60 ms | 100 ms |
|
||||
| LightGlue | TRT FP16 | ≤ 30 ms | 60 ms |
|
||||
| AdHoP | TRT FP16 | ≤ 90 ms | 150 ms |
|
||||
| DISK | TRT FP16 | ≤ 50 ms | 90 ms |
|
||||
|
||||
---
|
||||
|
||||
### C7-PT-02: aggregate GPU memory budget
|
||||
|
||||
**Traces to**: AC-4.2
|
||||
|
||||
**Load scenario**: all production-default engines resident concurrently.
|
||||
|
||||
**Expected results**:
|
||||
|
||||
| Metric | Target | Failure Threshold |
|
||||
|--------|--------|-------------------|
|
||||
| GPU resident memory (all engines) | ≤ 4 GB | 5 GB |
|
||||
| System RAM (process resident) | ≤ 1.5 GB | 2 GB |
|
||||
|
||||
(remaining 8 GB shared LPDDR5 budget partition belongs to OS + ROS-equivalents + scratch; tracked at the system level by NFT-LIM-01.)
|
||||
|
||||
---
|
||||
|
||||
## Security Tests
|
||||
|
||||
### C7-ST-01: engine deserialise refuses files with no SHA-256 sidecar
|
||||
|
||||
**Summary**: per Helper `Sha256Sidecar`, every engine has a sidecar `.sha256` file; deserialising an engine without one is refused.
|
||||
|
||||
**Traces to**: D-C10-3 (defensive)
|
||||
|
||||
**Test procedure**:
|
||||
1. Delete the sidecar for one valid engine.
|
||||
2. Call `load_engine` on it.
|
||||
3. Assert refusal with `EngineSidecarMissingError`.
|
||||
|
||||
**Pass criteria**: refusal + no GPU memory allocated.
|
||||
**Fail criteria**: load succeeds.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
C7 has no operator-facing behaviour; covered transitively via NFT-PERF-01 / NFT-PERF-03.
|
||||
|
||||
---
|
||||
|
||||
## Test Data Management
|
||||
|
||||
| Data Set | Source | Size |
|
||||
|----------|--------|------|
|
||||
| Pre-built engine cache for Derkachi profile | C10 build artifact | ~600 MB |
|
||||
| Spoofed `jetson-stats` sysfs harness | scripted | <1 MB |
|
||||
| Corrupted-engine fixture | scripted | varies |
|
||||
|
||||
**Setup**: C10 must have built engines for SM 87 / JP 6.2 / TRT 10.3 / FP16 once before C7 tests can run on Tier-2.
|
||||
**Teardown**: read-only.
|
||||
**Data isolation**: per-test temp dirs.
|
||||
Reference in New Issue
Block a user