Revise acceptance criteria and restrictions documentation to clarify recent updates and specifications. Key changes include enhanced definitions for position accuracy, image processing quality, and operational parameters, as well as updates to camera specifications and validation requirements. This revision aims to improve clarity and ensure alignment with project goals.

2026-06-22 01:51:13 +00:00 · 2026-05-01 16:24:46 +03:00
parent 3f173c1bb7
commit 7e15868d39
62 changed files with 6878 additions and 13 deletions
@@ -0,0 +1,94 @@
+# Performance Tests
+
+### NFT-PERF-01: Per-Frame Latency On Project Still Images
+
+**Summary**: Validate end-to-end latency for processing project nadir frames through geolocation output.
+
+**Traces to**: AC-4.1, AC-4.4
+
+**Metric**: Capture-to-output latency p50/p95/p99 and dropped-frame rate.
+
+**Preconditions**:
+- Jetson Orin Nano Super or equivalent production target is running in the intended power mode.
+- `project_60_still_images` fixture is available.
+
+| Step | Consumer Action | Measurement |
+|------|-----------------|-------------|
+| 1 | Replay images at target 3 fps or faster stress rate | Measure latency from input timestamp to emitted estimate |
+| 2 | Record all frame drops | Measure dropped-frame percentage |
+
+**Pass criteria**: p95 latency <400 ms; dropped frames <=10% under sustained load; no batching delay.
+
+**Duration**: Minimum 20 minutes or full fixture loop repeated enough times to reach stable measurements.
+
+---
+
+### NFT-PERF-02: BASALT + Wrapper Replay Latency
+
+**Summary**: Validate relative VIO hot-path latency using synchronized public or representative camera/IMU data.
+
+**Traces to**: AC-2.1a, AC-4.1, AC-4.2
+
+**Metric**: Per-frame VIO latency, completion rate, and memory usage.
+
+**Preconditions**:
+- MUN-FRL/ALTO/EPFL/Kagaru or representative synchronized dataset slice is pinned.
+- OpenVINS reference replay is available for comparison.
+
+| Step | Consumer Action | Measurement |
+|------|-----------------|-------------|
+| 1 | Replay synchronized camera/IMU stream through BASALT + wrapper | Measure per-frame processing time |
+| 2 | Compare emitted trajectory against ground truth | Measure completion rate and error distribution |
+| 3 | Monitor memory | Track CPU/GPU shared memory peak |
+
+**Pass criteria**: Normal-frame VO registration >95%; p95 processing latency <400 ms for the hot path; memory <8 GB shared.
+
+**Duration**: Dataset-dependent; at least one normal segment and one challenging segment.
+
+---
+
+### NFT-PERF-03: Relocalization Trigger Path Latency
+
+**Summary**: Validate the heavy DINOv2-VLAD + FAISS + ALIKED/LightGlue path under bounded top-K settings.
+
+**Traces to**: AC-3.2, AC-3.3, AC-4.1, AC-8.6
+
+**Metric**: Trigger-to-anchor latency, top-K query time, local verification time, accepted/rejected anchor counts.
+
+**Preconditions**:
+- Precomputed descriptor index is loaded.
+- Dynamic K settings are configured: K=5 stable, K=20 active-conflict, K=50 fallback.
+
+| Step | Consumer Action | Measurement |
+|------|-----------------|-------------|
+| 1 | Trigger relocalization from cold start or sharp turn | Measure DINOv2 descriptor time and FAISS query time |
+| 2 | Verify top-K candidates | Measure ALIKED/LightGlue + RANSAC latency |
+| 3 | Emit accepted/rejected decision | Measure total trigger-to-decision latency |
+
+**Pass criteria**: Heavy path is conditional, never blocks steady-state frame output; accepted anchor carries MRE <2.5 px and valid covariance.
+
+**Duration**: 100 relocalization trials across stable and active-conflict sector fixtures.
+
+---
+
+### NFT-PERF-04: Cold Boot Time To First Fix
+
+**Summary**: Validate companion boot to first valid `GPS_INPUT`.
+
+**Traces to**: AC-NEW-1
+
+**Metric**: Time from service start/boot marker to first valid `GPS_INPUT`.
+
+**Preconditions**:
+- Engines/indexes are built before the run.
+- Cache/index is available locally.
+- FC state handoff is simulated or provided.
+
+| Step | Consumer Action | Measurement |
+|------|-----------------|-------------|
+| 1 | Start service from cold boot condition | Measure initialization stages |
+| 2 | Wait for first valid output | Measure first valid `GPS_INPUT` timestamp |
+
+**Pass criteria**: 95th percentile <30 s over 50 runs.
+
+**Duration**: 50 cold-start trials.