Revise skills documentation to incorporate updated directory structure and terminology. Replace references to integration tests with blackbox tests in SKILL.md files and templates. Adjust paths in planning and deployment documentation to align with the new _docs/02_document/ structure, ensuring consistency and clarity throughout the documentation.

2026-04-22 22:06:37 +00:00 · 2026-03-25 06:35:41 +02:00
parent 1c6e8f47b1
commit 531a1301d5
12 changed files with 2190 additions and 0 deletions
@@ -0,0 +1,61 @@
+frame_index,image,expected_lat,expected_lon,max_error_m,threshold_50m_applies,threshold_20m_applies
+1,AD000001.jpg,48.275292,37.385220,100,yes,yes
+2,AD000002.jpg,48.275001,37.382922,100,yes,yes
+3,AD000003.jpg,48.274520,37.381657,100,yes,yes
+4,AD000004.jpg,48.274956,37.379004,100,yes,yes
+5,AD000005.jpg,48.273997,37.379828,100,yes,yes
+6,AD000006.jpg,48.272538,37.380294,100,yes,yes
+7,AD000007.jpg,48.272408,37.379153,100,yes,yes
+8,AD000008.jpg,48.271992,37.377572,100,yes,yes
+9,AD000009.jpg,48.271376,37.376671,100,yes,yes
+10,AD000010.jpg,48.271233,37.374806,100,yes,yes
+11,AD000011.jpg,48.270334,37.374442,100,yes,yes
+12,AD000012.jpg,48.269922,37.373284,100,yes,yes
+13,AD000013.jpg,48.269366,37.372134,100,yes,yes
+14,AD000014.jpg,48.268759,37.370940,100,yes,yes
+15,AD000015.jpg,48.268291,37.369815,100,yes,yes
+16,AD000016.jpg,48.267719,37.368469,100,yes,yes
+17,AD000017.jpg,48.267461,37.367255,100,yes,yes
+18,AD000018.jpg,48.266663,37.365888,100,yes,yes
+19,AD000019.jpg,48.266135,37.365460,100,yes,yes
+20,AD000020.jpg,48.265574,37.364211,100,yes,yes
+21,AD000021.jpg,48.264892,37.362998,100,yes,yes
+22,AD000022.jpg,48.264393,37.361086,100,yes,yes
+23,AD000023.jpg,48.263803,37.361028,100,yes,yes
+24,AD000024.jpg,48.263014,37.359878,100,yes,yes
+25,AD000025.jpg,48.262635,37.358277,100,yes,yes
+26,AD000026.jpg,48.261819,37.357116,100,yes,yes
+27,AD000027.jpg,48.261182,37.355907,100,yes,yes
+28,AD000028.jpg,48.260727,37.354723,100,yes,yes
+29,AD000029.jpg,48.260117,37.353469,100,yes,yes
+30,AD000030.jpg,48.259677,37.352165,100,yes,yes
+31,AD000031.jpg,48.258881,37.351376,100,yes,yes
+32,AD000032.jpg,48.258425,37.349964,100,yes,yes
+33,AD000033.jpg,48.258653,37.347004,100,yes,yes
+34,AD000034.jpg,48.257879,37.347711,100,yes,yes
+35,AD000035.jpg,48.256777,37.348444,100,yes,yes
+36,AD000036.jpg,48.255756,37.348098,100,yes,yes
+37,AD000037.jpg,48.255375,37.346549,100,yes,yes
+38,AD000038.jpg,48.254799,37.345603,100,yes,yes
+39,AD000039.jpg,48.254557,37.344566,100,yes,yes
+40,AD000040.jpg,48.254380,37.344375,100,yes,yes
+41,AD000041.jpg,48.253722,37.343093,100,yes,yes
+42,AD000042.jpg,48.254205,37.340532,100,yes,yes
+43,AD000043.jpg,48.252380,37.342112,100,yes,yes
+44,AD000044.jpg,48.251489,37.343079,100,yes,yes
+45,AD000045.jpg,48.251085,37.346128,100,yes,yes
+46,AD000046.jpg,48.250413,37.344034,100,yes,yes
+47,AD000047.jpg,48.249414,37.343296,100,yes,yes
+48,AD000048.jpg,48.249114,37.346895,100,yes,yes
+49,AD000049.jpg,48.250241,37.347741,100,yes,yes
+50,AD000050.jpg,48.250974,37.348379,100,yes,yes
+51,AD000051.jpg,48.251528,37.349468,100,yes,yes
+52,AD000052.jpg,48.251873,37.350485,100,yes,yes
+53,AD000053.jpg,48.252161,37.351491,100,yes,yes
+54,AD000054.jpg,48.252685,37.352343,100,yes,yes
+55,AD000055.jpg,48.253268,37.353119,100,yes,yes
+56,AD000056.jpg,48.253767,37.354246,100,yes,yes
+57,AD000057.jpg,48.254329,37.354946,100,yes,yes
+58,AD000058.jpg,48.254874,37.355765,100,yes,yes
+59,AD000059.jpg,48.255481,37.356501,100,yes,yes
+60,AD000060.jpg,48.256246,37.357485,100,yes,yes
@@ -0,0 +1,166 @@
+# Expected Results
+
+Maps every input data item to its quantifiable expected result.
+Tests use this mapping to compare actual system output against known-correct answers.
+
+## Result Format Legend
+
+| Result Type | When to Use | Example |
+|-------------|-------------|---------|
+| Exact value | Output must match precisely | `fix_type: 3`, `satellites_visible: 10` |
+| Tolerance range | Numeric output with acceptable variance | `lat: 48.275292 ± 50m` |
+| Threshold | Output must exceed or stay below a limit | `latency < 400ms`, `memory < 8GB` |
+| Pattern match | Output must match a string/regex pattern | `RELOC_REQ: last_lat=.* last_lon=.* uncertainty=.*m` |
+| File reference | Complex output compared against a reference file | `match expected_results/position_accuracy.csv` |
+| Set/count | Output must contain specific items or counts | `registered_frames / total_frames > 0.95` |
+
+## Comparison Methods
+
+| Method | Description | Tolerance Syntax |
+|--------|-------------|-----------------|
+| `numeric_tolerance` | abs(actual - expected) ≤ tolerance | `± <value>` |
+| `threshold_min` | actual ≥ threshold | `≥ <value>` |
+| `threshold_max` | actual ≤ threshold | `≤ <value>` |
+| `percentage` | percentage of items meeting criterion | `≥ N%` |
+| `exact` | actual == expected | N/A |
+| `regex` | actual matches regex pattern | regex string |
+| `file_reference` | compare against reference file | file path |
+
+## Input → Expected Result Mapping
+
+### Position Accuracy (60-image flight sequence)
+
+Ground truth GPS coordinates for each frame are in `coordinates.csv`. The system processes these frames sequentially (simulating a real flight) with corresponding IMU data (200Hz, from SITL ArduPilot or synthetic generation from trajectory) and satellite tile matches. The system outputs estimated GPS coordinates per frame. Expected results compare estimated positions against ground truth.
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 1 | `coordinates.csv` (all 60 frames) | Sequential flight images with ground truth GPS | ≥ 80% of frames have position error < 50m from ground truth | percentage | ≥ 80% of frames within 50m | `expected_results/position_accuracy.csv` |
+| 2 | `coordinates.csv` (all 60 frames) | Sequential flight images with ground truth GPS | ≥ 60% of frames have position error < 20m from ground truth | percentage | ≥ 60% of frames within 20m | `expected_results/position_accuracy.csv` |
+| 3 | `coordinates.csv` (all 60 frames) | Sequential flight images with ground truth GPS | Per-frame position output in WGS84 (lat, lon) | numeric_tolerance | each frame ± 100m max (no single frame exceeds 100m error) | `expected_results/position_accuracy.csv` |
+| 4 | `coordinates.csv` (all 60 frames) | Sequential flight images with ground truth GPS | Cumulative VO drift between satellite anchors < 100m | threshold_max | ≤ 100m drift between anchors | N/A |
+
+### GPS_INPUT Message Correctness
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 5 | Single frame + IMU data | Normal tracking frame with recent satellite match | `fix_type: 3`, `horiz_accuracy: 5-20m`, `satellites_visible: 10`, lat/lon populated | exact (fix_type, sat), numeric_tolerance (accuracy) | fix_type == 3, horiz_accuracy ∈ [1, 50] | N/A |
+| 6 | Frame sequence, no satellite match for >30s | VO-only tracking, no recent satellite anchor | `fix_type: 3`, `horiz_accuracy: 20-50m` | exact (fix_type), range (accuracy) | fix_type == 3, horiz_accuracy ∈ [20, 100] | N/A |
+| 7 | Frame sequence, VO lost + no satellite | IMU-only dead reckoning | `fix_type: 2`, `horiz_accuracy: 50-200m+` (growing over time) | exact (fix_type), threshold_min (accuracy) | fix_type == 2, horiz_accuracy ≥ 50 | N/A |
+| 8 | VO lost + 3 consecutive satellite failures | Total position failure | `fix_type: 0`, `horiz_accuracy: 999.0` | exact | fix_type == 0, horiz_accuracy == 999.0 | N/A |
+| 9 | Any valid frame | GPS_INPUT output rate | GPS_INPUT messages at 5-10Hz continuous | range | 5 ≤ rate_hz ≤ 10 | N/A |
+
+### Confidence Tier Transitions
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 10 | Frame with satellite match <30s ago, covariance <400m² | HIGH confidence conditions | Confidence tier: HIGH, SSE confidence: "HIGH" | exact | N/A | N/A |
+| 11 | Frame with cuVSLAM OK, no satellite match >30s | MEDIUM confidence conditions | Confidence tier: MEDIUM, SSE confidence: "MEDIUM" | exact | N/A | N/A |
+| 12 | Frame with cuVSLAM lost, IMU-only | LOW confidence conditions | Confidence tier: LOW, SSE confidence: "LOW" | exact | N/A | N/A |
+| 13 | 3+ consecutive total failures | FAILED conditions | Confidence tier: FAILED, SSE confidence: "FAILED", fix_type: 0 | exact | N/A | N/A |
+
+### Image Registration & Visual Odometry
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 14 | 60 sequential flight images | Normal flight (no sharp turns) | Image registration rate ≥ 95% (≥ 57 of 60 registered) | percentage | ≥ 95% | N/A |
+| 15 | 60 sequential flight images | Normal flight images | Mean reprojection error < 1.0 pixels | threshold_max | MRE < 1.0 px | N/A |
+
+### Disconnected Route Segments & Sharp Turns
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 16 | Frames 32-43 from coordinates.csv | Trajectory with direction change (turn area) | System continues producing position estimates through the turn | threshold_min | ≥ 1 position output per frame | N/A |
+| 17 | Simulated consecutive frames with 350m gap | Outlier between 2 consecutive photos due to tilt | System handles outlier, position estimate not corrupted (error < 100m for next valid frame) | threshold_max | ≤ 100m error after recovery | N/A |
+| 18 | Simulated sharp turn (no overlap, <5% overlap, <70° angle, <200m drift) | Sharp turn where VO fails | Satellite re-localization triggers, position recovered within 3 frames after turn | threshold_max | position error ≤ 50m after re-localization | N/A |
+| 19 | Simulated VO loss + satellite match success | Tracking loss → re-localization | cuVSLAM restarts, ESKF position corrected, tracking_state returns to NORMAL | exact | tracking_state == NORMAL after recovery | N/A |
+
+### 3-Consecutive-Failure Re-Localization
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 20 | Simulated VO loss + 3 satellite match failures | Cannot determine position by any means | Re-localization request sent: `RELOC_REQ: last_lat=.* last_lon=.* uncertainty=.*m` | regex | message matches pattern | N/A |
+| 21 | Re-localization request active | System waiting for operator | GPS_INPUT fix_type=0, system continues IMU prediction, continues satellite matching attempts | exact (fix_type) | fix_type == 0 | N/A |
+| 22 | Operator sends approximate coordinates (lat, lon) | Operator re-localization hint | System uses hint as ESKF measurement (high covariance ~500m), attempts satellite match in new area | threshold_max | position error ≤ 500m initially, ≤ 50m after satellite match | N/A |
+
+### Startup & Handoff
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 23 | System boot with GLOBAL_POSITION_INT available | Normal startup | System reads initial position, initializes ESKF, starts GPS_INPUT output | exact | GPS_INPUT output begins within 60s of boot | N/A |
+| 24 | System boot + first satellite match | Startup validation | First satellite match validates initial position, position error drops | threshold_max | position error ≤ 50m after first satellite match | N/A |
+
+### Mid-Flight Reboot Recovery
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 25 | System process killed mid-flight | Companion computer reboot | System recovers: reads FC position, inits ESKF with high uncertainty, loads TRT engines, starts cuVSLAM, performs satellite match | threshold_max | total recovery time ≤ 70s | N/A |
+| 26 | Post-reboot first satellite match | Recovery validation | Position accuracy restored after first satellite match | threshold_max | position error ≤ 50m after first satellite match | N/A |
+
+### Object Localization
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 27 | POST /objects/locate with pixel_x, pixel_y, gimbal angles, zoom, known UAV position | Object at known ground GPS | Response: `{ lat, lon, alt, accuracy_m, confidence }` with lat/lon matching ground truth | numeric_tolerance | lat/lon within accuracy_m of ground truth (consistent with frame-center accuracy) | N/A |
+| 28 | POST /objects/locate with invalid pixel coordinates | Out-of-frame pixel | HTTP 422 or error response indicating invalid input | exact | HTTP status 422 | N/A |
+
+### Coordinate Transform Chain
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 29 | Known GPS → NED → pixel → GPS round-trip | Coordinate transform validation | Round-trip error < 0.1m | threshold_max | ≤ 0.1m | N/A |
+
+### API & Communication
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 30 | GET /health | Health check endpoint | HTTP 200, JSON with memory_mb, gpu_temp_c, status fields | exact (status code), regex (body) | status == 200, body contains `"status"` | N/A |
+| 31 | POST /sessions | Start session | HTTP 200/201 with session ID | exact | status ∈ {200, 201} | N/A |
+| 32 | GET /sessions/{id}/stream | SSE position stream | SSE events at ~1Hz with fields: type, timestamp, lat, lon, alt, accuracy_h, confidence, vo_status | regex | each event matches SSE schema | N/A |
+| 33 | Unauthenticated request to /sessions | No JWT token | HTTP 401 Unauthorized | exact | status == 401 | N/A |
+
+### Performance Thresholds
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 34 | Single camera frame (6252x4168) | End-to-end processing time | Total pipeline latency < 400ms (capture → GPS coordinate output) | threshold_max | ≤ 400ms | N/A |
+| 35 | 30-minute sustained operation | Memory usage over time | Peak memory < 8GB, no memory leaks (growth < 50MB over 30min) | threshold_max | peak < 8192MB, growth ≤ 50MB | N/A |
+| 36 | 30-minute sustained operation | GPU thermal | SoC junction temperature stays below 80°C (no throttling) | threshold_max | ≤ 80°C | N/A |
+| 37 | cuVSLAM single frame | VO processing time | cuVSLAM inference ≤ 20ms per frame | threshold_max | ≤ 20ms | N/A |
+| 38 | Satellite matching single frame | Satellite matching time (async) | LiteSAM/XFeat inference ≤ 330ms | threshold_max | ≤ 330ms | N/A |
+| 39 | TRT engine load | Engine initialization time | All TRT engines loaded within 10s total | threshold_max | ≤ 10s | N/A |
+
+### Satellite Tile Management
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 40 | Mission area definition (200km path, ±2km buffer, zoom 18) | Tile storage calculation | Total storage 500-800MB for zoom 18 + zoom 19 flight path | range | [300MB, 1000MB] | N/A |
+| 41 | ESKF position ± 3σ search radius | Tile selection | Tiles covering search area loaded, mosaic assembled, covers at least 500m radius | threshold_min | coverage radius ≥ 500m | N/A |
+
+### TRT Engine Validation
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 42 | LiteSAM PyTorch model → ONNX → TRT FP16 | TRT engine conversion | Engine builds successfully on Jetson Orin Nano Super | exact | exit_code == 0 | N/A |
+| 43 | TRT engine output vs PyTorch reference (same input) | Inference correctness | Max L1 error between TRT and PyTorch output < 0.01 | threshold_max | L1_max < 0.01 | N/A |
+| 44 | LiteSAM MinGRU operations | TRT compatibility check | All MinGRU ops supported in TRT 10.3 (polygraphy inspect) | exact | unsupported_ops == 0 | N/A |
+
+### Telemetry
+
+| # | Input | Input Description | Expected Result | Comparison | Tolerance | Reference File |
+|---|-------|-------------------|-----------------|------------|-----------|---------------|
+| 45 | Normal operation | Telemetry output rate | NAMED_VALUE_FLOAT messages at 1Hz (gps_conf, gps_drift, gps_hacc) | numeric_tolerance | rate: 1Hz ± 0.2Hz | N/A |
+| 46 | VO tracking lost + 3 satellite failures | Re-localization telemetry | STATUSTEXT with RELOC_REQ sent to ground station | regex | message matches `RELOC_REQ:.*` | N/A |
+
+## Expected Result Reference Files
+
+### position_accuracy.csv
+
+Reference file: `expected_results/position_accuracy.csv`
+
+Contains the ground truth GPS coordinate for each frame in the 60-image test sequence (copied from `coordinates.csv`) plus the acceptance thresholds. Test harness computes haversine distance between estimated and ground truth positions, then applies aggregate criteria.
+
+Thresholds applied to the full 60-frame sequence:
+- ≥ 80% of frames: error < 50m
+- ≥ 60% of frames: error < 20m
+- 0% of frames: error > 100m (no single frame exceeds 100m)
+- Cumulative VO drift between satellite anchors: < 100m