mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-04-22 11:36:37 +00:00
Revise skills documentation to incorporate updated directory structure and terminology. Replace references to integration tests with blackbox tests in SKILL.md files and templates. Adjust paths in planning and deployment documentation to align with the new _docs/02_document/ structure, ensuring consistency and clarity throughout the documentation.
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,169 @@
|
||||
# Resilience Tests
|
||||
|
||||
### NFT-RES-01: Mid-Flight Reboot Recovery
|
||||
|
||||
**Summary**: Validate the system recovers from a companion computer reboot within 70 seconds and restores position accuracy.
|
||||
**Traces to**: AC-12 (mid-flight reboot recovery)
|
||||
|
||||
**Preconditions**:
|
||||
- System running in steady state with good position accuracy
|
||||
- SITL ArduPilot continues running (FC stays up during companion computer reboot)
|
||||
|
||||
**Fault injection**:
|
||||
- Kill gps-denied-system process (docker stop or SIGKILL)
|
||||
- Restart after 5s delay (simulates Jetson reboot time)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Record current position accuracy and confidence | Baseline metrics |
|
||||
| 2 | Kill gps-denied-system container | GPS_INPUT messages stop |
|
||||
| 3 | Verify SITL continues running (heartbeat present) | FC still alive, using IMU dead reckoning |
|
||||
| 4 | Restart gps-denied-system container after 5s | System starts recovery sequence |
|
||||
| 5 | Monitor time from restart to first GPS_INPUT | ≤ 70s |
|
||||
| 6 | Wait for first satellite match | Position accuracy restored |
|
||||
| 7 | Verify position error after recovery | Error ≤ 50m after first satellite match |
|
||||
|
||||
**Pass criteria**: Recovery time ≤ 70s; post-recovery position error ≤ 50m after satellite match
|
||||
**Duration**: 120s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-02: Tracking Loss and Satellite Re-Localization
|
||||
|
||||
**Summary**: Validate the system recovers from cuVSLAM tracking loss via satellite-based re-localization.
|
||||
**Traces to**: AC-07 (disconnected segments), AC-06 (sharp turn handling)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking (HIGH confidence)
|
||||
- Satellite tiles available
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay sends featureless/blurred frames (simulates VO tracking loss from sharp turn)
|
||||
- Then resumes normal frames
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking established | confidence: HIGH, vo_status: tracking |
|
||||
| 2 | Camera-replay serves 3 featureless frames | cuVSLAM reports tracking_lost |
|
||||
| 3 | System enters TRACKING_LOST state | Satellite matching switches to every frame |
|
||||
| 4 | Camera-replay resumes normal frames | Satellite match succeeds |
|
||||
| 5 | Monitor SSE: vo_status returns to "tracking" | cuVSLAM restarted |
|
||||
| 6 | Monitor SSE: confidence returns to HIGH | Position re-anchored |
|
||||
| 7 | Verify position accuracy after recovery | Error ≤ 50m |
|
||||
|
||||
**Pass criteria**: Recovery within 5 frames after normal frames resume; position error ≤ 50m post-recovery
|
||||
**Duration**: 30s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-03: Sustained IMU-Only Operation
|
||||
|
||||
**Summary**: Validate the system continues producing position estimates during extended IMU-only periods without crashing.
|
||||
**Traces to**: AC-08 (system continues during failure), AC-12 (failsafe)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Pause both camera-replay (no VO) and satellite-tile-server (no satellite matching)
|
||||
- Duration: 30s
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Establish normal tracking baseline | GPS_INPUT at 5-10Hz, confidence HIGH |
|
||||
| 2 | Pause camera-replay and satellite-tile-server | VO and satellite inputs stop |
|
||||
| 3 | Monitor GPS_INPUT for 30s | Messages continue at 5-10Hz (IMU-driven ESKF prediction) |
|
||||
| 4 | Verify horiz_accuracy grows over time | accuracy increases monotonically |
|
||||
| 5 | Verify fix_type transitions to 2 | Degraded but present |
|
||||
| 6 | Verify confidence transitions to LOW | Reflects IMU-only state |
|
||||
| 7 | Resume camera-replay and satellite-tile-server | System recovers to normal tracking |
|
||||
| 8 | Verify recovery to HIGH confidence | Satellite match re-anchors position |
|
||||
|
||||
**Pass criteria**: GPS_INPUT never stops during 30s IMU-only period; system recovers when inputs resume
|
||||
**Duration**: 60s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-04: Satellite Tile Server Failure
|
||||
|
||||
**Summary**: Validate the system continues operating when satellite tile server becomes unavailable, with graceful accuracy degradation.
|
||||
**Traces to**: AC-07 (resilience), solution risk: Google Maps quality
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Stop satellite-tile-server container (simulates tile unavailability)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking with satellite corrections | confidence: HIGH |
|
||||
| 2 | Stop satellite-tile-server | Satellite matching returns errors |
|
||||
| 3 | Monitor for 60s | System falls back to VO-only; confidence drops to MEDIUM after 30s |
|
||||
| 4 | Verify GPS_INPUT continues | Messages at 5-10Hz, fix_type remains 3 (VO tracking OK) |
|
||||
| 5 | Restart satellite-tile-server | Satellite matching resumes |
|
||||
| 6 | Verify confidence returns to HIGH | Position re-anchored |
|
||||
|
||||
**Pass criteria**: No crash or hang; GPS_INPUT continues; confidence degrades gracefully and recovers when tiles return
|
||||
**Duration**: 90s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-05: Corrupted Camera Frame
|
||||
|
||||
**Summary**: Validate the system handles a corrupted camera frame without crashing.
|
||||
**Traces to**: AC-06 (outlier tolerance)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay injects a truncated/corrupted JPEG between normal frames
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking for 5 frames | Baseline established |
|
||||
| 2 | Camera-replay sends corrupted JPEG | System logs warning, skips frame |
|
||||
| 3 | Camera-replay sends next normal frame | VO continues processing |
|
||||
| 4 | Verify no crash, no hang | GPS_INPUT continues at 5-10Hz |
|
||||
| 5 | Verify position accuracy on next valid frame | Error < 50m |
|
||||
|
||||
**Pass criteria**: System skips corrupted frame gracefully; no crash; next frame processed normally
|
||||
**Duration**: 15s
|
||||
|
||||
---
|
||||
|
||||
### NFT-RES-06: Camera Feed Interruption (No Frames for 10s)
|
||||
|
||||
**Summary**: Validate the system survives a 10-second camera feed interruption.
|
||||
**Traces to**: AC-12 (failsafe — N seconds no estimate), AC-08 (continued operation)
|
||||
|
||||
**Preconditions**:
|
||||
- System in normal tracking
|
||||
|
||||
**Fault injection**:
|
||||
- Camera-replay pauses for 10s (no frames delivered)
|
||||
|
||||
**Steps**:
|
||||
|
||||
| Step | Action | Expected Behavior |
|
||||
|------|--------|------------------|
|
||||
| 1 | Normal tracking baseline | GPS_INPUT at 5-10Hz |
|
||||
| 2 | Pause camera-replay for 10s | No new camera frames |
|
||||
| 3 | Monitor GPS_INPUT | Messages continue via IMU prediction |
|
||||
| 4 | Monitor confidence | Transitions to LOW after VO timeout |
|
||||
| 5 | Resume camera-replay | VO restarts, satellite matching resumes |
|
||||
| 6 | Verify recovery | confidence returns to HIGH within 10 frames |
|
||||
|
||||
**Pass criteria**: GPS_INPUT never stops; recovery within 10 frames after camera feed resumes
|
||||
**Duration**: 30s
|
||||
Reference in New Issue
Block a user