mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 20:21:17 +00:00
Oleksandr Bezdieniezhnykh
cab7b5d020
- Modified the Docker Compose configuration to include an input root for replay tests and added an environment variable for enabling SITL. - Enhanced documentation for various testing processes, including the addition of a Runtime Completeness Decomposition Gate and clarifications on internal module testing requirements. - Updated the implementation completeness report to reflect the current state and added new test cases for performance and resilience scenarios. Co-authored-by: Cursor <cursoragent@cursor.com>
108 lines
4.6 KiB
Markdown
108 lines
4.6 KiB
Markdown
# Resilience Tests
|
|
|
|
### NFT-RES-01: Total Visual Blackout With GPS Spoofing
|
|
|
|
**Summary**: Validate degraded-mode behavior when the camera feed is totally occluded/blacked out and real GPS is spoofed or denied.
|
|
|
|
**Traces to**: AC-3.5, AC-5.2, AC-NEW-8
|
|
|
|
**Preconditions**:
|
|
- Plane SITL or replay trace is emitting normal telemetry.
|
|
- System has a recent trusted visual/satellite anchor.
|
|
|
|
**Fault injection**:
|
|
- Full camera blackout/total occlusion for 5 s, 15 s, and 35 s while spoofed GPS is present.
|
|
|
|
| Step | Action | Expected Behavior |
|
|
|------|--------|-------------------|
|
|
| 1 | Inject total occlusion/blackout and spoofed GPS | Camera gate reports `usable_for_vio=false`, BASALT is bypassed, and system switches to `dead_reckoned` within <=1 processed frame or <=400 ms |
|
|
| 2 | Continue blackout | IMU-only covariance grows monotonically and spoofed GPS is ignored |
|
|
| 3 | Exceed 30 s or covariance >500 m | System emits no-fix/failsafe fields and QGC `VISUAL_BLACKOUT_FAILSAFE` |
|
|
|
|
**Pass criteria**: All pre-VIO occlusion gate, timing, covariance, `fix_type`, `horiz_accuracy`, and status thresholds match AC-NEW-8.
|
|
|
|
---
|
|
|
|
### NFT-RES-02: Sharp Turn And Disconnected Segment Relocalization
|
|
|
|
**Summary**: Validate recovery when frame-to-frame overlap drops below the VO threshold.
|
|
|
|
**Traces to**: AC-3.2, AC-3.3, AC-3.4, AC-8.6
|
|
|
|
**Preconditions**:
|
|
- Public or representative replay contains sharp-turn/disconnected segment cases, or equivalent synthetic sequence is generated from mapped imagery.
|
|
|
|
**Fault injection**:
|
|
- Sequence transition with <5% overlap, heading change <70°, and drift <200 m.
|
|
|
|
| Step | Action | Expected Behavior |
|
|
|------|--------|-------------------|
|
|
| 1 | Replay normal segment | BASALT + wrapper emits normal `vo_extrapolated` estimates |
|
|
| 2 | Inject sharp-turn/disconnected transition | VO failure is expected; system triggers VPR relocalization |
|
|
| 3 | Continue next segment | System connects segment through verified satellite anchor or reports degraded status |
|
|
|
|
**Pass criteria**: Relocalization request is issued when no position is available for >=3 consecutive frames and >=2 s; verified anchor reconnects the segment or output remains degraded with growing covariance.
|
|
|
|
---
|
|
|
|
### NFT-RES-03: Companion Computer Restart Mid-Flight
|
|
|
|
**Summary**: Validate reboot recovery from flight-controller state and preloaded cache.
|
|
|
|
**Traces to**: AC-5.3, AC-NEW-1
|
|
|
|
**Preconditions**:
|
|
- Replay/SITL mission is in progress.
|
|
- FDR has current segment logs.
|
|
|
|
**Fault injection**:
|
|
- Kill and restart the GPS-denied service during a GPS-denied segment.
|
|
|
|
| Step | Action | Expected Behavior |
|
|
|------|--------|-------------------|
|
|
| 1 | Kill service | FC continues on last known/IMU-extrapolated state |
|
|
| 2 | Restart service | Service reloads cache/index and uses FC state handoff |
|
|
| 3 | Observe first valid output | First valid `GPS_INPUT` emitted within <30 s |
|
|
|
|
**Pass criteria**: No raw frames are required for recovery; first valid fix <30 s p95; failure is logged in FDR.
|
|
|
|
---
|
|
|
|
### NFT-RES-04: Tile Cache Freshness Degradation
|
|
|
|
**Summary**: Validate graceful behavior when the only available tile candidates are stale.
|
|
|
|
**Traces to**: AC-8.2, AC-NEW-6
|
|
|
|
**Fault injection**:
|
|
- Mark cache tiles older than 6 months for active-conflict sector and older than 12 months for stable sector.
|
|
|
|
| Step | Action | Expected Behavior |
|
|
|------|--------|-------------------|
|
|
| 1 | Replay frame requiring satellite anchor | Stale tiles are rejected or down-confidence weighted |
|
|
| 2 | Inspect emitted estimate | No stale tile produces `satellite_anchored` label past hard rejection threshold |
|
|
|
|
**Pass criteria**: Freshness decay and hard rejection match AC-NEW-6.
|
|
|
|
---
|
|
|
|
### NFT-RES-INFRA: Replay/SITL Prerequisite Smoke
|
|
|
|
**Summary**: Validate that the Docker replay environment can execute the resilience scenario group with deterministic SITL/QGC stubs.
|
|
|
|
**Traces to**: AZ-237 AC-1, AZ-237 AC-4, AZ-233 AC-1, AZ-233 AC-3
|
|
|
|
**Preconditions**:
|
|
- `ardupilot-plane-sitl` and `qgc-observer` services are started by `docker-compose.test.yml`.
|
|
- `GPSD_ENABLE_SITL=1` is set only for the Docker replay stub environment.
|
|
|
|
**Fault injection**:
|
|
- Run the blackout/restart control smoke scenario through the replay consumer.
|
|
|
|
| Step | Action | Expected Behavior |
|
|
|------|--------|-------------------|
|
|
| 1 | Start Docker replay services | SITL and QGC observer stubs are reachable to the replay consumer |
|
|
| 2 | Execute the resilience smoke scenario | The report records a `pass` result instead of a missing-SITL prerequisite block |
|
|
|
|
**Pass criteria**: `NFT-RES-INFRA` reports `pass` in Docker replay mode; live SITL release-candidate scenarios remain covered by `NFT-RES-01` and `FT-N-02`.
|