Oleksandr Bezdieniezhnykh
cab7b5d020
- Modified the Docker Compose configuration to include an input root for replay tests and added an environment variable for enabling SITL. - Enhanced documentation for various testing processes, including the addition of a Runtime Completeness Decomposition Gate and clarifications on internal module testing requirements. - Updated the implementation completeness report to reflect the current state and added new test cases for performance and resilience scenarios. Co-authored-by: Cursor <cursoragent@cursor.com>
4.6 KiB
Resilience Tests
NFT-RES-01: Total Visual Blackout With GPS Spoofing
Summary: Validate degraded-mode behavior when the camera feed is totally occluded/blacked out and real GPS is spoofed or denied.
Traces to: AC-3.5, AC-5.2, AC-NEW-8
Preconditions:
- Plane SITL or replay trace is emitting normal telemetry.
- System has a recent trusted visual/satellite anchor.
Fault injection:
- Full camera blackout/total occlusion for 5 s, 15 s, and 35 s while spoofed GPS is present.
| Step | Action | Expected Behavior |
|---|---|---|
| 1 | Inject total occlusion/blackout and spoofed GPS | Camera gate reports usable_for_vio=false, BASALT is bypassed, and system switches to dead_reckoned within <=1 processed frame or <=400 ms |
| 2 | Continue blackout | IMU-only covariance grows monotonically and spoofed GPS is ignored |
| 3 | Exceed 30 s or covariance >500 m | System emits no-fix/failsafe fields and QGC VISUAL_BLACKOUT_FAILSAFE |
Pass criteria: All pre-VIO occlusion gate, timing, covariance, fix_type, horiz_accuracy, and status thresholds match AC-NEW-8.
NFT-RES-02: Sharp Turn And Disconnected Segment Relocalization
Summary: Validate recovery when frame-to-frame overlap drops below the VO threshold.
Traces to: AC-3.2, AC-3.3, AC-3.4, AC-8.6
Preconditions:
- Public or representative replay contains sharp-turn/disconnected segment cases, or equivalent synthetic sequence is generated from mapped imagery.
Fault injection:
- Sequence transition with <5% overlap, heading change <70°, and drift <200 m.
| Step | Action | Expected Behavior |
|---|---|---|
| 1 | Replay normal segment | BASALT + wrapper emits normal vo_extrapolated estimates |
| 2 | Inject sharp-turn/disconnected transition | VO failure is expected; system triggers VPR relocalization |
| 3 | Continue next segment | System connects segment through verified satellite anchor or reports degraded status |
Pass criteria: Relocalization request is issued when no position is available for >=3 consecutive frames and >=2 s; verified anchor reconnects the segment or output remains degraded with growing covariance.
NFT-RES-03: Companion Computer Restart Mid-Flight
Summary: Validate reboot recovery from flight-controller state and preloaded cache.
Traces to: AC-5.3, AC-NEW-1
Preconditions:
- Replay/SITL mission is in progress.
- FDR has current segment logs.
Fault injection:
- Kill and restart the GPS-denied service during a GPS-denied segment.
| Step | Action | Expected Behavior |
|---|---|---|
| 1 | Kill service | FC continues on last known/IMU-extrapolated state |
| 2 | Restart service | Service reloads cache/index and uses FC state handoff |
| 3 | Observe first valid output | First valid GPS_INPUT emitted within <30 s |
Pass criteria: No raw frames are required for recovery; first valid fix <30 s p95; failure is logged in FDR.
NFT-RES-04: Tile Cache Freshness Degradation
Summary: Validate graceful behavior when the only available tile candidates are stale.
Traces to: AC-8.2, AC-NEW-6
Fault injection:
- Mark cache tiles older than 6 months for active-conflict sector and older than 12 months for stable sector.
| Step | Action | Expected Behavior |
|---|---|---|
| 1 | Replay frame requiring satellite anchor | Stale tiles are rejected or down-confidence weighted |
| 2 | Inspect emitted estimate | No stale tile produces satellite_anchored label past hard rejection threshold |
Pass criteria: Freshness decay and hard rejection match AC-NEW-6.
NFT-RES-INFRA: Replay/SITL Prerequisite Smoke
Summary: Validate that the Docker replay environment can execute the resilience scenario group with deterministic SITL/QGC stubs.
Traces to: AZ-237 AC-1, AZ-237 AC-4, AZ-233 AC-1, AZ-233 AC-3
Preconditions:
ardupilot-plane-sitlandqgc-observerservices are started bydocker-compose.test.yml.GPSD_ENABLE_SITL=1is set only for the Docker replay stub environment.
Fault injection:
- Run the blackout/restart control smoke scenario through the replay consumer.
| Step | Action | Expected Behavior |
|---|---|---|
| 1 | Start Docker replay services | SITL and QGC observer stubs are reachable to the replay consumer |
| 2 | Execute the resilience smoke scenario | The report records a pass result instead of a missing-SITL prerequisite block |
Pass criteria: NFT-RES-INFRA reports pass in Docker replay mode; live SITL release-candidate scenarios remain covered by NFT-RES-01 and FT-N-02.