gps-denied-onboard/_docs/02_document/tests/resilience-tests.md

Resilience Tests

NFT-RES-01: FC IMU-only fallback after >3 s without estimate

Summary: Validates AC-5.2 — on >3 s without an estimate, the FC falls back to IMU-only dead reckoning AND the SUT logs the failure. Traces to: AC-5.2

Preconditions:

• SUT in satellite_anchored steady state on Derkachi replay.
• 4 s outage injector primed (replay paused for 4 s of wall-clock).

Fault injection:

• Pause frame source for 4 s of wall-clock while FC IMU stream continues.

Steps:

Step	Action	Expected Behavior
1	Mid-replay, halt frame delivery for 4 s	SUT continues emitting dead_reckoned estimates from FC IMU/attitude propagation
2	After 3 s without an emit (i.e. SUT internally fails to update for >3 s), SUT logs NO_ESTIMATE_TIMEOUT	FDR contains the log entry
3	Observe FC EKF source-set transition	EKF source-set transitions to internal IMU-only on the FC side per the FC's own failsafe logic (AP EKF_FAILSAFE or equivalent on iNav)
4	Resume frame delivery	SUT recovers; FC EKF source-set returns to companion-GPS source

Pass criteria:

• NO_ESTIMATE_TIMEOUT logged within 200 ms of the 3 s mark.
• FC EKF reflects the transition.
• Recovery on resume happens within 5 emit cycles.

---

NFT-RES-02: Companion mid-flight reboot

Summary: Validates AC-5.3 — on companion reboot mid-flight, SUT re-initializes from FC's current IMU-extrapolated position. Traces to: AC-5.3

Preconditions:

• SUT in steady state on Derkachi replay.
• FC SITL has been running long enough to have a stable IMU-extrapolated pose.

Fault injection:

• docker compose restart gps-denied-onboard mid-replay (or systemctl restart on Tier-2).

Steps:

Step	Action	Expected Behavior
1	At t=120 s of replay, restart SUT container	SUT goes down and back up
2	Wait for first post-restart GPS_INPUT / MSP2_SENSOR_GPS arrival	First emit lat/lon within ±100 m of FC's IMU-extrapolated pose at boot-complete time
3	Observe TTFF post-reboot	Within AC-NEW-1 budget (<30 s p95)

Pass criteria:

• First post-restart emit ±100 m of FC pose at boot-complete.
• Cold-restart TTFF < 30 s.
• No FC-side EKF divergence event during the gap.

---

NFT-RES-03: False-position safety budget Monte Carlo

Summary: Validates AC-NEW-4 false-position safety budget (P(error > 500 m) < 0.1%, P(error > 1 km) < 0.01%) on the available data + synthesis. PARTIAL — multi-flight statistics constrained by single Derkachi flight + 60 stills (see traceability matrix flag). Traces to: AC-NEW-4 (PARTIAL)

Preconditions:

• Tier-1 acceptable (statistical rather than hardware-bound).
• Pull together: 60 still-image runs (60 frames) + Derkachi replay (~14,700 frames at 30 fps OR resampled to ~870 frames at 3 Hz target). Total ≥930 frames per Monte Carlo iteration.
• Run M=50 Monte Carlo iterations with synthetic perturbations (camera-pose noise, IMU bias drift, randomized tile sub-selection).

Fault injection:

• Add per-iteration synthetic perturbations to mimic a population of independent flights.

Steps:

Step	Action	Expected Behavior
1	Run M iterations end-to-end	Per-iteration error distribution captured
2	Aggregate across all iterations × frames	Per-frame error CDF
3	Read off P(error > 500 m) and P(error > 1 km) from CDF	Both values

Pass criteria (PARTIAL):

• P(error > 500 m) < 0.1%.
• P(error > 1 km) < 0.01%.
• Test FAILS-OPEN with explicit "PARTIAL" annotation in CSV report when iteration count is below the AC-NEW-4-implied ≥100 flights — noted as reduced confidence pending D-PROJ-3 (AerialVL S03 + own multi-flight data).

---

NFT-RES-04: Visual blackout + spoof degraded-mode escalation

Summary: Validates the AC-NEW-8 escalation ladder (5 s, 15 s, 35 s blackouts paired with spoof) including the 100 m / 500 m covariance thresholds and the 10 s GPS-health gate before recovery. Traces to: AC-NEW-8 (twin of FT-N-04 with extended duration window and covariance assertions)

Preconditions: Same as FT-N-04; Tier-1 acceptable.

Fault injection: blackout-spoof-derkachi 5 s / 15 s / 35 s windows + spoofed FC GPS for the same windows.

Steps:

Step	Action	Expected Behavior
1	Begin 5 s window	Mode transition ≤ 400 ms; covariance grows monotonically; spoofed GPS rejected
2	At end of 5 s window, attempt recovery	Recovery only after FC GPS-health stable + non-spoofed for ≥10 s AND visual/satellite consistency check succeeds (gate enforced)
3	Begin 15 s window	Same as step 1 plus when 95% covariance crosses 100 m: outbound MAVLink fix-quality degraded to "2D fix or worse"
4	Begin 35 s window	Plus when 95% covariance crosses 500 m OR blackout exceeds 30 s: horiz_accuracy=999.0 + VISUAL_BLACKOUT_FAILSAFE STATUSTEXT emitted

Pass criteria:

• All four assertions fire at the right thresholds.
• Recovery gate is honored — early recovery attempts (FC GPS healthy for <10 s) MUST NOT promote spoofed GPS back into the estimator.

Duration: ~10 min total for three windows.