[AZ-424] [AZ-425] [AZ-426] Implement negatives set (FT-N-01/03/04)

Adds three pure-logic evaluators + scenarios + unit tests covering the
project's failure-mode robustness ladder (AC-3.1, AC-3.4, AC-3.5,
AC-NEW-8):

* outlier_tolerance_evaluator (AZ-424 / FT-N-01): per-event 50 m drift
  bound + 3-frame covariance-monotonic window over the AZ-408 outlier
  injector's medium-density manifest.
* outage_request_evaluator (AZ-425 / FT-N-03): detects 3+ consecutive
  missing-frame windows; validates OPERATOR_RELOC_REQUEST STATUSTEXT
  arrives at 2 s ±500 ms, dead_reckoned label during outage, and no
  FC EKF divergence.
* blackout_spoof_evaluator (AZ-426 / FT-N-04): eight-AC ladder across
  the 5 s / 15 s / 35 s sub-windows — switch latency, spoof rejection,
  monotonic covariance, honest horiz_accuracy, STATUSTEXT 1-2 Hz,
  35 s escalation thresholds, and recovery gate.

Each scenario is skip-gated on the AZ-441 / AZ-407 / AZ-416 replay /
SITL / mavproxy helpers; unit tests (14 + 18 + 29 = 61) cover the
AC logic today. Full e2e unit-test suite: 527 passed (+67).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-05-17 08:26:16 +03:00
parent a644debdb7
commit 2d6d44af5d
16 changed files with 3343 additions and 1 deletions
+141
View File
@@ -0,0 +1,141 @@
# Batch 73 Report — Test Implementation (cycle 1, batch 7 of test phase)
**Batch**: 73
**Date**: 2026-05-17
**Context**: Test implementation (greenfield Step 10 — Implement Tests)
**Tasks**: AZ-424 (3pt), AZ-425 (3pt), AZ-426 (5pt) — 11 cp / 3 tasks
**Cycle**: 1
**Verdict**: COMPLETE — PASS (self-reviewed; see `reviews/batch_73_review.md`)
## Summary
The negatives set — FT-N-01 / FT-N-03 / FT-N-04 — the project's
failure-mode robustness suite (AC-3.1, AC-3.4, AC-3.5, AC-NEW-8).
Same pattern as the prior batches in this phase:
* Pure-logic evaluator under `e2e/runner/helpers/` (everything the
scenario can express without docker-bound SITL access).
* Scenario file under `e2e/tests/negative/`, parameterised across
conftest fixtures, skip-gated on upstream replay / FDR / mavproxy
/ SITL observer helpers (auto-activates when AZ-441 + AZ-407 +
AZ-416 leftovers land).
* Helper-driven unit test file under `e2e/_unit_tests/helpers/`.
### AZ-424 — FT-N-01 350 m outlier injection tolerance (3pt)
* **`runner/helpers/outlier_tolerance_evaluator.py`** — three
invariants:
- AC-1: count gate — `MIN_OUTLIER_COUNT = 10` outliers across the
Derkachi 8-min `--density medium` replay (the AC-3.1 envelope).
- AC-2: per-event drift bound — `error_after_outlier
error_before_outlier ≤ DRIFT_BUDGET_M = 50.0`. `before` / `after`
are the immediate neighbour frames in the outbound stream;
`distance_m` is the shared Vincenty helper.
- AC-3: covariance monotonic across the 3-frame window centred on
the outlier (`COVARIANCE_WINDOW_FRAMES = 3`).
- Plus `load_outlier_manifest` (reads the AZ-408 injector's
`manifest.csv`) and `write_csv_evidence`.
* **`tests/negative/test_ft_n_01_outlier_tolerance.py`** — scenario
indirect-parametrises `outlier_injection_derkachi` at
`density="medium", seed=0`, drives replay, collects FDR
`outbound_estimate` records, joins them to per-frame GT, evaluates,
asserts per-event `passes_drift` + `passes_covariance` plus the
aggregate `passes_count`. Records NFR metrics
`ft_n_01.total_outliers`, `ft_n_01.failed_event_count`, per-event
`drift_m` + `cov_non_decreasing`.
* **14 unit tests** in `test_outlier_tolerance_evaluator.py`.
### AZ-425 — FT-N-03 Extended outage triggers operator re-loc request (3pt)
* **`runner/helpers/outage_request_evaluator.py`** — first detects
outage windows from frame-index gaps (≥`MIN_OUTAGE_FRAMES = 3`
consecutive missing frames), then per-window evaluates:
- AC-2: STATUSTEXT `OPERATOR_RELOC_REQUEST` observed at
`[OUTAGE_THRESHOLD_S TOLERANCE_S, OUTAGE_THRESHOLD_S +
TOLERANCE_S] = [1.5, 2.5] s` after outage onset.
- AC-3: at least one `source_label = dead_reckoned` outbound
emission inside the window.
- AC-4: zero FC-side EKF divergence events inside the window
(observable via SITL state read).
- Plus `detect_outage_windows` (with explicit handling for trailing
windows + multi-window flights) and `write_csv_evidence`.
* **`tests/negative/test_ft_n_03_outage_reloc.py`** — scenario drives
replay with a 3-frame outage injector (a future thin extension of
the AZ-408 outlier injector), reads FDR `frame_received` +
`outbound_estimate` records to reconstruct
`expected_frame_indices` and the estimate stream, walks the
mavproxy `.tlog` for STATUSTEXT, and pulls EKF divergence events
via `sitl_observer.read_ekf_divergence_events()`. Records per-window
NFR metrics with AC IDs (`length_frames`, `statustext_offset_ms`,
`dead_reckoned_count`, `ekf_divergence_count`).
* **18 unit tests** in `test_outage_request_evaluator.py`.
### AZ-426 — FT-N-04 Visual blackout + spoofed GPS combined failsafe (5pt)
* **`runner/helpers/blackout_spoof_evaluator.py`** — the most ladder-
heavy evaluator in the project: eight per-AC sub-reports stitched
into one `BlackoutSpoofReport`. Constants pulled into the module
header so the spec can be diffed against code in one place:
`SWITCH_LATENCY_MS = 400` (AC-1),
`HONEST_ACCURACY_RATIO = 0.95` (AC-4),
`STATUSTEXT_RATE_MIN_HZ = 1.0` / `STATUSTEXT_RATE_MAX_HZ = 2.0` (AC-5),
`ESCALATION_COV_2D_M = 100.0` (AC-6),
`ESCALATION_COV_FAILSAFE_M = 500.0`, `ESCALATION_DURATION_FAILSAFE_S = 30.0`,
`ESCALATION_LATENCY_MS = 500` (AC-7),
`RECOVERY_STABLE_S = 10.0` (AC-8).
Per-AC analysers:
- `evaluate_switch_latency`: budget = `min(SWITCH_LATENCY_MS,
frame_period_ms)` — the spec's "≤1 frame OR ≤400 ms (whichever is
shorter)" wording, made explicit.
- `evaluate_spoof_rejection`: requires both ≥1 FDR
`spoof-rejected` event AND zero `satellite_anchored` emissions
inside the window (so the SUT cannot silently re-promote on a
spoofed lock).
- `evaluate_covariance_monotonic`: first non-decreasing violation
timestamp + binary pass.
- `evaluate_honest_accuracy`: per-sample `horiz_accuracy ≥ 0.95 ×
cov_semi_major_m`. Boundary test pins the spec budget.
- `evaluate_statustext_rate`: `VISUAL_BLACKOUT_IMU_ONLY` rate over
the window must land in [1, 2] Hz.
- `evaluate_escalation` (35 s window only): AC-6 fix_type degrades
on the first cov-100 m crossing; AC-7 triggers on the earliest
of cov-500 m crossing OR 30 s duration. Non-35 s windows pass
vacuously — they aren't expected to hit either threshold.
- `evaluate_recovery_gate`: AC-8 — ≥10 s of healthy + non-spoofed
FC GPS + a consistency-check pass before re-promoting to
`satellite_anchored` post-window.
* **`tests/negative/test_ft_n_04_blackout_spoof.py`** — scenario
indirect-parametrises `blackout_spoof_derkachi` over
`_WINDOW_LADDER_S = (5.0, 15.0, 35.0)` with ids `["5s", "15s",
"35s"]`. Collects FDR `outbound_estimate` + `spoof_rejected`,
mavproxy STATUSTEXT, and SITL GPS-health + consistency-check
samples. Asserts each AC with a descriptive failure message that
surfaces the relevant sub-report fields.
* **29 unit tests** in `test_blackout_spoof_evaluator.py`.
## Layout invariant
`e2e/_unit_tests/test_directory_layout.py` now lists the three new
evaluators and the three new scenario files.
## Test Results
* New unit tests: 14 + 18 + 29 = **61**.
* Plus 6 new entries in `test_required_path_exists` parametrize
(3 helpers + 3 scenarios).
* Full `e2e/_unit_tests` suite: **527 passed in 130 s** (previous
cumulative: 460 → +67 net).
* Scenario collection across the three negatives: 48 items
parametrized; the session-end `/e2e-results/evidence/per-nfr`
teardown error is the same pre-existing `nfr_recorder` wart
documented in batches 69-72 — not a regression of this batch and
not blocking unit-suite collection.
## State
* Specs moved: `_docs/02_tasks/todo/AZ-{424,425,426}_*.md` →
`_docs/02_tasks/done/`.
* `_docs/_autodev_state.md` advanced to
`last_completed_batch: 73`.
* Cumulative review window: `last_cumulative_review = batches_70-72`;
the next K=3 cumulative review fires at the end of batch 75.
@@ -0,0 +1,173 @@
# Code Review Report
**Batch**: 73 — AZ-424, AZ-425, AZ-426
**Date**: 2026-05-17
**Verdict**: PASS
## Findings
(none)
## Findings Sweep
### Phase 1 — Context Loading
Loaded specs `AZ-424_ft_n_01_outlier_tolerance.md`,
`AZ-425_ft_n_03_outage_reloc.md`, `AZ-426_ft_n_04_blackout_spoof.md`.
Re-read injector surfaces touched by the new evaluators:
`e2e/fixtures/injectors/outlier.py` (manifest.csv schema +
`OutlierInjectionReport.out_root`), `e2e/fixtures/injectors/blackout_spoof.py`
(`BlackoutSpoofPlan`, `BlackoutSpoofSchedule.window_start_ms / window_end_ms`,
spoofed-GPS cadence + AC-NEW-8 200-500 m delta bounds). Re-read existing
fixture wiring in `e2e/runner/helpers/injector_fixtures.py` to confirm
`outlier_injection_derkachi` and `blackout_spoof_derkachi` parametrize
on `density` / `window_seconds`. Re-read the scenario template used in
batch 71/72 (`tests/positive/test_ft_p_10_smoothing_lookback.py`,
`tests/negative/test_ft_n_02_sharp_turn_failure.py`) for the
`_harness_helpers_implemented` gate pattern and the FDR / mavproxy /
sitl_observer access conventions.
### Phase 2 — Spec Compliance
**AZ-424 (FT-N-01)**
| AC | Coverage | Status |
|----|----------|--------|
| AC-1 (medium-density injection; ≥10 outliers) | `test_constants_match_spec`, `test_evaluate_count_below_minimum_fails`, `test_evaluate_count_at_minimum_passes_count_gate`, scenario assertion via `MIN_OUTLIER_COUNT` | Covered |
| AC-2 (drift bound ≤50 m per outlier) | `test_evaluate_event_drift_within_budget`, `test_evaluate_event_drift_exceeds_budget_fails`, `test_evaluate_event_missing_neighbour_drift_none`, scenario per-event assertion via `OutlierEventReport.passes_drift` | Covered |
| AC-3 (covariance monotonic across 3-frame window) | `test_evaluate_event_cov_monotonic_passes`, `test_evaluate_event_cov_decreasing_fails`, `test_evaluate_event_cov_flat_window_passes`, scenario assertion via `passes_covariance` | Covered |
| AC-4 (parameterization per fc_adapter × vio_strategy) | scenario uses conftest `fc_adapter`/`vio_strategy` fixtures + indirect `outlier_injection_derkachi` (density=medium, seed=0) | Covered |
| CSV evidence | `test_write_csv_evidence_round_trips`, scenario writes `ft-n-01-{fc_adapter}-{vio_strategy}.csv` | Covered |
**AZ-425 (FT-N-03)**
| AC | Coverage | Status |
|----|----------|--------|
| AC-1 (≥3 consecutive missing frames) | `test_detect_no_outage_returns_empty`, `test_detect_run_below_min_length_ignored`, `test_detect_single_outage_window`, `test_detect_multiple_windows`, `test_detect_trailing_outage_window`, scenario assertion via `passes_min_length` | Covered |
| AC-2 (STATUSTEXT `OPERATOR_RELOC_REQUEST` within 2 s ±500 ms of onset) | `test_statustext_within_tolerance_passes`, `test_statustext_within_tolerance_late_passes`, `test_statustext_too_early_fails`, `test_statustext_too_late_fails`, `test_statustext_missing_fails`, `test_statustext_payload_mismatch_fails`, scenario assertion via `passes_statustext` | Covered |
| AC-3 (dead_reckoned label during outage) | `test_dead_reckoned_during_window_passes`, `test_dead_reckoned_absent_fails`, scenario assertion via `passes_dead_reckoned` | Covered |
| AC-4 (no FC EKF divergence event during outage) | `test_ekf_divergence_during_window_fails`, `test_ekf_divergence_outside_window_ignored`, scenario assertion via `passes_ekf` | Covered |
| AC-5 (parameterization) | scenario uses conftest `fc_adapter`/`vio_strategy` fixtures | Covered |
| CSV evidence | `test_write_csv_evidence_round_trips`, scenario writes `ft-n-03-{fc_adapter}-{vio_strategy}.csv` | Covered |
**AZ-426 (FT-N-04)**
| AC | Coverage | Status |
|----|----------|--------|
| AC-1 (switch latency ≤1 frame OR ≤400 ms) | `test_switch_latency_within_400_ms_passes` (validates `min(400, frame_period_ms)` budget), `test_switch_latency_within_one_frame_passes`, `test_switch_latency_at_one_frame_boundary_passes`, `test_switch_latency_missing_dead_reckoned_fails`, scenario assertion via `switch_latency.passes` | Covered |
| AC-2 (spoof-rejected events AND no satellite re-anchor inside window) | `test_spoof_rejection_pass`, `test_spoof_rejection_no_events_fails`, `test_spoof_rejection_label_returns_to_satellite_fails`, scenario assertion via `spoof_rejection.passes` | Covered |
| AC-3 (covariance monotonic) | `test_covariance_monotonic_pass`, `test_covariance_monotonic_decreasing_fails`, scenario assertion via `covariance_monotonic.passes` | Covered |
| AC-4 (`horiz_accuracy ≥ 0.95 × cov_semi_major_m`) | `test_honest_accuracy_pass`, `test_honest_accuracy_boundary_pass`, `test_honest_accuracy_violation_fails`, scenario assertion via `honest_accuracy.passes` | Covered |
| AC-5 (`VISUAL_BLACKOUT_IMU_ONLY` rate ∈ [1, 2] Hz) | `test_statustext_rate_pass_at_1hz`, `test_statustext_rate_pass_at_2hz`, `test_statustext_rate_too_slow_fails`, `test_statustext_rate_too_fast_fails`, scenario assertion via `statustext_rate.passes` | Covered |
| AC-6 (35 s only: cov 100 m → fix_type ≤2D) | `test_escalation_non_35s_window_passes_vacuously`, `test_escalation_35s_ac6_fix_type_degraded_passes`, `test_escalation_35s_ac6_fix_type_not_degraded_fails`, scenario assertion gated on `is_35s` via `escalation.passes_ac6` | Covered |
| AC-7 (35 s only: cov 500 m OR 30 s duration → `horiz=999`, `VISUAL_BLACKOUT_FAILSAFE` within 500 ms) | `test_escalation_35s_no_crossings_passes` (vacuous on duration-only path), `test_escalation_35s_ac7_horiz_not_999_fails`, scenario assertion gated on `is_35s` via `escalation.passes_ac7` | Covered |
| AC-8 (recovery gate: ≥10 s stable + consistency check pass) | `test_recovery_gate_pass`, `test_recovery_gate_unstable_fails`, `test_recovery_gate_spoofed_fails`, `test_recovery_gate_no_consistency_check_fails`, `test_recovery_gate_no_recovery_attempt_vacuous_pass`, scenario assertion via `recovery_gate.passes` | Covered |
| AC-9 (parameterization × 3 windows) | scenario indirect-parametrizes `blackout_spoof_derkachi` over `_WINDOW_LADDER_S = (5.0, 15.0, 35.0)` with ids `["5s", "15s", "35s"]`; conftest `fc_adapter`/`vio_strategy` adds 6 variants = 18 collected items per fc_adapter pair | Covered |
| CSV evidence | `test_write_csv_evidence_round_trips`, scenario writes `ft-n-04-{window_s}s-{fc_adapter}-{vio_strategy}.csv` | Covered |
### Phase 3 — Code Quality
* **Single responsibility**: each evaluator is one module with one
responsibility — `outlier_tolerance_evaluator` aggregates per-event
AC-2/AC-3 reports; `outage_request_evaluator` detects outage windows
and evaluates AC-1..AC-4 per window; `blackout_spoof_evaluator`
evaluates the AC-1..AC-8 ladder against one `BlackoutWindow`. None
of the three pulls in scenario-specific helpers (drive replay /
collect samples) — those live in the scenario test files.
* **Method naming**: per-AC evaluators are named after the AC concern
(`evaluate_switch_latency`, `evaluate_spoof_rejection`,
`evaluate_covariance_monotonic`, `evaluate_honest_accuracy`,
`evaluate_statustext_rate`, `evaluate_escalation`,
`evaluate_recovery_gate`). The aggregate `evaluate(...)` in each
module composes the per-AC reports into a single dataclass.
* **No suppressed errors**: `load_outlier_manifest` raises on missing
file and missing columns; the manifest writer raises naturally on
ENOENT; the evaluator helpers raise no exceptions of their own.
No bare `except`, no `2>/dev/null`-equivalents.
* **AAA comment discipline**: every test uses `# Arrange / # Act /
# Assert`; sections are omitted when not needed (e.g. constant
invariant tests just have `# Assert`).
* **Public boundary**: confirmed all three evaluators import only from
the `e2e.runner.helpers.geo` symbol (when needed) and dataclasses /
stdlib. No `from gps_denied_onboard ...`. Confirmed via grep.
### Phase 4 — Security
* **No new secrets, credentials, or network paths**. All three
evaluators are pure-logic over already-collected samples / events.
* **Spoof rejection (AC-2)** is the project's primary anti-spoof
invariant; the evaluator does not bypass it — it asserts the FDR
recorded the rejection AND that the source-label state machine did
not silently re-promote to `satellite_anchored` inside the window.
* **Honest accuracy (AC-4)** ensures the SUT cannot under-report
uncertainty to the FC. The evaluator's check is `horiz_accuracy ≥
0.95 × cov_semi_major_m` per the spec; we explicitly cover the
boundary in `test_honest_accuracy_boundary_pass` so a future
implementation cannot pass by emitting `horiz = cov` while the spec
budget is `0.95 × cov`.
### Phase 5 — Performance
All three evaluators are O(N) over their input sequences (single
pass over estimates, single pass over events, single pass over
statustexts). No nested scans beyond the bounded 3-frame window in
`outlier_tolerance_evaluator.evaluate_event`. CSV writes use
buffered `csv.writer`. No file I/O at module import time.
### Phase 6 — Cross-Task Consistency
* **Shared `geo.distance_m`** is the single point-to-point distance
helper used by `outlier_tolerance_evaluator`. Matches the
`accuracy_evaluator`, `multi_segment_evaluator`,
`smoothing_evaluator`, `cold_start_evaluator` conventions.
* **Shared `_harness_helpers_implemented` skip gate**: all three new
scenarios use the same probe pattern as `test_ft_p_10_*`,
`test_ft_p_11_*`, `test_ft_n_02_*``NotImplementedError` on
`frame_source_replay`, `fdr_reader`, `imu_replay`,
`mavproxy_tlog_reader`, `sitl_observer` collapses to a single
`pytest.skip(...)` with a pointer to the relevant unit test.
* **Constants centralised inside each module**: `MIN_OUTLIER_COUNT`,
`DRIFT_BUDGET_M`, `SWITCH_LATENCY_MS`, `STATUSTEXT_RATE_*_HZ`,
`ESCALATION_*` all sit at the top of their respective modules and
are imported as named constants in the unit tests. No magic numbers
inline.
* **Source-label vocabulary**: `dead_reckoned` / `satellite_anchored`
are spelled identically across the three new evaluators and match
the prior batches (`sharp_turn_detector.ALLOWED_DURING_TURN_LABELS`,
`multi_segment_evaluator`, FDR schema in batch 67-68).
* **STATUSTEXT regex strings**: `OPERATOR_RELOC_REQUEST` (FT-N-03),
`VISUAL_BLACKOUT_IMU_ONLY` (FT-N-04 AC-5),
`VISUAL_BLACKOUT_FAILSAFE` (FT-N-04 AC-7) match the spec verbatim;
unit-tested for substring presence + payload mismatch.
### Phase 7 — Architecture Compliance
* **Module placement**: all three evaluators live in
`e2e/runner/helpers/`; their unit tests in
`e2e/_unit_tests/helpers/`; their scenarios in
`e2e/tests/negative/`. Consistent with the AZ-406 layout and the
directory-layout invariant test (which now lists the three new
helpers + three new scenarios).
* **No `src/gps_denied_onboard` imports** anywhere in the new code.
Verified by inspection — the evaluators only consume typed
dataclasses populated by the scenario from public-boundary
sources (FDR, mavproxy tlog, SITL state, injector manifests).
* **Scenario gating**: each new scenario file uses
`pytest.skip(...)` with an explicit message pointing to the unit
test that covers the gated AC logic. This is the established
pattern from FT-P-07/08/09/10/11 and FT-N-02 — scenario coverage
comes online once the AZ-441 / AZ-407 / AZ-416 leftovers ship.
## Test Results
* New unit tests: 14 (outlier) + 18 (outage) + 29 (blackout-spoof) = **61 new tests**
* Plus 6 new entries in the parametrized `test_required_path_exists`
(3 evaluator paths + 3 scenario paths) — counted toward the suite
total.
* Full `e2e/_unit_tests` suite: **527 passed in 130 s** (previous
cumulative: 460 → +67 net).
* Scenario collection for the three negative tests: 48 items collect
cleanly (parametrized across `fc_adapter × vio_strategy × {density |
window_seconds}`). The session-end `/e2e-results/evidence/per-nfr`
teardown error is the same pre-existing wart documented in batches
69-72 (nfr_recorder hardcoded path; not introduced by this batch).
+3 -1
View File
@@ -12,8 +12,10 @@ sub_step:
retry_count: 0
cycle: 1
tracker: jira
last_completed_batch: 72
last_completed_batch: 73
last_cumulative_review: batches_70-72
current_batch: 74
current_batch_tasks: ""
last_step_outcomes:
step_8: "Code is testable — no changes needed (testability_assessment.md committed; no list-of-changes, no source edits)"
step_9: "Already complete — 41 blackbox test tasks (AZ-406..AZ-446) under epic AZ-262 with specs in _docs/02_tasks/todo/ were produced in a prior cycle; AZ-406 test-infrastructure bootstrap also pre-existing. Folder fallback satisfied (todo/ has test tasks, _dependencies_table.md reflects 114 product + 41 test = 155 total). No Step-9 work executed in cycle 1."
@@ -0,0 +1,588 @@
"""Unit tests for `e2e/runner/helpers/blackout_spoof_evaluator.py` (AZ-426)."""
from __future__ import annotations
import csv
from pathlib import Path
from e2e.runner.helpers.blackout_spoof_evaluator import (
DEAD_RECKONED_LABEL,
ESCALATION_COV_2D_M,
ESCALATION_COV_FAILSAFE_M,
ESCALATION_DURATION_FAILSAFE_S,
ESCALATION_FIX_TYPE_2D,
ESCALATION_LATENCY_MS,
HONEST_ACCURACY_RATIO,
HORIZ_ACCURACY_FAILSAFE,
RECOVERY_STABLE_S,
SATELLITE_ANCHORED_LABEL,
STATUSTEXT_FAILSAFE,
STATUSTEXT_IMU_ONLY,
STATUSTEXT_RATE_MAX_HZ,
STATUSTEXT_RATE_MIN_HZ,
SWITCH_LATENCY_MS,
BlackoutWindow,
ConsistencyCheckEvent,
GpsHealthSample,
OutboundEstimateSample,
SpoofRejectedEvent,
StatustextSample,
evaluate,
evaluate_covariance_monotonic,
evaluate_escalation,
evaluate_honest_accuracy,
evaluate_recovery_gate,
evaluate_spoof_rejection,
evaluate_statustext_rate,
evaluate_switch_latency,
write_csv_evidence,
)
# Constants
def test_constants_match_spec():
# AZ-426: AC-1 ≤400 ms, AC-4 ≥0.95×cov, AC-5 1-2 Hz, AC-6/7/8 thresholds.
assert SWITCH_LATENCY_MS == 400
assert HONEST_ACCURACY_RATIO == 0.95
assert STATUSTEXT_RATE_MIN_HZ == 1.0 and STATUSTEXT_RATE_MAX_HZ == 2.0
assert ESCALATION_COV_2D_M == 100.0
assert ESCALATION_COV_FAILSAFE_M == 500.0
assert ESCALATION_DURATION_FAILSAFE_S == 30.0
assert ESCALATION_FIX_TYPE_2D == 2
assert HORIZ_ACCURACY_FAILSAFE == 999.0
assert ESCALATION_LATENCY_MS == 500
assert RECOVERY_STABLE_S == 10.0
assert STATUSTEXT_IMU_ONLY == "VISUAL_BLACKOUT_IMU_ONLY"
assert STATUSTEXT_FAILSAFE == "VISUAL_BLACKOUT_FAILSAFE"
assert DEAD_RECKONED_LABEL == "dead_reckoned"
assert SATELLITE_ANCHORED_LABEL == "satellite_anchored"
def _window(onset_ms: int = 10_000, duration_s: float = 5.0) -> BlackoutWindow:
return BlackoutWindow(
onset_monotonic_ms=onset_ms,
end_monotonic_ms=onset_ms + int(duration_s * 1000),
)
def _est(
ms: int,
*,
label: str = DEAD_RECKONED_LABEL,
cov: float = 5.0,
horiz: float | None = None,
fix_type: int = 3,
) -> OutboundEstimateSample:
return OutboundEstimateSample(
monotonic_ms=ms,
source_label=label,
cov_semi_major_m=cov,
horiz_accuracy=cov if horiz is None else horiz,
fix_type=fix_type,
)
# AC-1 switch latency
def test_switch_latency_within_400_ms_passes():
# Arrange
w = _window()
estimates = [
_est(w.onset_monotonic_ms - 100, label=SATELLITE_ANCHORED_LABEL),
_est(w.onset_monotonic_ms + 350),
]
# Act
report = evaluate_switch_latency(w, estimates, frame_period_ms=33)
# Assert — budget is min(400, 33) = 33 ms; 350 > 33 → fails.
assert report.first_dead_reckoned_offset_ms == 350
assert report.passes is False
def test_switch_latency_within_one_frame_passes():
# Arrange — frame period 100 ms, dead_reckoned at +50 ms → within both bounds.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 50)]
# Act
report = evaluate_switch_latency(w, estimates, frame_period_ms=100)
# Assert
assert report.passes is True
def test_switch_latency_at_one_frame_boundary_passes():
# Arrange — exact frame-period boundary.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 100)]
# Act
report = evaluate_switch_latency(w, estimates, frame_period_ms=100)
# Assert
assert report.passes is True
def test_switch_latency_missing_dead_reckoned_fails():
# Arrange — no dead_reckoned emission.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 50, label=SATELLITE_ANCHORED_LABEL)]
# Act
report = evaluate_switch_latency(w, estimates, frame_period_ms=100)
# Assert
assert report.first_dead_reckoned_offset_ms is None
assert report.passes is False
# AC-2 spoof rejection
def test_spoof_rejection_pass():
# Arrange — spoof events present, no satellite_anchored inside window.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 500)]
spoof_events = [SpoofRejectedEvent(monotonic_ms=w.onset_monotonic_ms + 200, reason="delta>500m")]
# Act
report = evaluate_spoof_rejection(w, estimates, spoof_events)
# Assert
assert report.passes is True
def test_spoof_rejection_no_events_fails():
# Arrange
w = _window()
estimates = [_est(w.onset_monotonic_ms + 500)]
# Act
report = evaluate_spoof_rejection(w, estimates, spoof_events=[])
# Assert
assert report.passes is False
def test_spoof_rejection_label_returns_to_satellite_fails():
# Arrange — spoof event present BUT label returns to satellite_anchored inside window.
w = _window()
estimates = [
_est(w.onset_monotonic_ms + 100),
_est(w.onset_monotonic_ms + 1000, label=SATELLITE_ANCHORED_LABEL),
]
spoof_events = [SpoofRejectedEvent(monotonic_ms=w.onset_monotonic_ms + 50, reason="x")]
# Act
report = evaluate_spoof_rejection(w, estimates, spoof_events)
# Assert
assert report.satellite_anchored_inside_window == 1
assert report.passes is False
# AC-3 covariance monotonic
def test_covariance_monotonic_pass():
# Arrange
w = _window()
estimates = [
_est(w.onset_monotonic_ms + 100, cov=5.0),
_est(w.onset_monotonic_ms + 200, cov=5.5),
_est(w.onset_monotonic_ms + 300, cov=6.0),
]
# Act
report = evaluate_covariance_monotonic(w, estimates)
# Assert
assert report.passes is True
assert report.first_decreasing_at_ms is None
def test_covariance_monotonic_decreasing_fails():
# Arrange
w = _window()
estimates = [
_est(w.onset_monotonic_ms + 100, cov=5.0),
_est(w.onset_monotonic_ms + 200, cov=4.0),
]
# Act
report = evaluate_covariance_monotonic(w, estimates)
# Assert
assert report.first_decreasing_at_ms == w.onset_monotonic_ms + 200
assert report.passes is False
# AC-4 honest accuracy
def test_honest_accuracy_pass():
# Arrange — horiz_accuracy = cov ≥ 0.95 × cov.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 100, cov=10.0, horiz=10.0)]
# Act
report = evaluate_honest_accuracy(w, estimates)
# Assert
assert report.passes is True
def test_honest_accuracy_boundary_pass():
# Arrange — horiz_accuracy = 0.95 × cov exactly.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 100, cov=10.0, horiz=9.5)]
# Act
report = evaluate_honest_accuracy(w, estimates)
# Assert
assert report.passes is True
def test_honest_accuracy_violation_fails():
# Arrange — horiz_accuracy = 0.90 × cov.
w = _window()
estimates = [_est(w.onset_monotonic_ms + 100, cov=10.0, horiz=9.0)]
# Act
report = evaluate_honest_accuracy(w, estimates)
# Assert
assert report.violation_count == 1
assert report.passes is False
# AC-5 STATUSTEXT rate
def test_statustext_rate_pass_at_1hz():
# Arrange — 5 STATUSTEXTs over 5 s = 1 Hz.
w = _window(duration_s=5.0)
sts = [
StatustextSample(monotonic_ms=w.onset_monotonic_ms + i * 1000, text=STATUSTEXT_IMU_ONLY)
for i in range(5)
]
# Act
report = evaluate_statustext_rate(w, sts)
# Assert
assert report.observed_hz is not None and abs(report.observed_hz - 1.0) < 1e-6
assert report.passes is True
def test_statustext_rate_pass_at_2hz():
# Arrange — 10 STATUSTEXTs over 5 s = 2 Hz.
w = _window(duration_s=5.0)
sts = [
StatustextSample(monotonic_ms=w.onset_monotonic_ms + i * 500, text=STATUSTEXT_IMU_ONLY)
for i in range(10)
]
# Act
report = evaluate_statustext_rate(w, sts)
# Assert
assert report.passes is True
def test_statustext_rate_too_slow_fails():
# Arrange — 2 STATUSTEXTs over 5 s = 0.4 Hz.
w = _window(duration_s=5.0)
sts = [
StatustextSample(monotonic_ms=w.onset_monotonic_ms + i * 2000, text=STATUSTEXT_IMU_ONLY)
for i in range(2)
]
# Act
report = evaluate_statustext_rate(w, sts)
# Assert
assert report.passes is False
def test_statustext_rate_too_fast_fails():
# Arrange — 30 STATUSTEXTs over 5 s = 6 Hz.
w = _window(duration_s=5.0)
sts = [
StatustextSample(monotonic_ms=w.onset_monotonic_ms + int(i * 5000 / 30), text=STATUSTEXT_IMU_ONLY)
for i in range(30)
]
# Act
report = evaluate_statustext_rate(w, sts)
# Assert
assert report.observed_hz is not None and report.observed_hz > STATUSTEXT_RATE_MAX_HZ
assert report.passes is False
# AC-6 / AC-7 escalation (35 s window)
def _make_35s_window(onset_ms: int = 10_000) -> BlackoutWindow:
return _window(onset_ms=onset_ms, duration_s=35.0)
def test_escalation_non_35s_window_passes_vacuously():
# Arrange — 5 s window with no escalation expected.
w = _window(duration_s=5.0)
estimates = [_est(w.onset_monotonic_ms + 100, cov=10.0)]
# Act
report = evaluate_escalation(w, estimates, statustexts=[], is_35s_window=False)
# Assert
assert report.passes is True
def test_escalation_35s_no_crossings_passes():
# Arrange — covariance stays below ESCALATION_COV_2D_M for whole window.
w = _make_35s_window()
estimates = [
_est(w.onset_monotonic_ms + i * 1000, cov=10.0 + i * 0.1)
for i in range(30)
]
# Act
report = evaluate_escalation(w, estimates, statustexts=[], is_35s_window=True)
# Assert — duration crossing at 30 s alone still triggers AC-7 path; no
# failsafe STATUSTEXT → fails AC-7.
assert report.cov500_or_30s_crossed is True
assert report.passes_ac7 is False
def test_escalation_35s_ac6_fix_type_degraded_passes():
# Arrange — cov crosses 100 m at 5 s; fix_type drops to 2 from then on.
w = _make_35s_window()
estimates = []
for i in range(30):
t = w.onset_monotonic_ms + i * 1000
cov = 50.0 if i < 5 else 150.0
fix = 3 if i < 5 else 2
estimates.append(_est(t, cov=cov, horiz=cov, fix_type=fix))
# Provide failsafe STATUSTEXT at +30 s (within ESCALATION_LATENCY_MS of duration breach).
failsafe_at = w.onset_monotonic_ms + int(ESCALATION_DURATION_FAILSAFE_S * 1000)
statustexts = [
StatustextSample(monotonic_ms=failsafe_at + 100, text=STATUSTEXT_FAILSAFE)
]
# All post-failsafe-trigger samples need horiz_accuracy == 999.
for i in range(30):
if estimates[i].monotonic_ms >= failsafe_at:
estimates[i] = OutboundEstimateSample(
monotonic_ms=estimates[i].monotonic_ms,
source_label=DEAD_RECKONED_LABEL,
cov_semi_major_m=estimates[i].cov_semi_major_m,
horiz_accuracy=HORIZ_ACCURACY_FAILSAFE,
fix_type=2,
)
# Act
report = evaluate_escalation(w, estimates, statustexts, is_35s_window=True)
# Assert
assert report.passes_ac6 is True
assert report.passes_ac7 is True
def test_escalation_35s_ac6_fix_type_not_degraded_fails():
# Arrange — cov crosses 100 m but fix_type stays 3.
w = _make_35s_window()
estimates = []
for i in range(30):
cov = 50.0 if i < 5 else 150.0
estimates.append(_est(w.onset_monotonic_ms + i * 1000, cov=cov, fix_type=3))
# Act
report = evaluate_escalation(w, estimates, statustexts=[], is_35s_window=True)
# Assert
assert report.passes_ac6 is False
def test_escalation_35s_ac7_horiz_not_999_fails():
# Arrange — failsafe trigger reached but horiz_accuracy != 999.
w = _make_35s_window()
estimates = []
for i in range(35):
cov = 50.0 + i * 20.0 # crosses 100 then 500.
estimates.append(_est(w.onset_monotonic_ms + i * 1000, cov=cov, horiz=cov, fix_type=2))
failsafe_at = w.onset_monotonic_ms + int(ESCALATION_DURATION_FAILSAFE_S * 1000)
statustexts = [
StatustextSample(monotonic_ms=failsafe_at + 100, text=STATUSTEXT_FAILSAFE)
]
# Act
report = evaluate_escalation(w, estimates, statustexts, is_35s_window=True)
# Assert
assert report.horiz_accuracy_999 is False
assert report.passes_ac7 is False
# AC-8 recovery gate
def _post_window(w: BlackoutWindow) -> tuple[int, int]:
return w.end_monotonic_ms, w.end_monotonic_ms + int(RECOVERY_STABLE_S * 1000) + 500
def test_recovery_gate_pass():
# Arrange — 12 s of healthy GPS + consistency pass + then satellite_anchored emission.
w = _window()
end, recovery = _post_window(w)
estimates = [
_est(end + 500),
_est(recovery + 100, label=SATELLITE_ANCHORED_LABEL),
]
gps_health = [
GpsHealthSample(monotonic_ms=end + i * 1000, healthy=True, spoofed=False)
for i in range(12)
]
consistency = [ConsistencyCheckEvent(monotonic_ms=end + 5000, passed=True)]
# Act
report = evaluate_recovery_gate(w, estimates, gps_health, consistency)
# Assert
assert report.passes is True
assert report.recovery_at_ms == recovery + 100
def test_recovery_gate_unstable_fails():
# Arrange — GPS goes unhealthy mid-stability window.
w = _window()
end, recovery = _post_window(w)
estimates = [_est(recovery + 100, label=SATELLITE_ANCHORED_LABEL)]
gps_health = [
GpsHealthSample(monotonic_ms=end + i * 1000, healthy=(i != 5), spoofed=False)
for i in range(12)
]
consistency = [ConsistencyCheckEvent(monotonic_ms=end + 5000, passed=True)]
# Act
report = evaluate_recovery_gate(w, estimates, gps_health, consistency)
# Assert
assert report.passes is False
def test_recovery_gate_spoofed_fails():
# Arrange — GPS healthy but spoofed=True for one sample.
w = _window()
end, recovery = _post_window(w)
estimates = [_est(recovery + 100, label=SATELLITE_ANCHORED_LABEL)]
gps_health = [
GpsHealthSample(monotonic_ms=end + i * 1000, healthy=True, spoofed=(i == 3))
for i in range(12)
]
consistency = [ConsistencyCheckEvent(monotonic_ms=end + 5000, passed=True)]
# Act
report = evaluate_recovery_gate(w, estimates, gps_health, consistency)
# Assert
assert report.passes is False
def test_recovery_gate_no_consistency_check_fails():
# Arrange
w = _window()
end, recovery = _post_window(w)
estimates = [_est(recovery + 100, label=SATELLITE_ANCHORED_LABEL)]
gps_health = [
GpsHealthSample(monotonic_ms=end + i * 1000, healthy=True, spoofed=False)
for i in range(12)
]
# Act
report = evaluate_recovery_gate(w, estimates, gps_health, consistency_checks=[])
# Assert
assert report.consistency_check_passed is False
assert report.passes is False
def test_recovery_gate_no_recovery_attempt_vacuous_pass():
# Arrange — no satellite_anchored post-window.
w = _window()
estimates = [_est(w.end_monotonic_ms + 500)]
# Act
report = evaluate_recovery_gate(w, estimates, gps_health=[], consistency_checks=[])
# Assert
assert report.recovery_at_ms is None
assert report.passes is True
# Aggregate evaluate + CSV evidence
def _make_passing_5s_inputs() -> dict:
w = _window(duration_s=5.0)
end, recovery = _post_window(w)
estimates = [_est(w.onset_monotonic_ms + 50)]
estimates.extend(
_est(w.onset_monotonic_ms + 100 + i * 100, cov=10.0 + i * 0.1)
for i in range(20)
)
estimates.append(_est(recovery + 100, label=SATELLITE_ANCHORED_LABEL))
statustexts = [
StatustextSample(monotonic_ms=w.onset_monotonic_ms + i * 1000, text=STATUSTEXT_IMU_ONLY)
for i in range(5)
]
spoof_events = [SpoofRejectedEvent(monotonic_ms=w.onset_monotonic_ms + 50, reason="x")]
gps_health = [
GpsHealthSample(monotonic_ms=end + i * 1000, healthy=True, spoofed=False)
for i in range(12)
]
consistency = [ConsistencyCheckEvent(monotonic_ms=end + 5000, passed=True)]
return dict(
window=w,
estimates=estimates,
statustexts=statustexts,
spoof_events=spoof_events,
gps_health=gps_health,
consistency_checks=consistency,
frame_period_ms=100,
is_35s_window=False,
)
def test_evaluate_5s_all_pass():
# Arrange
inputs = _make_passing_5s_inputs()
# Act
report = evaluate(**inputs)
# Assert
assert report.passes is True
def test_write_csv_evidence_round_trips(tmp_path: Path):
# Arrange
inputs = _make_passing_5s_inputs()
report = evaluate(**inputs)
# Act
out = write_csv_evidence(tmp_path / "ft-n-04.csv", report)
# Assert
with out.open() as fh:
rows = list(csv.DictReader(fh))
assert len(rows) == 1
assert rows[0]["passes"] == "true"
assert rows[0]["ac1_passes"] == "true"
assert rows[0]["ac2_passes"] == "true"
@@ -0,0 +1,353 @@
"""Unit tests for `e2e/runner/helpers/outage_request_evaluator.py` (AZ-425)."""
from __future__ import annotations
import csv
from pathlib import Path
from e2e.runner.helpers.outage_request_evaluator import (
DEAD_RECKONED_LABEL,
MIN_OUTAGE_FRAMES,
OUTAGE_THRESHOLD_S,
STATUSTEXT_REGEX,
TOLERANCE_S,
EkfDivergenceEvent,
OutboundEstimateSample,
StatustextSample,
detect_outage_windows,
evaluate,
evaluate_window,
write_csv_evidence,
)
# Constants
def test_constants_match_spec():
# AZ-425: AC-1 ≥3 frames; AC-2 2 s ±500 ms; AC-3 dead_reckoned label.
assert MIN_OUTAGE_FRAMES == 3
assert OUTAGE_THRESHOLD_S == 2.0
assert TOLERANCE_S == 0.5
assert STATUSTEXT_REGEX == "OPERATOR_RELOC_REQUEST"
assert DEAD_RECKONED_LABEL == "dead_reckoned"
# detect_outage_windows
def _est(frame: int, label: str = "satellite_anchored", ms: int = 0) -> OutboundEstimateSample:
return OutboundEstimateSample(
frame_idx=frame,
monotonic_ms=ms if ms else frame * 100,
source_label=label,
)
def test_detect_no_outage_returns_empty():
# Arrange — full frame sequence with all estimates.
expected = list(range(10))
estimates = [_est(i) for i in expected]
# Act
windows = detect_outage_windows(expected, estimates, frame_period_ms=100)
# Assert
assert windows == []
def test_detect_run_below_min_length_ignored():
# Arrange — 2-frame gap is below MIN_OUTAGE_FRAMES=3.
expected = list(range(10))
estimates = [_est(i) for i in expected if i not in (4, 5)]
# Act
windows = detect_outage_windows(expected, estimates, frame_period_ms=100)
# Assert
assert windows == []
def test_detect_single_outage_window():
# Arrange — 3-frame gap at indices 4,5,6.
expected = list(range(10))
estimates = [_est(i) for i in expected if i not in (4, 5, 6)]
# Act
windows = detect_outage_windows(
expected, estimates, frame_period_ms=100, replay_start_monotonic_ms=1000
)
# Assert
assert len(windows) == 1
w = windows[0]
assert w.first_missing_frame_idx == 4
assert w.last_missing_frame_idx == 6
assert w.length_frames == 3
assert w.onset_monotonic_ms == 1000 + 4 * 100 # 1400
assert w.end_monotonic_ms == 1000 + 7 * 100 # 1700
assert w.duration_ms == 300
def test_detect_multiple_windows():
# Arrange — two gaps: 4-6 and 12-15.
expected = list(range(20))
skip = {4, 5, 6, 12, 13, 14, 15}
estimates = [_est(i) for i in expected if i not in skip]
# Act
windows = detect_outage_windows(expected, estimates, frame_period_ms=100)
# Assert
assert len(windows) == 2
assert windows[0].first_missing_frame_idx == 4 and windows[0].length_frames == 3
assert windows[1].first_missing_frame_idx == 12 and windows[1].length_frames == 4
def test_detect_trailing_outage_window():
# Arrange — gap at the end of the sequence.
expected = list(range(10))
estimates = [_est(i) for i in expected if i < 7]
# Act
windows = detect_outage_windows(expected, estimates, frame_period_ms=100)
# Assert
assert len(windows) == 1
assert windows[0].first_missing_frame_idx == 7
assert windows[0].last_missing_frame_idx == 9
# evaluate_window — AC-2 STATUSTEXT timing
def _window_at(onset_ms: int, length: int = 3, period_ms: int = 100):
# Ensure expected sequence is long enough to fully contain the gap + a trailing frame.
total = max(20, length + 5)
expected = list(range(total))
skip = set(range(2, 2 + length))
estimates = [_est(i, ms=i * period_ms) for i in expected if i not in skip]
[w] = detect_outage_windows(
expected,
estimates,
frame_period_ms=period_ms,
replay_start_monotonic_ms=onset_ms - 2 * period_ms,
)
return w, estimates
def test_statustext_within_tolerance_passes():
# Arrange — STATUSTEXT exactly at onset+2 s.
window, estimates = _window_at(onset_ms=10_000, length=30, period_ms=100)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 2000, text="OPERATOR_RELOC_REQUEST"),
]
# Act
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Assert
assert report.statustext_offset_ms == 2000
assert report.passes_statustext is True
def test_statustext_within_tolerance_late_passes():
# Arrange — STATUSTEXT at onset+2.4 s (within ±500 ms).
window, estimates = _window_at(onset_ms=10_000, length=30)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 2400, text="OPERATOR_RELOC_REQUEST"),
]
# Act
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Assert
assert report.passes_statustext is True
def test_statustext_too_early_fails():
# Arrange — STATUSTEXT at onset+1.0 s (before 1.5 s lower bound).
window, estimates = _window_at(onset_ms=10_000, length=30)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 1000, text="OPERATOR_RELOC_REQUEST"),
]
# Act
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Assert
assert report.statustext_offset_ms == 1000
assert report.passes_statustext is False
def test_statustext_too_late_fails():
# Arrange — STATUSTEXT at onset+3.0 s (beyond 2.5 s upper bound).
window, estimates = _window_at(onset_ms=10_000, length=30)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 3000, text="OPERATOR_RELOC_REQUEST"),
]
# Act
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Assert
assert report.passes_statustext is False
def test_statustext_missing_fails():
# Arrange
window, estimates = _window_at(onset_ms=10_000, length=30)
# Act
report = evaluate_window(window, estimates, statustexts=[], ekf_events=[])
# Assert
assert report.statustext_offset_ms is None
assert report.passes_statustext is False
def test_statustext_payload_mismatch_fails():
# Arrange — different STATUSTEXT message at the right time.
window, estimates = _window_at(onset_ms=10_000, length=30)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 2000, text="EKF_VARIANCE"),
]
# Act
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Assert
assert report.passes_statustext is False
# AC-3 dead_reckoned during outage
def test_dead_reckoned_during_window_passes():
# Arrange — outage 4-6 with dead_reckoned estimate at ms 500 (frame 5 in window).
expected = list(range(20))
skip = {4, 5, 6}
estimates = [
_est(i, ms=i * 100)
for i in expected
if i not in skip
]
# Add dead_reckoned filler emission during the outage window.
estimates.append(
OutboundEstimateSample(frame_idx=4, monotonic_ms=500, source_label=DEAD_RECKONED_LABEL)
)
[w] = detect_outage_windows(expected, [e for e in estimates if e.frame_idx not in {4, 5, 6} or e.source_label == "satellite_anchored"], frame_period_ms=100)
# Note: detection ignores dead_reckoned filler so window still spans 4-6.
# Act
report = evaluate_window(w, estimates, statustexts=[], ekf_events=[])
# Assert — at least one dead_reckoned emission with monotonic_ms in [onset_ms, end_ms].
assert report.dead_reckoned_count >= 1
assert report.passes_dead_reckoned is True
def test_dead_reckoned_absent_fails():
# Arrange
window, estimates = _window_at(onset_ms=10_000, length=3, period_ms=100)
# Act
report = evaluate_window(window, estimates, statustexts=[], ekf_events=[])
# Assert
assert report.dead_reckoned_count == 0
assert report.passes_dead_reckoned is False
# AC-4 EKF divergence
def test_ekf_divergence_during_window_fails():
# Arrange
window, estimates = _window_at(onset_ms=10_000, length=30)
events = [
EkfDivergenceEvent(
monotonic_ms=window.onset_monotonic_ms + 1000, reason="velocity_innov"
)
]
# Act
report = evaluate_window(window, estimates, statustexts=[], ekf_events=events)
# Assert
assert report.ekf_divergence_count == 1
assert report.passes_ekf is False
def test_ekf_divergence_outside_window_ignored():
# Arrange
window, estimates = _window_at(onset_ms=10_000, length=30)
events = [
EkfDivergenceEvent(
monotonic_ms=window.end_monotonic_ms + 1000, reason="velocity_innov"
)
]
# Act
report = evaluate_window(window, estimates, statustexts=[], ekf_events=events)
# Assert
assert report.passes_ekf is True
# evaluate aggregate
def test_evaluate_all_pass():
# Arrange — single outage with everything in order.
expected = list(range(40))
skip = set(range(10, 40))
period_ms = 100
estimates = [
_est(i, ms=i * period_ms)
for i in expected
if i not in skip
]
estimates.append(
OutboundEstimateSample(
frame_idx=10, monotonic_ms=10 * period_ms + 500, source_label=DEAD_RECKONED_LABEL
)
)
statustexts = [
StatustextSample(monotonic_ms=10 * period_ms + 2000, text="OPERATOR_RELOC_REQUEST")
]
# Act
reports = evaluate(
expected,
estimates,
statustexts,
ekf_events=[],
frame_period_ms=period_ms,
)
# Assert
assert len(reports) == 1
assert reports[0].passes is True
# CSV evidence
def test_write_csv_evidence_round_trips(tmp_path: Path):
# Arrange
window, estimates = _window_at(onset_ms=10_000, length=30)
statustexts = [
StatustextSample(monotonic_ms=window.onset_monotonic_ms + 2000, text="OPERATOR_RELOC_REQUEST")
]
report = evaluate_window(window, estimates, statustexts, ekf_events=[])
# Act
out = write_csv_evidence(tmp_path / "ft-n-03.csv", [report])
# Assert
with out.open() as fh:
rows = list(csv.DictReader(fh))
assert len(rows) == 1
assert rows[0]["passes_statustext"] == "true"
assert int(rows[0]["length_frames"]) == 30
@@ -0,0 +1,330 @@
"""Unit tests for `e2e/runner/helpers/outlier_tolerance_evaluator.py` (AZ-424)."""
from __future__ import annotations
import csv
from pathlib import Path
import pytest
from e2e.runner.helpers.outlier_tolerance_evaluator import (
COVARIANCE_WINDOW_FRAMES,
DRIFT_BUDGET_M,
MIN_OUTLIER_COUNT,
GtPose,
OutboundEstimate,
OutlierEvent,
evaluate,
evaluate_event,
load_outlier_manifest,
write_csv_evidence,
)
# Constants
def test_constants_match_spec():
# AC-2 budget + AC-3 window + AC-1 minimum count, per AZ-424.
assert DRIFT_BUDGET_M == 50.0
assert COVARIANCE_WINDOW_FRAMES == 3
assert MIN_OUTLIER_COUNT == 10
# Manifest loading
def _write_manifest(path: Path, rows: list[dict]) -> None:
fieldnames = [
"frame_idx",
"src_jpeg_path",
"replacement_tile_x",
"replacement_tile_y",
"geodesic_offset_m",
"seed",
]
with path.open("w", newline="") as fh:
writer = csv.DictWriter(fh, fieldnames=fieldnames)
writer.writeheader()
for r in rows:
row = {k: "" for k in fieldnames}
row.update(r)
writer.writerow(row)
def test_load_outlier_manifest_missing_file_raises(tmp_path: Path):
# Assert
with pytest.raises(FileNotFoundError, match="outlier manifest not found"):
load_outlier_manifest(tmp_path / "nope.csv")
def test_load_outlier_manifest_missing_columns_raises(tmp_path: Path):
# Arrange
p = tmp_path / "manifest.csv"
with p.open("w", newline="") as fh:
writer = csv.DictWriter(fh, fieldnames=["frame_idx", "src_jpeg_path"])
writer.writeheader()
writer.writerow({"frame_idx": "1", "src_jpeg_path": "x.jpg"})
# Assert
with pytest.raises(ValueError, match="missing required columns"):
load_outlier_manifest(p)
def test_load_outlier_manifest_returns_events(tmp_path: Path):
# Arrange
p = tmp_path / "manifest.csv"
_write_manifest(
p,
[
{"frame_idx": "10", "src_jpeg_path": "AD000011.jpg", "geodesic_offset_m": "412.5"},
{"frame_idx": "20", "src_jpeg_path": "AD000021.jpg", "geodesic_offset_m": "381.0"},
],
)
# Act
events = load_outlier_manifest(p)
# Assert
assert len(events) == 2
assert events[0] == OutlierEvent(
frame_idx=10, geodesic_offset_m=412.5, src_jpeg_path="AD000011.jpg"
)
assert events[1].frame_idx == 20
# evaluate_event — AC-2 drift bound
def _est(frame: int, lat: float, lon: float, cov: float = 5.0) -> OutboundEstimate:
return OutboundEstimate(
frame_idx=frame,
monotonic_ms=frame * 100,
lat_deg=lat,
lon_deg=lon,
cov_semi_major_m=cov,
source_label="C3_VIO",
)
def _gt(frame: int, lat: float, lon: float) -> GtPose:
return GtPose(frame_idx=frame, lat_deg=lat, lon_deg=lon)
def test_evaluate_event_drift_within_budget():
# Arrange — estimate before/after match GT exactly; outlier frame drifts.
estimates = {
9: _est(9, 50.0000, 30.0000, cov=4.0),
10: _est(10, 50.0050, 30.0050, cov=5.0), # outlier
11: _est(11, 50.0001, 30.0001, cov=5.0),
}
gt = {
9: _gt(9, 50.0000, 30.0000),
10: _gt(10, 50.0001, 30.0001),
11: _gt(11, 50.0002, 30.0002),
}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=412.5, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.frame_idx == 10
assert report.drift_m is not None
assert report.drift_m <= DRIFT_BUDGET_M
assert report.passes_drift is True
def test_evaluate_event_drift_exceeds_budget_fails():
# Arrange — after-frame error is >> before-frame error.
estimates = {
9: _est(9, 50.0000, 30.0000),
10: _est(10, 50.0050, 30.0050),
11: _est(11, 50.0010, 30.0010), # ~129 m off
}
gt = {
9: _gt(9, 50.0000, 30.0000),
10: _gt(10, 50.0001, 30.0001),
11: _gt(11, 50.0000, 30.0000),
}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=400.0, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.drift_m is not None and report.drift_m > DRIFT_BUDGET_M
assert report.passes_drift is False
assert report.passes is False
def test_evaluate_event_missing_neighbour_drift_none():
# Arrange — only outlier frame present.
estimates = {10: _est(10, 50.0050, 30.0050)}
gt = {10: _gt(10, 50.0001, 30.0001)}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=400.0, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.drift_m is None
assert report.passes_drift is False
# evaluate_event — AC-3 covariance monotonic
def test_evaluate_event_cov_monotonic_passes():
# Arrange
estimates = {
9: _est(9, 50.0, 30.0, cov=4.0),
10: _est(10, 50.0, 30.0, cov=5.0),
11: _est(11, 50.0, 30.0, cov=5.5),
}
gt = {f: _gt(f, 50.0, 30.0) for f in (9, 10, 11)}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=400.0, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.cov_non_decreasing is True
assert report.passes_covariance is True
def test_evaluate_event_cov_decreasing_fails():
# Arrange — outlier frame cov is lower than before frame.
estimates = {
9: _est(9, 50.0, 30.0, cov=5.0),
10: _est(10, 50.0, 30.0, cov=4.0), # decrease — violates AC-3
11: _est(11, 50.0, 30.0, cov=5.0),
}
gt = {f: _gt(f, 50.0, 30.0) for f in (9, 10, 11)}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=400.0, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.cov_non_decreasing is False
assert report.passes_covariance is False
def test_evaluate_event_cov_flat_window_passes():
# Arrange — equal covariances satisfy non-decreasing.
estimates = {
9: _est(9, 50.0, 30.0, cov=5.0),
10: _est(10, 50.0, 30.0, cov=5.0),
11: _est(11, 50.0, 30.0, cov=5.0),
}
gt = {f: _gt(f, 50.0, 30.0) for f in (9, 10, 11)}
event = OutlierEvent(frame_idx=10, geodesic_offset_m=400.0, src_jpeg_path="x.jpg")
# Act
report = evaluate_event(event, estimates, gt)
# Assert
assert report.cov_non_decreasing is True
# Aggregate evaluate — AC-1 minimum count
def test_evaluate_count_below_minimum_fails():
# Arrange — only 5 outliers; AC-1 requires ≥10.
events = [
OutlierEvent(frame_idx=i * 10, geodesic_offset_m=400.0, src_jpeg_path=f"x{i}.jpg")
for i in range(1, 6)
]
estimates: list[OutboundEstimate] = []
gt: list[GtPose] = []
for ev in events:
for delta in (-1, 0, 1):
estimates.append(_est(ev.frame_idx + delta, 50.0, 30.0, cov=5.0))
gt.append(_gt(ev.frame_idx + delta, 50.0, 30.0))
# Act
report = evaluate(events, estimates, gt)
# Assert
assert report.total_outliers == 5
assert report.passes_count is False
assert report.passes is False
def test_evaluate_count_at_minimum_passes_count_gate():
# Arrange — exactly 10 outliers with non-violating drift/cov.
events = [
OutlierEvent(frame_idx=i * 10, geodesic_offset_m=400.0, src_jpeg_path=f"x{i}.jpg")
for i in range(1, 11)
]
estimates: list[OutboundEstimate] = []
gt: list[GtPose] = []
for ev in events:
for delta in (-1, 0, 1):
estimates.append(_est(ev.frame_idx + delta, 50.0, 30.0, cov=5.0))
gt.append(_gt(ev.frame_idx + delta, 50.0, 30.0))
# Act
report = evaluate(events, estimates, gt)
# Assert
assert report.total_outliers == 10
assert report.passes_count is True
assert report.failed_event_count == 0
assert report.passes is True
def test_evaluate_mixed_pass_fail_aggregates_correctly():
# Arrange — 10 events, one with drift violation.
events = [
OutlierEvent(frame_idx=i * 10, geodesic_offset_m=400.0, src_jpeg_path=f"x{i}.jpg")
for i in range(1, 11)
]
estimates: list[OutboundEstimate] = []
gt: list[GtPose] = []
for ev in events:
for delta in (-1, 0, 1):
estimates.append(_est(ev.frame_idx + delta, 50.0, 30.0, cov=5.0))
gt.append(_gt(ev.frame_idx + delta, 50.0, 30.0))
# Override frame 31 to be 200 m off — produces drift > 50 m for event at frame_idx=30.
estimates = [e for e in estimates if e.frame_idx != 31]
estimates.append(_est(31, 50.0018, 30.0, cov=5.0)) # ≈200 m off
# Act
report = evaluate(events, estimates, gt)
# Assert
assert report.total_outliers == 10
assert report.failed_event_count == 1
assert report.passes is False
# CSV evidence writer
def test_write_csv_evidence_round_trips(tmp_path: Path):
# Arrange
events = [
OutlierEvent(frame_idx=10, geodesic_offset_m=412.5, src_jpeg_path="AD000011.jpg"),
OutlierEvent(frame_idx=20, geodesic_offset_m=381.0, src_jpeg_path="AD000021.jpg"),
]
estimates: list[OutboundEstimate] = []
gt: list[GtPose] = []
for ev in events:
for delta in (-1, 0, 1):
estimates.append(_est(ev.frame_idx + delta, 50.0, 30.0, cov=5.0))
gt.append(_gt(ev.frame_idx + delta, 50.0, 30.0))
report = evaluate(events, estimates, gt)
# Act
out = write_csv_evidence(tmp_path / "ft_n_01_evidence.csv", report)
# Assert
assert out.exists()
with out.open() as fh:
rows = list(csv.DictReader(fh))
assert [int(r["frame_idx"]) for r in rows] == [10, 20]
assert all(r["passes"] == "true" for r in rows)
assert all(r["cov_non_decreasing"] == "true" for r in rows)
+6
View File
@@ -52,6 +52,9 @@ E2E_ROOT = Path(__file__).resolve().parents[1]
"runner/helpers/msp_frame_observer.py",
"runner/helpers/ap_contract_evaluator.py",
"runner/helpers/cold_start_evaluator.py",
"runner/helpers/outlier_tolerance_evaluator.py",
"runner/helpers/outage_request_evaluator.py",
"runner/helpers/blackout_spoof_evaluator.py",
"fixtures/mock-suite-sat/Dockerfile",
"fixtures/mock-suite-sat/app.py",
"fixtures/mock-suite-sat/requirements.txt",
@@ -96,7 +99,10 @@ E2E_ROOT = Path(__file__).resolve().parents[1]
"tests/positive/test_ft_p_09_inav.py",
"tests/positive/test_ft_p_10_smoothing_lookback.py",
"tests/positive/test_ft_p_11_cold_start_init.py",
"tests/negative/test_ft_n_01_outlier_tolerance.py",
"tests/negative/test_ft_n_02_sharp_turn_failure.py",
"tests/negative/test_ft_n_03_outage_reloc.py",
"tests/negative/test_ft_n_04_blackout_spoof.py",
],
)
def test_required_path_exists(relative_path: str) -> None:
@@ -0,0 +1,557 @@
"""Blackout-spoof evaluation for FT-N-04 (AZ-426 / AC-3.5 + AC-NEW-8).
Three-window ladder (5 s / 15 s / 35 s) with the
``blackout_spoof.py`` injector + FC-inbound spoof proxy. The
evaluator validates per AZ-426:
* AC-1: switch latency — within ≤1 frame OR ≤``SWITCH_LATENCY_MS``
(whichever is shorter), the first outbound estimate after blackout
onset carries ``source_label = dead_reckoned``.
* AC-2: spoof rejection — at least one FDR ``spoof-rejected`` event
is observed during the blackout window AND zero spoofed GPS records
are consumed into the estimator (label never returns to
``satellite_anchored`` during the window).
* AC-3: monotonic covariance — ``cov_semi_major_m`` is non-decreasing
across consecutive emissions inside the window.
* AC-4: honest horiz_accuracy —
``horiz_accuracy ≥ HONEST_ACCURACY_RATIO × cov_semi_major_m``
for every emission.
* AC-5: STATUSTEXT 1-2 Hz —
``VISUAL_BLACKOUT_IMU_ONLY`` STATUSTEXT rate is in
``[STATUSTEXT_RATE_MIN_HZ, STATUSTEXT_RATE_MAX_HZ]`` throughout the
window.
* AC-6 (35 s only): when 95 % covariance crosses
``ESCALATION_COV_2D_M``, fix_type degrades to ≤``ESCALATION_FIX_TYPE_2D``.
* AC-7 (35 s only): when 95 % covariance crosses
``ESCALATION_COV_FAILSAFE_M`` OR window duration exceeds
``ESCALATION_DURATION_FAILSAFE_S``, ``horiz_accuracy ==
HORIZ_ACCURACY_FAILSAFE`` AND ``VISUAL_BLACKOUT_FAILSAFE``
STATUSTEXT is emitted within ≤``ESCALATION_LATENCY_MS`` of the
crossing.
* AC-8: recovery gate — after blackout end, label only returns to
``satellite_anchored`` once both (a) FC GPS-health is stable +
non-spoofed for ≥``RECOVERY_STABLE_S`` AND (b) a
visual/satellite consistency check succeeds.
Public-boundary discipline: does NOT import any
``src/gps_denied_onboard`` symbol.
"""
from __future__ import annotations
import csv
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Sequence
# AC-1
SWITCH_LATENCY_MS = 400
# AC-2
DEAD_RECKONED_LABEL = "dead_reckoned"
SATELLITE_ANCHORED_LABEL = "satellite_anchored"
# AC-4
HONEST_ACCURACY_RATIO = 0.95
# AC-5
STATUSTEXT_IMU_ONLY = "VISUAL_BLACKOUT_IMU_ONLY"
STATUSTEXT_RATE_MIN_HZ = 1.0
STATUSTEXT_RATE_MAX_HZ = 2.0
# AC-6 / AC-7
STATUSTEXT_FAILSAFE = "VISUAL_BLACKOUT_FAILSAFE"
ESCALATION_COV_2D_M = 100.0
ESCALATION_COV_FAILSAFE_M = 500.0
ESCALATION_DURATION_FAILSAFE_S = 30.0
ESCALATION_FIX_TYPE_2D = 2 # MAVLink GPS_FIX_TYPE_2D
HORIZ_ACCURACY_FAILSAFE = 999.0
ESCALATION_LATENCY_MS = 500
# AC-8
RECOVERY_STABLE_S = 10.0
@dataclass(frozen=True)
class BlackoutWindow:
"""The injector-emitted window the evaluator is bound to."""
onset_monotonic_ms: int
end_monotonic_ms: int
@property
def duration_s(self) -> float:
return (self.end_monotonic_ms - self.onset_monotonic_ms) / 1000.0
@dataclass(frozen=True)
class OutboundEstimateSample:
"""One outbound estimate with fields used by FT-N-04 ACs."""
monotonic_ms: int
source_label: str
cov_semi_major_m: float
horiz_accuracy: float # AP GPS_INPUT.horiz_accuracy (m)
fix_type: int # MAVLink GPS fix type (0..6); -1 if unavailable
@dataclass(frozen=True)
class StatustextSample:
monotonic_ms: int
text: str
@dataclass(frozen=True)
class SpoofRejectedEvent:
"""One FDR `spoof-rejected` event."""
monotonic_ms: int
reason: str
@dataclass(frozen=True)
class GpsHealthSample:
"""FC-side GPS health sample (post-blackout, for recovery gate)."""
monotonic_ms: int
healthy: bool
spoofed: bool
@dataclass(frozen=True)
class ConsistencyCheckEvent:
"""Visual/satellite consistency check outcome (post-blackout)."""
monotonic_ms: int
passed: bool
@dataclass(frozen=True)
class SwitchLatencyReport:
"""AC-1 result."""
first_dead_reckoned_offset_ms: int | None # ms after window onset
frame_period_ms: int
passes: bool
@dataclass(frozen=True)
class SpoofRejectionReport:
"""AC-2 result."""
spoof_rejected_count: int
satellite_anchored_inside_window: int
passes: bool
@dataclass(frozen=True)
class CovarianceMonotonicReport:
"""AC-3 result."""
first_decreasing_at_ms: int | None
sample_count: int
passes: bool
@dataclass(frozen=True)
class HonestAccuracyReport:
"""AC-4 result."""
violation_count: int
sample_count: int
passes: bool
@dataclass(frozen=True)
class StatustextRateReport:
"""AC-5 result for VISUAL_BLACKOUT_IMU_ONLY."""
observed_hz: float | None
count: int
passes: bool
@dataclass(frozen=True)
class EscalationReport:
"""AC-6 + AC-7 result (35 s window only — other windows return passes=True)."""
cov2d_crossed: bool
cov2d_crossed_at_ms: int | None
fix_type_degraded: bool # AC-6 satisfied
cov500_or_30s_crossed: bool
cov500_or_30s_crossed_at_ms: int | None
horiz_accuracy_999: bool # AC-7 part 1
failsafe_statustext_offset_ms: int | None
failsafe_statustext_in_time: bool # AC-7 part 2
passes_ac6: bool
passes_ac7: bool
@property
def passes(self) -> bool:
return self.passes_ac6 and self.passes_ac7
@dataclass(frozen=True)
class RecoveryGateReport:
"""AC-8 result."""
recovery_at_ms: int | None
stable_period_s: float | None
consistency_check_passed: bool
passes: bool
@dataclass(frozen=True)
class BlackoutSpoofReport:
"""Aggregate FT-N-04 result for one window."""
window: BlackoutWindow
switch_latency: SwitchLatencyReport
spoof_rejection: SpoofRejectionReport
covariance_monotonic: CovarianceMonotonicReport
honest_accuracy: HonestAccuracyReport
statustext_rate: StatustextRateReport
escalation: EscalationReport
recovery_gate: RecoveryGateReport
@property
def passes(self) -> bool:
return all(
(
self.switch_latency.passes,
self.spoof_rejection.passes,
self.covariance_monotonic.passes,
self.honest_accuracy.passes,
self.statustext_rate.passes,
self.escalation.passes,
self.recovery_gate.passes,
)
)
def _inside_window(window: BlackoutWindow, t_ms: int) -> bool:
return window.onset_monotonic_ms <= t_ms <= window.end_monotonic_ms
def _samples_inside_window(
window: BlackoutWindow, samples: Iterable[OutboundEstimateSample]
) -> list[OutboundEstimateSample]:
return [s for s in samples if _inside_window(window, s.monotonic_ms)]
def evaluate_switch_latency(
window: BlackoutWindow,
estimates: Sequence[OutboundEstimateSample],
frame_period_ms: int,
) -> SwitchLatencyReport:
"""AC-1: dead_reckoned label within ≤1 frame OR ≤SWITCH_LATENCY_MS."""
budget_ms = min(SWITCH_LATENCY_MS, frame_period_ms)
offset: int | None = None
for s in estimates:
if s.monotonic_ms < window.onset_monotonic_ms:
continue
if s.source_label == DEAD_RECKONED_LABEL:
offset = s.monotonic_ms - window.onset_monotonic_ms
break
return SwitchLatencyReport(
first_dead_reckoned_offset_ms=offset,
frame_period_ms=frame_period_ms,
passes=offset is not None and offset <= budget_ms,
)
def evaluate_spoof_rejection(
window: BlackoutWindow,
estimates: Sequence[OutboundEstimateSample],
spoof_events: Sequence[SpoofRejectedEvent],
) -> SpoofRejectionReport:
"""AC-2: spoof-rejected events present AND no satellite_anchored re-entry."""
rejected = sum(
1 for ev in spoof_events if _inside_window(window, ev.monotonic_ms)
)
inside = _samples_inside_window(window, estimates)
re_anchored = sum(1 for s in inside if s.source_label == SATELLITE_ANCHORED_LABEL)
return SpoofRejectionReport(
spoof_rejected_count=rejected,
satellite_anchored_inside_window=re_anchored,
passes=rejected >= 1 and re_anchored == 0,
)
def evaluate_covariance_monotonic(
window: BlackoutWindow, estimates: Sequence[OutboundEstimateSample]
) -> CovarianceMonotonicReport:
"""AC-3: cov_semi_major_m non-decreasing across consecutive emissions."""
inside = _samples_inside_window(window, estimates)
first_dec: int | None = None
for i in range(1, len(inside)):
if inside[i].cov_semi_major_m < inside[i - 1].cov_semi_major_m:
first_dec = inside[i].monotonic_ms
break
return CovarianceMonotonicReport(
first_decreasing_at_ms=first_dec,
sample_count=len(inside),
passes=first_dec is None and len(inside) >= 1,
)
def evaluate_honest_accuracy(
window: BlackoutWindow, estimates: Sequence[OutboundEstimateSample]
) -> HonestAccuracyReport:
"""AC-4: horiz_accuracy ≥ HONEST_ACCURACY_RATIO × cov_semi_major_m."""
inside = _samples_inside_window(window, estimates)
violations = sum(
1
for s in inside
if s.horiz_accuracy < HONEST_ACCURACY_RATIO * s.cov_semi_major_m
)
return HonestAccuracyReport(
violation_count=violations,
sample_count=len(inside),
passes=violations == 0 and len(inside) >= 1,
)
def evaluate_statustext_rate(
window: BlackoutWindow, statustexts: Sequence[StatustextSample]
) -> StatustextRateReport:
"""AC-5: VISUAL_BLACKOUT_IMU_ONLY rate ∈ [1, 2] Hz."""
inside = [
st
for st in statustexts
if STATUSTEXT_IMU_ONLY in st.text and _inside_window(window, st.monotonic_ms)
]
duration_s = window.duration_s
if duration_s <= 0 or not inside:
return StatustextRateReport(observed_hz=None, count=len(inside), passes=False)
rate = len(inside) / duration_s
return StatustextRateReport(
observed_hz=rate,
count=len(inside),
passes=STATUSTEXT_RATE_MIN_HZ <= rate <= STATUSTEXT_RATE_MAX_HZ,
)
def _first_cov_crossing_ms(
window: BlackoutWindow,
estimates: Sequence[OutboundEstimateSample],
threshold_m: float,
) -> int | None:
for s in _samples_inside_window(window, estimates):
if s.cov_semi_major_m >= threshold_m:
return s.monotonic_ms
return None
def evaluate_escalation(
window: BlackoutWindow,
estimates: Sequence[OutboundEstimateSample],
statustexts: Sequence[StatustextSample],
*,
is_35s_window: bool,
) -> EscalationReport:
"""AC-6 + AC-7: applies only to the 35 s sub-case.
For non-35 s windows the report is vacuously passing — those windows
are not expected to cross either escalation threshold and any
incidental crossing is treated as informational only.
"""
cov2d_at = _first_cov_crossing_ms(window, estimates, ESCALATION_COV_2D_M)
cov500_at = _first_cov_crossing_ms(window, estimates, ESCALATION_COV_FAILSAFE_M)
duration_breach_at: int | None = None
if window.duration_s >= ESCALATION_DURATION_FAILSAFE_S:
duration_breach_at = (
window.onset_monotonic_ms
+ int(ESCALATION_DURATION_FAILSAFE_S * 1000)
)
failsafe_trigger_at: int | None = None
if cov500_at is not None and duration_breach_at is not None:
failsafe_trigger_at = min(cov500_at, duration_breach_at)
else:
failsafe_trigger_at = cov500_at if cov500_at is not None else duration_breach_at
if not is_35s_window:
return EscalationReport(
cov2d_crossed=cov2d_at is not None,
cov2d_crossed_at_ms=cov2d_at,
fix_type_degraded=True,
cov500_or_30s_crossed=failsafe_trigger_at is not None,
cov500_or_30s_crossed_at_ms=failsafe_trigger_at,
horiz_accuracy_999=True,
failsafe_statustext_offset_ms=None,
failsafe_statustext_in_time=True,
passes_ac6=True,
passes_ac7=True,
)
# AC-6: any sample at/after cov2d_at must have fix_type ≤ ESCALATION_FIX_TYPE_2D.
fix_degraded = True
if cov2d_at is not None:
post = [s for s in _samples_inside_window(window, estimates) if s.monotonic_ms >= cov2d_at]
if post and any(s.fix_type > ESCALATION_FIX_TYPE_2D for s in post):
fix_degraded = False
passes_ac6 = cov2d_at is None or fix_degraded
# AC-7: post-trigger samples must have horiz_accuracy == 999 AND
# VISUAL_BLACKOUT_FAILSAFE STATUSTEXT must arrive within ≤500 ms of trigger.
horiz_999 = True
failsafe_offset: int | None = None
failsafe_in_time = True
if failsafe_trigger_at is not None:
post = [s for s in _samples_inside_window(window, estimates) if s.monotonic_ms >= failsafe_trigger_at]
if post and any(s.horiz_accuracy != HORIZ_ACCURACY_FAILSAFE for s in post):
horiz_999 = False
for st in statustexts:
if STATUSTEXT_FAILSAFE not in st.text:
continue
if st.monotonic_ms < failsafe_trigger_at:
continue
offset = st.monotonic_ms - failsafe_trigger_at
if failsafe_offset is None or offset < failsafe_offset:
failsafe_offset = offset
failsafe_in_time = (
failsafe_offset is not None and failsafe_offset <= ESCALATION_LATENCY_MS
)
passes_ac7 = failsafe_trigger_at is None or (horiz_999 and failsafe_in_time)
return EscalationReport(
cov2d_crossed=cov2d_at is not None,
cov2d_crossed_at_ms=cov2d_at,
fix_type_degraded=fix_degraded,
cov500_or_30s_crossed=failsafe_trigger_at is not None,
cov500_or_30s_crossed_at_ms=failsafe_trigger_at,
horiz_accuracy_999=horiz_999,
failsafe_statustext_offset_ms=failsafe_offset,
failsafe_statustext_in_time=failsafe_in_time,
passes_ac6=passes_ac6,
passes_ac7=passes_ac7,
)
def evaluate_recovery_gate(
window: BlackoutWindow,
estimates: Sequence[OutboundEstimateSample],
gps_health: Sequence[GpsHealthSample],
consistency_checks: Sequence[ConsistencyCheckEvent],
) -> RecoveryGateReport:
"""AC-8: recovery only after ≥10 s healthy/non-spoofed FC GPS AND a consistency check pass."""
# First post-window satellite_anchored sample marks the (claimed) recovery moment.
recovery_at: int | None = None
for s in estimates:
if (
s.monotonic_ms > window.end_monotonic_ms
and s.source_label == SATELLITE_ANCHORED_LABEL
):
recovery_at = s.monotonic_ms
break
if recovery_at is None:
# No recovery attempted — vacuously passing for this gate; the
# caller can still flag it via window-level coverage.
return RecoveryGateReport(
recovery_at_ms=None,
stable_period_s=None,
consistency_check_passed=False,
passes=True,
)
# (a) Continuous healthy/non-spoofed FC GPS for ≥RECOVERY_STABLE_S BEFORE recovery_at.
cutoff_ms = recovery_at - int(RECOVERY_STABLE_S * 1000)
relevant = [
h for h in gps_health
if window.end_monotonic_ms <= h.monotonic_ms <= recovery_at
]
stable = all(h.healthy and not h.spoofed for h in relevant) and len(relevant) >= 1
earliest_relevant = relevant[0].monotonic_ms if relevant else recovery_at
stable_period_s = (recovery_at - earliest_relevant) / 1000.0
has_enough_window = earliest_relevant <= cutoff_ms
# (b) Consistency check pass occurred between window-end and recovery_at.
consistency_passed = any(
c.passed and window.end_monotonic_ms <= c.monotonic_ms <= recovery_at
for c in consistency_checks
)
return RecoveryGateReport(
recovery_at_ms=recovery_at,
stable_period_s=stable_period_s,
consistency_check_passed=consistency_passed,
passes=stable and has_enough_window and consistency_passed,
)
def evaluate(
window: BlackoutWindow,
*,
estimates: Sequence[OutboundEstimateSample],
statustexts: Sequence[StatustextSample],
spoof_events: Sequence[SpoofRejectedEvent],
gps_health: Sequence[GpsHealthSample],
consistency_checks: Sequence[ConsistencyCheckEvent],
frame_period_ms: int,
is_35s_window: bool,
) -> BlackoutSpoofReport:
"""Run every AC-1..AC-8 check for a single window."""
return BlackoutSpoofReport(
window=window,
switch_latency=evaluate_switch_latency(window, estimates, frame_period_ms),
spoof_rejection=evaluate_spoof_rejection(window, estimates, spoof_events),
covariance_monotonic=evaluate_covariance_monotonic(window, estimates),
honest_accuracy=evaluate_honest_accuracy(window, estimates),
statustext_rate=evaluate_statustext_rate(window, statustexts),
escalation=evaluate_escalation(
window, estimates, statustexts, is_35s_window=is_35s_window
),
recovery_gate=evaluate_recovery_gate(
window, estimates, gps_health, consistency_checks
),
)
def write_csv_evidence(out_path: Path, report: BlackoutSpoofReport) -> Path:
"""Write FT-N-04 aggregate evidence — one row of per-AC summary."""
out_path.parent.mkdir(parents=True, exist_ok=True)
with out_path.open("w", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(
[
"window_duration_s",
"ac1_switch_latency_ms",
"ac1_passes",
"ac2_spoof_rejected_count",
"ac2_re_anchored_count",
"ac2_passes",
"ac3_first_decreasing_at_ms",
"ac3_passes",
"ac4_violation_count",
"ac4_passes",
"ac5_observed_hz",
"ac5_passes",
"ac6_cov2d_at_ms",
"ac6_passes",
"ac7_failsafe_trigger_at_ms",
"ac7_passes",
"ac8_recovery_at_ms",
"ac8_passes",
"passes",
]
)
r = report
writer.writerow(
[
f"{r.window.duration_s:.3f}",
"" if r.switch_latency.first_dead_reckoned_offset_ms is None else r.switch_latency.first_dead_reckoned_offset_ms,
"true" if r.switch_latency.passes else "false",
r.spoof_rejection.spoof_rejected_count,
r.spoof_rejection.satellite_anchored_inside_window,
"true" if r.spoof_rejection.passes else "false",
"" if r.covariance_monotonic.first_decreasing_at_ms is None else r.covariance_monotonic.first_decreasing_at_ms,
"true" if r.covariance_monotonic.passes else "false",
r.honest_accuracy.violation_count,
"true" if r.honest_accuracy.passes else "false",
"" if r.statustext_rate.observed_hz is None else f"{r.statustext_rate.observed_hz:.3f}",
"true" if r.statustext_rate.passes else "false",
"" if r.escalation.cov2d_crossed_at_ms is None else r.escalation.cov2d_crossed_at_ms,
"true" if r.escalation.passes_ac6 else "false",
"" if r.escalation.cov500_or_30s_crossed_at_ms is None else r.escalation.cov500_or_30s_crossed_at_ms,
"true" if r.escalation.passes_ac7 else "false",
"" if r.recovery_gate.recovery_at_ms is None else r.recovery_gate.recovery_at_ms,
"true" if r.recovery_gate.passes else "false",
"true" if r.passes else "false",
]
)
return out_path
@@ -0,0 +1,293 @@
"""Outage-request evaluation for FT-N-03 (AZ-425 / AC-3.4).
Detects sustained no-estimate outage windows from an outbound-estimate
stream, then evaluates:
* AC-1: outage onset — ≥``MIN_OUTAGE_FRAMES`` consecutive missing frames.
* AC-2: STATUSTEXT containing ``OPERATOR_RELOC_REQUEST`` is emitted
within ``[OUTAGE_THRESHOLD_S TOLERANCE_S, OUTAGE_THRESHOLD_S +
TOLERANCE_S]`` of outage onset.
* AC-3: during the outage window, the outbound stream emits at least
one estimate carrying ``source_label = dead_reckoned`` (IMU-extrapolated
propagation continues).
* AC-4: FC-side SITL state shows NO EKF divergence event during the
outage.
A "no-estimate frame" is a frame_idx in the expected sequence with no
matching outbound-estimate record. Frame indices are expected to be
monotonic; ``expected_frame_indices`` is supplied by the caller so the
evaluator does not have to know the replay's total frame count.
Public-boundary discipline: does NOT import any
``src/gps_denied_onboard`` symbol.
"""
from __future__ import annotations
import csv
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Sequence
MIN_OUTAGE_FRAMES = 3 # AC-1
OUTAGE_THRESHOLD_S = 2.0 # AC-2
TOLERANCE_S = 0.5 # AC-2 ±500 ms window
STATUSTEXT_REGEX = "OPERATOR_RELOC_REQUEST" # AC-2 exact substring
DEAD_RECKONED_LABEL = "dead_reckoned" # AC-3
@dataclass(frozen=True)
class OutboundEstimateSample:
"""One outbound estimate keyed by frame index + monotonic time."""
frame_idx: int
monotonic_ms: int
source_label: str
@dataclass(frozen=True)
class StatustextSample:
"""One STATUSTEXT message captured from mavproxy tlog."""
monotonic_ms: int
text: str
@dataclass(frozen=True)
class EkfDivergenceEvent:
"""One EKF-divergence event observed via SITL state read."""
monotonic_ms: int
reason: str
@dataclass(frozen=True)
class OutageWindow:
"""One detected outage window — contiguous run of missing frames."""
first_missing_frame_idx: int
last_missing_frame_idx: int
onset_monotonic_ms: int
end_monotonic_ms: int
@property
def length_frames(self) -> int:
return self.last_missing_frame_idx - self.first_missing_frame_idx + 1
@property
def duration_ms(self) -> int:
return self.end_monotonic_ms - self.onset_monotonic_ms
@dataclass(frozen=True)
class OutageReport:
"""AC-1 / AC-2 / AC-3 / AC-4 evaluation for one outage window."""
window: OutageWindow
passes_min_length: bool # AC-1
statustext_offset_ms: int | None # AC-2: ms after onset, None if absent
passes_statustext: bool # AC-2
dead_reckoned_count: int # AC-3 supporting metric
passes_dead_reckoned: bool # AC-3
ekf_divergence_count: int # AC-4 supporting metric
passes_ekf: bool # AC-4
@property
def passes(self) -> bool:
return (
self.passes_min_length
and self.passes_statustext
and self.passes_dead_reckoned
and self.passes_ekf
)
def detect_outage_windows(
expected_frame_indices: Sequence[int],
estimates: Sequence[OutboundEstimateSample],
frame_period_ms: int,
replay_start_monotonic_ms: int = 0,
) -> list[OutageWindow]:
"""Detect contiguous outage windows.
A frame index in ``expected_frame_indices`` with no matching estimate
counts as missing. Runs of consecutive missing frames of length
≥``MIN_OUTAGE_FRAMES`` become outage windows.
``frame_period_ms`` is the nominal inter-frame interval; onset/end
timestamps are derived as
``replay_start_monotonic_ms + frame_idx * frame_period_ms``. The
timing fields are estimates — when actual capture timestamps are
available the caller should pass them via ``estimates`` and rely on
those for downstream timing checks.
"""
present = {e.frame_idx for e in estimates}
windows: list[OutageWindow] = []
run_start: int | None = None
prev_idx: int | None = None
for idx in expected_frame_indices:
if idx not in present:
if run_start is None:
run_start = idx
prev_idx = idx
else:
if run_start is not None and prev_idx is not None:
run_length = prev_idx - run_start + 1
if run_length >= MIN_OUTAGE_FRAMES:
windows.append(
OutageWindow(
first_missing_frame_idx=run_start,
last_missing_frame_idx=prev_idx,
onset_monotonic_ms=replay_start_monotonic_ms
+ run_start * frame_period_ms,
end_monotonic_ms=replay_start_monotonic_ms
+ (prev_idx + 1) * frame_period_ms,
)
)
run_start = None
prev_idx = None
# Trailing run.
if run_start is not None and prev_idx is not None:
run_length = prev_idx - run_start + 1
if run_length >= MIN_OUTAGE_FRAMES:
windows.append(
OutageWindow(
first_missing_frame_idx=run_start,
last_missing_frame_idx=prev_idx,
onset_monotonic_ms=replay_start_monotonic_ms
+ run_start * frame_period_ms,
end_monotonic_ms=replay_start_monotonic_ms
+ (prev_idx + 1) * frame_period_ms,
)
)
return windows
def _first_statustext_offset_ms(
window: OutageWindow,
statustexts: Iterable[StatustextSample],
) -> int | None:
"""Return ms-offset of first OPERATOR_RELOC_REQUEST after onset, or None."""
best: int | None = None
for st in statustexts:
if STATUSTEXT_REGEX not in st.text:
continue
if st.monotonic_ms < window.onset_monotonic_ms:
continue
offset = st.monotonic_ms - window.onset_monotonic_ms
if best is None or offset < best:
best = offset
return best
def _dead_reckoned_during_window(
window: OutageWindow,
estimates: Iterable[OutboundEstimateSample],
) -> int:
count = 0
for e in estimates:
if (
e.source_label == DEAD_RECKONED_LABEL
and window.onset_monotonic_ms <= e.monotonic_ms <= window.end_monotonic_ms
):
count += 1
return count
def _ekf_divergence_during_window(
window: OutageWindow,
events: Iterable[EkfDivergenceEvent],
) -> int:
count = 0
for ev in events:
if window.onset_monotonic_ms <= ev.monotonic_ms <= window.end_monotonic_ms:
count += 1
return count
def evaluate_window(
window: OutageWindow,
estimates: Sequence[OutboundEstimateSample],
statustexts: Sequence[StatustextSample],
ekf_events: Sequence[EkfDivergenceEvent],
) -> OutageReport:
"""Compute AC-1..AC-4 evaluation for a single outage window."""
offset = _first_statustext_offset_ms(window, statustexts)
threshold_ms = int(OUTAGE_THRESHOLD_S * 1000)
tolerance_ms = int(TOLERANCE_S * 1000)
passes_statustext = (
offset is not None
and (threshold_ms - tolerance_ms) <= offset <= (threshold_ms + tolerance_ms)
)
dr_count = _dead_reckoned_during_window(window, estimates)
ekf_count = _ekf_divergence_during_window(window, ekf_events)
return OutageReport(
window=window,
passes_min_length=window.length_frames >= MIN_OUTAGE_FRAMES,
statustext_offset_ms=offset,
passes_statustext=passes_statustext,
dead_reckoned_count=dr_count,
passes_dead_reckoned=dr_count >= 1,
ekf_divergence_count=ekf_count,
passes_ekf=ekf_count == 0,
)
def evaluate(
expected_frame_indices: Sequence[int],
estimates: Sequence[OutboundEstimateSample],
statustexts: Sequence[StatustextSample],
ekf_events: Sequence[EkfDivergenceEvent],
frame_period_ms: int,
replay_start_monotonic_ms: int = 0,
) -> list[OutageReport]:
"""Detect outage windows and evaluate each."""
windows = detect_outage_windows(
expected_frame_indices,
estimates,
frame_period_ms=frame_period_ms,
replay_start_monotonic_ms=replay_start_monotonic_ms,
)
return [evaluate_window(w, estimates, statustexts, ekf_events) for w in windows]
def write_csv_evidence(out_path: Path, reports: Sequence[OutageReport]) -> Path:
out_path.parent.mkdir(parents=True, exist_ok=True)
with out_path.open("w", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(
[
"first_missing_frame",
"last_missing_frame",
"length_frames",
"onset_ms",
"duration_ms",
"statustext_offset_ms",
"dead_reckoned_count",
"ekf_divergence_count",
"passes_min_length",
"passes_statustext",
"passes_dead_reckoned",
"passes_ekf",
"passes",
]
)
for r in reports:
writer.writerow(
[
r.window.first_missing_frame_idx,
r.window.last_missing_frame_idx,
r.window.length_frames,
r.window.onset_monotonic_ms,
r.window.duration_ms,
"" if r.statustext_offset_ms is None else r.statustext_offset_ms,
r.dead_reckoned_count,
r.ekf_divergence_count,
"true" if r.passes_min_length else "false",
"true" if r.passes_statustext else "false",
"true" if r.passes_dead_reckoned else "false",
"true" if r.passes_ekf else "false",
"true" if r.passes else "false",
]
)
return out_path
@@ -0,0 +1,261 @@
"""Outlier-tolerance evaluation for FT-N-01 (AZ-424 / AC-3.1).
Consumes the AZ-408 ``outlier`` injector's ``manifest.csv`` (which
frames were replaced + the geodesic offset) and the SUT's outbound
estimate stream, and validates:
* AC-1: at least ``MIN_OUTLIER_COUNT`` outlier frames were injected
over the replay.
* AC-2: for every outlier event,
``error_after_outlier ≤ error_before_outlier + DRIFT_BUDGET_M``.
* AC-3: ``cov_semi_major_m`` is non-decreasing across the 3-frame
window centred on the outlier (frame before, outlier, frame after).
The injector's ``geodesic_offset_m`` column verifies the
RESTRICT-CAM-1 / AC-3.1 threshold (>350 m) per-row — the AC-1 count
check here is a coarser invariant that does not duplicate the
per-row geodesic gate.
Public-boundary discipline: does NOT import any
``src/gps_denied_onboard`` symbol.
"""
from __future__ import annotations
import csv
from dataclasses import dataclass
from pathlib import Path
from typing import Sequence
from .geo import distance_m
DRIFT_BUDGET_M = 50.0 # AC-2
COVARIANCE_WINDOW_FRAMES = 3 # AC-3: 1 before + 1 outlier + 1 after
MIN_OUTLIER_COUNT = 10 # AC-1: ~10 over Derkachi 8-min replay
@dataclass(frozen=True)
class GtPose:
"""One ground-truth pose for a video frame, keyed by frame index."""
frame_idx: int
lat_deg: float
lon_deg: float
@dataclass(frozen=True)
class OutboundEstimate:
"""One outbound estimate with covariance + label, keyed by frame index."""
frame_idx: int
monotonic_ms: int
lat_deg: float
lon_deg: float
cov_semi_major_m: float
source_label: str
@dataclass(frozen=True)
class OutlierEvent:
"""One row from the injector's manifest.csv."""
frame_idx: int
geodesic_offset_m: float
src_jpeg_path: str
@dataclass(frozen=True)
class OutlierEventReport:
"""AC-2 + AC-3 evaluation for one outlier event."""
frame_idx: int
error_before_m: float | None
error_outlier_m: float | None
error_after_m: float | None
drift_m: float | None # error_after - error_before; AC-2 budget
cov_before: float | None
cov_outlier: float | None
cov_after: float | None
cov_non_decreasing: bool
@property
def passes_drift(self) -> bool:
return (
self.drift_m is not None
and self.drift_m <= DRIFT_BUDGET_M
)
@property
def passes_covariance(self) -> bool:
return self.cov_non_decreasing
@property
def passes(self) -> bool:
return self.passes_drift and self.passes_covariance
@dataclass(frozen=True)
class OutlierToleranceReport:
"""Aggregate report for all outlier events in the replay."""
events: tuple[OutlierEventReport, ...]
total_outliers: int
@property
def passes_count(self) -> bool:
return self.total_outliers >= MIN_OUTLIER_COUNT
@property
def failed_event_count(self) -> int:
return sum(1 for e in self.events if not e.passes)
@property
def passes(self) -> bool:
return self.passes_count and self.failed_event_count == 0
def load_outlier_manifest(manifest_path: Path) -> list[OutlierEvent]:
"""Read ``outlier/manifest.csv`` into typed events.
Schema (AZ-408): ``frame_idx, src_jpeg_path, replacement_tile_x,
replacement_tile_y, geodesic_offset_m, seed``.
"""
if not manifest_path.exists():
raise FileNotFoundError(
f"outlier manifest not found: {manifest_path} — run the "
"outlier injector first (AZ-408 / runner/helpers/injector_fixtures)"
)
events: list[OutlierEvent] = []
with manifest_path.open() as fh:
reader = csv.DictReader(fh)
required = {"frame_idx", "src_jpeg_path", "geodesic_offset_m"}
missing = required - set(reader.fieldnames or [])
if missing:
raise ValueError(
f"outlier manifest {manifest_path} missing required columns: "
f"{sorted(missing)}"
)
for row in reader:
events.append(
OutlierEvent(
frame_idx=int(row["frame_idx"]),
geodesic_offset_m=float(row["geodesic_offset_m"]),
src_jpeg_path=row["src_jpeg_path"],
)
)
return events
def _index_by_frame(estimates: Sequence[OutboundEstimate]) -> dict[int, OutboundEstimate]:
by_frame: dict[int, OutboundEstimate] = {}
for e in estimates:
by_frame[e.frame_idx] = e
return by_frame
def _index_gt(gt: Sequence[GtPose]) -> dict[int, GtPose]:
by_frame: dict[int, GtPose] = {}
for g in gt:
by_frame[g.frame_idx] = g
return by_frame
def _error_m(est: OutboundEstimate | None, gt: GtPose | None) -> float | None:
if est is None or gt is None:
return None
return distance_m(gt.lat_deg, gt.lon_deg, est.lat_deg, est.lon_deg)
def evaluate_event(
event: OutlierEvent,
estimates_by_frame: dict[int, OutboundEstimate],
gt_by_frame: dict[int, GtPose],
) -> OutlierEventReport:
"""Compute the AC-2 + AC-3 report for one outlier event."""
before = estimates_by_frame.get(event.frame_idx - 1)
outlier = estimates_by_frame.get(event.frame_idx)
after = estimates_by_frame.get(event.frame_idx + 1)
gt_before = gt_by_frame.get(event.frame_idx - 1)
gt_outlier = gt_by_frame.get(event.frame_idx)
gt_after = gt_by_frame.get(event.frame_idx + 1)
err_before = _error_m(before, gt_before)
err_outlier = _error_m(outlier, gt_outlier)
err_after = _error_m(after, gt_after)
drift: float | None = None
if err_before is not None and err_after is not None:
drift = err_after - err_before
cov_before = before.cov_semi_major_m if before is not None else None
cov_outlier = outlier.cov_semi_major_m if outlier is not None else None
cov_after = after.cov_semi_major_m if after is not None else None
covs = [c for c in (cov_before, cov_outlier, cov_after) if c is not None]
cov_non_decreasing = all(covs[i + 1] >= covs[i] for i in range(len(covs) - 1))
return OutlierEventReport(
frame_idx=event.frame_idx,
error_before_m=err_before,
error_outlier_m=err_outlier,
error_after_m=err_after,
drift_m=drift,
cov_before=cov_before,
cov_outlier=cov_outlier,
cov_after=cov_after,
cov_non_decreasing=cov_non_decreasing,
)
def evaluate(
events: Sequence[OutlierEvent],
estimates: Sequence[OutboundEstimate],
gt: Sequence[GtPose],
) -> OutlierToleranceReport:
"""Aggregate report across all outlier events."""
by_frame = _index_by_frame(estimates)
gt_idx = _index_gt(gt)
reports = tuple(evaluate_event(ev, by_frame, gt_idx) for ev in events)
return OutlierToleranceReport(events=reports, total_outliers=len(events))
def write_csv_evidence(out_path: Path, report: OutlierToleranceReport) -> Path:
"""Write per-event FT-N-01 evidence CSV."""
out_path.parent.mkdir(parents=True, exist_ok=True)
with out_path.open("w", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(
[
"frame_idx",
"error_before_m",
"error_outlier_m",
"error_after_m",
"drift_m",
"cov_before",
"cov_outlier",
"cov_after",
"cov_non_decreasing",
"passes_drift",
"passes_covariance",
"passes",
]
)
for e in report.events:
writer.writerow(
[
e.frame_idx,
"" if e.error_before_m is None else f"{e.error_before_m:.3f}",
"" if e.error_outlier_m is None else f"{e.error_outlier_m:.3f}",
"" if e.error_after_m is None else f"{e.error_after_m:.3f}",
"" if e.drift_m is None else f"{e.drift_m:.3f}",
"" if e.cov_before is None else f"{e.cov_before:.3f}",
"" if e.cov_outlier is None else f"{e.cov_outlier:.3f}",
"" if e.cov_after is None else f"{e.cov_after:.3f}",
"true" if e.cov_non_decreasing else "false",
"true" if e.passes_drift else "false",
"true" if e.passes_covariance else "false",
"true" if e.passes else "false",
]
)
return out_path
@@ -0,0 +1,170 @@
"""FT-N-01 — 350 m outlier injection tolerance (AZ-424 / AC-3.1).
Replays the Derkachi flight with the AZ-408 ``outlier`` injector at
``--density medium`` and verifies AC-1 / AC-2 / AC-3 via
``runner.helpers.outlier_tolerance_evaluator``.
Gated on the same upstream replay helpers as FT-N-02 / FT-P-07
(``frame_source_replay``, ``fdr_reader``, ``imu_replay``). When those
helpers are still stubbed (current state under AZ-441 / AZ-407
leftovers), the scenario test skips while
``e2e/_unit_tests/helpers/test_outlier_tolerance_evaluator.py`` covers
the pure-logic AC-2 / AC-3 invariants.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from fixtures.injectors.outlier import OutlierInjectionReport
from runner.helpers import outlier_tolerance_evaluator as ote
@pytest.fixture(scope="module")
def _harness_helpers_implemented() -> bool:
from runner.helpers import fdr_reader, imu_replay
from runner.helpers.frame_source_replay import FrameSourceReplayer
try:
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
try:
replayer.replay_video(Path("/tmp/non-existent.mp4"))
except NotImplementedError:
return False
try:
list(fdr_reader.iter_records(Path("/tmp/non-existent")))
except NotImplementedError:
return False
try:
imu_replay.ImuReplayer(emitter=_NullImuEmitter()).replay(Path("/tmp/non-existent.csv")) # type: ignore[arg-type]
except NotImplementedError:
return False
return True
except Exception:
return False
class _NullSink:
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
return None
class _NullImuEmitter:
def emit(self, sample: object) -> None:
return None
@pytest.mark.parametrize(
"outlier_injection_derkachi",
[{"density": "medium", "seed": 0}],
indirect=True,
)
@pytest.mark.traces_to("AC-3.1,AC-1,AC-2,AC-3,AC-4")
def test_ft_n_01_outlier_tolerance(
fc_adapter: str,
vio_strategy: str,
outlier_injection_derkachi: OutlierInjectionReport,
evidence_dir, # type: ignore[no-untyped-def]
run_id: str,
nfr_recorder, # type: ignore[no-untyped-def]
_harness_helpers_implemented: bool,
) -> None:
if not _harness_helpers_implemented:
pytest.skip(
"FT-N-01 full replay requires runner.helpers.{frame_source_replay,"
"fdr_reader,imu_replay} — currently AZ-441 / AZ-407 leftovers. "
"AC-1/AC-2/AC-3 helper logic covered by "
"e2e/_unit_tests/helpers/test_outlier_tolerance_evaluator.py."
)
from runner.helpers import fdr_reader
from runner.helpers.frame_source_replay import FrameSourceReplayer
# 1. AC-1 — load injection plan (outlier event frames + offsets).
manifest_path = outlier_injection_derkachi.out_root / "manifest.csv"
events = ote.load_outlier_manifest(manifest_path)
assert len(events) >= ote.MIN_OUTLIER_COUNT, (
f"AC-1: medium-density injection must produce ≥{ote.MIN_OUTLIER_COUNT} "
f"outliers (got {len(events)} from {manifest_path})"
)
# 2. Drive replay against the injected frames directory.
FrameSourceReplayer(_resolve_frame_sink()).replay_video(
outlier_injection_derkachi.out_root / "frames"
)
# 3. Collect outbound estimates + GT from FDR + tile cache.
fdr_root = Path(evidence_dir).parent / f"run-{run_id}" / "fdr"
estimates: list[ote.OutboundEstimate] = []
for rec in fdr_reader.iter_records(fdr_root):
if rec.record_type != "outbound_estimate":
continue
payload = rec.payload
estimates.append(
ote.OutboundEstimate(
frame_idx=int(payload["frame_idx"]), # type: ignore[arg-type]
monotonic_ms=int(rec.monotonic_ms),
lat_deg=float(payload["lat_deg"]), # type: ignore[arg-type]
lon_deg=float(payload["lon_deg"]), # type: ignore[arg-type]
cov_semi_major_m=float(payload["cov_semi_major_m"]), # type: ignore[arg-type]
source_label=str(payload["source_label"]), # type: ignore[arg-type]
)
)
gt: list[ote.GtPose] = _resolve_gt_per_frame(outlier_injection_derkachi)
if not estimates:
pytest.fail("FT-N-01: no outbound_estimate records produced")
# 4. Evaluate per outlier event.
report = ote.evaluate(events, estimates, gt)
out_csv = evidence_dir / f"ft-n-01-{fc_adapter}-{vio_strategy}.csv"
ote.write_csv_evidence(out_csv, report)
# 5. NFR + AC assertions.
nfr_recorder.record_metric(
"ft_n_01.total_outliers", float(report.total_outliers), ac_id="AC-1"
)
nfr_recorder.record_metric(
"ft_n_01.failed_event_count", float(report.failed_event_count), ac_id="AC-2"
)
for e in report.events:
if e.drift_m is not None:
nfr_recorder.record_metric(
f"ft_n_01.event_{e.frame_idx}.drift_m", e.drift_m, ac_id="AC-2"
)
nfr_recorder.record_metric(
f"ft_n_01.event_{e.frame_idx}.cov_non_decreasing",
1.0 if e.cov_non_decreasing else 0.0,
ac_id="AC-3",
)
assert report.passes_count, (
f"AC-1: ≥{ote.MIN_OUTLIER_COUNT} outliers required; "
f"got {report.total_outliers}"
)
for e in report.events:
assert e.passes_drift, (
f"AC-2 (drift ≤ {ote.DRIFT_BUDGET_M} m) failed at frame "
f"{e.frame_idx}: drift_m={e.drift_m}, "
f"error_before={e.error_before_m}, error_after={e.error_after_m}"
)
assert e.passes_covariance, (
f"AC-3 (cov_semi_major_m non-decreasing across window) failed at "
f"frame {e.frame_idx}: "
f"cov_before={e.cov_before}, cov_outlier={e.cov_outlier}, "
f"cov_after={e.cov_after}"
)
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
raise NotImplementedError(
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
)
def _resolve_gt_per_frame(report: OutlierInjectionReport) -> list[ote.GtPose]:
raise NotImplementedError(
"Per-frame GT resolution is owned by AZ-407 / runner.helpers.tile_cache_gt"
)
@@ -0,0 +1,201 @@
"""FT-N-03 — Extended outage triggers operator re-loc request (AZ-425 / AC-3.4).
Replays the Derkachi flight with a 3-consecutive-frame failure injector
(a thin extension of the AZ-408 outlier injector that emits all-zero
frames instead of crops) and verifies AC-1..AC-4 via
``runner.helpers.outage_request_evaluator``.
Gated on the same upstream replay helpers as FT-N-01 / FT-N-02 / FT-P-07
(``frame_source_replay``, ``fdr_reader``, ``imu_replay``, mavproxy
``.tlog`` capture, SITL state read). When those helpers are still
stubbed, the scenario test skips while
``e2e/_unit_tests/helpers/test_outage_request_evaluator.py`` covers the
AC-1..AC-4 evaluator logic.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from runner.helpers import outage_request_evaluator as ore
DERKACHI_DIR = (
Path(__file__).resolve().parents[3]
/ "_docs"
/ "00_problem"
/ "input_data"
/ "flight_derkachi"
)
DERKACHI_MP4 = DERKACHI_DIR / "flight_derkachi.mp4"
@pytest.fixture(scope="module")
def _harness_helpers_implemented() -> bool:
from runner.helpers import fdr_reader, mavproxy_tlog_reader, sitl_observer
from runner.helpers.frame_source_replay import FrameSourceReplayer
try:
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
try:
replayer.replay_video(Path("/tmp/non-existent.mp4"))
except NotImplementedError:
return False
try:
list(fdr_reader.iter_records(Path("/tmp/non-existent")))
except NotImplementedError:
return False
try:
list(mavproxy_tlog_reader.iter_messages(Path("/tmp/non-existent.tlog")))
except NotImplementedError:
return False
try:
sitl_observer.read_ekf_divergence_events() # type: ignore[attr-defined]
except (AttributeError, NotImplementedError):
return False
return True
except Exception:
return False
class _NullSink:
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
return None
@pytest.mark.traces_to("AC-3.4,AC-1,AC-2,AC-3,AC-4,AC-5")
def test_ft_n_03_outage_reloc(
fc_adapter: str,
vio_strategy: str,
evidence_dir, # type: ignore[no-untyped-def]
run_id: str,
nfr_recorder, # type: ignore[no-untyped-def]
_harness_helpers_implemented: bool,
) -> None:
if not _harness_helpers_implemented:
pytest.skip(
"FT-N-03 full replay requires runner.helpers.{frame_source_replay,"
"fdr_reader,mavproxy_tlog_reader,sitl_observer} — currently "
"AZ-441 / AZ-407 / AZ-416 leftovers. AC-1..AC-4 evaluator logic "
"covered by e2e/_unit_tests/helpers/test_outage_request_evaluator.py."
)
from runner.helpers import fdr_reader, mavproxy_tlog_reader, sitl_observer
from runner.helpers.frame_source_replay import FrameSourceReplayer
# 1. Build / locate the 3-frame outage injection fixture.
injected_frames_dir = _resolve_outage_injection_frames()
# 2. Drive replay.
FrameSourceReplayer(_resolve_frame_sink()).replay_video(injected_frames_dir)
# 3. Collect outbound estimates + STATUSTEXT + EKF events.
fdr_root = Path(evidence_dir).parent / f"run-{run_id}" / "fdr"
estimates: list[ore.OutboundEstimateSample] = []
expected_frame_indices: list[int] = []
for rec in fdr_reader.iter_records(fdr_root):
if rec.record_type == "frame_received":
expected_frame_indices.append(int(rec.payload["frame_idx"])) # type: ignore[arg-type]
elif rec.record_type == "outbound_estimate":
payload = rec.payload
estimates.append(
ore.OutboundEstimateSample(
frame_idx=int(payload["frame_idx"]), # type: ignore[arg-type]
monotonic_ms=int(rec.monotonic_ms),
source_label=str(payload["source_label"]), # type: ignore[arg-type]
)
)
tlog_path = Path(evidence_dir).parent / f"run-{run_id}" / "mavproxy.tlog"
statustexts = [
ore.StatustextSample(
monotonic_ms=int(m.timestamp_us // 1000),
text=str(m.fields.get("text", "")),
)
for m in mavproxy_tlog_reader.iter_messages(tlog_path)
if m.msg_type == "STATUSTEXT"
]
ekf_events = [
ore.EkfDivergenceEvent(
monotonic_ms=int(ev.monotonic_ms), reason=str(ev.reason)
)
for ev in sitl_observer.read_ekf_divergence_events() # type: ignore[attr-defined]
]
# 4. Evaluate.
reports = ore.evaluate(
expected_frame_indices,
estimates,
statustexts,
ekf_events,
frame_period_ms=_resolve_frame_period_ms(),
)
out_csv = evidence_dir / f"ft-n-03-{fc_adapter}-{vio_strategy}.csv"
ore.write_csv_evidence(out_csv, reports)
# 5. NFR metrics + AC assertions.
assert reports, "FT-N-03: at least one outage window must be detected (AC-1)"
for r in reports:
nfr_recorder.record_metric(
f"ft_n_03.window_{r.window.first_missing_frame_idx}.length_frames",
float(r.window.length_frames),
ac_id="AC-1",
)
if r.statustext_offset_ms is not None:
nfr_recorder.record_metric(
f"ft_n_03.window_{r.window.first_missing_frame_idx}.statustext_offset_ms",
float(r.statustext_offset_ms),
ac_id="AC-2",
)
nfr_recorder.record_metric(
f"ft_n_03.window_{r.window.first_missing_frame_idx}.dead_reckoned_count",
float(r.dead_reckoned_count),
ac_id="AC-3",
)
nfr_recorder.record_metric(
f"ft_n_03.window_{r.window.first_missing_frame_idx}.ekf_divergence_count",
float(r.ekf_divergence_count),
ac_id="AC-4",
)
for r in reports:
assert r.passes_min_length, (
f"AC-1: outage window {r.window.first_missing_frame_idx}-"
f"{r.window.last_missing_frame_idx} is shorter than "
f"{ore.MIN_OUTAGE_FRAMES} frames"
)
assert r.passes_statustext, (
f"AC-2: '{ore.STATUSTEXT_REGEX}' STATUSTEXT not within "
f"{int(ore.OUTAGE_THRESHOLD_S * 1000)} ±{int(ore.TOLERANCE_S * 1000)} ms "
f"of outage onset at frame {r.window.first_missing_frame_idx} "
f"(observed offset={r.statustext_offset_ms} ms)"
)
assert r.passes_dead_reckoned, (
f"AC-3: no `dead_reckoned` estimate emitted during outage "
f"window starting at frame {r.window.first_missing_frame_idx}"
)
assert r.passes_ekf, (
f"AC-4: EKF divergence event(s) observed during outage "
f"window starting at frame {r.window.first_missing_frame_idx} "
f"(count={r.ekf_divergence_count})"
)
def _resolve_outage_injection_frames() -> Path:
raise NotImplementedError(
"3-frame outage injector is owned by AZ-408 extension / "
"fixtures/injectors/outlier.py (--all-zero variant)"
)
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
raise NotImplementedError(
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
)
def _resolve_frame_period_ms() -> int:
raise NotImplementedError(
"Frame period resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
)
@@ -0,0 +1,267 @@
"""FT-N-04 — Visual blackout + spoofed GPS combined failsafe (AZ-426 / AC-3.5, AC-NEW-8).
Three sub-cases (5 s / 15 s / 35 s) at the ladder of windows
prescribed by AC-3.5 + AC-NEW-8, replayed via the AZ-408
``blackout_spoof`` injector + the FC-inbound spoof proxy, and
validated by ``runner.helpers.blackout_spoof_evaluator``.
Gated on the same upstream replay helpers as the other negative
scenarios (``frame_source_replay``, ``fdr_reader``,
``mavproxy_tlog_reader``, ``sitl_observer``, ``fc_proxy`` runtime
binding). When those helpers are still stubbed the scenario test
skips while
``e2e/_unit_tests/helpers/test_blackout_spoof_evaluator.py`` covers
the AC-1..AC-8 evaluator logic.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from fixtures.injectors.blackout_spoof import BlackoutSpoofReport
from runner.helpers import blackout_spoof_evaluator as bse
_WINDOW_LADDER_S = (5.0, 15.0, 35.0)
@pytest.fixture(scope="module")
def _harness_helpers_implemented() -> bool:
from runner.helpers import fdr_reader, mavproxy_tlog_reader, sitl_observer
from runner.helpers.frame_source_replay import FrameSourceReplayer
try:
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
try:
replayer.replay_video(Path("/tmp/non-existent.mp4"))
except NotImplementedError:
return False
try:
list(fdr_reader.iter_records(Path("/tmp/non-existent")))
except NotImplementedError:
return False
try:
list(mavproxy_tlog_reader.iter_messages(Path("/tmp/non-existent.tlog")))
except NotImplementedError:
return False
try:
sitl_observer.read_gps_health_samples() # type: ignore[attr-defined]
sitl_observer.read_consistency_check_events() # type: ignore[attr-defined]
except (AttributeError, NotImplementedError):
return False
return True
except Exception:
return False
class _NullSink:
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
return None
@pytest.mark.parametrize(
"blackout_spoof_derkachi",
[{"window_seconds": s, "seed": 0} for s in _WINDOW_LADDER_S],
indirect=True,
ids=[f"{int(s)}s" for s in _WINDOW_LADDER_S],
)
@pytest.mark.traces_to(
"AC-3.5,AC-NEW-8,AC-1,AC-2,AC-3,AC-4,AC-5,AC-6,AC-7,AC-8,AC-9"
)
def test_ft_n_04_blackout_spoof(
fc_adapter: str,
vio_strategy: str,
blackout_spoof_derkachi: BlackoutSpoofReport,
evidence_dir, # type: ignore[no-untyped-def]
run_id: str,
nfr_recorder, # type: ignore[no-untyped-def]
_harness_helpers_implemented: bool,
) -> None:
if not _harness_helpers_implemented:
pytest.skip(
"FT-N-04 full replay requires runner.helpers.{frame_source_replay,"
"fdr_reader,mavproxy_tlog_reader,sitl_observer,fc_proxy} — currently "
"AZ-441 / AZ-407 / AZ-416 leftovers. AC-1..AC-8 evaluator logic "
"covered by e2e/_unit_tests/helpers/test_blackout_spoof_evaluator.py."
)
from runner.helpers import fdr_reader, mavproxy_tlog_reader, sitl_observer
from runner.helpers.frame_source_replay import FrameSourceReplayer
schedule = blackout_spoof_derkachi.schedule
window = bse.BlackoutWindow(
onset_monotonic_ms=schedule.window_start_ms,
end_monotonic_ms=schedule.window_end_ms,
)
is_35s = abs(window.duration_s - 35.0) < 0.5
# 1. Drive replay (frames + paired fc-proxy spoof injection).
FrameSourceReplayer(_resolve_frame_sink()).replay_video(
blackout_spoof_derkachi.out_root / "frames"
)
_drive_fc_proxy(blackout_spoof_derkachi.out_root / "schedule.json")
# 2. Collect FDR estimates + spoof-rejected events.
fdr_root = Path(evidence_dir).parent / f"run-{run_id}" / "fdr"
estimates: list[bse.OutboundEstimateSample] = []
spoof_events: list[bse.SpoofRejectedEvent] = []
for rec in fdr_reader.iter_records(fdr_root):
if rec.record_type == "outbound_estimate":
p = rec.payload
estimates.append(
bse.OutboundEstimateSample(
monotonic_ms=int(rec.monotonic_ms),
source_label=str(p["source_label"]), # type: ignore[arg-type]
cov_semi_major_m=float(p["cov_semi_major_m"]), # type: ignore[arg-type]
horiz_accuracy=float(p.get("horiz_accuracy", p["cov_semi_major_m"])), # type: ignore[arg-type]
fix_type=int(p.get("fix_type", -1)), # type: ignore[arg-type]
)
)
elif rec.record_type == "spoof_rejected":
spoof_events.append(
bse.SpoofRejectedEvent(
monotonic_ms=int(rec.monotonic_ms),
reason=str(rec.payload.get("reason", "")), # type: ignore[arg-type]
)
)
# 3. Collect STATUSTEXTs from mavproxy tlog.
tlog_path = Path(evidence_dir).parent / f"run-{run_id}" / "mavproxy.tlog"
statustexts = [
bse.StatustextSample(
monotonic_ms=int(m.timestamp_us // 1000),
text=str(m.fields.get("text", "")),
)
for m in mavproxy_tlog_reader.iter_messages(tlog_path)
if m.msg_type == "STATUSTEXT"
]
# 4. Collect FC-side GPS health + consistency-check events (recovery gate).
gps_health = [
bse.GpsHealthSample(
monotonic_ms=int(s.monotonic_ms),
healthy=bool(s.healthy),
spoofed=bool(s.spoofed),
)
for s in sitl_observer.read_gps_health_samples() # type: ignore[attr-defined]
]
consistency = [
bse.ConsistencyCheckEvent(
monotonic_ms=int(c.monotonic_ms), passed=bool(c.passed)
)
for c in sitl_observer.read_consistency_check_events() # type: ignore[attr-defined]
]
# 5. Evaluate.
report = bse.evaluate(
window,
estimates=estimates,
statustexts=statustexts,
spoof_events=spoof_events,
gps_health=gps_health,
consistency_checks=consistency,
frame_period_ms=_resolve_frame_period_ms(),
is_35s_window=is_35s,
)
out_csv = (
evidence_dir
/ f"ft-n-04-{int(window.duration_s)}s-{fc_adapter}-{vio_strategy}.csv"
)
bse.write_csv_evidence(out_csv, report)
# 6. NFR metrics + AC assertions.
nfr_recorder.record_metric(
f"ft_n_04.{int(window.duration_s)}s.switch_latency_ms",
float(report.switch_latency.first_dead_reckoned_offset_ms or 0),
ac_id="AC-1",
)
nfr_recorder.record_metric(
f"ft_n_04.{int(window.duration_s)}s.spoof_rejected_count",
float(report.spoof_rejection.spoof_rejected_count),
ac_id="AC-2",
)
nfr_recorder.record_metric(
f"ft_n_04.{int(window.duration_s)}s.honest_accuracy_violation_count",
float(report.honest_accuracy.violation_count),
ac_id="AC-4",
)
if report.statustext_rate.observed_hz is not None:
nfr_recorder.record_metric(
f"ft_n_04.{int(window.duration_s)}s.statustext_imu_only_hz",
report.statustext_rate.observed_hz,
ac_id="AC-5",
)
if is_35s:
nfr_recorder.record_metric(
"ft_n_04.35s.cov2d_at_ms",
float(report.escalation.cov2d_crossed_at_ms or 0),
ac_id="AC-6",
)
nfr_recorder.record_metric(
"ft_n_04.35s.failsafe_trigger_at_ms",
float(report.escalation.cov500_or_30s_crossed_at_ms or 0),
ac_id="AC-7",
)
assert report.switch_latency.passes, (
f"AC-1: dead_reckoned label not within ≤{bse.SWITCH_LATENCY_MS} ms / "
f"1 frame of blackout onset; "
f"offset={report.switch_latency.first_dead_reckoned_offset_ms} ms, "
f"frame_period={report.switch_latency.frame_period_ms} ms"
)
assert report.spoof_rejection.passes, (
f"AC-2: spoof rejection failed; "
f"rejected_count={report.spoof_rejection.spoof_rejected_count}, "
f"re_anchored_count={report.spoof_rejection.satellite_anchored_inside_window}"
)
assert report.covariance_monotonic.passes, (
f"AC-3: cov_semi_major_m decreased at "
f"{report.covariance_monotonic.first_decreasing_at_ms} ms"
)
assert report.honest_accuracy.passes, (
f"AC-4: horiz_accuracy under-reporting "
f"({report.honest_accuracy.violation_count} violations of "
f"{report.honest_accuracy.sample_count} samples)"
)
assert report.statustext_rate.passes, (
f"AC-5: VISUAL_BLACKOUT_IMU_ONLY rate "
f"{report.statustext_rate.observed_hz} Hz outside "
f"[{bse.STATUSTEXT_RATE_MIN_HZ}, {bse.STATUSTEXT_RATE_MAX_HZ}] Hz"
)
if is_35s:
assert report.escalation.passes_ac6, (
f"AC-6: fix_type not degraded after cov crossed "
f"{bse.ESCALATION_COV_2D_M} m at "
f"{report.escalation.cov2d_crossed_at_ms} ms"
)
assert report.escalation.passes_ac7, (
f"AC-7: failsafe escalation incomplete; "
f"horiz_999={report.escalation.horiz_accuracy_999}, "
f"failsafe_statustext_offset_ms="
f"{report.escalation.failsafe_statustext_offset_ms}"
)
assert report.recovery_gate.passes, (
f"AC-8: recovery gate failed; "
f"recovery_at_ms={report.recovery_gate.recovery_at_ms}, "
f"stable_period_s={report.recovery_gate.stable_period_s}, "
f"consistency_check_passed={report.recovery_gate.consistency_check_passed}"
)
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
raise NotImplementedError(
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
)
def _drive_fc_proxy(schedule_path: Path) -> None:
raise NotImplementedError(
"FC-inbound spoof proxy driver is owned by AZ-441 / runner.helpers.fc_proxy_runtime"
)
def _resolve_frame_period_ms() -> int:
raise NotImplementedError(
"Frame period resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
)