mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 08:31:13 +00:00
[AZ-409] [AZ-412] [AZ-413] Batch 70: FT-P-01/04/05/06 scenarios
AZ-409 (3pt) — FT-P-01 still-image frame-center accuracy: - accuracy_evaluator.py: GT loader + Vincenty error + AC-2/AC-3 pass-counts - test_ft_p_01_still_image_accuracy.py: scenario gated on frame_source_replay + sitl_observer NotImplementedError; AC-4 timeout discipline AZ-412 (3pt) — FT-P-04 Derkachi f2f registration >=95% on normal segments: - registration_classifier.py: accel-derived attitude + overlap heuristic + success ratio with AC-3 sharp-turn exclusion - test_ft_p_04_derkachi_f2f_registration.py: scenario gated on frame_source_replay + imu_replay + fdr_reader AZ-413 (3pt) — FT-P-05 + FT-P-06 cross-domain MRE budgets: - mre_evaluator.py: per-image budget (strict <2.5px) + 95th-percentile via numpy linear interp + combined report - test_ft_p_05_sat_anchor.py: cross-domain scenario, reuses accuracy_evaluator for geodesic join - test_ft_p_06_mre_budgets.py: pure piggyback on FT-P-04 + FT-P-05 CSV evidence; skips when either upstream CSV missing Tests: 325 unit tests pass (+77 vs batch 69). Reports: batch_70_report.md, batch_70_review.md (PASS). Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Oleksandr Bezdieniezhnykh
@@ -0,0 +1,209 @@
|
||||
# Batch 70 Report — Test Implementation (cycle 1, batch 4 of test phase)
|
||||
|
||||
**Batch**: 70
|
||||
**Date**: 2026-05-16
|
||||
**Context**: Test implementation (greenfield Step 10 — Implement Tests)
|
||||
**Tasks**: AZ-409 (3pt), AZ-412 (3pt), AZ-413 (3pt) — 9 cp / 3 tasks
|
||||
**Cycle**: 1
|
||||
**Verdict**: COMPLETE — PASS (self-reviewed; see `reviews/batch_70_review.md`)
|
||||
|
||||
## Summary
|
||||
|
||||
Three pure-positive scenarios on the same Derkachi + still-image fixtures
|
||||
that AZ-407 / AZ-408 set up. Each follows the now-established
|
||||
batch-69 pattern:
|
||||
|
||||
* A pure-logic helper module under `e2e/runner/helpers/` (everything the
|
||||
scenario needs except docker-bound replay + observation).
|
||||
* A scenario file under `e2e/tests/positive/` parameterized across
|
||||
`(fc_adapter, vio_strategy)` and skip-gated on upstream helper
|
||||
`NotImplementedError` (auto-activates when the harness lands).
|
||||
* A unit-test file under `e2e/_unit_tests/helpers/` that drives the
|
||||
helper directly with synthetic + real-fixture data.
|
||||
|
||||
### AZ-409 — FT-P-01 still-image frame-center accuracy (3pt)
|
||||
|
||||
* **`runner/helpers/accuracy_evaluator.py`** — `load_gt_coordinates`
|
||||
parses `_docs/00_problem/input_data/coordinates.csv`; `evaluate`
|
||||
joins by `image_id`, computes Vincenty geodesic distance via
|
||||
`geo.distance_m`, and produces per-image + aggregate reports. The
|
||||
three thresholds are exposed as module constants
|
||||
(`PASS_COUNT_50M_REQUIRED=48`, `PASS_COUNT_20M_REQUIRED=30`,
|
||||
`TOTAL_IMAGES_REQUIRED=60`) so a future spec change has exactly one
|
||||
place to flip. `AggregateReport.overall_pass` is the boolean the
|
||||
scenario asserts.
|
||||
* **`tests/positive/test_ft_p_01_still_image_accuracy.py`** — pytest
|
||||
scenario, gated on `frame_source_replay.replay_image_directory` +
|
||||
`sitl_observer.get_observer`. Pushes one image at a time with a 5 s
|
||||
per-image timeout; timeouts are recorded as `(inf, inf)` and propagate
|
||||
to `pass_50m=false`, `pass_20m=false`, `error_m=inf` per AC-4.
|
||||
* **20 unit tests** in `test_accuracy_evaluator.py`.
|
||||
|
||||
### AZ-412 — FT-P-04 Derkachi frame-to-frame registration ≥95 % (3pt)
|
||||
|
||||
* **`runner/helpers/registration_classifier.py`** — derives bank +
|
||||
pitch from SCALED_IMU2 accelerometer (spec-mandated; AC-1 prohibits
|
||||
internal SUT attitude). The classifier expands each 10 Hz IMU row
|
||||
into 3 video-frame indices (30 fps / 10 Hz = 3), classifies each
|
||||
frame as normal iff bank/pitch ∈ ±10° AND inferred prior-frame
|
||||
overlap ≥40 %, then exposes a `compute_success_ratio(classifications,
|
||||
registration_success_by_frame)` that returns a typed `SuccessReport`
|
||||
with `excluded_by_{attitude,overlap,missing_metric}` counts so AC-3
|
||||
diagnostics survive in the run report.
|
||||
* **Inferred-overlap heuristic** — translation = horizontal velocity ×
|
||||
(1/30 s); overlap = `1 - translation / ground_footprint_m` clamped to
|
||||
[0, 1]; default ground footprint = 147 m (derived from the camera_info.md
|
||||
~141 m altitude × 55° HFOV). The heuristic is explicitly an upper bound;
|
||||
the docstring records the assumption so a future calibration change has
|
||||
the tunable in one place.
|
||||
* **`tests/positive/test_ft_p_04_derkachi_f2f_registration.py`** —
|
||||
gated on `frame_source_replay`, `imu_replay`, `fdr_reader`. Reads
|
||||
per-frame `registration_success` from `frame_metric` FDR records;
|
||||
emits `ft-p-04-{fc_adapter}-{vio_strategy}.csv`; asserts AC-2.
|
||||
* **26 unit tests** in `test_registration_classifier.py` (including
|
||||
attitude round-trips for ±30° roll/pitch, the reproducibility check
|
||||
on the real first 100 IMU rows, and the boundary ratio cases).
|
||||
|
||||
### AZ-413 — FT-P-05 + FT-P-06 cross-domain MRE budgets (3pt)
|
||||
|
||||
* **`runner/helpers/mre_evaluator.py`** — three independent reports:
|
||||
`PerImageBudgetReport` (FT-P-05 AC-2: every MRE < 2.5 px, strict <),
|
||||
`P95Report` (single-domain p95 < budget), `CombinedP95Report` (FT-P-06
|
||||
AC-4: both domains pass). The 95th percentile uses
|
||||
`numpy.percentile(..., method='linear')` — exactly what the spec
|
||||
mandates. `load_frame_to_frame_csv` raises `ValueError` if the
|
||||
FT-P-04 CSV lacks an `mre_px` column (forces the failure to surface
|
||||
at the SUT-contract layer rather than silently passing).
|
||||
* **`tests/positive/test_ft_p_05_sat_anchor.py`** — gated scenario that
|
||||
pushes the 60 images, joins MRE with GT-error via
|
||||
`accuracy_evaluator.evaluate`, emits `ft-p-05.csv`, asserts AC-2 + AC-3.
|
||||
* **`tests/positive/test_ft_p_06_mre_budgets.py`** — pure piggyback that
|
||||
reads `ft-p-04-*.csv` + `ft-p-05-*.csv` from the same run and asserts
|
||||
AC-4. Skips (does NOT fail) if either upstream CSV is missing — that
|
||||
failure mode is the FT-P-04 / FT-P-05 scenario's responsibility.
|
||||
* **22 unit tests** in `test_mre_evaluator.py`.
|
||||
|
||||
## Files added / modified
|
||||
|
||||
### Added (9)
|
||||
|
||||
AZ-409:
|
||||
* `e2e/runner/helpers/accuracy_evaluator.py`
|
||||
* `e2e/tests/positive/test_ft_p_01_still_image_accuracy.py`
|
||||
* `e2e/_unit_tests/helpers/test_accuracy_evaluator.py`
|
||||
|
||||
AZ-412:
|
||||
* `e2e/runner/helpers/registration_classifier.py`
|
||||
* `e2e/tests/positive/test_ft_p_04_derkachi_f2f_registration.py`
|
||||
* `e2e/_unit_tests/helpers/test_registration_classifier.py`
|
||||
|
||||
AZ-413:
|
||||
* `e2e/runner/helpers/mre_evaluator.py`
|
||||
* `e2e/tests/positive/test_ft_p_05_sat_anchor.py`
|
||||
* `e2e/tests/positive/test_ft_p_06_mre_budgets.py`
|
||||
* `e2e/_unit_tests/helpers/test_mre_evaluator.py`
|
||||
|
||||
### Modified (2)
|
||||
|
||||
* `e2e/_unit_tests/test_directory_layout.py` — added 3 helper paths and
|
||||
4 scenario paths (the FT-P-01/04/05/06 scenarios; FT-P-02 + FT-P-03/14
|
||||
were added in batch 69).
|
||||
* `_docs/_autodev_state.md` — batch 70 pointer.
|
||||
|
||||
## Spec / module-layout drift notes
|
||||
|
||||
* **AZ-409 AC-5 says "four times" (the 4-variant matrix);** the conftest
|
||||
currently parameterises `(fc_adapter, vio_strategy)` as 2 × 3 = 6
|
||||
variants (`vins_mono` was added in AZ-406 alongside `okvis2` and
|
||||
`klt_ransac`). AC-5 reads "the conftest's `(fc_adapter, vio_strategy)`
|
||||
parameterization" first, with the 4-variant list as an example — so
|
||||
the conftest is authoritative. No code change needed; flagged here so
|
||||
the audit trail sees the discrepancy.
|
||||
* **AZ-412 / AZ-413 same observation** — both ACs say "per
|
||||
parameterization" without pinning a count; the conftest's 6-variant
|
||||
matrix is what runs.
|
||||
* **AZ-412 attitude convention** — the helper docstring records the
|
||||
Z-down + accel-decomposition assumption explicitly (the SCALED_IMU2
|
||||
wire format doesn't ship attitude). Roll/pitch ±30° round-trips are
|
||||
tested to confirm the decomposition.
|
||||
* **AZ-412 ground footprint** — default 147 m is derived from
|
||||
`camera_info.md` (~141 m alt, ~55° HFOV). Recorded as a module
|
||||
constant + classifier kwarg so a future re-calibration touches one
|
||||
place.
|
||||
* **AZ-413 strict `<` boundary** — AC-2 says "MRE < 2.5 px"; the helper
|
||||
uses `<` (not `≤`), and the unit test
|
||||
`test_evaluate_per_image_budget_single_fail_fails_overall` proves a
|
||||
2.5 px reading FAILS. Removes the boundary ambiguity.
|
||||
|
||||
## Test Results
|
||||
|
||||
### Focused tests (Step 6.4)
|
||||
|
||||
`pytest e2e/_unit_tests/` — **325 passed in 172.07s** (was 248 at end
|
||||
of batch 69; +77 new tests across this batch).
|
||||
|
||||
Breakdown of new tests:
|
||||
|
||||
* AZ-409 — 20 tests
|
||||
* AZ-412 — 26 tests
|
||||
* AZ-413 — 22 tests
|
||||
* AZ-409/412/413 directory_layout entries — 9 new parametrize cases
|
||||
|
||||
Scenario collection: 6 scenario files × parametrize matrix yields 42
|
||||
collected items in `e2e/tests/positive/` (all 4 new scenario files plus
|
||||
the 2 from batch 69). Every scenario file remains correctly skip-gated;
|
||||
no premature activation.
|
||||
|
||||
### No full-project pytest run
|
||||
|
||||
Per the implement skill's Test-Run Cadence, Step 16 owns the only
|
||||
full-project suite invocation; batches run focused tests only.
|
||||
|
||||
## AC Test Coverage
|
||||
|
||||
See `reviews/batch_70_review.md` for the per-AC traceability table. In
|
||||
summary: every unit-testable AC is covered; every runtime-only AC
|
||||
(end-to-end harness loop) is documented as gated and auto-activating
|
||||
when the upstream helpers land.
|
||||
|
||||
## Code Review Verdict
|
||||
|
||||
Self-reviewed — PASS. See `reviews/batch_70_review.md` for the full
|
||||
sweep (no Critical / High / Medium / Low findings).
|
||||
|
||||
## Auto-Fix Attempts
|
||||
|
||||
0. No code-review failures — auto-fix gate was not entered.
|
||||
|
||||
## Stuck Agents
|
||||
|
||||
None.
|
||||
|
||||
## Deferred follow-ups
|
||||
|
||||
Unchanged from batch 69 (same list, same owners):
|
||||
|
||||
* `runner.helpers.frame_source_replay.FrameSourceReplayer.{replay_video,
|
||||
replay_image_directory}` — owned by AZ-441.
|
||||
* `runner.helpers.fdr_reader.iter_records` — owned by AZ-441.
|
||||
* `runner.helpers.imu_replay.ImuReplayer.replay` — owned by AZ-407
|
||||
per scaffold docstring (not landed yet).
|
||||
* `runner.helpers.sitl_observer.get_observer` — owned by AZ-416 / AZ-417.
|
||||
* `runner.helpers.mavproxy_tlog_reader.iter_messages` — owned by AZ-416.
|
||||
|
||||
This batch did not introduce any new debt.
|
||||
|
||||
## Next Batch
|
||||
|
||||
Batch 71 candidate set (all are 3pt scenario tasks unblocked by this
|
||||
batch's helpers + existing AZ-407 fixtures):
|
||||
|
||||
* AZ-414 (FT-P-07 + FT-N-02 — sharp-turn behaviour)
|
||||
* AZ-415 (FT-P-08 — multi-segment relocalisation)
|
||||
* AZ-418 (FT-P-10 — smoothing lookback) — 3pt
|
||||
|
||||
Likely composition: ~9 cp across 3 tasks, same shape as batches 69–70.
|
||||
|
||||
The next milestone after batches 71–72 will be the K=3 cumulative
|
||||
review covering batches 70, 71, 72 (the current `last_cumulative_review`
|
||||
is `batches_67-69`).
|
||||
@@ -0,0 +1,131 @@
|
||||
# Code Review Report
|
||||
|
||||
**Batch**: 70 — AZ-409, AZ-412, AZ-413
|
||||
**Date**: 2026-05-16
|
||||
**Verdict**: PASS
|
||||
|
||||
## Findings
|
||||
|
||||
(none)
|
||||
|
||||
## Findings Sweep
|
||||
|
||||
### Phase 1 — Context Loading
|
||||
|
||||
Loaded specs `AZ-409_ft_p_01_still_image_accuracy.md`,
|
||||
`AZ-412_ft_p_04_derkachi_f2f_registration.md`,
|
||||
`AZ-413_ft_p_05_06_sat_anchor_mre.md`,
|
||||
`_docs/00_problem/input_data/expected_results/results_report.md`
|
||||
(authoritative Pass/Fail Rules), plus the existing `geo.py`,
|
||||
`anchor_pair_detector.py`, `estimate_schema.py` helpers for pattern
|
||||
re-use.
|
||||
|
||||
### Phase 2 — Spec Compliance
|
||||
|
||||
**AZ-409 (FT-P-01)**
|
||||
|
||||
| AC | Test | Status |
|
||||
|----|------|--------|
|
||||
| AC-1 (per-image distance computed) | `test_evaluate_all_pass_yields_overall_pass`, `test_evaluate_full_timeout_run_produces_zero_pass_counts` | Covered |
|
||||
| AC-2 (≥48/60 within 50 m) | `test_evaluate_boundary_threshold_holds`, `test_evaluate_below_50m_threshold_fails_overall` | Covered |
|
||||
| AC-3 (≥30/60 within 20 m) | `test_evaluate_boundary_threshold_holds`, `test_evaluate_below_20m_threshold_fails_overall` | Covered |
|
||||
| AC-4 (timeout discipline) | `test_compute_per_image_timeout_sets_inf_and_false_flags`, `test_evaluate_missing_estimate_recorded_as_timeout` | Covered |
|
||||
| AC-5 (parametrization 6 variants) | Verified via `pytest --collect-only` — 6 variants collected | Covered |
|
||||
| Runtime push-to-SITL end-to-end | gated by `_harness_helpers_implemented` on `frame_source_replay` + `sitl_observer` | NOT COVERED (harness-loop, same pattern as batch 69 AZ-410/AZ-411) |
|
||||
|
||||
**AZ-412 (FT-P-04)**
|
||||
|
||||
| AC | Test | Status |
|
||||
|----|------|--------|
|
||||
| AC-1 (classification reproducibility) | `test_classify_frames_is_reproducible_ac1` (uses real Derkachi data_imu.csv first 100 rows) | Covered |
|
||||
| AC-2 (success ratio ≥ 0.95) | `test_compute_success_ratio_perfect_run_passes`, `test_compute_success_ratio_at_95_pct_passes`, `test_compute_success_ratio_below_95_pct_fails` | Covered |
|
||||
| AC-3 (sharp-turn frames excluded from denominator) | `test_classify_frames_excludes_sharp_roll`, `test_compute_success_ratio_excludes_sharp_turn_from_denominator_ac3`, `test_compute_success_ratio_handles_missing_metric_separately` | Covered |
|
||||
| AC-4 (parametrization 6 variants) | Verified via `pytest --collect-only` | Covered |
|
||||
| Runtime full Derkachi replay | gated by `_harness_helpers_implemented` on `frame_source_replay`, `imu_replay`, `fdr_reader` | NOT COVERED (harness-loop) |
|
||||
|
||||
**AZ-413 (FT-P-05 + FT-P-06)**
|
||||
|
||||
| AC | Test | Status |
|
||||
|----|------|--------|
|
||||
| AC-1 (per-image MRE captured) | `test_evaluate_per_image_budget_all_pass` (covers the captured-list path); `test_write_cross_domain_csv_round_trip` (CSV column shape) | Covered |
|
||||
| AC-2 (cross-domain MRE < 2.5 px, all 60) | `test_evaluate_per_image_budget_single_fail_fails_overall`, `test_evaluate_per_image_budget_above_boundary_fails` (strict < 2.5 boundary explicitly tested) | Covered |
|
||||
| AC-3 (accuracy alongside MRE) | Delegated to `accuracy_evaluator` (already covered by AZ-409 tests); FT-P-05 scenario wires both via `evaluate()` | Covered by reuse |
|
||||
| AC-4 (95th-percentile budgets) | `test_evaluate_p95_uses_numpy_linear_interpolation`, `test_evaluate_combined_p95_both_pass`, `test_evaluate_combined_p95_fails_when_frame_to_frame_fails`, `test_evaluate_combined_p95_fails_when_cross_domain_fails` | Covered |
|
||||
| AC-5 (parametrization 6 variants per scenario file) | Verified via `pytest --collect-only` — 12 items between FT-P-05 (6) + FT-P-06 (6) | Covered |
|
||||
| Runtime push-to-SITL end-to-end | gated by `_harness_helpers_implemented` on `frame_source_replay`, `sitl_observer`, `fdr_reader` | NOT COVERED (harness-loop) |
|
||||
|
||||
No Spec-Gap findings.
|
||||
|
||||
### Phase 3 — Code Quality
|
||||
|
||||
- **SRP** respected per task:
|
||||
- `accuracy_evaluator` owns geodesic distance + pass-count rules only.
|
||||
- `registration_classifier` owns attitude derivation + overlap heuristic + success ratio only.
|
||||
- `mre_evaluator` owns per-image budget + p95 budget only.
|
||||
- **Error handling** consistent: every loader raises `FileNotFoundError` on missing input and `ValueError` on header/column drift (matches the AZ-410 / AZ-411 helper pattern).
|
||||
- **Naming**: dataclass + function names follow the project's snake_case / CamelCase convention.
|
||||
- **Complexity**: longest function is `classify_frames` at ~50 lines (linear pipeline). All others under 30.
|
||||
- **Tests assert behaviour**, not just "no exception": geodesic round-trips against real distances, boundary conditions (exactly 48/60, exactly 0.95 ratio, exactly 2.5 px) are explicitly tested.
|
||||
- **Spec drift guard**: each helper has a `test_constants_match_spec` test that fails if the public constants drift from the AC text (catches a renamer that touches code but forgets the spec).
|
||||
- **Boundary strictness**: AC-2 of FT-P-05 says "MRE < 2.5 px"; the helper uses strict `<` and the test `test_evaluate_per_image_budget_single_fail_fails_overall` proves a 2.5 px reading FAILS. This is the kind of boundary the spec would otherwise be ambiguous on.
|
||||
|
||||
### Phase 4 — Security
|
||||
|
||||
No SQL, no subprocess, no credentials. CSV loaders validate header columns explicitly; numeric coercion via `float()` / `int()` raises on garbage input.
|
||||
|
||||
### Phase 5 — Performance
|
||||
|
||||
- All three helpers operate on per-flight-sized data (60 images, ≤14700 frames, ≤4900 IMU rows). Pure-Python loops are fine.
|
||||
- `mre_evaluator.evaluate_p95` uses `numpy.percentile` (vectorised).
|
||||
- No new I/O patterns beyond CSV read/write.
|
||||
|
||||
### Phase 6 — Cross-Task Consistency
|
||||
|
||||
- **API stability**: the three new helpers share the same shape pattern as AZ-410's `anchor_pair_detector` and AZ-411's `estimate_schema` — typed `@dataclass(frozen=True)` records, a `load_…` reader, an `evaluate(…)` / `compute_…` core, a `write_csv_evidence` emitter. The FT-P-05 scenario reuses `accuracy_evaluator.evaluate()` (AZ-409) to compute per-image error_m → demonstrates the cross-task consistency in action.
|
||||
- **No duplicate symbols across batches**: each helper module owns disjoint public names; the only shared dependency is `runner.helpers.geo.distance_m`.
|
||||
- **Scenario-file skip pattern**: all 4 new scenario files (`test_ft_p_01_*`, `test_ft_p_04_*`, `test_ft_p_05_*`, `test_ft_p_06_*`) reuse the `_harness_helpers_implemented` gate pattern from batch 69. Consistent.
|
||||
- **Within-batch dep (AZ-413 → AZ-412)**: FT-P-06 reads FT-P-04's CSV (the f2f MRE column). The mre_evaluator's `load_frame_to_frame_csv` explicitly validates that the `mre_px` column is present; if absent (FT-P-04 evidence not yet carrying MRE), FT-P-06 fails with a clear message pointing at the SUT contract (AC-NEW-3 FDR schema). This is the safest failure mode for an inter-task dep.
|
||||
|
||||
### Phase 7 — Architecture Compliance
|
||||
|
||||
1. **Layer direction**: every new file under `e2e/**`. The `test_no_sut_imports.py` invariant (passes after the run) confirms zero `gps_denied_onboard` imports across all 14 new files.
|
||||
2. **Public API respect**: only public names imported across modules (`runner.helpers.{geo,accuracy_evaluator,mre_evaluator}` etc.). No leading-underscore cross-module imports.
|
||||
3. **No new cyclic dependencies**: import graph:
|
||||
- `accuracy_evaluator` → `geo`
|
||||
- `registration_classifier` → (none)
|
||||
- `mre_evaluator` → (numpy + stdlib)
|
||||
- `tests.positive.test_ft_p_01_*` → `accuracy_evaluator`
|
||||
- `tests.positive.test_ft_p_04_*` → `registration_classifier`
|
||||
- `tests.positive.test_ft_p_05_*` → `accuracy_evaluator` + `mre_evaluator`
|
||||
- `tests.positive.test_ft_p_06_*` → `mre_evaluator`
|
||||
Linear DAG.
|
||||
4. **Duplicate symbols across components**: none.
|
||||
5. **Cross-cutting concerns**: pytest plugin registration unchanged from batch 69 (the new helpers don't need a plugin — they're called from scenario test bodies).
|
||||
|
||||
No Architecture findings.
|
||||
|
||||
Baseline delta section omitted (no `architecture_compliance_baseline.md` for this project).
|
||||
|
||||
## AC Test Coverage Summary
|
||||
|
||||
| Task | ACs Covered (unit) | NOT COVERED (harness-loop) | Test File |
|
||||
|------|---------------------|----------------------------|-----------|
|
||||
| AZ-409 | 1, 2, 3, 4, 5 | Runtime push-to-SITL end-to-end | `test_accuracy_evaluator.py` (20 tests) |
|
||||
| AZ-412 | 1, 2, 3, 4 | Runtime full Derkachi replay | `test_registration_classifier.py` (26 tests) |
|
||||
| AZ-413 | 1, 2, 3, 4, 5 | Runtime push-to-SITL end-to-end | `test_mre_evaluator.py` (22 tests) |
|
||||
|
||||
## Verdict: PASS
|
||||
|
||||
No Critical, High, Medium, or Low findings. Unit-test layer is complete
|
||||
and consistent across the three tasks; runtime end-to-end paths are
|
||||
correctly gated and documented as hardware-loop ACs pending the upstream
|
||||
`frame_source_replay` / `sitl_observer` / `fdr_reader` / `imu_replay`
|
||||
helpers landing.
|
||||
|
||||
## Auto-Fix Attempts: 0
|
||||
|
||||
No failures — auto-fix gate not entered.
|
||||
|
||||
## Stuck Agents: 0
|
||||
|
||||
None.
|
||||
@@ -12,7 +12,7 @@ sub_step:
|
||||
retry_count: 0
|
||||
cycle: 1
|
||||
tracker: jira
|
||||
last_completed_batch: 69
|
||||
last_completed_batch: 70
|
||||
last_cumulative_review: batches_67-69
|
||||
last_step_outcomes:
|
||||
step_8: "Code is testable — no changes needed (testability_assessment.md committed; no list-of-changes, no source edits)"
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
"""Unit tests for ``runner.helpers.accuracy_evaluator`` (FT-P-01 / AZ-409).
|
||||
|
||||
Covers AC-1 (per-image evaluation), AC-2 (50 m pass-count threshold ≥48),
|
||||
AC-3 (20 m pass-count threshold ≥30), AC-4 (timeout discipline) and the
|
||||
CSV evidence shape.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers.accuracy_evaluator import (
|
||||
PASS_COUNT_20M_REQUIRED,
|
||||
PASS_COUNT_50M_REQUIRED,
|
||||
TOTAL_IMAGES_REQUIRED,
|
||||
AggregateReport,
|
||||
EstimateInput,
|
||||
GtCoordinate,
|
||||
PerImageResult,
|
||||
compute_per_image,
|
||||
evaluate,
|
||||
load_gt_coordinates,
|
||||
write_csv_evidence,
|
||||
)
|
||||
from runner.helpers.geo import distance_m, offset
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
GT_CSV = REPO_ROOT / "_docs" / "00_problem" / "input_data" / "coordinates.csv"
|
||||
|
||||
|
||||
def test_load_gt_coordinates_parses_repo_csv() -> None:
|
||||
"""The shipped ``coordinates.csv`` must parse cleanly into 60 rows."""
|
||||
# Act
|
||||
rows = load_gt_coordinates(GT_CSV)
|
||||
|
||||
# Assert
|
||||
assert len(rows) == TOTAL_IMAGES_REQUIRED
|
||||
assert rows[0].image_id == "AD000001.jpg"
|
||||
assert rows[0].lat_deg == pytest.approx(48.275292, abs=1e-6)
|
||||
assert rows[0].lon_deg == pytest.approx(37.385220, abs=1e-6)
|
||||
assert rows[-1].image_id == "AD000060.jpg"
|
||||
|
||||
|
||||
def test_load_gt_coordinates_rejects_missing_file(tmp_path: Path) -> None:
|
||||
"""Explicit FileNotFoundError, not a silent empty list."""
|
||||
# Act / Assert
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_gt_coordinates(tmp_path / "missing.csv")
|
||||
|
||||
|
||||
def test_load_gt_coordinates_rejects_wrong_header(tmp_path: Path) -> None:
|
||||
# Arrange
|
||||
bad = tmp_path / "bad.csv"
|
||||
bad.write_text("img_name,latitude,longitude\nx,1,2\n")
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="header mismatch"):
|
||||
load_gt_coordinates(bad)
|
||||
|
||||
|
||||
def test_compute_per_image_zero_error_for_exact_match() -> None:
|
||||
"""Exact GT → estimate match yields error_m ≈ 0 and both pass flags True."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275292, 37.385220)
|
||||
est = EstimateInput("AD000001.jpg", 48.275292, 37.385220)
|
||||
|
||||
# Act
|
||||
result = compute_per_image(gt, est)
|
||||
|
||||
# Assert
|
||||
assert result.error_m == pytest.approx(0.0, abs=1e-6)
|
||||
assert result.pass_50m is True
|
||||
assert result.pass_20m is True
|
||||
|
||||
|
||||
def test_compute_per_image_15m_north_passes_both() -> None:
|
||||
"""15 m north of GT — below both 50 m and 20 m budgets."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275292, 37.385220)
|
||||
new_lat, new_lon = offset(gt.lat_deg, gt.lon_deg, bearing_deg=0.0, distance_m=15.0)
|
||||
est = EstimateInput("AD000001.jpg", new_lat, new_lon)
|
||||
|
||||
# Act
|
||||
result = compute_per_image(gt, est)
|
||||
|
||||
# Assert
|
||||
assert result.error_m == pytest.approx(15.0, abs=0.5)
|
||||
assert result.pass_50m is True
|
||||
assert result.pass_20m is True
|
||||
|
||||
|
||||
def test_compute_per_image_35m_east_passes_50_only() -> None:
|
||||
"""35 m east of GT — passes 50 m budget, fails 20 m budget."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275292, 37.385220)
|
||||
new_lat, new_lon = offset(gt.lat_deg, gt.lon_deg, bearing_deg=90.0, distance_m=35.0)
|
||||
est = EstimateInput("AD000001.jpg", new_lat, new_lon)
|
||||
|
||||
# Act
|
||||
result = compute_per_image(gt, est)
|
||||
|
||||
# Assert
|
||||
assert result.error_m == pytest.approx(35.0, abs=0.5)
|
||||
assert result.pass_50m is True
|
||||
assert result.pass_20m is False
|
||||
|
||||
|
||||
def test_compute_per_image_120m_south_fails_both() -> None:
|
||||
"""120 m south of GT — fails both budgets."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275292, 37.385220)
|
||||
new_lat, new_lon = offset(gt.lat_deg, gt.lon_deg, bearing_deg=180.0, distance_m=120.0)
|
||||
est = EstimateInput("AD000001.jpg", new_lat, new_lon)
|
||||
|
||||
# Act
|
||||
result = compute_per_image(gt, est)
|
||||
|
||||
# Assert
|
||||
assert result.error_m == pytest.approx(120.0, abs=0.5)
|
||||
assert result.pass_50m is False
|
||||
assert result.pass_20m is False
|
||||
|
||||
|
||||
def test_compute_per_image_timeout_sets_inf_and_false_flags() -> None:
|
||||
"""AC-4: inf estimate → error_m = inf, both flags False; no crash."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275292, 37.385220)
|
||||
est = EstimateInput("AD000001.jpg", math.inf, math.inf)
|
||||
|
||||
# Act
|
||||
result = compute_per_image(gt, est)
|
||||
|
||||
# Assert
|
||||
assert math.isinf(result.error_m)
|
||||
assert result.pass_50m is False
|
||||
assert result.pass_20m is False
|
||||
|
||||
|
||||
def test_compute_per_image_rejects_image_id_mismatch() -> None:
|
||||
"""compute_per_image refuses to silently join across image_ids."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.0, 37.0)
|
||||
est = EstimateInput("AD000002.jpg", 48.0, 37.0)
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="image_id mismatch"):
|
||||
compute_per_image(gt, est)
|
||||
|
||||
|
||||
def _make_gt_with_offsets(offsets_m: list[float]) -> tuple[list[GtCoordinate], list[EstimateInput]]:
|
||||
"""Build GT + estimates: each estimate is `offsets_m[i]` meters north of GT."""
|
||||
base_lat, base_lon = 48.275, 37.385
|
||||
gt_rows: list[GtCoordinate] = []
|
||||
estimates: list[EstimateInput] = []
|
||||
for i, off in enumerate(offsets_m, start=1):
|
||||
image_id = f"AD{i:06d}.jpg"
|
||||
gt_lat = base_lat + i * 1e-4
|
||||
gt_lon = base_lon
|
||||
gt_rows.append(GtCoordinate(image_id, gt_lat, gt_lon))
|
||||
est_lat, est_lon = offset(gt_lat, gt_lon, bearing_deg=0.0, distance_m=off)
|
||||
estimates.append(EstimateInput(image_id, est_lat, est_lon))
|
||||
return gt_rows, estimates
|
||||
|
||||
|
||||
def test_evaluate_all_pass_yields_overall_pass() -> None:
|
||||
"""60 images all <20 m: AC-2 + AC-3 both pass."""
|
||||
# Arrange
|
||||
offsets = [5.0] * TOTAL_IMAGES_REQUIRED
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
|
||||
# Act
|
||||
results, aggregate = evaluate(gt_rows, estimates)
|
||||
|
||||
# Assert
|
||||
assert len(results) == TOTAL_IMAGES_REQUIRED
|
||||
assert aggregate.pass_count_50m == 60
|
||||
assert aggregate.pass_count_20m == 60
|
||||
assert aggregate.timeout_count == 0
|
||||
assert aggregate.overall_pass is True
|
||||
|
||||
|
||||
def test_evaluate_boundary_threshold_holds() -> None:
|
||||
"""Exactly 48 within 50 m + 30 within 20 m → overall_pass = True."""
|
||||
# Arrange — 30 images at 10m (pass both), 18 images at 35m (pass 50 only),
|
||||
# 12 images at 120m (fail both).
|
||||
offsets = [10.0] * 30 + [35.0] * 18 + [120.0] * 12
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
|
||||
# Act
|
||||
_, aggregate = evaluate(gt_rows, estimates)
|
||||
|
||||
# Assert
|
||||
assert aggregate.pass_count_50m == 48
|
||||
assert aggregate.pass_count_20m == 30
|
||||
assert aggregate.pass_ac2 is True
|
||||
assert aggregate.pass_ac3 is True
|
||||
assert aggregate.overall_pass is True
|
||||
|
||||
|
||||
def test_evaluate_below_50m_threshold_fails_overall() -> None:
|
||||
"""47/60 within 50 m → AC-2 fails → overall_pass False."""
|
||||
# Arrange — 30 at 10m, 17 at 35m (47 within 50m), 13 at 120m.
|
||||
offsets = [10.0] * 30 + [35.0] * 17 + [120.0] * 13
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
|
||||
# Act
|
||||
_, aggregate = evaluate(gt_rows, estimates)
|
||||
|
||||
# Assert
|
||||
assert aggregate.pass_count_50m == 47
|
||||
assert aggregate.pass_ac2 is False
|
||||
assert aggregate.overall_pass is False
|
||||
|
||||
|
||||
def test_evaluate_below_20m_threshold_fails_overall() -> None:
|
||||
"""All 60 within 50 m but only 29 within 20 m → AC-3 fails."""
|
||||
# Arrange
|
||||
offsets = [10.0] * 29 + [35.0] * 31
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
|
||||
# Act
|
||||
_, aggregate = evaluate(gt_rows, estimates)
|
||||
|
||||
# Assert
|
||||
assert aggregate.pass_count_50m == 60
|
||||
assert aggregate.pass_count_20m == 29
|
||||
assert aggregate.pass_ac3 is False
|
||||
assert aggregate.overall_pass is False
|
||||
|
||||
|
||||
def test_evaluate_missing_estimate_recorded_as_timeout() -> None:
|
||||
"""GT row without estimate → timeout (inf, both False) and aggregate counts it."""
|
||||
# Arrange
|
||||
offsets = [5.0] * TOTAL_IMAGES_REQUIRED
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
# Drop the 7th estimate to simulate a SITL timeout for AD000007.jpg.
|
||||
dropped_index = 6
|
||||
estimates_with_gap = [e for i, e in enumerate(estimates) if i != dropped_index]
|
||||
|
||||
# Act
|
||||
results, aggregate = evaluate(gt_rows, estimates_with_gap)
|
||||
|
||||
# Assert
|
||||
assert len(results) == TOTAL_IMAGES_REQUIRED
|
||||
assert aggregate.timeout_count == 1
|
||||
assert results[dropped_index].image_id == "AD000007.jpg"
|
||||
assert math.isinf(results[dropped_index].error_m)
|
||||
assert results[dropped_index].pass_50m is False
|
||||
|
||||
|
||||
def test_evaluate_rejects_duplicate_estimate_image_id() -> None:
|
||||
"""Two estimates for the same image_id → ValueError (programming error)."""
|
||||
# Arrange
|
||||
offsets = [5.0] * 2
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
duplicate = EstimateInput(estimates[0].image_id, estimates[0].est_lat_deg, estimates[0].est_lon_deg)
|
||||
estimates.append(duplicate)
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="duplicate estimate image_ids"):
|
||||
evaluate(gt_rows, estimates)
|
||||
|
||||
|
||||
def test_evaluate_rejects_stranger_estimate_image_id() -> None:
|
||||
"""Estimate for an image not in GT → ValueError (programming error)."""
|
||||
# Arrange
|
||||
offsets = [5.0] * 2
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
estimates.append(EstimateInput("AD999999.jpg", 48.0, 37.0))
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="not in GT"):
|
||||
evaluate(gt_rows, estimates)
|
||||
|
||||
|
||||
def test_evaluate_full_timeout_run_produces_zero_pass_counts() -> None:
|
||||
"""All 60 timed out → pass counts 0, overall_pass False."""
|
||||
# Arrange
|
||||
gt_rows = [GtCoordinate(f"AD{i:06d}.jpg", 48.275 + i * 1e-4, 37.385) for i in range(1, 61)]
|
||||
estimates: list[EstimateInput] = []
|
||||
|
||||
# Act
|
||||
results, aggregate = evaluate(gt_rows, estimates)
|
||||
|
||||
# Assert
|
||||
assert aggregate.timeout_count == 60
|
||||
assert aggregate.pass_count_50m == 0
|
||||
assert aggregate.pass_count_20m == 0
|
||||
assert aggregate.overall_pass is False
|
||||
assert all(math.isinf(r.error_m) for r in results)
|
||||
|
||||
|
||||
def test_aggregate_report_thresholds_match_results_report() -> None:
|
||||
"""The thresholds in code must match results_report.md (48 / 30 / 60)."""
|
||||
# Assert
|
||||
assert PASS_COUNT_50M_REQUIRED == 48
|
||||
assert PASS_COUNT_20M_REQUIRED == 30
|
||||
assert TOTAL_IMAGES_REQUIRED == 60
|
||||
|
||||
|
||||
def test_write_csv_evidence_round_trip(tmp_path: Path) -> None:
|
||||
"""CSV row count + header + numeric round-trip on the evidence file."""
|
||||
# Arrange
|
||||
offsets = [5.0, 35.0, 120.0]
|
||||
gt_rows, estimates = _make_gt_with_offsets(offsets)
|
||||
results, _ = evaluate(gt_rows, estimates)
|
||||
out_path = tmp_path / "ft-p-01.csv"
|
||||
|
||||
# Act
|
||||
written = write_csv_evidence(out_path, results)
|
||||
|
||||
# Assert
|
||||
assert written == out_path
|
||||
rows = list(csv.reader(out_path.open()))
|
||||
assert rows[0] == [
|
||||
"image_id",
|
||||
"gt_lat",
|
||||
"gt_lon",
|
||||
"est_lat",
|
||||
"est_lon",
|
||||
"error_m",
|
||||
"pass_50m",
|
||||
"pass_20m",
|
||||
]
|
||||
assert len(rows) == 1 + len(offsets)
|
||||
# AD000003 had a 120 m offset → pass_50m=false, pass_20m=false
|
||||
far_row = rows[3]
|
||||
assert far_row[0] == "AD000003.jpg"
|
||||
assert far_row[6] == "false"
|
||||
assert far_row[7] == "false"
|
||||
|
||||
|
||||
def test_write_csv_evidence_serializes_timeout_as_inf(tmp_path: Path) -> None:
|
||||
"""Timeout rows are written with the literal 'inf' for est_lat/est_lon/error_m."""
|
||||
# Arrange
|
||||
gt = GtCoordinate("AD000001.jpg", 48.275, 37.385)
|
||||
timeout = PerImageResult(
|
||||
image_id="AD000001.jpg",
|
||||
gt_lat=gt.lat_deg,
|
||||
gt_lon=gt.lon_deg,
|
||||
est_lat=math.inf,
|
||||
est_lon=math.inf,
|
||||
error_m=math.inf,
|
||||
pass_50m=False,
|
||||
pass_20m=False,
|
||||
)
|
||||
out_path = tmp_path / "ft-p-01.csv"
|
||||
|
||||
# Act
|
||||
write_csv_evidence(out_path, [timeout])
|
||||
|
||||
# Assert
|
||||
rows = list(csv.reader(out_path.open()))
|
||||
assert rows[1][3] == "inf"
|
||||
assert rows[1][4] == "inf"
|
||||
assert rows[1][5] == "inf"
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Unit tests for ``runner.helpers.mre_evaluator`` (FT-P-05 + FT-P-06 / AZ-413).
|
||||
|
||||
Covers AC-2 of FT-P-05 (every cross-domain MRE < 2.5 px), AC-3 of FT-P-05
|
||||
(accuracy alongside MRE — delegated to ``accuracy_evaluator``), and AC-4
|
||||
of FT-P-06 (95th-percentile MRE budgets per domain).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from runner.helpers.mre_evaluator import (
|
||||
MRE_P95_CROSS_DOMAIN_BUDGET_PX,
|
||||
MRE_P95_FRAME_TO_FRAME_BUDGET_PX,
|
||||
MRE_PER_IMAGE_BUDGET_PX,
|
||||
CombinedP95Report,
|
||||
CrossDomainRecord,
|
||||
FrameToFrameRecord,
|
||||
PerImageBudgetReport,
|
||||
P95Report,
|
||||
evaluate_combined_p95,
|
||||
evaluate_p95,
|
||||
evaluate_per_image_budget,
|
||||
load_cross_domain_csv,
|
||||
load_frame_to_frame_csv,
|
||||
summarize_mre_distribution,
|
||||
write_cross_domain_csv,
|
||||
)
|
||||
|
||||
|
||||
def test_constants_match_spec() -> None:
|
||||
"""The three budgets must match the AC text."""
|
||||
# Assert
|
||||
assert MRE_PER_IMAGE_BUDGET_PX == 2.5
|
||||
assert MRE_P95_FRAME_TO_FRAME_BUDGET_PX == 1.0
|
||||
assert MRE_P95_CROSS_DOMAIN_BUDGET_PX == 2.5
|
||||
|
||||
|
||||
def test_evaluate_per_image_budget_all_pass() -> None:
|
||||
"""All MREs under 2.5 → AC-2 passes."""
|
||||
# Arrange
|
||||
records = [CrossDomainRecord(f"AD{i:06d}.jpg", mre_px=1.5, error_m=10.0) for i in range(60)]
|
||||
|
||||
# Act
|
||||
report = evaluate_per_image_budget(records)
|
||||
|
||||
# Assert
|
||||
assert report.total_images == 60
|
||||
assert report.pass_count == 60
|
||||
assert report.fail_image_ids == ()
|
||||
assert report.max_mre_px == 1.5
|
||||
assert report.passes is True
|
||||
|
||||
|
||||
def test_evaluate_per_image_budget_single_fail_fails_overall() -> None:
|
||||
"""One MRE at the boundary → fails (strict < 2.5)."""
|
||||
# Arrange — 59 pass, 1 at exactly 2.5
|
||||
records = [CrossDomainRecord(f"AD{i:06d}.jpg", mre_px=1.0, error_m=5.0) for i in range(59)]
|
||||
records.append(CrossDomainRecord("AD000060.jpg", mre_px=2.5, error_m=5.0))
|
||||
|
||||
# Act
|
||||
report = evaluate_per_image_budget(records)
|
||||
|
||||
# Assert
|
||||
assert report.pass_count == 59
|
||||
assert report.fail_image_ids == ("AD000060.jpg",)
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_evaluate_per_image_budget_above_boundary_fails() -> None:
|
||||
"""An MRE strictly above 2.5 fails."""
|
||||
# Arrange
|
||||
records = [
|
||||
CrossDomainRecord("a", mre_px=1.0, error_m=5.0),
|
||||
CrossDomainRecord("b", mre_px=3.0, error_m=15.0),
|
||||
]
|
||||
|
||||
# Act
|
||||
report = evaluate_per_image_budget(records)
|
||||
|
||||
# Assert
|
||||
assert report.fail_image_ids == ("b",)
|
||||
assert report.passes is False
|
||||
assert report.max_mre_px == 3.0
|
||||
|
||||
|
||||
def test_evaluate_per_image_budget_empty_list_does_not_pass() -> None:
|
||||
"""Zero records → does NOT pass (no positive evidence of compliance)."""
|
||||
# Act
|
||||
report = evaluate_per_image_budget([])
|
||||
|
||||
# Assert
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_evaluate_per_image_budget_rejects_zero_budget() -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="budget_px must be > 0"):
|
||||
evaluate_per_image_budget([], budget_px=0.0)
|
||||
|
||||
|
||||
def test_evaluate_p95_uses_numpy_linear_interpolation() -> None:
|
||||
"""Spec mandates numpy's default percentile algorithm; verify match."""
|
||||
# Arrange — 20 samples uniformly from 0.1 to 2.0.
|
||||
samples = [round(0.1 * i, 2) for i in range(1, 21)]
|
||||
expected_p95 = float(np.percentile(np.asarray(samples, dtype=float), 95))
|
||||
|
||||
# Act
|
||||
report = evaluate_p95(samples, budget_px=2.5)
|
||||
|
||||
# Assert
|
||||
assert report.sample_count == 20
|
||||
assert report.p95_px == pytest.approx(expected_p95)
|
||||
assert report.passes is True
|
||||
|
||||
|
||||
def test_evaluate_p95_passes_when_below_budget() -> None:
|
||||
"""p95 < 1.0 → passes for the frame-to-frame budget."""
|
||||
# Arrange — 100 samples mostly below 1.0
|
||||
samples = [0.5] * 95 + [0.9] * 5 # p95 = 0.5 (linear interp)
|
||||
|
||||
# Act
|
||||
report = evaluate_p95(samples, budget_px=MRE_P95_FRAME_TO_FRAME_BUDGET_PX)
|
||||
|
||||
# Assert
|
||||
assert report.passes is True
|
||||
|
||||
|
||||
def test_evaluate_p95_fails_when_above_budget() -> None:
|
||||
"""p95 ≥ 1.0 → fails."""
|
||||
# Arrange
|
||||
samples = [0.5] * 90 + [1.5] * 10 # p95 ≈ 1.5
|
||||
|
||||
# Act
|
||||
report = evaluate_p95(samples, budget_px=MRE_P95_FRAME_TO_FRAME_BUDGET_PX)
|
||||
|
||||
# Assert
|
||||
assert report.passes is False
|
||||
assert report.p95_px == pytest.approx(1.5, abs=1e-6)
|
||||
|
||||
|
||||
def test_evaluate_p95_empty_input_does_not_pass() -> None:
|
||||
"""Zero samples → NaN p95, does not pass."""
|
||||
# Act
|
||||
report = evaluate_p95([], budget_px=2.5)
|
||||
|
||||
# Assert
|
||||
assert report.sample_count == 0
|
||||
assert math.isnan(report.p95_px)
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_evaluate_p95_rejects_zero_budget() -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="budget_px must be > 0"):
|
||||
evaluate_p95([1.0], budget_px=0.0)
|
||||
|
||||
|
||||
def test_evaluate_combined_p95_both_pass() -> None:
|
||||
"""Both domains below their budgets → combined report passes."""
|
||||
# Arrange
|
||||
f2f = [FrameToFrameRecord(frame_index=i, mre_px=0.4) for i in range(100)]
|
||||
xd = [CrossDomainRecord(f"AD{i:06d}.jpg", mre_px=1.0, error_m=5.0) for i in range(60)]
|
||||
|
||||
# Act
|
||||
report = evaluate_combined_p95(f2f, xd)
|
||||
|
||||
# Assert
|
||||
assert report.frame_to_frame.passes is True
|
||||
assert report.cross_domain.passes is True
|
||||
assert report.passes is True
|
||||
|
||||
|
||||
def test_evaluate_combined_p95_fails_when_frame_to_frame_fails() -> None:
|
||||
"""f2f p95 ≥ 1.0 → combined fails even if cross-domain passes."""
|
||||
# Arrange — f2f p95 ≈ 1.5, cross-domain p95 ≈ 1.0
|
||||
f2f = [FrameToFrameRecord(frame_index=i, mre_px=0.5) for i in range(90)] + [
|
||||
FrameToFrameRecord(frame_index=i, mre_px=1.5) for i in range(90, 100)
|
||||
]
|
||||
xd = [CrossDomainRecord(f"a{i}", mre_px=1.0, error_m=5.0) for i in range(60)]
|
||||
|
||||
# Act
|
||||
report = evaluate_combined_p95(f2f, xd)
|
||||
|
||||
# Assert
|
||||
assert report.frame_to_frame.passes is False
|
||||
assert report.cross_domain.passes is True
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_evaluate_combined_p95_fails_when_cross_domain_fails() -> None:
|
||||
"""cross-domain p95 ≥ 2.5 → combined fails even if f2f passes."""
|
||||
# Arrange
|
||||
f2f = [FrameToFrameRecord(frame_index=i, mre_px=0.5) for i in range(100)]
|
||||
xd = [CrossDomainRecord(f"a{i}", mre_px=1.0, error_m=5.0) for i in range(54)] + [
|
||||
CrossDomainRecord(f"b{i}", mre_px=3.0, error_m=5.0) for i in range(6)
|
||||
]
|
||||
|
||||
# Act
|
||||
report = evaluate_combined_p95(f2f, xd)
|
||||
|
||||
# Assert
|
||||
assert report.cross_domain.passes is False
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_write_cross_domain_csv_round_trip(tmp_path: Path) -> None:
|
||||
"""write + read returns the same records."""
|
||||
# Arrange
|
||||
records = [
|
||||
CrossDomainRecord("AD000001.jpg", mre_px=1.234, error_m=12.345),
|
||||
CrossDomainRecord("AD000002.jpg", mre_px=2.6, error_m=200.0),
|
||||
]
|
||||
out = tmp_path / "ft-p-05.csv"
|
||||
|
||||
# Act
|
||||
write_cross_domain_csv(out, records)
|
||||
loaded = load_cross_domain_csv(out)
|
||||
|
||||
# Assert
|
||||
assert len(loaded) == 2
|
||||
assert loaded[0].image_id == "AD000001.jpg"
|
||||
assert loaded[0].mre_px == pytest.approx(1.234, abs=1e-3)
|
||||
assert loaded[1].mre_px == pytest.approx(2.6, abs=1e-3)
|
||||
|
||||
|
||||
def test_write_cross_domain_csv_emits_pass_mre_column(tmp_path: Path) -> None:
|
||||
"""Each row's pass_mre cell reflects the < 2.5 strict comparison."""
|
||||
# Arrange
|
||||
records = [
|
||||
CrossDomainRecord("a", mre_px=1.0, error_m=5.0),
|
||||
CrossDomainRecord("b", mre_px=2.5, error_m=5.0),
|
||||
CrossDomainRecord("c", mre_px=2.499, error_m=5.0),
|
||||
]
|
||||
out = tmp_path / "ft-p-05.csv"
|
||||
|
||||
# Act
|
||||
write_cross_domain_csv(out, records)
|
||||
rows = list(csv.reader(out.open()))
|
||||
|
||||
# Assert
|
||||
assert rows[1][7] == "true" # a (1.0 px)
|
||||
assert rows[2][7] == "false" # b (2.5 px — strict <)
|
||||
assert rows[3][7] == "true" # c (2.499 px)
|
||||
|
||||
|
||||
def test_load_cross_domain_csv_rejects_missing_file(tmp_path: Path) -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_cross_domain_csv(tmp_path / "missing.csv")
|
||||
|
||||
|
||||
def test_load_cross_domain_csv_rejects_missing_columns(tmp_path: Path) -> None:
|
||||
# Arrange
|
||||
bad = tmp_path / "bad.csv"
|
||||
bad.write_text("image_id,mre_px\nx,1.0\n")
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="missing columns"):
|
||||
load_cross_domain_csv(bad)
|
||||
|
||||
|
||||
def test_load_frame_to_frame_csv_rejects_missing_mre_column(tmp_path: Path) -> None:
|
||||
"""If FT-P-04 evidence lacks mre_px, FT-P-06 must fail loudly."""
|
||||
# Arrange
|
||||
bad = tmp_path / "ft-p-04.csv"
|
||||
bad.write_text(
|
||||
"frame_index,imu_row_index,bank_deg,pitch_deg,translation_m,overlap_fraction,is_normal,excluded_reason,registration_success\n"
|
||||
"0,0,0.0,0.0,0.0,1.0,true,,true\n"
|
||||
)
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="mre_px"):
|
||||
load_frame_to_frame_csv(bad)
|
||||
|
||||
|
||||
def test_load_frame_to_frame_csv_round_trip(tmp_path: Path) -> None:
|
||||
"""When mre_px is present, records parse correctly."""
|
||||
# Arrange
|
||||
good = tmp_path / "ft-p-04.csv"
|
||||
good.write_text(
|
||||
"frame_index,mre_px\n0,0.5\n1,0.7\n2,\n3,1.1\n"
|
||||
)
|
||||
|
||||
# Act
|
||||
records = load_frame_to_frame_csv(good)
|
||||
|
||||
# Assert — blank mre_px rows are skipped.
|
||||
assert [r.frame_index for r in records] == [0, 1, 3]
|
||||
assert records[0].mre_px == 0.5
|
||||
|
||||
|
||||
def test_summarize_mre_distribution_basic_stats() -> None:
|
||||
"""median / p95 / max / count for a tiny sample."""
|
||||
# Arrange
|
||||
records = [FrameToFrameRecord(frame_index=i, mre_px=float(i)) for i in range(10)]
|
||||
|
||||
# Act
|
||||
summary = summarize_mre_distribution(records)
|
||||
|
||||
# Assert
|
||||
assert summary["count"] == 10
|
||||
assert summary["median"] == pytest.approx(4.5)
|
||||
assert summary["max"] == 9.0
|
||||
assert summary["p95"] == pytest.approx(np.percentile(np.arange(10, dtype=float), 95))
|
||||
|
||||
|
||||
def test_summarize_mre_distribution_empty_returns_nan() -> None:
|
||||
# Act
|
||||
summary = summarize_mre_distribution([])
|
||||
|
||||
# Assert
|
||||
assert summary["count"] == 0
|
||||
assert math.isnan(summary["median"])
|
||||
assert math.isnan(summary["p95"])
|
||||
@@ -0,0 +1,411 @@
|
||||
"""Unit tests for ``runner.helpers.registration_classifier`` (FT-P-04 / AZ-412).
|
||||
|
||||
Covers AC-1 (normal-segment classification reproducibility), AC-2
|
||||
(success ratio ≥0.95), AC-3 (sharp-turn exclusion from denominator),
|
||||
and the CSV evidence shape.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers.registration_classifier import (
|
||||
ATTITUDE_LIMIT_DEG,
|
||||
DEFAULT_GROUND_FOOTPRINT_M,
|
||||
IMU_HZ,
|
||||
SUCCESS_RATIO_REQUIRED,
|
||||
TARGET_OVERLAP_FRACTION,
|
||||
VIDEO_FPS,
|
||||
VIDEO_FRAMES_PER_IMU_ROW,
|
||||
FrameAttitude,
|
||||
FrameClassification,
|
||||
ImuTelemetryRow,
|
||||
SuccessReport,
|
||||
classify_frames,
|
||||
compute_attitude,
|
||||
compute_overlap_fraction,
|
||||
compute_success_ratio,
|
||||
compute_translation_m,
|
||||
load_imu_telemetry,
|
||||
write_csv_evidence,
|
||||
)
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
DERKACHI_IMU_CSV = REPO_ROOT / "_docs" / "00_problem" / "input_data" / "flight_derkachi" / "data_imu.csv"
|
||||
|
||||
|
||||
def _level_row(time_s: float = 0.0) -> ImuTelemetryRow:
|
||||
"""A cruise/level row: gravity is z=-1000mg, cruise velocity 10 m/s east."""
|
||||
return ImuTelemetryRow(
|
||||
timestamp_ms=time_s * 1000.0,
|
||||
time_s=time_s,
|
||||
xacc=0,
|
||||
yacc=0,
|
||||
zacc=-1000,
|
||||
vx_cms=1000.0,
|
||||
vy_cms=0.0,
|
||||
vz_cms=0.0,
|
||||
)
|
||||
|
||||
|
||||
def _rolled_row(time_s: float, roll_deg: float) -> ImuTelemetryRow:
|
||||
"""A row with the given roll about +x; uses the accel decomposition."""
|
||||
rad = math.radians(roll_deg)
|
||||
return ImuTelemetryRow(
|
||||
timestamp_ms=time_s * 1000.0,
|
||||
time_s=time_s,
|
||||
xacc=0,
|
||||
yacc=int(round(-1000.0 * math.sin(rad))),
|
||||
zacc=int(round(-1000.0 * math.cos(rad))),
|
||||
vx_cms=1000.0,
|
||||
vy_cms=0.0,
|
||||
vz_cms=0.0,
|
||||
)
|
||||
|
||||
|
||||
def _pitched_row(time_s: float, pitch_deg: float) -> ImuTelemetryRow:
|
||||
"""A row pitched nose-down by ``pitch_deg``; ``+pitch_deg`` = nose down."""
|
||||
rad = math.radians(pitch_deg)
|
||||
return ImuTelemetryRow(
|
||||
timestamp_ms=time_s * 1000.0,
|
||||
time_s=time_s,
|
||||
xacc=int(round(-1000.0 * math.sin(rad))),
|
||||
yacc=0,
|
||||
zacc=int(round(-1000.0 * math.cos(rad))),
|
||||
vx_cms=1000.0,
|
||||
vy_cms=0.0,
|
||||
vz_cms=0.0,
|
||||
)
|
||||
|
||||
|
||||
def test_load_imu_telemetry_parses_repo_csv() -> None:
|
||||
"""The shipped ``data_imu.csv`` parses cleanly into ≈4900 rows."""
|
||||
# Act
|
||||
rows = load_imu_telemetry(DERKACHI_IMU_CSV)
|
||||
|
||||
# Assert — results_report.md says "4,900 nonblank rows".
|
||||
assert len(rows) == 4900
|
||||
assert rows[0].time_s == pytest.approx(0.0, abs=1e-9)
|
||||
# The first row's accel components match the file header we inspected.
|
||||
assert rows[0].xacc == 21
|
||||
assert rows[0].yacc == -3
|
||||
assert rows[0].zacc == -984
|
||||
|
||||
|
||||
def test_load_imu_telemetry_rejects_missing_file(tmp_path: Path) -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(FileNotFoundError):
|
||||
load_imu_telemetry(tmp_path / "missing.csv")
|
||||
|
||||
|
||||
def test_load_imu_telemetry_rejects_missing_columns(tmp_path: Path) -> None:
|
||||
# Arrange
|
||||
bad = tmp_path / "bad.csv"
|
||||
bad.write_text("timestamp(ms),Time\n100,0.1\n")
|
||||
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="missing columns"):
|
||||
load_imu_telemetry(bad)
|
||||
|
||||
|
||||
def test_compute_attitude_level_row_within_one_degree() -> None:
|
||||
"""Repo's first row (≈level cruise) → bank + pitch both within ±1°."""
|
||||
# Act
|
||||
attitude = compute_attitude(_level_row())
|
||||
|
||||
# Assert
|
||||
assert abs(attitude.bank_deg) < 1.0
|
||||
assert abs(attitude.pitch_deg) < 1.0
|
||||
|
||||
|
||||
def test_compute_attitude_right_roll_30_deg_round_trip() -> None:
|
||||
"""A row constructed with 30° right roll → bank ≈ +30°."""
|
||||
# Act
|
||||
attitude = compute_attitude(_rolled_row(time_s=0.1, roll_deg=30.0))
|
||||
|
||||
# Assert
|
||||
assert attitude.bank_deg == pytest.approx(30.0, abs=0.5)
|
||||
assert abs(attitude.pitch_deg) < 0.5
|
||||
|
||||
|
||||
def test_compute_attitude_left_roll_30_deg_round_trip() -> None:
|
||||
"""30° left roll → bank ≈ -30°."""
|
||||
# Act
|
||||
attitude = compute_attitude(_rolled_row(time_s=0.1, roll_deg=-30.0))
|
||||
|
||||
# Assert
|
||||
assert attitude.bank_deg == pytest.approx(-30.0, abs=0.5)
|
||||
|
||||
|
||||
def test_compute_attitude_pitch_down_15_deg_round_trip() -> None:
|
||||
"""Pitched nose-down 15° → pitch ≈ +15°."""
|
||||
# Act
|
||||
attitude = compute_attitude(_pitched_row(time_s=0.1, pitch_deg=15.0))
|
||||
|
||||
# Assert
|
||||
assert attitude.pitch_deg == pytest.approx(15.0, abs=0.5)
|
||||
|
||||
|
||||
def test_compute_translation_m_uses_per_frame_dt() -> None:
|
||||
"""Translation = horizontal_speed * (1/30s) per video frame."""
|
||||
# Arrange — 10 m/s east cruise.
|
||||
row = ImuTelemetryRow(0.0, 0.0, 0, 0, -1000, vx_cms=1000.0, vy_cms=0.0, vz_cms=0.0)
|
||||
|
||||
# Act
|
||||
translation = compute_translation_m(row, prev_row=None)
|
||||
|
||||
# Assert — 10 m/s × (1/30 s) ≈ 0.333 m
|
||||
assert translation == pytest.approx(10.0 / 30.0, rel=1e-6)
|
||||
|
||||
|
||||
def test_compute_overlap_fraction_full_overlap_when_translation_zero() -> None:
|
||||
# Act
|
||||
overlap = compute_overlap_fraction(translation_m=0.0, ground_footprint_m=147.0)
|
||||
|
||||
# Assert
|
||||
assert overlap == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_compute_overlap_fraction_half_overlap_at_half_footprint() -> None:
|
||||
"""Translating by half the footprint → 50% overlap."""
|
||||
# Act
|
||||
overlap = compute_overlap_fraction(translation_m=73.5, ground_footprint_m=147.0)
|
||||
|
||||
# Assert
|
||||
assert overlap == pytest.approx(0.5, abs=1e-6)
|
||||
|
||||
|
||||
def test_compute_overlap_fraction_clamped_at_zero() -> None:
|
||||
"""Translating further than the footprint → 0% (clamped, never negative)."""
|
||||
# Act
|
||||
overlap = compute_overlap_fraction(translation_m=300.0, ground_footprint_m=147.0)
|
||||
|
||||
# Assert
|
||||
assert overlap == 0.0
|
||||
|
||||
|
||||
def test_compute_overlap_fraction_rejects_zero_footprint() -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="ground_footprint_m must be > 0"):
|
||||
compute_overlap_fraction(translation_m=1.0, ground_footprint_m=0.0)
|
||||
|
||||
|
||||
def test_classify_frames_expands_each_imu_row_to_three_video_frames() -> None:
|
||||
"""VIDEO_FRAMES_PER_IMU_ROW = 3; classify_frames respects it."""
|
||||
# Arrange
|
||||
rows = [_level_row(time_s=0.0), _level_row(time_s=0.1)]
|
||||
|
||||
# Act
|
||||
classifications = classify_frames(rows)
|
||||
|
||||
# Assert
|
||||
assert len(classifications) == 2 * VIDEO_FRAMES_PER_IMU_ROW == 6
|
||||
assert [c.frame_index for c in classifications] == [0, 1, 2, 3, 4, 5]
|
||||
assert [c.imu_row_index for c in classifications] == [0, 0, 0, 1, 1, 1]
|
||||
|
||||
|
||||
def test_classify_frames_marks_level_cruise_as_normal() -> None:
|
||||
"""Level cruise rows (±10° attitude, low translation) are all normal."""
|
||||
# Arrange — 10 rows of level cruise.
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(10)]
|
||||
|
||||
# Act
|
||||
classifications = classify_frames(rows)
|
||||
|
||||
# Assert
|
||||
assert all(c.is_normal for c in classifications)
|
||||
assert all(c.excluded_reason == "" for c in classifications)
|
||||
|
||||
|
||||
def test_classify_frames_excludes_sharp_roll() -> None:
|
||||
"""A 25° roll row is excluded; the level rows around it stay normal."""
|
||||
# Arrange — 3 level + 1 sharp roll + 3 level
|
||||
rows = (
|
||||
[_level_row(time_s=0.1 * i) for i in range(3)]
|
||||
+ [_rolled_row(time_s=0.3, roll_deg=25.0)]
|
||||
+ [_level_row(time_s=0.1 * i) for i in range(4, 7)]
|
||||
)
|
||||
|
||||
# Act
|
||||
classifications = classify_frames(rows)
|
||||
|
||||
# Assert
|
||||
sharp_frames = [c for c in classifications if c.imu_row_index == 3]
|
||||
other_frames = [c for c in classifications if c.imu_row_index != 3]
|
||||
assert len(sharp_frames) == VIDEO_FRAMES_PER_IMU_ROW
|
||||
assert all(not c.is_normal for c in sharp_frames)
|
||||
assert all(c.excluded_reason == "attitude_exceeds_limit" for c in sharp_frames)
|
||||
assert all(c.is_normal for c in other_frames)
|
||||
|
||||
|
||||
def test_classify_frames_is_reproducible_ac1() -> None:
|
||||
"""AC-1: same input → same classification across two runs."""
|
||||
# Arrange — pull a real chunk of Derkachi telemetry.
|
||||
rows = load_imu_telemetry(DERKACHI_IMU_CSV)[:100]
|
||||
|
||||
# Act
|
||||
a = classify_frames(rows)
|
||||
b = classify_frames(rows)
|
||||
|
||||
# Assert
|
||||
assert a == b
|
||||
|
||||
|
||||
def test_classify_frames_rejects_invalid_overlap_threshold() -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="min_overlap_fraction"):
|
||||
classify_frames([_level_row()], min_overlap_fraction=1.5)
|
||||
|
||||
|
||||
def test_classify_frames_rejects_invalid_attitude_limit() -> None:
|
||||
# Act / Assert
|
||||
with pytest.raises(ValueError, match="attitude_limit_deg"):
|
||||
classify_frames([_level_row()], attitude_limit_deg=0.0)
|
||||
|
||||
|
||||
def test_compute_success_ratio_perfect_run_passes() -> None:
|
||||
"""100 normal frames + 100 success metrics → ratio 1.0; passes."""
|
||||
# Arrange
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(34)] # 34 × 3 = 102 frames
|
||||
classifications = classify_frames(rows)
|
||||
success_map = {c.frame_index: True for c in classifications}
|
||||
|
||||
# Act
|
||||
report = compute_success_ratio(classifications, success_map)
|
||||
|
||||
# Assert
|
||||
assert report.denominator == len(classifications)
|
||||
assert report.success_count == len(classifications)
|
||||
assert report.ratio == 1.0
|
||||
assert report.passes is True
|
||||
assert report.excluded_count == 0
|
||||
|
||||
|
||||
def test_compute_success_ratio_at_95_pct_passes() -> None:
|
||||
"""Exactly 95% success → AC-2 passes."""
|
||||
# Arrange — 20 normal frames, 1 failure → 19/20 = 0.95.
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(7)] # 7 × 3 = 21 frames; trim to 20.
|
||||
classifications = classify_frames(rows)[:20]
|
||||
success_map = {c.frame_index: (i != 0) for i, c in enumerate(classifications)}
|
||||
|
||||
# Act
|
||||
report = compute_success_ratio(classifications, success_map)
|
||||
|
||||
# Assert
|
||||
assert report.denominator == 20
|
||||
assert report.success_count == 19
|
||||
assert report.ratio == pytest.approx(0.95)
|
||||
assert report.passes is True
|
||||
|
||||
|
||||
def test_compute_success_ratio_below_95_pct_fails() -> None:
|
||||
"""94% success → AC-2 fails."""
|
||||
# Arrange — 100 normal frames, 6 failures → 94/100 = 0.94.
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(34)]
|
||||
classifications = classify_frames(rows)[:100]
|
||||
success_map = {c.frame_index: (i >= 6) for i, c in enumerate(classifications)}
|
||||
|
||||
# Act
|
||||
report = compute_success_ratio(classifications, success_map)
|
||||
|
||||
# Assert
|
||||
assert report.denominator == 100
|
||||
assert report.ratio == pytest.approx(0.94)
|
||||
assert report.passes is False
|
||||
|
||||
|
||||
def test_compute_success_ratio_excludes_sharp_turn_from_denominator_ac3() -> None:
|
||||
"""AC-3: sharp-turn frames are NOT counted in the denominator."""
|
||||
# Arrange — 5 normal + 5 sharp + 5 normal IMU rows = 45 frames total.
|
||||
rows = (
|
||||
[_level_row(time_s=0.1 * i) for i in range(5)]
|
||||
+ [_rolled_row(time_s=0.1 * (5 + i), roll_deg=30.0) for i in range(5)]
|
||||
+ [_level_row(time_s=0.1 * (10 + i)) for i in range(5)]
|
||||
)
|
||||
classifications = classify_frames(rows)
|
||||
success_map = {c.frame_index: True for c in classifications}
|
||||
|
||||
# Act
|
||||
report = compute_success_ratio(classifications, success_map)
|
||||
|
||||
# Assert — 30 normal video frames; 15 excluded by attitude.
|
||||
assert report.denominator == 30
|
||||
assert report.excluded_by_attitude == 15
|
||||
assert report.excluded_by_overlap == 0
|
||||
assert report.excluded_by_missing_metric == 0
|
||||
|
||||
|
||||
def test_compute_success_ratio_handles_missing_metric_separately() -> None:
|
||||
"""A normal frame without a success-map entry is excluded as 'missing'."""
|
||||
# Arrange
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(5)]
|
||||
classifications = classify_frames(rows)
|
||||
# Drop the first three frames from the success map.
|
||||
success_map = {c.frame_index: True for c in classifications[3:]}
|
||||
|
||||
# Act
|
||||
report = compute_success_ratio(classifications, success_map)
|
||||
|
||||
# Assert
|
||||
assert report.excluded_by_missing_metric == 3
|
||||
assert report.denominator == len(classifications) - 3
|
||||
|
||||
|
||||
def test_constants_match_spec() -> None:
|
||||
"""The constants exposed by the module must match the AC text."""
|
||||
# Assert
|
||||
assert ATTITUDE_LIMIT_DEG == 10.0
|
||||
assert TARGET_OVERLAP_FRACTION == 0.40
|
||||
assert SUCCESS_RATIO_REQUIRED == 0.95
|
||||
assert VIDEO_FPS == 30
|
||||
assert IMU_HZ == 10
|
||||
assert VIDEO_FRAMES_PER_IMU_ROW == 3
|
||||
assert DEFAULT_GROUND_FOOTPRINT_M > 0
|
||||
|
||||
|
||||
def test_write_csv_evidence_round_trip(tmp_path: Path) -> None:
|
||||
"""CSV header + per-frame row written exactly as specified."""
|
||||
# Arrange
|
||||
rows = [_level_row(time_s=0.1 * i) for i in range(2)]
|
||||
classifications = classify_frames(rows)
|
||||
success_map = {0: True, 1: False, 2: True, 3: True, 4: True, 5: True}
|
||||
out_path = tmp_path / "ft-p-04.csv"
|
||||
|
||||
# Act
|
||||
write_csv_evidence(out_path, classifications, success_map)
|
||||
|
||||
# Assert
|
||||
written = list(csv.reader(out_path.open()))
|
||||
assert written[0] == [
|
||||
"frame_index",
|
||||
"imu_row_index",
|
||||
"bank_deg",
|
||||
"pitch_deg",
|
||||
"translation_m",
|
||||
"overlap_fraction",
|
||||
"is_normal",
|
||||
"excluded_reason",
|
||||
"registration_success",
|
||||
]
|
||||
assert len(written) == 1 + len(classifications)
|
||||
# frame 1 must have registration_success=false written.
|
||||
assert written[2][8] == "false"
|
||||
|
||||
|
||||
def test_write_csv_evidence_omits_metric_when_missing(tmp_path: Path) -> None:
|
||||
"""Frames without a success-map entry emit an empty registration_success cell."""
|
||||
# Arrange
|
||||
rows = [_level_row(time_s=0.0)]
|
||||
classifications = classify_frames(rows)
|
||||
out_path = tmp_path / "ft-p-04-empty.csv"
|
||||
|
||||
# Act
|
||||
write_csv_evidence(out_path, classifications, {})
|
||||
|
||||
# Assert
|
||||
written = list(csv.reader(out_path.open()))
|
||||
assert all(row[8] == "" for row in written[1:])
|
||||
@@ -43,6 +43,9 @@ E2E_ROOT = Path(__file__).resolve().parents[1]
|
||||
"runner/helpers/geo.py",
|
||||
"runner/helpers/anchor_pair_detector.py",
|
||||
"runner/helpers/estimate_schema.py",
|
||||
"runner/helpers/accuracy_evaluator.py",
|
||||
"runner/helpers/registration_classifier.py",
|
||||
"runner/helpers/mre_evaluator.py",
|
||||
"fixtures/mock-suite-sat/Dockerfile",
|
||||
"fixtures/mock-suite-sat/app.py",
|
||||
"fixtures/mock-suite-sat/requirements.txt",
|
||||
@@ -75,8 +78,12 @@ E2E_ROOT = Path(__file__).resolve().parents[1]
|
||||
"tests/security/__init__.py",
|
||||
"tests/resource_limit/__init__.py",
|
||||
"tests/positive/test_smoke.py",
|
||||
"tests/positive/test_ft_p_01_still_image_accuracy.py",
|
||||
"tests/positive/test_ft_p_02_derkachi_drift.py",
|
||||
"tests/positive/test_ft_p_03_14_schema_wgs84.py",
|
||||
"tests/positive/test_ft_p_04_derkachi_f2f_registration.py",
|
||||
"tests/positive/test_ft_p_05_sat_anchor.py",
|
||||
"tests/positive/test_ft_p_06_mre_budgets.py",
|
||||
],
|
||||
)
|
||||
def test_required_path_exists(relative_path: str) -> None:
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Per-image accuracy evaluation for FT-P-01 (AZ-409 — AC-1.1, AC-1.2).
|
||||
|
||||
Consumes a list of ``(image_id, est_lat, est_lon)`` estimates produced by
|
||||
the SUT during a 60-image still-image push, joins against the ground-truth
|
||||
``coordinates.csv`` shipped with the project, computes Vincenty geodesic
|
||||
distance per image, and reports the AC-2 / AC-3 pass-counts.
|
||||
|
||||
The helper is **transport-agnostic**: the scenario test reads the per-image
|
||||
estimates from the SITL observer (or post-run FDR archive) and hands a
|
||||
typed list to ``evaluate()`` — no SUT import.
|
||||
|
||||
The pass-count thresholds come from the spec's
|
||||
``expected_results/results_report.md`` Pass/Fail Rules:
|
||||
|
||||
* AC-2 (50 m budget): ≥48 / 60 images pass (80 %).
|
||||
* AC-3 (20 m budget): ≥30 / 60 images pass (50 %).
|
||||
|
||||
Timeout discipline (AC-4): when the SITL listener times out for an image,
|
||||
the scenario passes ``est_lat = est_lon = float('inf')``; ``evaluate()``
|
||||
records ``error_m = inf``, ``pass_50m = False``, ``pass_20m = False`` for
|
||||
that image. The aggregate may still pass if other images carry the count.
|
||||
|
||||
Public-boundary discipline: this module does NOT import any
|
||||
``src/gps_denied_onboard`` symbol.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Sequence
|
||||
|
||||
from .geo import distance_m
|
||||
|
||||
PASS_COUNT_50M_REQUIRED = 48
|
||||
PASS_COUNT_20M_REQUIRED = 30
|
||||
TOTAL_IMAGES_REQUIRED = 60
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GtCoordinate:
|
||||
"""Ground-truth WGS84 frame-center coordinate for one still image."""
|
||||
|
||||
image_id: str
|
||||
lat_deg: float
|
||||
lon_deg: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EstimateInput:
|
||||
"""One outbound estimate observed at the SITL listener.
|
||||
|
||||
For a timed-out image (no message received within the scenario's 5 s
|
||||
budget) the scenario passes ``est_lat = est_lon = float('inf')``;
|
||||
``evaluate()`` records ``error_m = inf`` and both pass flags False.
|
||||
"""
|
||||
|
||||
image_id: str
|
||||
est_lat_deg: float
|
||||
est_lon_deg: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PerImageResult:
|
||||
"""Per-image evaluation row written to ``ft-p-01.csv``."""
|
||||
|
||||
image_id: str
|
||||
gt_lat: float
|
||||
gt_lon: float
|
||||
est_lat: float
|
||||
est_lon: float
|
||||
error_m: float
|
||||
pass_50m: bool
|
||||
pass_20m: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AggregateReport:
|
||||
"""Aggregate pass-count over a 60-image run; drives the scenario assertion."""
|
||||
|
||||
total_images: int
|
||||
pass_count_50m: int
|
||||
pass_count_20m: int
|
||||
timeout_count: int
|
||||
pass_50m_required: int = PASS_COUNT_50M_REQUIRED
|
||||
pass_20m_required: int = PASS_COUNT_20M_REQUIRED
|
||||
|
||||
@property
|
||||
def pass_ac2(self) -> bool:
|
||||
"""AC-2: ≥48 / 60 pass the 50 m budget."""
|
||||
return self.pass_count_50m >= self.pass_50m_required
|
||||
|
||||
@property
|
||||
def pass_ac3(self) -> bool:
|
||||
"""AC-3: ≥30 / 60 pass the 20 m budget."""
|
||||
return self.pass_count_20m >= self.pass_20m_required
|
||||
|
||||
@property
|
||||
def overall_pass(self) -> bool:
|
||||
"""Scenario passes iff both AC-2 and AC-3 hold."""
|
||||
return self.pass_ac2 and self.pass_ac3
|
||||
|
||||
|
||||
def load_gt_coordinates(csv_path: Path) -> list[GtCoordinate]:
|
||||
"""Parse the project's ``coordinates.csv``.
|
||||
|
||||
Header format: ``image, lat, lon`` (with the project's whitespace
|
||||
around commas — tolerated).
|
||||
"""
|
||||
if not csv_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"coordinates.csv not found at {csv_path} — check the bind-mount or repo path"
|
||||
)
|
||||
rows: list[GtCoordinate] = []
|
||||
with csv_path.open() as fh:
|
||||
reader = csv.reader(fh)
|
||||
header = next(reader)
|
||||
normalised_header = [c.strip() for c in header]
|
||||
expected = ["image", "lat", "lon"]
|
||||
if normalised_header != expected:
|
||||
raise ValueError(
|
||||
f"coordinates.csv header mismatch: expected {expected}, got {normalised_header}"
|
||||
)
|
||||
for raw in reader:
|
||||
if not raw:
|
||||
continue
|
||||
image_id, lat_str, lon_str = (c.strip() for c in raw)
|
||||
rows.append(
|
||||
GtCoordinate(
|
||||
image_id=image_id,
|
||||
lat_deg=float(lat_str),
|
||||
lon_deg=float(lon_str),
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def _is_timeout(value: float) -> bool:
|
||||
"""An est_lat or est_lon of inf marks an AC-4 timeout."""
|
||||
return math.isinf(value)
|
||||
|
||||
|
||||
def compute_per_image(
|
||||
gt: GtCoordinate, estimate: EstimateInput
|
||||
) -> PerImageResult:
|
||||
"""Compute error_m + AC-2/AC-3 pass flags for one image."""
|
||||
if gt.image_id != estimate.image_id:
|
||||
raise ValueError(
|
||||
f"image_id mismatch: gt='{gt.image_id}' estimate='{estimate.image_id}'"
|
||||
)
|
||||
if _is_timeout(estimate.est_lat_deg) or _is_timeout(estimate.est_lon_deg):
|
||||
return PerImageResult(
|
||||
image_id=gt.image_id,
|
||||
gt_lat=gt.lat_deg,
|
||||
gt_lon=gt.lon_deg,
|
||||
est_lat=estimate.est_lat_deg,
|
||||
est_lon=estimate.est_lon_deg,
|
||||
error_m=math.inf,
|
||||
pass_50m=False,
|
||||
pass_20m=False,
|
||||
)
|
||||
err = distance_m(gt.lat_deg, gt.lon_deg, estimate.est_lat_deg, estimate.est_lon_deg)
|
||||
return PerImageResult(
|
||||
image_id=gt.image_id,
|
||||
gt_lat=gt.lat_deg,
|
||||
gt_lon=gt.lon_deg,
|
||||
est_lat=estimate.est_lat_deg,
|
||||
est_lon=estimate.est_lon_deg,
|
||||
error_m=err,
|
||||
pass_50m=err <= 50.0,
|
||||
pass_20m=err <= 20.0,
|
||||
)
|
||||
|
||||
|
||||
def evaluate(
|
||||
gt_rows: Sequence[GtCoordinate],
|
||||
estimates: Sequence[EstimateInput],
|
||||
) -> tuple[list[PerImageResult], AggregateReport]:
|
||||
"""Join GT + estimates by image_id, compute per-image + aggregate.
|
||||
|
||||
The GT order is authoritative — the resulting list is in GT order so
|
||||
the CSV column is stable across runs. An estimate without a matching
|
||||
GT row is an error (the scenario should not push a stranger image);
|
||||
a GT row without a matching estimate is a timeout (recorded with inf).
|
||||
"""
|
||||
by_id = {e.image_id: e for e in estimates}
|
||||
if len(by_id) != len(estimates):
|
||||
seen: set[str] = set()
|
||||
dupes: list[str] = []
|
||||
for e in estimates:
|
||||
if e.image_id in seen:
|
||||
dupes.append(e.image_id)
|
||||
seen.add(e.image_id)
|
||||
raise ValueError(f"duplicate estimate image_ids: {sorted(set(dupes))}")
|
||||
stranger_ids = sorted(set(by_id) - {g.image_id for g in gt_rows})
|
||||
if stranger_ids:
|
||||
raise ValueError(
|
||||
f"estimate(s) for image_id(s) not in GT: {stranger_ids}"
|
||||
)
|
||||
|
||||
results: list[PerImageResult] = []
|
||||
timeout_count = 0
|
||||
for gt in gt_rows:
|
||||
est = by_id.get(gt.image_id)
|
||||
if est is None:
|
||||
est = EstimateInput(image_id=gt.image_id, est_lat_deg=math.inf, est_lon_deg=math.inf)
|
||||
timeout_count += 1
|
||||
elif _is_timeout(est.est_lat_deg) or _is_timeout(est.est_lon_deg):
|
||||
timeout_count += 1
|
||||
results.append(compute_per_image(gt, est))
|
||||
|
||||
aggregate = AggregateReport(
|
||||
total_images=len(results),
|
||||
pass_count_50m=sum(1 for r in results if r.pass_50m),
|
||||
pass_count_20m=sum(1 for r in results if r.pass_20m),
|
||||
timeout_count=timeout_count,
|
||||
)
|
||||
return results, aggregate
|
||||
|
||||
|
||||
def write_csv_evidence(out_path: Path, results: Iterable[PerImageResult]) -> Path:
|
||||
"""Write the FT-P-01 per-image evidence CSV.
|
||||
|
||||
Header: ``image_id, gt_lat, gt_lon, est_lat, est_lon, error_m, pass_50m, pass_20m``.
|
||||
"""
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with out_path.open("w", newline="") as fh:
|
||||
writer = csv.writer(fh)
|
||||
writer.writerow(
|
||||
[
|
||||
"image_id",
|
||||
"gt_lat",
|
||||
"gt_lon",
|
||||
"est_lat",
|
||||
"est_lon",
|
||||
"error_m",
|
||||
"pass_50m",
|
||||
"pass_20m",
|
||||
]
|
||||
)
|
||||
for r in results:
|
||||
writer.writerow(
|
||||
[
|
||||
r.image_id,
|
||||
f"{r.gt_lat:.6f}",
|
||||
f"{r.gt_lon:.6f}",
|
||||
"inf" if math.isinf(r.est_lat) else f"{r.est_lat:.6f}",
|
||||
"inf" if math.isinf(r.est_lon) else f"{r.est_lon:.6f}",
|
||||
"inf" if math.isinf(r.error_m) else f"{r.error_m:.3f}",
|
||||
"true" if r.pass_50m else "false",
|
||||
"true" if r.pass_20m else "false",
|
||||
]
|
||||
)
|
||||
return out_path
|
||||
@@ -0,0 +1,284 @@
|
||||
"""MRE budget evaluation for FT-P-05 / FT-P-06 (AZ-413 / AC-2.1b, AC-2.2).
|
||||
|
||||
The SUT exposes per-frame **MRE** (Mean Reprojection Error, in pixels)
|
||||
for both:
|
||||
|
||||
* **Frame-to-frame** registrations — produced during the Derkachi replay
|
||||
(FT-P-04 scope; the MRE per frame is recorded in the FDR archive
|
||||
alongside the boolean success metric).
|
||||
* **Cross-domain** registrations — produced when the satellite-anchor
|
||||
pipeline matches a UAV frame against a satellite tile (FT-P-05 scope;
|
||||
one MRE per still-image push).
|
||||
|
||||
FT-P-05 binds:
|
||||
* AC-2 (per-image cross-domain): every image's MRE < 2.5 px.
|
||||
* AC-3 (accuracy alongside MRE): inherits FT-P-01 thresholds (≥80 % at
|
||||
50 m, ≥50 % at 20 m) but on the same image set; the helper reuses
|
||||
``accuracy_evaluator`` for the geodesic part.
|
||||
|
||||
FT-P-06 binds AC-4: the 95th percentile MRE bound — < 1.0 px frame-to-frame
|
||||
AND < 2.5 px cross-domain. The 95th percentile is computed with numpy's
|
||||
default linear-interpolation algorithm (which the spec explicitly names).
|
||||
|
||||
Public-boundary discipline: this module does NOT import any
|
||||
``src/gps_denied_onboard`` symbol.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from statistics import median
|
||||
from typing import Iterable, Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
MRE_PER_IMAGE_BUDGET_PX = 2.5
|
||||
MRE_P95_FRAME_TO_FRAME_BUDGET_PX = 1.0
|
||||
MRE_P95_CROSS_DOMAIN_BUDGET_PX = 2.5
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CrossDomainRecord:
|
||||
"""One observation per still-image push (FT-P-05)."""
|
||||
|
||||
image_id: str
|
||||
mre_px: float
|
||||
error_m: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FrameToFrameRecord:
|
||||
"""One observation per video frame (FT-P-04 evidence reused by FT-P-06)."""
|
||||
|
||||
frame_index: int
|
||||
mre_px: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PerImageBudgetReport:
|
||||
"""FT-P-05 AC-2: every image MRE < 2.5 px."""
|
||||
|
||||
total_images: int
|
||||
pass_count: int
|
||||
fail_image_ids: tuple[str, ...]
|
||||
max_mre_px: float
|
||||
budget_px: float = MRE_PER_IMAGE_BUDGET_PX
|
||||
|
||||
@property
|
||||
def passes(self) -> bool:
|
||||
return self.pass_count == self.total_images > 0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class P95Report:
|
||||
"""FT-P-06 AC-4: 95th-percentile budget."""
|
||||
|
||||
sample_count: int
|
||||
p95_px: float
|
||||
budget_px: float
|
||||
|
||||
@property
|
||||
def passes(self) -> bool:
|
||||
return self.sample_count > 0 and self.p95_px < self.budget_px
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CombinedP95Report:
|
||||
"""FT-P-06 combined assertion across both domains."""
|
||||
|
||||
frame_to_frame: P95Report
|
||||
cross_domain: P95Report
|
||||
|
||||
@property
|
||||
def passes(self) -> bool:
|
||||
return self.frame_to_frame.passes and self.cross_domain.passes
|
||||
|
||||
|
||||
def evaluate_per_image_budget(
|
||||
records: Sequence[CrossDomainRecord],
|
||||
*,
|
||||
budget_px: float = MRE_PER_IMAGE_BUDGET_PX,
|
||||
) -> PerImageBudgetReport:
|
||||
"""AC-2 of FT-P-05: every cross-domain MRE strictly below ``budget_px``.
|
||||
|
||||
Strictness: the spec text "MRE < 2.5 px for all images" reads as a
|
||||
strict less-than. A record at exactly 2.5 px FAILS (the matcher must
|
||||
be inside the budget, not on the boundary).
|
||||
"""
|
||||
if budget_px <= 0:
|
||||
raise ValueError(f"budget_px must be > 0, got {budget_px}")
|
||||
fail_ids: list[str] = []
|
||||
pass_count = 0
|
||||
max_mre = 0.0
|
||||
for r in records:
|
||||
max_mre = max(max_mre, r.mre_px)
|
||||
if r.mre_px < budget_px:
|
||||
pass_count += 1
|
||||
else:
|
||||
fail_ids.append(r.image_id)
|
||||
return PerImageBudgetReport(
|
||||
total_images=len(records),
|
||||
pass_count=pass_count,
|
||||
fail_image_ids=tuple(fail_ids),
|
||||
max_mre_px=max_mre,
|
||||
budget_px=budget_px,
|
||||
)
|
||||
|
||||
|
||||
def evaluate_p95(
|
||||
mre_samples: Sequence[float],
|
||||
*,
|
||||
budget_px: float,
|
||||
) -> P95Report:
|
||||
"""AC-4 of FT-P-06: 95th-percentile MRE strictly below ``budget_px``.
|
||||
|
||||
Percentile computed via ``numpy.percentile`` with the default
|
||||
``method='linear'`` (linear interpolation between adjacent ranks).
|
||||
The spec explicitly names that method.
|
||||
"""
|
||||
if budget_px <= 0:
|
||||
raise ValueError(f"budget_px must be > 0, got {budget_px}")
|
||||
n = len(mre_samples)
|
||||
if n == 0:
|
||||
return P95Report(sample_count=0, p95_px=float("nan"), budget_px=budget_px)
|
||||
p95 = float(np.percentile(np.asarray(mre_samples, dtype=float), 95))
|
||||
return P95Report(sample_count=n, p95_px=p95, budget_px=budget_px)
|
||||
|
||||
|
||||
def evaluate_combined_p95(
|
||||
frame_to_frame: Sequence[FrameToFrameRecord],
|
||||
cross_domain: Sequence[CrossDomainRecord],
|
||||
) -> CombinedP95Report:
|
||||
"""FT-P-06 combined assertion using per-domain budgets."""
|
||||
f2f = evaluate_p95(
|
||||
[r.mre_px for r in frame_to_frame],
|
||||
budget_px=MRE_P95_FRAME_TO_FRAME_BUDGET_PX,
|
||||
)
|
||||
xd = evaluate_p95(
|
||||
[r.mre_px for r in cross_domain],
|
||||
budget_px=MRE_P95_CROSS_DOMAIN_BUDGET_PX,
|
||||
)
|
||||
return CombinedP95Report(frame_to_frame=f2f, cross_domain=xd)
|
||||
|
||||
|
||||
def load_cross_domain_csv(csv_path: Path) -> list[CrossDomainRecord]:
|
||||
"""Read ``ft-p-05.csv`` back into typed records (used by FT-P-06)."""
|
||||
if not csv_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"FT-P-05 evidence not found at {csv_path} — run FT-P-05 first."
|
||||
)
|
||||
records: list[CrossDomainRecord] = []
|
||||
with csv_path.open() as fh:
|
||||
reader = csv.DictReader(fh)
|
||||
needed = {"image_id", "mre_px", "error_m"}
|
||||
missing = needed - set(reader.fieldnames or [])
|
||||
if missing:
|
||||
raise ValueError(f"FT-P-05 CSV missing columns: {sorted(missing)}")
|
||||
for row in reader:
|
||||
records.append(
|
||||
CrossDomainRecord(
|
||||
image_id=row["image_id"],
|
||||
mre_px=float(row["mre_px"]),
|
||||
error_m=float(row["error_m"]) if row["error_m"] != "inf" else float("inf"),
|
||||
)
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def load_frame_to_frame_csv(csv_path: Path) -> list[FrameToFrameRecord]:
|
||||
"""Read frame-to-frame MRE from the FT-P-04 evidence CSV.
|
||||
|
||||
The FT-P-04 CSV currently includes ``registration_success`` per frame
|
||||
but NOT MRE; that column will be added when the SUT exposes it
|
||||
(AC-NEW-3 FDR schema). This loader expects a ``mre_px`` column —
|
||||
raises ValueError if absent so the FT-P-06 scenario fails loudly.
|
||||
"""
|
||||
if not csv_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"FT-P-04 evidence not found at {csv_path} — run FT-P-04 first."
|
||||
)
|
||||
records: list[FrameToFrameRecord] = []
|
||||
with csv_path.open() as fh:
|
||||
reader = csv.DictReader(fh)
|
||||
if "mre_px" not in (reader.fieldnames or []):
|
||||
raise ValueError(
|
||||
"FT-P-04 evidence is missing the 'mre_px' column required by FT-P-06. "
|
||||
"The SUT must emit per-frame MRE in the FDR archive (AC-NEW-3)."
|
||||
)
|
||||
for row in reader:
|
||||
mre_str = row["mre_px"].strip()
|
||||
if not mre_str:
|
||||
continue
|
||||
records.append(
|
||||
FrameToFrameRecord(
|
||||
frame_index=int(row["frame_index"]),
|
||||
mre_px=float(mre_str),
|
||||
)
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def write_cross_domain_csv(
|
||||
out_path: Path,
|
||||
records: Iterable[CrossDomainRecord],
|
||||
*,
|
||||
pass_50m: dict[str, bool] | None = None,
|
||||
pass_20m: dict[str, bool] | None = None,
|
||||
) -> Path:
|
||||
"""Write the FT-P-05 per-image evidence CSV.
|
||||
|
||||
Header: ``image_id, est_lat, est_lon, error_m, mre_px, pass_50m,
|
||||
pass_20m, pass_mre``. The lat/lon columns are emitted as blanks here
|
||||
(the scenario file fills them via ``write_csv_evidence`` from
|
||||
``accuracy_evaluator`` — this writer is for the FT-P-06-relevant
|
||||
columns only).
|
||||
"""
|
||||
pass_50m = pass_50m or {}
|
||||
pass_20m = pass_20m or {}
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with out_path.open("w", newline="") as fh:
|
||||
writer = csv.writer(fh)
|
||||
writer.writerow(
|
||||
[
|
||||
"image_id",
|
||||
"est_lat",
|
||||
"est_lon",
|
||||
"error_m",
|
||||
"mre_px",
|
||||
"pass_50m",
|
||||
"pass_20m",
|
||||
"pass_mre",
|
||||
]
|
||||
)
|
||||
for r in records:
|
||||
writer.writerow(
|
||||
[
|
||||
r.image_id,
|
||||
"",
|
||||
"",
|
||||
"inf" if r.error_m == float("inf") else f"{r.error_m:.3f}",
|
||||
f"{r.mre_px:.4f}",
|
||||
"true" if pass_50m.get(r.image_id, False) else "false",
|
||||
"true" if pass_20m.get(r.image_id, False) else "false",
|
||||
"true" if r.mre_px < MRE_PER_IMAGE_BUDGET_PX else "false",
|
||||
]
|
||||
)
|
||||
return out_path
|
||||
|
||||
|
||||
def summarize_mre_distribution(records: Sequence[FrameToFrameRecord | CrossDomainRecord]) -> dict[str, float]:
|
||||
"""Summary stats for diagnostic logging (median, p95, max).
|
||||
|
||||
Convenience helper; not used by the AC assertions themselves.
|
||||
"""
|
||||
if not records:
|
||||
return {"count": 0.0, "median": float("nan"), "p95": float("nan"), "max": float("nan")}
|
||||
samples = [r.mre_px for r in records]
|
||||
return {
|
||||
"count": float(len(samples)),
|
||||
"median": float(median(samples)),
|
||||
"p95": float(np.percentile(np.asarray(samples, dtype=float), 95)),
|
||||
"max": float(max(samples)),
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
"""Normal-segment classification + success-ratio for FT-P-04 (AZ-412 / AC-2.1a).
|
||||
|
||||
The SUT exposes a per-frame ``registration_success`` boolean (either via
|
||||
``NAMED_VALUE_FLOAT`` MAVLink messages or via the post-run FDR archive).
|
||||
This helper:
|
||||
|
||||
1. Reads the Derkachi ``data_imu.csv`` (SCALED_IMU2 + GLOBAL_POSITION_INT
|
||||
columns) and derives a per-row attitude approximation from accelerometer
|
||||
readings (the spec's AC-1 explicitly says attitude is
|
||||
``SCALED_IMU2``-derived, NOT internal SUT state).
|
||||
2. Classifies each video frame as **normal** when both:
|
||||
* bank/pitch within ±10° of nadir, AND
|
||||
* inferred prior-frame overlap ≥40 % (heuristic from translation magnitude).
|
||||
3. Computes the success ratio over the **normal** set only — sharp-turn
|
||||
frames are excluded from the denominator per AC-3.
|
||||
4. Asserts the ratio meets the AC-2 budget (≥0.95).
|
||||
|
||||
The video is 30 fps; the IMU/telemetry CSV is 10 Hz (one row per 100 ms,
|
||||
i.e. 3 video frames per row). The classifier expands each telemetry row
|
||||
to 3 video-frame indices (the same row drives 3 consecutive frames).
|
||||
|
||||
Public-boundary discipline: this module does NOT import any
|
||||
``src/gps_denied_onboard`` symbol.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Mapping, Sequence
|
||||
|
||||
ATTITUDE_LIMIT_DEG = 10.0
|
||||
TARGET_OVERLAP_FRACTION = 0.40
|
||||
SUCCESS_RATIO_REQUIRED = 0.95
|
||||
VIDEO_FPS = 30
|
||||
IMU_HZ = 10
|
||||
VIDEO_FRAMES_PER_IMU_ROW = VIDEO_FPS // IMU_HZ
|
||||
# Derkachi nadir camera: the camera_info.md fixture records ~141 m altitude
|
||||
# AGL and ~55° horizontal FOV. The "ground footprint width" at nadir is
|
||||
# 2 * alt * tan(FOV/2) ≈ 2 * 141 * tan(27.5°) ≈ 147 m. We use a single
|
||||
# scenario-wide ground footprint to keep the heuristic transparent.
|
||||
DEFAULT_GROUND_FOOTPRINT_M = 147.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ImuTelemetryRow:
|
||||
"""One row of ``data_imu.csv`` distilled to the columns the classifier needs.
|
||||
|
||||
Velocity fields are floats (cm/s) because the shipped ``data_imu.csv``
|
||||
stores them in scientific notation (e.g. ``-4.44E-16`` near hover).
|
||||
Acceleration fields stay int per the SCALED_IMU2 wire format.
|
||||
"""
|
||||
|
||||
timestamp_ms: float
|
||||
time_s: float
|
||||
xacc: int
|
||||
yacc: int
|
||||
zacc: int
|
||||
vx_cms: float
|
||||
vy_cms: float
|
||||
vz_cms: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FrameAttitude:
|
||||
"""Bank + pitch derived from accel; used for the ±10° gate.
|
||||
|
||||
Accel-only attitude assumes the platform is in near-equilibrium
|
||||
flight (the dominant accel is gravity). Valid for the cropped
|
||||
nadir-cruise segments AC-2.1a targets; explicitly NOT valid during
|
||||
aggressive manoeuvres — but those are exactly the frames AC-2.1a
|
||||
wants to EXCLUDE from the denominator. So the limitation matches
|
||||
the AC intent.
|
||||
"""
|
||||
|
||||
bank_deg: float
|
||||
pitch_deg: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FrameClassification:
|
||||
"""Per-video-frame classification used to build the FT-P-04 denominator."""
|
||||
|
||||
frame_index: int
|
||||
imu_row_index: int
|
||||
attitude: FrameAttitude
|
||||
translation_m: float
|
||||
overlap_fraction: float
|
||||
is_normal: bool
|
||||
excluded_reason: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SuccessReport:
|
||||
"""Aggregate report consumed by the scenario assertion.
|
||||
|
||||
``ratio = success_count / denominator`` where ``denominator`` is the
|
||||
count of normal frames (sharp-turn / low-overlap frames are excluded
|
||||
per AC-3 — they are counted in ``excluded_count`` for diagnostic
|
||||
clarity).
|
||||
"""
|
||||
|
||||
success_count: int
|
||||
denominator: int
|
||||
ratio: float
|
||||
excluded_count: int
|
||||
excluded_by_attitude: int
|
||||
excluded_by_overlap: int
|
||||
excluded_by_missing_metric: int
|
||||
ratio_required: float = SUCCESS_RATIO_REQUIRED
|
||||
|
||||
@property
|
||||
def passes(self) -> bool:
|
||||
return self.denominator > 0 and self.ratio >= self.ratio_required
|
||||
|
||||
|
||||
def load_imu_telemetry(csv_path: Path) -> list[ImuTelemetryRow]:
|
||||
"""Read ``data_imu.csv`` and return one row per non-blank entry.
|
||||
|
||||
Only the columns the classifier needs are kept. Other columns are
|
||||
ignored to keep the classifier independent of upstream column churn.
|
||||
"""
|
||||
if not csv_path.exists():
|
||||
raise FileNotFoundError(
|
||||
f"data_imu.csv not found at {csv_path} — bind-mount the Derkachi fixture"
|
||||
)
|
||||
needed = {
|
||||
"timestamp(ms)",
|
||||
"Time",
|
||||
"SCALED_IMU2.xacc",
|
||||
"SCALED_IMU2.yacc",
|
||||
"SCALED_IMU2.zacc",
|
||||
"GLOBAL_POSITION_INT.vx",
|
||||
"GLOBAL_POSITION_INT.vy",
|
||||
"GLOBAL_POSITION_INT.vz",
|
||||
}
|
||||
rows: list[ImuTelemetryRow] = []
|
||||
with csv_path.open() as fh:
|
||||
reader = csv.DictReader(fh)
|
||||
missing = needed - set(reader.fieldnames or [])
|
||||
if missing:
|
||||
raise ValueError(f"data_imu.csv missing columns: {sorted(missing)}")
|
||||
for raw in reader:
|
||||
if not raw["timestamp(ms)"].strip():
|
||||
continue
|
||||
rows.append(
|
||||
ImuTelemetryRow(
|
||||
timestamp_ms=float(raw["timestamp(ms)"]),
|
||||
time_s=float(raw["Time"]),
|
||||
xacc=int(float(raw["SCALED_IMU2.xacc"])),
|
||||
yacc=int(float(raw["SCALED_IMU2.yacc"])),
|
||||
zacc=int(float(raw["SCALED_IMU2.zacc"])),
|
||||
vx_cms=float(raw["GLOBAL_POSITION_INT.vx"]),
|
||||
vy_cms=float(raw["GLOBAL_POSITION_INT.vy"]),
|
||||
vz_cms=float(raw["GLOBAL_POSITION_INT.vz"]),
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def compute_attitude(row: ImuTelemetryRow) -> FrameAttitude:
|
||||
"""Derive bank + pitch from accelerometer (gravity-as-down assumption).
|
||||
|
||||
SCALED_IMU2 acc components are in mg (milli-g). Sign convention:
|
||||
body-frame +x forward, +y right, +z down. With dominant gravity on
|
||||
+z the resting attitude has xacc=0, yacc=0, zacc=-1000 (negative
|
||||
because the body frame measures the reaction force pointing UP).
|
||||
|
||||
pitch = atan2(-xacc, sqrt(yacc² + zacc²))
|
||||
bank = atan2(yacc, zacc)
|
||||
"""
|
||||
x = float(row.xacc)
|
||||
y = float(row.yacc)
|
||||
z = float(row.zacc)
|
||||
pitch_rad = math.atan2(-x, math.sqrt(y * y + z * z))
|
||||
bank_rad = math.atan2(y, z)
|
||||
# The atan2(y, z) convention puts level flight at ±π (since z is
|
||||
# negative gravity); we want level = 0, so subtract π and wrap.
|
||||
bank_deg_raw = math.degrees(bank_rad)
|
||||
if bank_deg_raw > 90.0:
|
||||
bank_deg = bank_deg_raw - 180.0
|
||||
elif bank_deg_raw < -90.0:
|
||||
bank_deg = bank_deg_raw + 180.0
|
||||
else:
|
||||
bank_deg = bank_deg_raw
|
||||
return FrameAttitude(bank_deg=bank_deg, pitch_deg=math.degrees(pitch_rad))
|
||||
|
||||
|
||||
def compute_translation_m(row: ImuTelemetryRow, prev_row: ImuTelemetryRow | None) -> float:
|
||||
"""Ground-plane translation between consecutive frames in meters.
|
||||
|
||||
Uses the GLOBAL_POSITION_INT velocity (vx, vy in cm/s); vz is
|
||||
excluded because vertical motion mostly affects scale, not overlap.
|
||||
Per-frame dt = 1/30 s. With telemetry at 10 Hz, the same velocity
|
||||
drives 3 consecutive frames.
|
||||
"""
|
||||
vx_ms = row.vx_cms / 100.0
|
||||
vy_ms = row.vy_cms / 100.0
|
||||
horizontal_speed = math.hypot(vx_ms, vy_ms)
|
||||
dt_s = 1.0 / VIDEO_FPS
|
||||
return horizontal_speed * dt_s
|
||||
|
||||
|
||||
def compute_overlap_fraction(
|
||||
translation_m: float, ground_footprint_m: float
|
||||
) -> float:
|
||||
"""Fraction of ground footprint that overlaps with the prior frame.
|
||||
|
||||
Approximation: assume a square ground footprint of side
|
||||
``ground_footprint_m``. After translating by ``translation_m`` in
|
||||
the horizontal plane, the overlap is
|
||||
``max(0, 1 - translation_m / ground_footprint_m)``.
|
||||
|
||||
This is an upper bound — diagonal motion or rotation eats more
|
||||
overlap. The ±10° attitude gate rules out the rotation-heavy
|
||||
frames; pure translation is what survives, and this approximation
|
||||
is tight for cruise flight.
|
||||
"""
|
||||
if ground_footprint_m <= 0:
|
||||
raise ValueError(f"ground_footprint_m must be > 0, got {ground_footprint_m}")
|
||||
fraction = 1.0 - translation_m / ground_footprint_m
|
||||
return max(0.0, min(1.0, fraction))
|
||||
|
||||
|
||||
def classify_frames(
|
||||
imu_rows: Sequence[ImuTelemetryRow],
|
||||
*,
|
||||
attitude_limit_deg: float = ATTITUDE_LIMIT_DEG,
|
||||
min_overlap_fraction: float = TARGET_OVERLAP_FRACTION,
|
||||
ground_footprint_m: float = DEFAULT_GROUND_FOOTPRINT_M,
|
||||
video_frames_per_imu_row: int = VIDEO_FRAMES_PER_IMU_ROW,
|
||||
) -> list[FrameClassification]:
|
||||
"""Build the per-video-frame classification list.
|
||||
|
||||
Each ``ImuTelemetryRow`` drives ``video_frames_per_imu_row``
|
||||
consecutive video frames (3 frames per IMU row by default). Frame
|
||||
indices are 0-based and contiguous.
|
||||
|
||||
Determinism: this function depends only on the input rows + tunables
|
||||
— same input → same output.
|
||||
"""
|
||||
if attitude_limit_deg <= 0:
|
||||
raise ValueError(f"attitude_limit_deg must be > 0, got {attitude_limit_deg}")
|
||||
if not 0.0 < min_overlap_fraction < 1.0:
|
||||
raise ValueError(
|
||||
f"min_overlap_fraction must be in (0, 1), got {min_overlap_fraction}"
|
||||
)
|
||||
|
||||
classifications: list[FrameClassification] = []
|
||||
prev_row: ImuTelemetryRow | None = None
|
||||
frame_index = 0
|
||||
for imu_row_index, row in enumerate(imu_rows):
|
||||
attitude = compute_attitude(row)
|
||||
translation_m = compute_translation_m(row, prev_row)
|
||||
overlap_fraction = compute_overlap_fraction(translation_m, ground_footprint_m)
|
||||
attitude_ok = (
|
||||
abs(attitude.bank_deg) <= attitude_limit_deg
|
||||
and abs(attitude.pitch_deg) <= attitude_limit_deg
|
||||
)
|
||||
overlap_ok = overlap_fraction >= min_overlap_fraction
|
||||
is_normal = attitude_ok and overlap_ok
|
||||
if not attitude_ok:
|
||||
reason = "attitude_exceeds_limit"
|
||||
elif not overlap_ok:
|
||||
reason = "overlap_below_threshold"
|
||||
else:
|
||||
reason = ""
|
||||
for _ in range(video_frames_per_imu_row):
|
||||
classifications.append(
|
||||
FrameClassification(
|
||||
frame_index=frame_index,
|
||||
imu_row_index=imu_row_index,
|
||||
attitude=attitude,
|
||||
translation_m=translation_m,
|
||||
overlap_fraction=overlap_fraction,
|
||||
is_normal=is_normal,
|
||||
excluded_reason=reason,
|
||||
)
|
||||
)
|
||||
frame_index += 1
|
||||
prev_row = row
|
||||
return classifications
|
||||
|
||||
|
||||
def compute_success_ratio(
|
||||
classifications: Sequence[FrameClassification],
|
||||
registration_success_by_frame: Mapping[int, bool],
|
||||
) -> SuccessReport:
|
||||
"""Compute the success ratio over the normal-frame denominator.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
classifications : the per-frame classification list (output of
|
||||
``classify_frames``).
|
||||
registration_success_by_frame : dict[frame_index, bool] — populated
|
||||
from ``NAMED_VALUE_FLOAT`` listener output or post-run FDR read.
|
||||
Frames missing from this dict are excluded from the denominator
|
||||
and counted in ``excluded_by_missing_metric`` (the SUT failed to
|
||||
emit the metric — AC-2 of the AC-NEW-3 FDR-schema spec covers
|
||||
the schema; this scenario complains if the metric is absent).
|
||||
|
||||
Returns
|
||||
-------
|
||||
SuccessReport
|
||||
"""
|
||||
success = 0
|
||||
denominator = 0
|
||||
excluded_by_attitude = 0
|
||||
excluded_by_overlap = 0
|
||||
excluded_by_missing_metric = 0
|
||||
for cls in classifications:
|
||||
if not cls.is_normal:
|
||||
if cls.excluded_reason == "attitude_exceeds_limit":
|
||||
excluded_by_attitude += 1
|
||||
elif cls.excluded_reason == "overlap_below_threshold":
|
||||
excluded_by_overlap += 1
|
||||
continue
|
||||
metric = registration_success_by_frame.get(cls.frame_index)
|
||||
if metric is None:
|
||||
excluded_by_missing_metric += 1
|
||||
continue
|
||||
denominator += 1
|
||||
if metric:
|
||||
success += 1
|
||||
excluded_count = excluded_by_attitude + excluded_by_overlap + excluded_by_missing_metric
|
||||
ratio = (success / denominator) if denominator > 0 else 0.0
|
||||
return SuccessReport(
|
||||
success_count=success,
|
||||
denominator=denominator,
|
||||
ratio=ratio,
|
||||
excluded_count=excluded_count,
|
||||
excluded_by_attitude=excluded_by_attitude,
|
||||
excluded_by_overlap=excluded_by_overlap,
|
||||
excluded_by_missing_metric=excluded_by_missing_metric,
|
||||
)
|
||||
|
||||
|
||||
def write_csv_evidence(
|
||||
out_path: Path,
|
||||
classifications: Iterable[FrameClassification],
|
||||
registration_success_by_frame: Mapping[int, bool],
|
||||
) -> Path:
|
||||
"""Write the FT-P-04 per-frame evidence CSV.
|
||||
|
||||
Header: ``frame_index, imu_row_index, bank_deg, pitch_deg,
|
||||
translation_m, overlap_fraction, is_normal, excluded_reason,
|
||||
registration_success``.
|
||||
"""
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with out_path.open("w", newline="") as fh:
|
||||
writer = csv.writer(fh)
|
||||
writer.writerow(
|
||||
[
|
||||
"frame_index",
|
||||
"imu_row_index",
|
||||
"bank_deg",
|
||||
"pitch_deg",
|
||||
"translation_m",
|
||||
"overlap_fraction",
|
||||
"is_normal",
|
||||
"excluded_reason",
|
||||
"registration_success",
|
||||
]
|
||||
)
|
||||
for cls in classifications:
|
||||
metric = registration_success_by_frame.get(cls.frame_index)
|
||||
writer.writerow(
|
||||
[
|
||||
cls.frame_index,
|
||||
cls.imu_row_index,
|
||||
f"{cls.attitude.bank_deg:.3f}",
|
||||
f"{cls.attitude.pitch_deg:.3f}",
|
||||
f"{cls.translation_m:.4f}",
|
||||
f"{cls.overlap_fraction:.4f}",
|
||||
"true" if cls.is_normal else "false",
|
||||
cls.excluded_reason,
|
||||
"" if metric is None else ("true" if metric else "false"),
|
||||
]
|
||||
)
|
||||
return out_path
|
||||
@@ -0,0 +1,175 @@
|
||||
"""FT-P-01 — Still-image set-60 frame-center accuracy (AC-1.1, AC-1.2).
|
||||
|
||||
The full scenario:
|
||||
|
||||
1. Push each ``AD0000NN.jpg`` from ``still-image-set-60`` to the SUT's
|
||||
frame source, one at a time.
|
||||
2. Wait up to 5 s for the SITL listener to receive the SUT's outbound
|
||||
``GPS_INPUT`` (ArduPilot) or ``MSP2_SENSOR_GPS`` (iNav) message.
|
||||
3. Compute Vincenty geodesic distance between the SUT estimate and the
|
||||
per-image GT from ``_docs/00_problem/input_data/coordinates.csv``.
|
||||
4. Emit ``e2e-results/run-${RUN_ID}/ft-p-01-{fc_adapter}-{vio_strategy}.csv``
|
||||
with one row per image.
|
||||
5. Assert AC-2 (≥48/60 within 50 m) and AC-3 (≥30/60 within 20 m) per
|
||||
``expected_results/results_report.md`` Pass/Fail Rules.
|
||||
|
||||
What this file owns:
|
||||
|
||||
* The AC-1 / AC-2 / AC-3 / AC-4 / AC-5 wiring above.
|
||||
* CSV evidence emission via the AZ-409-owned ``accuracy_evaluator``.
|
||||
|
||||
What this file does NOT own:
|
||||
|
||||
* The frame-source push → ``runner.helpers.frame_source_replay`` (stub;
|
||||
owned by AZ-441) — skip-gated.
|
||||
* The SITL message receipt → ``runner.helpers.sitl_observer`` (stub;
|
||||
owned by AZ-416/AZ-417) — skip-gated.
|
||||
|
||||
When both upstream helpers land, this file's runtime path activates
|
||||
automatically — the skip is keyed off the ``NotImplementedError`` from
|
||||
the helper imports, not off a hard-coded marker.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers import accuracy_evaluator as ae
|
||||
|
||||
GT_CSV = Path(__file__).resolve().parents[3] / "_docs" / "00_problem" / "input_data" / "coordinates.csv"
|
||||
STILL_IMAGES_DIR = GT_CSV.parent
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def _harness_helpers_implemented() -> bool:
|
||||
"""True iff the upstream replay + SITL-observation helpers are real.
|
||||
|
||||
Same auto-detect pattern as FT-P-02 / FT-P-03 — the gate flips when
|
||||
the helpers stop raising NotImplementedError, so no marker churn.
|
||||
"""
|
||||
from runner.helpers import frame_source_replay, sitl_observer
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
try:
|
||||
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
|
||||
try:
|
||||
replayer.replay_image_directory(Path("/tmp/non-existent"))
|
||||
except NotImplementedError:
|
||||
return False
|
||||
try:
|
||||
sitl_observer.get_observer(fc_adapter="ardupilot", host="sitl-ardupilot")
|
||||
except NotImplementedError:
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class _NullSink:
|
||||
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _ft_p_01_image_paths() -> list[Path]:
|
||||
"""The 60 AD0000NN.jpg images, sorted lexicographically (AD000001..AD000060)."""
|
||||
return sorted(STILL_IMAGES_DIR.glob("AD??????.jpg"))
|
||||
|
||||
|
||||
@pytest.mark.traces_to("AC-1.1,AC-1.2,AC-1,AC-2,AC-3,AC-4,AC-5")
|
||||
def test_ft_p_01_still_image_accuracy(
|
||||
fc_adapter: str,
|
||||
vio_strategy: str,
|
||||
evidence_dir, # type: ignore[no-untyped-def]
|
||||
run_id: str,
|
||||
nfr_recorder, # type: ignore[no-untyped-def]
|
||||
_harness_helpers_implemented: bool,
|
||||
) -> None:
|
||||
"""Full FT-P-01 scenario (AC-1.1, AC-1.2).
|
||||
|
||||
AC-1: per-image distance computed for all 60 images.
|
||||
AC-2: ``pass_count(error_m ≤ 50) ≥ 48``.
|
||||
AC-3: ``pass_count(error_m ≤ 20) ≥ 30``.
|
||||
AC-4: per-image timeout → ``error_m=∞``; aggregate continues.
|
||||
AC-5: parametrized across ``(fc_adapter, vio_strategy)`` (4 variants).
|
||||
"""
|
||||
if not _harness_helpers_implemented:
|
||||
pytest.skip(
|
||||
"FT-P-01 still-image push requires runner.helpers.{frame_source_replay,"
|
||||
"sitl_observer} — currently AZ-441 + AZ-416/AZ-417 leftovers. "
|
||||
"Pure-logic ACs covered by e2e/_unit_tests/helpers/test_accuracy_evaluator.py."
|
||||
)
|
||||
|
||||
from runner.helpers import frame_source_replay, sitl_observer
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
# 1. Resolve GT + image inventory once.
|
||||
gt_rows = ae.load_gt_coordinates(GT_CSV)
|
||||
image_paths = _ft_p_01_image_paths()
|
||||
if len(image_paths) != ae.TOTAL_IMAGES_REQUIRED:
|
||||
pytest.fail(
|
||||
f"FT-P-01 expects {ae.TOTAL_IMAGES_REQUIRED} images in {STILL_IMAGES_DIR}, "
|
||||
f"found {len(image_paths)}"
|
||||
)
|
||||
|
||||
# 2. Resolve the SITL listener for the requested FC adapter.
|
||||
sitl_host = "sitl-ardupilot" if fc_adapter == "ardupilot" else "sitl-inav"
|
||||
observer = sitl_observer.get_observer(fc_adapter=fc_adapter, host=sitl_host)
|
||||
sink = _resolve_frame_sink()
|
||||
replayer = FrameSourceReplayer(sink)
|
||||
|
||||
# 3. Push images one at a time, capturing per-image estimates.
|
||||
estimates: list[ae.EstimateInput] = []
|
||||
per_image_timeout_s = 5.0
|
||||
for path in image_paths:
|
||||
image_id = path.name
|
||||
replayer.replay_image(path)
|
||||
try:
|
||||
msg = observer.wait_for_outbound(timeout_s=per_image_timeout_s)
|
||||
estimates.append(
|
||||
ae.EstimateInput(
|
||||
image_id=image_id,
|
||||
est_lat_deg=float(msg.lat_deg),
|
||||
est_lon_deg=float(msg.lon_deg),
|
||||
)
|
||||
)
|
||||
except TimeoutError:
|
||||
estimates.append(
|
||||
ae.EstimateInput(image_id=image_id, est_lat_deg=math.inf, est_lon_deg=math.inf)
|
||||
)
|
||||
|
||||
# 4. Evaluate + emit CSV evidence.
|
||||
results, aggregate = ae.evaluate(gt_rows, estimates)
|
||||
out_csv = evidence_dir / f"ft-p-01-{fc_adapter}-{vio_strategy}.csv"
|
||||
ae.write_csv_evidence(out_csv, results)
|
||||
|
||||
# 5. Record NFR metrics for the run report.
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_01.pass_count_50m", float(aggregate.pass_count_50m), ac_id="AC-2"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_01.pass_count_20m", float(aggregate.pass_count_20m), ac_id="AC-3"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_01.timeout_count", float(aggregate.timeout_count), ac_id="AC-4"
|
||||
)
|
||||
|
||||
# 6. AC assertions.
|
||||
assert aggregate.pass_ac2, (
|
||||
f"AC-2 (50 m budget) failed: {aggregate.pass_count_50m}/60 "
|
||||
f"< required {ae.PASS_COUNT_50M_REQUIRED}; "
|
||||
f"timeouts={aggregate.timeout_count}"
|
||||
)
|
||||
assert aggregate.pass_ac3, (
|
||||
f"AC-3 (20 m budget) failed: {aggregate.pass_count_20m}/60 "
|
||||
f"< required {ae.PASS_COUNT_20M_REQUIRED}"
|
||||
)
|
||||
|
||||
|
||||
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
|
||||
"""Stub helper resolved when the underlying replayer lands."""
|
||||
raise NotImplementedError(
|
||||
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
|
||||
)
|
||||
@@ -0,0 +1,189 @@
|
||||
"""FT-P-04 — Derkachi frame-to-frame registration ≥95% on normal segments (AC-2.1a).
|
||||
|
||||
The full scenario:
|
||||
|
||||
1. Replay the Derkachi MP4 + IMU through the SUT.
|
||||
2. Collect per-video-frame ``registration_success`` from
|
||||
``NAMED_VALUE_FLOAT`` OR from the post-run FDR archive (whichever the
|
||||
SUT emits — both are public-boundary artefacts per AC-NEW-3).
|
||||
3. Derive "normal" segment classification from ``data_imu.csv`` only —
|
||||
AC-1 explicitly requires SCALED_IMU2-derived attitude (no internal
|
||||
SUT state).
|
||||
4. Compute success ratio over the normal denominator (AC-3 excludes
|
||||
sharp-turn frames).
|
||||
5. Emit ``ft-p-04-{fc_adapter}-{vio_strategy}.csv`` with one row per
|
||||
video frame for evidence.
|
||||
6. Assert AC-2 (ratio ≥ 0.95).
|
||||
|
||||
What this file owns:
|
||||
|
||||
* The AC-1 / AC-2 / AC-3 / AC-4 wiring above.
|
||||
* CSV evidence emission via the AZ-412-owned ``registration_classifier``.
|
||||
|
||||
What this file does NOT own:
|
||||
|
||||
* The MP4 video-replay path → ``runner.helpers.frame_source_replay``
|
||||
(stub; AZ-441) — skip-gated.
|
||||
* The IMU CSV replay → ``runner.helpers.imu_replay`` (stub; AZ-407
|
||||
leftover) — skip-gated.
|
||||
* The FDR-archive iteration → ``runner.helpers.fdr_reader`` (stub;
|
||||
AZ-441) — skip-gated.
|
||||
|
||||
When all three upstream helpers land, this file's runtime path activates
|
||||
automatically — the skip is keyed off the ``NotImplementedError`` from
|
||||
the helper imports, not off a hard-coded marker.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers import registration_classifier as rc
|
||||
|
||||
DERKACHI_DIR = (
|
||||
Path(__file__).resolve().parents[3]
|
||||
/ "_docs"
|
||||
/ "00_problem"
|
||||
/ "input_data"
|
||||
/ "flight_derkachi"
|
||||
)
|
||||
DERKACHI_IMU_CSV = DERKACHI_DIR / "data_imu.csv"
|
||||
DERKACHI_MP4 = DERKACHI_DIR / "flight_derkachi.mp4"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def _harness_helpers_implemented() -> bool:
|
||||
"""True iff every upstream helper FT-P-04 needs has a real impl."""
|
||||
from runner.helpers import fdr_reader, frame_source_replay, imu_replay
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
try:
|
||||
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
|
||||
try:
|
||||
replayer.replay_video(Path("/tmp/non-existent.mp4"))
|
||||
except NotImplementedError:
|
||||
return False
|
||||
try:
|
||||
list(fdr_reader.iter_records(Path("/tmp/non-existent")))
|
||||
except NotImplementedError:
|
||||
return False
|
||||
try:
|
||||
imu_replay.ImuReplayer(emitter=_NullImuEmitter()).replay(Path("/tmp/non-existent.csv")) # type: ignore[arg-type]
|
||||
except NotImplementedError:
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class _NullSink:
|
||||
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
|
||||
return None
|
||||
|
||||
|
||||
class _NullImuEmitter:
|
||||
def emit(self, sample: object) -> None:
|
||||
return None
|
||||
|
||||
|
||||
@pytest.mark.traces_to("AC-2.1a,AC-1,AC-2,AC-3,AC-4")
|
||||
def test_ft_p_04_derkachi_f2f_registration(
|
||||
fc_adapter: str,
|
||||
vio_strategy: str,
|
||||
evidence_dir, # type: ignore[no-untyped-def]
|
||||
run_id: str,
|
||||
nfr_recorder, # type: ignore[no-untyped-def]
|
||||
_harness_helpers_implemented: bool,
|
||||
) -> None:
|
||||
"""Full FT-P-04 scenario.
|
||||
|
||||
AC-1: classification reproducibility — unit-tested via
|
||||
``test_classify_frames_is_reproducible_ac1``.
|
||||
AC-2: ``success_ratio_over_normal_segments ≥ 0.95``.
|
||||
AC-3: sharp-turn frames excluded from the denominator.
|
||||
AC-4: parametrized across ``(fc_adapter, vio_strategy)``.
|
||||
"""
|
||||
if not _harness_helpers_implemented:
|
||||
pytest.skip(
|
||||
"FT-P-04 full replay requires runner.helpers.{frame_source_replay,"
|
||||
"imu_replay,fdr_reader} — currently AZ-441 / AZ-407 leftovers. "
|
||||
"Pure-logic ACs covered by e2e/_unit_tests/helpers/test_registration_classifier.py."
|
||||
)
|
||||
|
||||
from runner.helpers import fdr_reader, imu_replay
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
# 1. Build the per-frame classification from data_imu.csv up-front.
|
||||
imu_rows = rc.load_imu_telemetry(DERKACHI_IMU_CSV)
|
||||
classifications = rc.classify_frames(imu_rows)
|
||||
|
||||
# 2. Drive the replay.
|
||||
sink = _resolve_frame_sink()
|
||||
emitter = _resolve_fc_inbound_emitter(fc_adapter)
|
||||
FrameSourceReplayer(sink).replay_video(DERKACHI_MP4)
|
||||
imu_replay.ImuReplayer(emitter).replay(DERKACHI_IMU_CSV)
|
||||
|
||||
# 3. Collect per-frame registration_success from the FDR archive.
|
||||
fdr_root = Path(evidence_dir).parent / f"run-{run_id}" / "fdr"
|
||||
registration_success_by_frame: dict[int, bool] = {}
|
||||
for rec in fdr_reader.iter_records(fdr_root):
|
||||
if rec.record_type == "frame_metric":
|
||||
payload = rec.payload
|
||||
if payload.get("metric") == "registration_success":
|
||||
frame_index = int(payload["frame_index"]) # type: ignore[arg-type]
|
||||
registration_success_by_frame[frame_index] = bool(payload["value"]) # type: ignore[arg-type]
|
||||
|
||||
if not registration_success_by_frame:
|
||||
pytest.fail(
|
||||
"FT-P-04: SUT did not emit any frame_metric records with "
|
||||
"metric='registration_success' (required by AC-NEW-3 FDR schema)."
|
||||
)
|
||||
|
||||
# 4. Compute success report + emit evidence.
|
||||
report = rc.compute_success_ratio(classifications, registration_success_by_frame)
|
||||
out_csv = evidence_dir / f"ft-p-04-{fc_adapter}-{vio_strategy}.csv"
|
||||
rc.write_csv_evidence(out_csv, classifications, registration_success_by_frame)
|
||||
|
||||
# 5. Record NFR metrics for the run report.
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_04.success_ratio", report.ratio, ac_id="AC-2"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_04.normal_denominator", float(report.denominator), ac_id="AC-3"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_04.excluded_by_attitude", float(report.excluded_by_attitude), ac_id="AC-3"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_04.excluded_by_overlap", float(report.excluded_by_overlap), ac_id="AC-3"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_04.excluded_by_missing_metric",
|
||||
float(report.excluded_by_missing_metric),
|
||||
ac_id="AC-2",
|
||||
)
|
||||
|
||||
# 6. AC-2 assertion.
|
||||
assert report.passes, (
|
||||
f"AC-2 (registration ≥{rc.SUCCESS_RATIO_REQUIRED:.0%}) failed: "
|
||||
f"ratio={report.ratio:.4f} over {report.denominator} normal frames "
|
||||
f"(excluded: attitude={report.excluded_by_attitude}, "
|
||||
f"overlap={report.excluded_by_overlap}, "
|
||||
f"missing_metric={report.excluded_by_missing_metric})"
|
||||
)
|
||||
|
||||
|
||||
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
|
||||
"""Stub helper resolved when the underlying replayer lands."""
|
||||
raise NotImplementedError(
|
||||
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
|
||||
)
|
||||
|
||||
|
||||
def _resolve_fc_inbound_emitter(fc_adapter: str): # type: ignore[no-untyped-def]
|
||||
"""Stub helper resolved when the FC inbound emitter lands."""
|
||||
raise NotImplementedError(
|
||||
"FC inbound emitter resolution is owned by AZ-416/AZ-417 / runner.helpers.imu_replay"
|
||||
)
|
||||
@@ -0,0 +1,195 @@
|
||||
"""FT-P-05 — Satellite-anchor cross-domain registration MRE + accuracy (AC-2.1b).
|
||||
|
||||
The full scenario:
|
||||
|
||||
1. Push each ``AD0000NN.jpg`` from ``still-image-set-60`` to the SUT.
|
||||
2. Wait for the SUT's outbound estimate (same path as FT-P-01) + record
|
||||
per-image MRE from ``NAMED_VALUE_FLOAT`` or post-run FDR.
|
||||
3. Compute geodesic error vs ``coordinates.csv`` GT (delegated to
|
||||
``accuracy_evaluator``).
|
||||
4. Emit ``ft-p-05-{fc_adapter}-{vio_strategy}.csv`` (image_id, est_lat,
|
||||
est_lon, error_m, mre_px, pass_50m, pass_20m, pass_mre).
|
||||
5. Assert AC-2 (every MRE < 2.5 px) AND AC-3 (≥80 % within 50 m AND
|
||||
≥50 % within 20 m — same image set as FT-P-01; this AC is
|
||||
"implied by FT-P-01" if FT-P-01 passes in the same run).
|
||||
|
||||
What this file owns:
|
||||
|
||||
* The AC-1 / AC-2 / AC-3 / AC-5 wiring above.
|
||||
* CSV evidence emission via the AZ-413-owned ``mre_evaluator``.
|
||||
|
||||
What this file does NOT own:
|
||||
|
||||
* The frame-source push → ``runner.helpers.frame_source_replay`` (stub;
|
||||
AZ-441) — skip-gated.
|
||||
* The SITL message receipt + MRE harvesting → ``runner.helpers.{sitl_observer,
|
||||
fdr_reader}`` (stubs; AZ-416/AZ-417, AZ-441) — skip-gated.
|
||||
|
||||
When the upstream helpers land, this file's runtime path activates
|
||||
automatically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers import accuracy_evaluator as ae
|
||||
from runner.helpers import mre_evaluator as me
|
||||
|
||||
GT_CSV = Path(__file__).resolve().parents[3] / "_docs" / "00_problem" / "input_data" / "coordinates.csv"
|
||||
STILL_IMAGES_DIR = GT_CSV.parent
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def _harness_helpers_implemented() -> bool:
|
||||
"""True iff replay + SITL observation + FDR helpers are all real."""
|
||||
from runner.helpers import fdr_reader, frame_source_replay, sitl_observer
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
try:
|
||||
replayer = FrameSourceReplayer(sink=_NullSink()) # type: ignore[arg-type]
|
||||
try:
|
||||
replayer.replay_image_directory(Path("/tmp/non-existent"))
|
||||
except NotImplementedError:
|
||||
return False
|
||||
try:
|
||||
sitl_observer.get_observer(fc_adapter="ardupilot", host="sitl-ardupilot")
|
||||
except NotImplementedError:
|
||||
return False
|
||||
try:
|
||||
list(fdr_reader.iter_records(Path("/tmp/non-existent")))
|
||||
except NotImplementedError:
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class _NullSink:
|
||||
def write_frame(self, jpeg_bytes: bytes, timestamp_ms: int) -> None:
|
||||
return None
|
||||
|
||||
|
||||
@pytest.mark.traces_to("AC-2.1b,AC-1,AC-2,AC-3,AC-5")
|
||||
def test_ft_p_05_sat_anchor(
|
||||
fc_adapter: str,
|
||||
vio_strategy: str,
|
||||
evidence_dir, # type: ignore[no-untyped-def]
|
||||
run_id: str,
|
||||
nfr_recorder, # type: ignore[no-untyped-def]
|
||||
_harness_helpers_implemented: bool,
|
||||
) -> None:
|
||||
"""Full FT-P-05 scenario.
|
||||
|
||||
AC-1: per-image MRE captured in ``ft-p-05.csv``.
|
||||
AC-2: every MRE < 2.5 px.
|
||||
AC-3: ≥80 % within 50 m AND ≥50 % within 20 m (same image set as FT-P-01).
|
||||
AC-5: parametrized across ``(fc_adapter, vio_strategy)``.
|
||||
"""
|
||||
if not _harness_helpers_implemented:
|
||||
pytest.skip(
|
||||
"FT-P-05 still-image push requires runner.helpers.{frame_source_replay,"
|
||||
"sitl_observer,fdr_reader} — currently AZ-441 + AZ-416/AZ-417 leftovers. "
|
||||
"Pure-logic ACs covered by e2e/_unit_tests/helpers/test_mre_evaluator.py."
|
||||
)
|
||||
|
||||
from runner.helpers import fdr_reader, frame_source_replay, sitl_observer
|
||||
from runner.helpers.frame_source_replay import FrameSourceReplayer
|
||||
|
||||
# 1. Resolve GT + image inventory.
|
||||
gt_rows = ae.load_gt_coordinates(GT_CSV)
|
||||
image_paths = sorted(STILL_IMAGES_DIR.glob("AD??????.jpg"))
|
||||
if len(image_paths) != ae.TOTAL_IMAGES_REQUIRED:
|
||||
pytest.fail(
|
||||
f"FT-P-05 expects {ae.TOTAL_IMAGES_REQUIRED} images, found {len(image_paths)}"
|
||||
)
|
||||
|
||||
# 2. Push images, collect (est_lat, est_lon, mre_px) per image.
|
||||
sitl_host = "sitl-ardupilot" if fc_adapter == "ardupilot" else "sitl-inav"
|
||||
observer = sitl_observer.get_observer(fc_adapter=fc_adapter, host=sitl_host)
|
||||
sink = _resolve_frame_sink()
|
||||
replayer = FrameSourceReplayer(sink)
|
||||
|
||||
estimates: list[ae.EstimateInput] = []
|
||||
mre_records: list[me.CrossDomainRecord] = []
|
||||
per_image_timeout_s = 5.0
|
||||
for path in image_paths:
|
||||
image_id = path.name
|
||||
replayer.replay_image(path)
|
||||
try:
|
||||
msg = observer.wait_for_outbound(timeout_s=per_image_timeout_s)
|
||||
estimates.append(
|
||||
ae.EstimateInput(
|
||||
image_id=image_id,
|
||||
est_lat_deg=float(msg.lat_deg),
|
||||
est_lon_deg=float(msg.lon_deg),
|
||||
)
|
||||
)
|
||||
mre_records.append(
|
||||
me.CrossDomainRecord(
|
||||
image_id=image_id,
|
||||
mre_px=float(msg.mre_px),
|
||||
error_m=0.0, # filled in once geodesic computed
|
||||
)
|
||||
)
|
||||
except TimeoutError:
|
||||
estimates.append(
|
||||
ae.EstimateInput(image_id=image_id, est_lat_deg=math.inf, est_lon_deg=math.inf)
|
||||
)
|
||||
mre_records.append(
|
||||
me.CrossDomainRecord(image_id=image_id, mre_px=math.inf, error_m=math.inf)
|
||||
)
|
||||
|
||||
# 3. Compute per-image error_m by joining with GT.
|
||||
per_image_results, accuracy_aggregate = ae.evaluate(gt_rows, estimates)
|
||||
pass_50m = {r.image_id: r.pass_50m for r in per_image_results}
|
||||
pass_20m = {r.image_id: r.pass_20m for r in per_image_results}
|
||||
error_by_image = {r.image_id: r.error_m for r in per_image_results}
|
||||
mre_records = [
|
||||
me.CrossDomainRecord(
|
||||
image_id=r.image_id, mre_px=r.mre_px, error_m=error_by_image[r.image_id]
|
||||
)
|
||||
for r in mre_records
|
||||
]
|
||||
|
||||
# 4. Emit FT-P-05 evidence.
|
||||
out_csv = evidence_dir / f"ft-p-05-{fc_adapter}-{vio_strategy}.csv"
|
||||
me.write_cross_domain_csv(out_csv, mre_records, pass_50m=pass_50m, pass_20m=pass_20m)
|
||||
|
||||
# 5. Evaluate AC-2 + record NFR metrics.
|
||||
mre_report = me.evaluate_per_image_budget(mre_records)
|
||||
nfr_recorder.record_metric("ft_p_05.max_mre_px", mre_report.max_mre_px, ac_id="AC-2")
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_05.mre_pass_count", float(mre_report.pass_count), ac_id="AC-2"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_05.pass_count_50m", float(accuracy_aggregate.pass_count_50m), ac_id="AC-3"
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_05.pass_count_20m", float(accuracy_aggregate.pass_count_20m), ac_id="AC-3"
|
||||
)
|
||||
|
||||
# 6. AC assertions.
|
||||
assert mre_report.passes, (
|
||||
f"AC-2 (cross-domain MRE < {me.MRE_PER_IMAGE_BUDGET_PX} px) failed: "
|
||||
f"{len(mre_report.fail_image_ids)} image(s) over budget; "
|
||||
f"max_mre={mre_report.max_mre_px:.4f} px; "
|
||||
f"failing image_ids={list(mre_report.fail_image_ids)[:5]}"
|
||||
)
|
||||
assert accuracy_aggregate.pass_ac2, (
|
||||
f"AC-3 (50 m budget — implied by FT-P-01) failed: "
|
||||
f"{accuracy_aggregate.pass_count_50m}/60 < {ae.PASS_COUNT_50M_REQUIRED}"
|
||||
)
|
||||
assert accuracy_aggregate.pass_ac3, (
|
||||
f"AC-3 (20 m budget — implied by FT-P-01) failed: "
|
||||
f"{accuracy_aggregate.pass_count_20m}/60 < {ae.PASS_COUNT_20M_REQUIRED}"
|
||||
)
|
||||
|
||||
|
||||
def _resolve_frame_sink(): # type: ignore[no-untyped-def]
|
||||
raise NotImplementedError(
|
||||
"frame sink resolution is owned by AZ-441 / runner.helpers.frame_source_replay"
|
||||
)
|
||||
@@ -0,0 +1,93 @@
|
||||
"""FT-P-06 — 95th-percentile MRE budgets (AC-2.2).
|
||||
|
||||
Piggyback test: depends on the FT-P-04 + FT-P-05 evidence CSVs produced
|
||||
in the same run. Reads both, aggregates per domain, asserts:
|
||||
|
||||
* Frame-to-frame p95 MRE < 1.0 px
|
||||
* Cross-domain p95 MRE < 2.5 px
|
||||
|
||||
What this file owns:
|
||||
|
||||
* The AC-4 assertion + the combined report.
|
||||
|
||||
What this file does NOT own:
|
||||
|
||||
* The FT-P-04 evidence collection — owned by ``test_ft_p_04_*``.
|
||||
* The FT-P-05 evidence collection — owned by ``test_ft_p_05_*``.
|
||||
* Both run as the same pytest session; this test depends on the
|
||||
artefacts they wrote to ``evidence_dir``.
|
||||
|
||||
Skip discipline: if either evidence CSV is missing, the test SKIPS with
|
||||
a clear reason (it cannot fail without the upstream evidence; that
|
||||
would mask the actual gate, which is whether FT-P-04 / FT-P-05 ran).
|
||||
The autodev / Tier-1 runner will only mark this test FAIL if it runs
|
||||
AND the evidence is present AND the p95 budgets are exceeded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from runner.helpers import mre_evaluator as me
|
||||
|
||||
|
||||
@pytest.mark.traces_to("AC-2.2,AC-4,AC-5")
|
||||
def test_ft_p_06_mre_budgets(
|
||||
fc_adapter: str,
|
||||
vio_strategy: str,
|
||||
evidence_dir, # type: ignore[no-untyped-def]
|
||||
nfr_recorder, # type: ignore[no-untyped-def]
|
||||
) -> None:
|
||||
"""AC-4: 95th-percentile MRE < 1.0 px f2f AND < 2.5 px cross-domain.
|
||||
|
||||
AC-5: parametrized across ``(fc_adapter, vio_strategy)``.
|
||||
|
||||
This test is a pure piggyback — it reads the FT-P-04 + FT-P-05 CSVs
|
||||
from the same run. If either is missing the test skips (without
|
||||
those, FT-P-06 has nothing to assert on).
|
||||
"""
|
||||
f2f_csv = evidence_dir / f"ft-p-04-{fc_adapter}-{vio_strategy}.csv"
|
||||
xd_csv = evidence_dir / f"ft-p-05-{fc_adapter}-{vio_strategy}.csv"
|
||||
|
||||
if not f2f_csv.exists() or not xd_csv.exists():
|
||||
missing = [str(p.name) for p in (f2f_csv, xd_csv) if not p.exists()]
|
||||
pytest.skip(
|
||||
f"FT-P-06 piggybacks on FT-P-04 + FT-P-05 evidence; missing in this run: {missing}. "
|
||||
"Pure-logic ACs covered by e2e/_unit_tests/helpers/test_mre_evaluator.py."
|
||||
)
|
||||
|
||||
# Both CSVs present — load and evaluate.
|
||||
try:
|
||||
f2f_records = me.load_frame_to_frame_csv(f2f_csv)
|
||||
except ValueError as exc:
|
||||
# mre_px column absent → FT-P-04 evidence does not yet carry MRE.
|
||||
# Per the FT-P-06 spec: "if absent, the test fails" — but at this
|
||||
# point the failure is on the SUT (it must expose per-frame MRE).
|
||||
pytest.fail(f"FT-P-04 evidence is missing per-frame MRE: {exc}")
|
||||
xd_records = me.load_cross_domain_csv(xd_csv)
|
||||
|
||||
combined = me.evaluate_combined_p95(f2f_records, xd_records)
|
||||
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_06.f2f_p95_mre_px",
|
||||
combined.frame_to_frame.p95_px,
|
||||
ac_id="AC-4",
|
||||
)
|
||||
nfr_recorder.record_metric(
|
||||
"ft_p_06.cross_domain_p95_mre_px",
|
||||
combined.cross_domain.p95_px,
|
||||
ac_id="AC-4",
|
||||
)
|
||||
|
||||
assert combined.frame_to_frame.passes, (
|
||||
f"AC-4 (frame-to-frame p95 MRE < {me.MRE_P95_FRAME_TO_FRAME_BUDGET_PX} px) "
|
||||
f"failed: p95={combined.frame_to_frame.p95_px:.4f} over "
|
||||
f"{combined.frame_to_frame.sample_count} samples"
|
||||
)
|
||||
assert combined.cross_domain.passes, (
|
||||
f"AC-4 (cross-domain p95 MRE < {me.MRE_P95_CROSS_DOMAIN_BUDGET_PX} px) "
|
||||
f"failed: p95={combined.cross_domain.p95_px:.4f} over "
|
||||
f"{combined.cross_domain.sample_count} samples"
|
||||
)
|
||||
Reference in New Issue
Block a user