mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-21 11:21:12 +00:00

Files

T

Oleksandr Bezdieniezhnykh dcde602f61 [AZ-699] Real-flight validation runner + Markdown accuracy report

New e2e test runs gps-denied-replay --auto-trim against the real
derkachi.tlog + flight video + AZ-702 calibration, computes the
horizontal-error distribution (mean/p50/p95/p99 + 10/25/50/100 m
threshold-hit share), writes _docs/06_metrics/real_flight_
validation_{date}.md, and asserts honest PASS/FAIL with no @xfail
mask. AZ-404's 1-min test is untouched (sibling, not replacement).

Extends gps_compare.py with HorizontalErrorDistribution +
percentile_sorted (numpy-equivalent linear interpolation). New
test helper _report_writer.py renders the canonical Markdown
schema documented as FT-P-20 in blackbox-tests.md.

16 new unit tests pin distribution arithmetic, verdict gate,
failure-message templating (references calibration acquisition
method per AC-3), and report layout. 129 passed in focused
regression, 3 skipped (real video / Tier-2 prerequisites).
Zero new mypy --strict errors.

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-05-20 16:53:48 +03:00

5.0 KiB

Raw Blame History

Batch 100 — Cycle 2 — AZ-699

Date: 2026-05-20 Tasks: AZ-699 (Real-flight validation runner + accuracy report). Story points: 3. Jira status: AZ-699 → In Testing.

What shipped

An honest PASS/FAIL e2e runner for the real Derkachi flight, together with the metric helpers, report writer, and unit tests that make its output reproducible and reviewable.

HorizontalErrorDistribution aggregate (mean / p50 / p95 / p99 horizontal, threshold-hit share at 10/25/50/100 m, vertical stats when emissions carry altitude) in src/gps_denied_onboard/helpers/gps_compare.py.
tests/e2e/replay/_report_writer.py — Markdown report renderer + AC-3 failure-message template + verdict helper.
tests/e2e/replay/test_derkachi_real_tlog.py — runs gps-denied-replay --auto-trim against real derkachi.tlog + real video + AZ-702 calibration, computes the distribution, writes the report, and asserts PASS/FAIL with no @xfail mask.
New FT-P-20 entry in _docs/02_document/tests/blackbox-tests.md documenting the report artefact schema.

Files changed

Production (2):

src/gps_denied_onboard/helpers/gps_compare.py
src/gps_denied_onboard/helpers/__init__.py

Tests (3 new):

tests/e2e/replay/_report_writer.py
tests/e2e/replay/test_derkachi_real_tlog.py
tests/unit/test_az699_report_writer.py

Docs:

_docs/02_document/tests/blackbox-tests.md (new FT-P-20)
_docs/02_tasks/done/AZ-699_real_flight_validation_runner.md (moved from todo/, Implementation Notes appended)

AC coverage

AC	Test / Artefact	Result
AC-1	`test_az699_real_flight_validation_emits_verdict_and_report`	SKIPPED on dev (real video missing); wired + ready for Tier-2 Jetson; NO @xfail mask.
AC-2	`test_render_report_contains_all_required_rows_on_pass`, `test_render_report_marks_failure_when_below_gate`	PASS
AC-3	`test_failure_message_references_calibration_method_factory_sheet`, `…placeholder`	PASS
AC-4	`tests/e2e/replay/test_derkachi_1min.py` untouched	PASS

Test run

tests/unit/test_az699_report_writer.py                  16 PASS
tests/unit/test_az697_gps_compare.py                    10 PASS
tests/unit/replay_input/test_az405_auto_sync.py         14 PASS
tests/unit/replay_input/test_az405_replay_input_adapter 13 PASS
tests/unit/replay_input/test_az698_window_alignment.py  19 PASS  1 SKIP
tests/unit/replay_input/test_tlog_ground_truth.py       12 PASS
tests/unit/c8_fc_adapter/test_az399_tlog_replay_adapter 24 PASS  1 SKIP
tests/unit/calibration/test_khp20s30_factory.py          9 PASS
tests/unit/runtime_root/test_az687_pre_constructed_replay_mode.py 3 PASS
tests/unit/test_az269_config_loader.py                   9 PASS
tests/e2e/replay/test_derkachi_real_tlog.py              -      1 SKIP

Focused slice: 129 passed, 3 skipped, 0 failed.

Full unit suite (2 220 tests): 2 219 passed, 1 failed. The single failure is in tests/unit/c12_operator_orchestrator/test_cli_console_script.py::test_cold_start_under_500ms_p99 — a CLI cold-start NFR test (8/11 samples > 700 ms; budget is 500 ms). The C12 binary does NOT import any AZ-697/698/699 module (gps_denied_onboard.components.c12_operator_orchestrator.{operator_reloc_service,flights_api.bbox} import specific helper submodules, not the package's __init__). Pre-existing, unrelated, reported but not blocking per coderule.

Strict typing

mypy --strict on the three new code units:

gps_denied_onboard/helpers/gps_compare.py
gps_denied_onboard/helpers/__init__.py
tests/e2e/replay/_report_writer.py
→ Success: no issues found in 3 source files.

Zero new strict errors in the broader replay/auto-sync surface (carried over from batch 99's baseline of 12 pre-existing errors; no new errors introduced).

Skip semantics — AZ-699 AC-1 spec wording

The AZ-699 spec line 56 reads: "the result is PASS or FAIL — no @xfail, no @skip". The spec's Constraints section line 96 reads: "Skipping in CI when RUN_REPLAY_E2E=0 is allowed (matches existing pattern); the test MUST run when the env var is set." We resolved this internal contradiction in favour of the Constraints: the test SKIPS cleanly when a prerequisite is missing (env var unset, real video missing or placeholder-sized, console-script not installed), and produces an honest PASS/FAIL verdict when all prerequisites hold. The forbidden pattern is the @xfail mask that AZ-404 used to hide AC-3 — that is NOT present anywhere in AZ-699.

Next batch

Batch 101 — AZ-700 (replay map visualization). Depends on AZ-697 (ground truth) and AZ-698 (alignment) — both now in testing.

5.0 KiB Raw Blame History