azaion/gps-denied-onboard
1
0
Fork 0
mirror of https://github.com/azaion/gps-denied-onboard.git synced 2026-06-22 07:31:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

[AZ-841] Remove xfail markers from Derkachi tests — environment segregation via tier2+RUN_REPLAY_E2E

Browse Source
This commit is contained in:
Oleksandr Bezdieniezhnykh
2026-06-10 05:35:01 +03:00
parent 201ec7cdd4
commit c1baef57be
4 changed files with 33 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_docs/02_tasks/backlog/AZ-841_unxfail_az777_tier2_tests.md
-71
View File
@@ -1,71 +0,0 @@
# Un-xfail AZ-777 AC-4 + AC-5 Tier-2 tests (AZ-835 C5)
> **Cycle-4 deferral (2026-05-26)**: moved to `backlog/` during cycle-4 Step 9
> scope review. Blocking issues:
> - **Conflict with AZ-895 AC-4**: AZ-895 (cycle-4 cleanup) explicitly states
> `test_derkachi_real_tlog.py` stays `@xfail` with the AZ-848-scoped reason
> in cycle 4. Un-xfailing this test here contradicts AZ-895 and will fail
> the Jetson run because AZ-848 (the underlying clock bug) is in backlog/.
> - **Partial overlap with AZ-894 AC-3**: the other un-xfail target
> (`test_derkachi_1min.py::AC3`) is the same test AZ-894 (cycle-4 CSV
> adapter) covers under its own AC-3 — re-doing the un-xfail in a
> separate ticket duplicates effort.
> - **Replay condition**: revisit when EITHER (a) AZ-848 is fixed and the
> tlog adapter path is restored, OR (b) cycle 4 lands and we rescope this
> ticket to only the CSV-path tests AZ-894 doesn't already cover.
**Task**: AZ-841_unxfail_az777_tier2_tests
**Name**: Un-xfail AZ-777 AC-4 + AC-5 Tier-2 tests once C3 fixture + C4 orchestrator land (AZ-835 C5)
**Description**: Fifth building block of Epic AZ-835. Once C3 (AZ-839, `operator_pre_flight_setup` real fixture) and C4 (AZ-840, e2e orchestrator test) land, remove the `@pytest.mark.xfail` markers from the AZ-777 Tier-2 tests. The verdict — PASS or FAIL — becomes the honest signal. Both tests remain gated by `RUN_REPLAY_E2E=1` + `@pytest.mark.tier2`.
**Complexity**: 1 SP
**Dependencies**: AZ-839 (C3, `operator_pre_flight_setup` real fixture — HARD); AZ-840 (C4, e2e orchestrator test — HARD); AZ-777 (being closed/superseded by this Epic; tests live in same file tree); AZ-835 (parent Epic)
**Component**: `tests/e2e/replay/test_derkachi_1min.py` (xfail removal) + `tests/e2e/replay/test_derkachi_real_tlog.py` (xfail removal)
**Tracker**: AZ-841 (https://denyspopov.atlassian.net/browse/AZ-841)
**Parent Epic**: AZ-835
Jira AZ-841 is the authoritative spec; this file is the in-workspace mirror.
## Targets
1. `tests/e2e/replay/test_derkachi_1min.py::test_ac3_within_100m_80pct_of_ticks` (AZ-777 AC-4) — remove `@pytest.mark.xfail`; verify `@pytest.mark.tier2` + `RUN_REPLAY_E2E` gating stays in place.
2. `tests/e2e/replay/test_derkachi_real_tlog.py::test_az699_real_flight_validation_emits_verdict_and_report` (AZ-777 AC-5) — remove `@pytest.mark.xfail`; verify gating stays in place.
## Verification
**On Tier-2 Jetson** (`RUN_REPLAY_E2E=1`):
- `test_ac3_within_100m_80pct_of_ticks` PASSES (≥ 80 % of ticks within 100 m of ground truth, log lines `replay.satellite_anchor_inserted` visible).
- `test_az699_real_flight_validation_emits_verdict_and_report` runs to completion within 15 min and emits `_docs/06_metrics/real_flight_validation_<YYYY-MM-DD>.md` with the honest distribution. PASS preferred but NOT required for AC-4 — emitting the honest report IS the success criterion.
**Locally** (no env):
- Both tests skip explicitly with a reason naming `RUN_REPLAY_E2E` — they MUST NOT pass as a side effect of being skipped.
## Acceptance criteria
| # | Criterion |
|---|-----------|
| AC-1 | `@pytest.mark.xfail` removed from both AZ-777 tests. |
| AC-2 | Both tests still gated by `@pytest.mark.tier2` + skip-unless-env(`RUN_REPLAY_E2E=1`). Skip reason names the missing env. |
| AC-3 | On Jetson with `RUN_REPLAY_E2E=1`, `test_ac3_within_100m_80pct_of_ticks` PASSES (≥ 80 % within 100 m). |
| AC-4 | On Jetson with `RUN_REPLAY_E2E=1`, `test_az699_real_flight_validation_emits_verdict_and_report` completes within 15 min and emits the verdict report. PASS preferred but not required for AC-4. |
| AC-5 | If either test FAILS on the metric (e.g. only 60 % within 100 m), the test reports FAIL honestly — no fallback to xfail or skip. Failure mode is a feature, not a bug. |
| AC-6 | Locally on a machine without `RUN_REPLAY_E2E`, both tests skip with an explicit reason. |
## Out of scope
- Modifying the airborne pipeline to improve metric performance (separate optimization tickets if AC-3 fails).
- Adding new test cases (this ticket only removes xfail; new cases belong to other tickets).
- Documentation updates (AZ-842 / C6).
- Modifying the verdict thresholds (AZ-696).
## Risks
**Risk 1 — Un-xfailed tests may FAIL on the metric.** If horizontal-error distribution comes in worse than the 80 % @ 100 m gate, this test reports FAIL. That outcome is in-scope for AC-5 (report honestly) and out-of-scope for this ticket's fix (file a separate optimization ticket).
## References
- Parent Epic: AZ-835 — https://denyspopov.atlassian.net/browse/AZ-835
- Hard deps: AZ-839 (C3), AZ-840 (C4)
- Tests: `tests/e2e/replay/test_derkachi_1min.py`, `tests/e2e/replay/test_derkachi_real_tlog.py`
- AZ-777 spec: `_docs/02_tasks/done/AZ-777_derkachi_c6_reference_fixture.md` (post-closure)
- Threshold spec: AZ-696 (≥ 80 % within 100 m)
- Verdict writer: `src/gps_denied_onboard/helpers/accuracy_report.py`