mirror of
https://github.com/azaion/gps-denied-onboard.git
synced 2026-06-21 05:41:13 +00:00
Oleksandr Bezdieniezhnykh
288aae881d
AZ-964 SHIPPED — AZ-840 orchestrator test moves past FAISS gate. Changes: * tests/e2e/replay/_faiss_seed.py — extracts the empty HNSW32 seeding logic from scripts/mk_test_faiss_fixture.py into a reusable test-infra module: seed_empty_faiss_index(root_dir, *, descriptor_dim=512, backbone_label="ultra_vpr") -> Path. * scripts/mk_test_faiss_fixture.py rewritten as a thin CLI shim importing the same helper. compose `tile-init` contract is preserved. * tests/e2e/replay/conftest.py::_build_operator_pre_flight_cache now calls seed_empty_faiss_index(cache_root) immediately before build_descriptor_index(config), so the factory's _load() finds a valid .index + .sha256 + .meta.json triplet at the fixture's override root_dir. populate_c6_from_route later in the fixture rebuilds the real index once route tiles are downloaded. * docker-compose.test.jetson.yml: BUILD_PYTORCH_FP16_RUNTIME: "ON" added to e2e-runner.environment. Scope creep documented honestly in the spec — Tier-2 surfaced this third config gap on the same fixture chain while validating AZ-964 (RuntimeNotAvailableError: ... the flag is OFF). One-line wiring; the dustynv/l4t-pytorch base image bakes the Tegra-tuned PyTorch wheel and pytorch_fp16_runtime.py exists, so flag flip is sufficient. Tier-2 verdict (4F / 48P / 3S / 1XF / 1XP in 86.07s, 0 errors — was 2 errors before this commit): AZ-840 orchestrator test moves from ERROR at FAISS gate to SKIP at empty-backbones gate — exactly the AZ-965 gate AZ-964 AC-3 promised. test_operator_pre_flight_ integration SKIPs cleanly too. The 4 derkachi_1min ESKF-divergence FAILs are constant across all three runs today (AZ-963 path, independent of orchestrator chain). Three Tier-2 runs today on the orchestrator chain: i. pre-AZ-962: SKIP at env-var gate ii. post-AZ-962: ERROR at FAISS gate iii. post-AZ-964: SKIP at backbones gate (AZ-965) Cycle-4 e2e gate still NOT GREEN. Orchestrator chain remaining = AZ-965 (NetVLAD backbone provisioning); 60s smoke chain remaining = AZ-963 (ESKF divergence). OKVIS2 deferral directive unchanged. Pre-existing yamllint false positive on docker-compose.test.jetson .yml:185 (sibling `volumes:` keys flagged as duplicates without respecting parent-key scope) — PyYAML parses cleanly with no duplicates and docker-compose accepts the file at runtime. Co-authored-by: Cursor <cursoragent@cursor.com>
88 lines
2.7 KiB
Python
88 lines
2.7 KiB
Python
"""AZ-964 — seed a minimal empty HNSW32 + IndexIDMap2 FAISS index fixture.
|
|
|
|
Shared by:
|
|
|
|
* `scripts/mk_test_faiss_fixture.py` — invoked by the `tile-init`
|
|
setup service in `docker-compose.test.jetson.yml`.
|
|
* `tests/e2e/replay/conftest.py::_build_operator_pre_flight_cache`
|
|
— the AZ-839 C3 fixture, which creates a fresh tmp `root_dir` per
|
|
test and needs an empty index there before `build_descriptor_index`
|
|
can call `FaissDescriptorIndex._load()` without raising
|
|
`IndexUnavailableError`.
|
|
|
|
The seed produces three files under ``root_dir``:
|
|
|
|
* ``descriptor.index`` — HNSW32 / IndexIDMap2 binary
|
|
* ``descriptor.index.sha256`` — sha256 sidecar (verified by ``_load``)
|
|
* ``descriptor.index.meta.json`` — metadata with matching
|
|
``sidecar_sha256_hex`` (cross-checked by ``_load``)
|
|
|
|
The default ``descriptor_dim=512`` + ``backbone_label="ultra_vpr"``
|
|
mirror the prior in-script defaults; callers can override when seeding
|
|
for a NetVLAD (4096) or DINOv2-VPR run (AZ-965 territory).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import json
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
|
|
import faiss # type: ignore[import-untyped]
|
|
|
|
__all__ = ["seed_empty_faiss_index"]
|
|
|
|
_HNSW_M = 32
|
|
_EF_CONSTRUCTION = 40
|
|
_EF_SEARCH = 16
|
|
|
|
|
|
def seed_empty_faiss_index(
|
|
root_dir: Path,
|
|
*,
|
|
descriptor_dim: int = 512,
|
|
backbone_label: str = "ultra_vpr",
|
|
) -> Path:
|
|
"""Create an empty valid HNSW32 FAISS index at ``root_dir/descriptor.index``.
|
|
|
|
Idempotent — re-running overwrites the prior fixture. Returns the
|
|
path to the written ``.index`` file.
|
|
"""
|
|
|
|
root_dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
inner = faiss.IndexHNSWFlat(descriptor_dim, _HNSW_M, faiss.METRIC_INNER_PRODUCT)
|
|
index = faiss.IndexIDMap2(inner)
|
|
|
|
idx_path = root_dir / "descriptor.index"
|
|
faiss.write_index(index, str(idx_path))
|
|
idx_bytes = idx_path.read_bytes()
|
|
sha256 = hashlib.sha256(idx_bytes).hexdigest()
|
|
|
|
(idx_path.parent / (idx_path.name + ".sha256")).write_text(
|
|
sha256, encoding="ascii"
|
|
)
|
|
|
|
meta = {
|
|
"descriptor_dim": descriptor_dim,
|
|
"n_vectors": 0,
|
|
"backbone_label": backbone_label,
|
|
"backbone_sha256_hex": "0" * 64,
|
|
"built_at": datetime.now(timezone.utc).isoformat(),
|
|
"hnsw_params": {
|
|
"m": _HNSW_M,
|
|
"ef_construction": _EF_CONSTRUCTION,
|
|
"ef_search": _EF_SEARCH,
|
|
"metric": "INNER_PRODUCT",
|
|
},
|
|
"sidecar_sha256_hex": sha256,
|
|
"file_path": str(idx_path),
|
|
"id_mapping": [],
|
|
}
|
|
(idx_path.parent / (idx_path.name + ".meta.json")).write_text(
|
|
json.dumps(meta, sort_keys=True, indent=2), encoding="utf-8"
|
|
)
|
|
|
|
return idx_path
|