# C6 FaissDescriptorIndex — HNSW Search + Atomic Rebuild + pybind11 Wrapper

**Task**: AZ-306_c6_faiss_descriptor_index
**Name**: C6 FaissDescriptorIndex
**Description**: Implement `FaissDescriptorIndex`, the production-default `DescriptorIndex` Protocol strategy. Owns the F1 pre-flight `rebuild_from_descriptors` path (atomic `.index` file write + sidecar via AZ-280), the F2 takeoff load (mmap with `IO_FLAG_MMAP_IFC`), the F3 hot-path `search_topk` (HNSW; ≤ 5 ms p95 warm; sole consumer is C2 VPR), the `index_metadata` sidecar block, and the `cpp/faiss_index/` pybind11 wrapper that links FAISS HEAD-pinned per Plan-phase under the `BUILD_FAISS_INDEX` flag.
**Complexity**: 5 points
**Dependencies**: AZ-303_c6_storage_interfaces, AZ-280_sha256_sidecar, AZ-263_initial_structure, AZ-269_config_loader, AZ-266_log_module
**Component**: c6_tile_cache (epic AZ-250 / E-C6)
**Tracker**: AZ-306
**Epic**: AZ-250 (E-C6)

### Document Dependencies

- `_docs/02_document/contracts/c6_tile_cache/descriptor_index.md` — Protocol this task implements; produced by AZ-303.
- `_docs/02_document/contracts/shared_helpers/sha256_sidecar.md` — atomic-write + sidecar pattern for the `.index` file.
- `_docs/02_document/contracts/shared_config/composition_root_protocol.md` — `config.tile_cache.descriptor_index_runtime`, `config.tile_cache.faiss_index_path`, `config.tile_cache.faiss_warmup_query` fields.
- `_docs/02_document/contracts/shared_logging/log_record_schema.md` — INFO/WARN log shapes for load + warm-up + corruption events.

## Problem

Without a real `FaissDescriptorIndex`:

- C2 VPR has no production retrieval path — `search_topk` is a hole; F3 hot path fails before C2.5.
- C10 CacheProvisioner has no production index builder — F1 pre-flight cannot persist a `.index` file; takeoff blocks.
- The F2 takeoff cold-start budget (AC-NEW-1 ≤ 12 s end-to-end) cannot be measured — without a real warm-up query, the first per-frame `search_topk` would pay the multi-second mmap page-in (description.md § 7).
- The `IndexUnavailableError` raise points (mismatched sidecar, dimension mismatch, mmap'd file replaced concurrently) are unenforced — silent corruption is possible.
- The `BUILD_FAISS_INDEX=OFF` Tier-0 dev path has no test surface — build matrix coverage is missing.

This task is the production-default impl. The Protocol, contract, and the pinned FAISS dependency are now ready; this is the integration point.

## Outcome

- A `FaissDescriptorIndex` class at `src/gps_denied_onboard/components/c6_tile_cache/faiss_descriptor_index.py` conforming to the `DescriptorIndex` Protocol from AZ-303.
- A pybind11 wrapper at `cpp/faiss_index/` (CMake `BUILD_FAISS_INDEX` flag) that exposes only the methods this task needs: `read_index_mmap(path)`, `write_index(path, index_handle)`, `add_with_ids(index, vectors, ids)`, `search(index, query, k)`, `set_ef_search(index, ef)`. The wrapper holds NO Python state — all state is in the Python `FaissDescriptorIndex` class.
- Constructor signature: `__init__(self, *, index_path: Path, sha256_sidecar: Sha256SidecarHelper, logger: Logger, warmup_query: Optional[np.ndarray] = None)`. The composition root wires the dependencies; the warm-up query is loaded from `config.tile_cache.faiss_warmup_query` at startup if present.
- `search_topk(query, k) -> list[tuple[TileId, float]]`:
  1. Validates `query.shape == (descriptor_dim,)`, `query.dtype == np.float32`, `query.flags.c_contiguous`. Mismatch → `IndexUnavailableError` (per `descriptor_index.md` § I-3).
  2. Calls `cpp/faiss_index/search` with `k`.
  3. Maps the returned int64 ids back to `TileId` via the in-memory `_id_to_tile_id` map (built at load time from the sidecar metadata block).
  4. Returns up to `k` `(TileId, float)` pairs ordered by ascending distance. Fewer-than-k results are tolerated per § I-2.
- `descriptor_dim() -> int`: returns the cached `IndexMetadata.descriptor_dim` from load time. Constant-time.
- `mmap_handle() -> Path`: returns the `index_path` constructor arg. Raises `IndexUnavailableError` if the index is not currently loaded (e.g., construction failed and the operator caught the exception).
- `rebuild_from_descriptors(descriptors, tile_ids, hnsw_params) -> None`:
  1. Validates `descriptors.shape == (len(tile_ids), descriptor_dim)`, `descriptors.dtype == np.float32`, `descriptors.flags.c_contiguous`, `len(tile_ids) > 0`. Mismatch → `IndexBuildError`.
  2. Builds the HNSW index in C++ via `cpp/faiss_index/add_with_ids` with the supplied params.
  3. Serialises to a temp path under `index_path.parent` via `cpp/faiss_index/write_index`.
  4. Writes the sidecar metadata block (a separate `<index_path>.meta.json` file carrying `IndexMetadata` JSON: `descriptor_dim`, `n_vectors`, `backbone_label`, `backbone_sha256_hex`, `built_at`, `hnsw_params`, plus the `tile_id` ↔ int64 mapping).
  5. Runs `sha256_sidecar.atomic_write_with_sidecar(index_path, temp_index_bytes)` for atomic rename of the `.index` file + `.sha256` sidecar.
  6. Reloads the in-memory index from the new file (so subsequent `search_topk` calls hit the fresh data).
  7. Emits an INFO log on success: `kind="c6.faiss.rebuilt"` with `n_vectors`, `descriptor_dim`, elapsed seconds.
- `index_metadata() -> IndexMetadata`: parses the `<index_path>.meta.json` sidecar; raises `IndexUnavailableError` if missing or corrupt.
- Load flow at construction:
  1. Validates `index_path` exists; if missing, raises `IndexUnavailableError` (composition root catches and decides — Tier-0 dev may proceed with thermal-aware paths disabled, similar to AZ-302's pattern).
  2. Reads `<index_path>.sha256` and validates it matches `sha256(<index_path>)`; mismatch → `IndexUnavailableError`.
  3. Reads `<index_path>.meta.json` and validates it parses to `IndexMetadata`; corruption → `IndexUnavailableError`.
  4. Calls `cpp/faiss_index/read_index_mmap(index_path)` with `IO_FLAG_MMAP_IFC` (FAISS's mmap-backed read path).
  5. Caches `descriptor_dim`, `n_vectors`, the `_id_to_tile_id` map, and the FAISS index handle.
  6. If `warmup_query` is supplied, runs ONE `search_topk(warmup_query, k=1)` to page in the mmap'd file.
- `cpp/faiss_index/` is a thin pybind11 module — no Python-level state, no GIL holds beyond what FAISS itself does. The build is gated by CMake `BUILD_FAISS_INDEX=ON`; with the flag off, the Python `FaissDescriptorIndex` class is not even importable (the `from cpp_faiss_index import ...` line at module top fails import-time, exactly as `BUILD_TENSORRT_RUNTIME=OFF` makes `tensorrt_runtime.py` unimportable).
- All third-party FAISS exceptions (C++ exceptions surfaced via pybind11 as `RuntimeError`) are caught and rewrapped into `IndexUnavailableError` (read path) or `IndexBuildError` (rebuild path).

## Scope

### Included

- `FaissDescriptorIndex` class implementation conforming to AZ-303's Protocol.
- `cpp/faiss_index/` pybind11 module with the five-method surface above.
- The `<index_path>.meta.json` sidecar format — a JSON document carrying `IndexMetadata` plus the `tile_id` ↔ int64 mapping.
- The HNSW int64-id assignment scheme: a stable, deterministic mapping from `TileId` (composite tuple) to int64 id at rebuild time. The mapping function is `int64(sha256(zoom|lat|lon|source).first8bytes)` — collisions are detected at rebuild time (rebuild raises `IndexBuildError` on collision).
- Construction-time mmap of the existing `.index` file (or `IndexUnavailableError` if absent / corrupted).
- Optional construction-time warm-up query (no warm-up if `warmup_query=None`).
- Lazy-import gating: the `cpp_faiss_index` import lives at module top, so `BUILD_FAISS_INDEX=OFF` makes the module unimportable. The composition-root factory's `if BUILD_FAISS_INDEX:` guard prevents the import attempt under the OFF flag.
- Diagnostic INFO log on construction with `n_vectors`, `descriptor_dim`, sidecar SHA-256, build timestamp; INFO on `rebuild_from_descriptors` start + end with elapsed seconds.
- Standalone CLI `python -m c6_tile_cache.faiss_descriptor_index inspect <index_path>` for operator post-flight inspection (prints `IndexMetadata` + the first 5 vectors' ids).

### Excluded

- The C10 CacheProvisioner orchestration that calls `rebuild_from_descriptors` — owned by E-C10. This task exposes the API; C10 calls it.
- The C2 VPR consumer wiring of `search_topk` — owned by E-C2.
- A second `DescriptorIndex` impl (e.g., `FlatDescriptorIndex` for unit tests that don't want HNSW overhead) — out of scope this cycle. Tests use a fake satisfying the Protocol.
- GPU FAISS variants — explicitly forbidden by AZ-303 § I-4.
- Incremental updates / online learning — F1 pre-flight is full-rebuild only per `descriptor_index.md` Non-Goals.
- Descriptor compression / PQ quantisation — out of scope this cycle (HNSW32 raw float32).
- Cross-flight `.index` sharing — parent-suite concern (D-PROJ-2).
- Backbone retraining — owned by E-C7 / E-C10.

## Acceptance Criteria

**AC-1: search_topk returns ordered ids on a known corpus**
Given a freshly-rebuilt index from 1000 known descriptors with deterministic int64 ids
When `search_topk(query=descriptors[0], k=5)` is called
Then the result is a list of 5 `(TileId, float)` pairs; the first pair's `TileId` matches `tile_ids[0]`; the first pair's distance is < 1e-6 (self-match); pairs are ordered by ascending distance

**AC-2: search_topk returns fewer-than-k when corpus is small**
Given a 3-vector corpus and `k=10`
When `search_topk(query, k=10)` is called
Then the result has length 3; every pair's `TileId` matches one of the 3 corpus tile_ids; no exception

**AC-3: search_topk rejects shape / dtype / contiguity mismatch**
Given a query with `shape=(descriptor_dim+1,)` (wrong dim), or `dtype=float64`, or `flags.c_contiguous=False`
When `search_topk(query, k=5)` is called
Then `IndexUnavailableError` is raised with a message naming the violation; no FAISS call is made (verifiable via the C++ wrapper's call counter staying flat)

**AC-4: rebuild_from_descriptors atomic on crash**
Given an existing valid `.index` and `.meta.json` and `.sha256` sidecars
When `rebuild_from_descriptors` is called and the test simulates `os._exit` AFTER the temp file is written but BEFORE the atomic rename
Then on next construction the original `.index` and sidecars are intact and loadable; the temp file is left behind for cleanup at next start (cleanup is the construction-time scan's responsibility)

**AC-5: rebuild_from_descriptors writes correct sidecars**
Given a successful rebuild
When the test inspects the resulting files
Then the `.index` file's sha256 matches the `.sha256` sidecar content; the `.meta.json` `descriptor_dim` matches `descriptors.shape[1]`; `n_vectors` matches `len(tile_ids)`; `built_at` is within 1 s of the call time; `hnsw_params` matches the input

**AC-6: Construction validates sidecar coherence**
Given an `.index` whose `.sha256` sidecar content is mutated to a wrong value
When `FaissDescriptorIndex(index_path=..., sha256_sidecar=..., ...)` is constructed
Then `IndexUnavailableError` is raised with a message naming the path; the FAISS handle is not loaded (verifiable via `mmap_handle()` raising `IndexUnavailableError` on the partially-constructed object)

**AC-7: Construction validates meta.json**
Given an `.index` whose `.meta.json` is missing or contains malformed JSON
When the index is constructed
Then `IndexUnavailableError` is raised; the FAISS handle is not loaded

**AC-8: Warm-up query pages the mmap on construction**
Given a freshly-loaded index whose mmap'd file is NOT in the OS page cache and a `warmup_query` is supplied
When the construction returns
Then a subsequent `search_topk` p95 < 5 ms (warm); without the warm-up, the first `search_topk` would be ≥ 100 ms (cold). The test fakes the cold-state by `posix_fadvise(POSIX_FADV_DONTNEED)` on the mapped file before construction.

**AC-9: search_topk p95 latency budget**
Given a 100k-vector corpus, page cache warm
When `search_topk` is called 1000 times with random queries
Then p95 ≤ 5 ms (failure threshold 50 ms — but this is a sanity bound, NOT the C2 budget; the canonical C2-PT-01 measurement is in C2's test phase)

**AC-10: BUILD_FAISS_INDEX=OFF makes the module unimportable**
Given a build with `BUILD_FAISS_INDEX=OFF` (the `cpp_faiss_index` shared lib is not built)
When `from gps_denied_onboard.components.c6_tile_cache import faiss_descriptor_index` is attempted
Then `ImportError` is raised at the `from cpp_faiss_index import ...` line; the composition-root factory's `if BUILD_FAISS_INDEX:` guard MUST prevent the import attempt. The factory raises `RuntimeNotAvailableError` instead.

**AC-11: int64-id collision detection at rebuild**
Given two `tile_ids` whose deterministic int64 mapping collides (synthetic test using a hash-seed mock)
When `rebuild_from_descriptors` is called
Then `IndexBuildError` is raised with a message naming both colliding tile_ids; no `.index` is written; the original index (if any) is untouched

**AC-12: index_metadata round-trip**
Given a rebuild with known `(descriptor_dim, n_vectors, backbone_label, backbone_sha256_hex, hnsw_params)`
When the post-rebuild `index_metadata()` is called
Then the returned `IndexMetadata` matches every field; `sidecar_sha256_hex` matches `sha256(.index)` content

## Non-Functional Requirements

**Performance**
- `search_topk` p95 ≤ 5 ms warm at 100k corpus (AC-9 / sanity bound; canonical budget is C2-PT-01).
- Construction with warm-up ≤ 10 s for a 100k-vector index (mmap page-in dominates; warm-up is a single search).
- `rebuild_from_descriptors` is bound by FAISS HNSW build time — minutes for 100k vectors. NOT a hot-path operation; F1 pre-flight only.

**Compatibility**
- FAISS HEAD pinned per Plan-phase (description.md § 5). No version negotiation.
- pybind11 stable ABI as already pinned by AZ-263 bootstrap.
- numpy float32 C-contiguous arrays only on the search surface.

**Reliability**
- All FAISS C++ exceptions are caught and rewrapped into `IndexUnavailableError` / `IndexBuildError`.
- The mmap'd file lifetime is bound to the `FaissDescriptorIndex` instance lifetime; the composition root holds the singleton for the flight.
- `rebuild_from_descriptors` is atomic — partial failure preserves the prior index.
- `.index` is never modified in place — always written to a temp path then atomically renamed.

**Concurrency**
- `search_topk` is NOT re-entrant per AZ-303 § I-8. The F3 hot path is single-threaded (description.md). Multi-threaded callers MUST use a per-thread instance (out of scope this cycle; documented as a constraint).
- `rebuild_from_descriptors` is offline; never runs concurrently with `search_topk` in the same process. F1 pre-flight is in C10's pre-flight binary; F3 is in the airborne binary.

## Unit Tests

| AC Ref | What to Test | Required Outcome |
|--------|-------------|-----------------|
| AC-1 | rebuild + search_topk on 1000 descriptors | First result self-matches at distance < 1e-6; ordered by distance |
| AC-2 | search_topk with k > corpus size | Returns corpus-size results; no exception |
| AC-3 | search_topk with wrong shape / dtype / non-contiguous | IndexUnavailableError; no FAISS call |
| AC-4 | rebuild crash mid-rename (simulated) | Original index intact on next load |
| AC-5 | Inspect post-rebuild sidecars | `.sha256` matches; `.meta.json` matches input |
| AC-6 | Sidecar content corrupted | IndexUnavailableError on construct |
| AC-7 | `.meta.json` missing/malformed | IndexUnavailableError on construct |
| AC-8 | Warm-up forces mmap page-in | Subsequent search p95 < 5 ms even after fadvise DONTNEED |
| AC-9 | Microbench search × 1000 on 100k corpus | p95 ≤ 5 ms |
| AC-10 | Build with BUILD_FAISS_INDEX=OFF | ImportError; factory raises RuntimeNotAvailableError |
| AC-11 | Two tile_ids whose int64 mapping collides | IndexBuildError; no `.index` written |
| AC-12 | Round-trip IndexMetadata after rebuild | Every field matches input |
| NFR-perf-rebuild | 100k vectors, time the rebuild | Wall ≤ 5 minutes (sanity bound; F1 pre-flight runs offline) |
| NFR-reliability-fascade-rewrap | Inject a FAISS C++ exception | Rewrapped into IndexUnavailableError; original message in __cause__ |

## Constraints

- FAISS HEAD pinned per Plan-phase (description.md § 5); no version-negotiation logic.
- The `cpp/faiss_index/` wrapper exposes EXACTLY the five methods listed in Outcome — adding methods is a separate task.
- The pybind11 module holds NO Python state — all state is in Python; the wrapper is a stateless façade.
- numpy float32 C-contiguous on all array surfaces; no auto-casting.
- HNSW only this cycle — no `IndexFlat`, no `IndexIVF*`, no GPU variants.
- `.index` files are NEVER modified in place — always temp + atomic-rename.
- The int64-id deterministic mapping `int64(sha256(zoom|lat|lon|source).first8bytes)` is a project convention; if a future task changes it, every prior `.index` is invalidated and the operator must rebuild.
- The `<index_path>.meta.json` sidecar is the source of truth for `tile_id` ↔ int64 mapping; the `.index` file alone is insufficient (FAISS HNSW stores int64 ids only).
- Lazy-import gating is mandatory — the `cpp_faiss_index` import at module top is the gate; the composition-root factory's `if BUILD_FAISS_INDEX:` block is what skips the import in OFF builds.
- This task adds no new third-party dependencies beyond FAISS HEAD (already pinned by description.md) and pybind11 (already pinned by AZ-263).
- The CLI inspect mode is for operators; not part of any consumer's public API.

## Risks & Mitigation

**Risk 1: FAISS HEAD breaks API across pin updates**
- *Risk*: An operator bumps FAISS pin; the C++ surface changes; the pybind11 wrapper fails to compile.
- *Mitigation*: FAISS pin is recorded in `description.md` § 5; the wrapper is the only place that depends on the C++ surface. Pin updates are a separate task with its own AC. Documented at the wrapper top.

**Risk 2: Mmap'd file is replaced concurrently**
- *Risk*: An out-of-band process renames the `.index` file mid-flight; the mmap reads now hit corrupted bytes.
- *Mitigation*: AZ-303 § I-1 forbids mid-flight modification. The composition root holds the singleton for the flight; out-of-band renames are operator-error. A future defensive task could add a periodic sidecar re-check; out of scope this cycle.

**Risk 3: Int64-id collision (cryptographic-hash) under adversarial inputs**
- *Risk*: With ~10k tiles per provisioning, the birthday-paradox collision probability for an 8-byte truncation of SHA-256 is ~10^-12; effectively zero, but adversarial inputs could engineer a collision.
- *Mitigation*: AC-11 detects collisions at rebuild time and aborts (raises `IndexBuildError`). Operator surfaces the error and either tweaks the corpus or bumps to a 16-byte id mapping — both are out-of-cycle, but the detection point is hard.

**Risk 4: HNSW first-query cold latency exceeds AC-NEW-1 budget**
- *Risk*: The 100k-vector index's mmap takes seconds to page in; without warm-up, the first F3 search blocks for ≥ 1 s.
- *Mitigation*: AC-8 forces a warm-up at construction; the operator's pre-flight `config.tile_cache.faiss_warmup_query` ensures it's not None in production. C10's pre-flight orchestrator is responsible for ensuring the warm-up query is supplied.

**Risk 5: pybind11 ABI mismatch between dev and CI**
- *Risk*: A developer compiles against a different Python minor than CI; the `.so` has a different ABI tag.
- *Mitigation*: AZ-263 pins Python minor + pybind11 version; CMake reads the same versions. The CI matrix's per-binary build job rebuilds the wrapper from source.

## Runtime Completeness

- **Named capability**: FAISS HNSW retrieval + atomic `.index` rebuild + sidecar coherence + mmap-backed read + pybind11 wrapper (description.md / E-C6 / NFT-LIM-01 / D-C10-3 / AC-NEW-1).
- **Production code that must exist**: real `FaissDescriptorIndex` Python class implementing AZ-303's Protocol; real `cpp/faiss_index/` pybind11 wrapper linking real FAISS; real HNSW build via FAISS's `add_with_ids`; real mmap'd read via `IO_FLAG_MMAP_IFC`; real atomic rename via the AZ-280 sidecar helper; real warm-up query at construction; real third-party-exception rewrap.
- **Allowed external stubs**: tests MAY use a fake `Sha256SidecarHelper` (where `atomic_write_with_sidecar` writes to a tmp path); production wiring uses the real AZ-280 helper. Tests MAY use synthetic descriptors and tile_ids; production uses real C10 CacheProvisioner output.
- **Unacceptable substitutes**: a Python-level fake "FAISS" that bypasses the C++ wrapper (would defeat AC-9 latency, the byte-identity of the `.index` file, and the mmap behaviour); a SciPy / scikit-learn `NearestNeighbors` shim "for testing" (different algorithm, different latency profile, different file format — would invalidate the rebuild contract); skipping the warm-up query "to keep construction fast" (would break AC-NEW-1 cold-start budget); an in-memory id map without the `.meta.json` sidecar (would lose the tile_id ↔ int64 mapping across process restarts); a non-rewrapping handler that lets FAISS C++ exceptions escape (would break the family invariant from AZ-303).

## Contract

This task implements the contract at `_docs/02_document/contracts/c6_tile_cache/descriptor_index.md`.
Consumers MUST read that file — not this task spec — to discover the interface.
