Refactor MPHF layer logic and evidence handling

This commit implements core lookup functionality for MPHF layers, including `hash` and `find_strict` methods. It introduces support for different evidence types (Exact, Hybrid, Approximate) within layers, clarifying lookup behavior and improving partition calculation logic in the cache. Additionally, dead code is removed and dependencies are updated.
This commit is contained in:
Eric Coissac committed 2026-10-15 12:22:45 +02:00
1 parent f9edefd135
commit 12afbe819c
10 files changed
+26 -794

No files matched your search

@@ -355,3 +355,13 @@ Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not th
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
## 21. Dead code removed (2026-10-15)
The items listed as unused in §4 were removed after Serena's `safe_delete_symbol` and the compiler confirmed no reference: `MphfLayer::find_strict`; the single-kmer `hash` chain (`MphfLayer::hash`, `TypedLayer::hash`, `KmerLayer::hash`, `IndexCache::hash`); `IndexCache::find`; the uncompiled `obikmerge/src/merge_layer/` directory (689 lines; `lib.rs` only declares `merge` and `partition_merge`); and `xxhash-rust` from `obidebruinj/Cargo.toml` (it still serves `obikseq`, `obifastwrite` and `obiksession`).
Side effect: `find_strict` was the only reader of the `unitigs` handle that `Hybrid` layers opened, so that field (and the extra `.idx` map and mmap per Hybrid layer) was dropped too. `find` on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what `convert` works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding `find_strict` and that field.
`ahash` in `obidebruinj/Cargo.toml` is not referenced by any source file of the crate either (not removed: not asked).
Checks after the removal: `cargo build --release` without warnings, `cargo test --workspace` 555 passed / 0 failed, `scripts/smoke_test_index.sh` PASS.