This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
23 KiB
MPHF batch queries — site inventory and expected gains
Investigation of 2026-10-14 (ptr_hash 2.1.2, epserde 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked [read] (verified in source), [computed] (derived from stated sizes) or [estimate] (reasoning, not measured). Measurements, once made, are in the last section.
1. What ptr_hash 2.1.2 actually offers [read]
Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>.
Vis the pilot array (one byte per bucket),Fis only the remap storage. A layer'smphf.bin(~2.4 bits/key [computed]: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetchpilots[bucket]— the right line. The remap (read only whenslot >= n, a small fraction of keys) is not prefetched.index_batch<K>(xs: [Q; K]) -> [usize; K]— stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (len % K) is the caller's job.index_stream<B>(keys: IntoIterator) -> impl Iterator— ring buffer of B keys; handles head and tail itself; accepts any iterator. Itsnext()isunimplemented!: only fold-based consumption works (for_each,fold,sum).collect(),zip,for x in itpanic. Consume withfor_eachinto a pre-sizedVec.index_batch_exactneeds the nightlyunstablefeature;index_batch_exact2is#[doc(hidden)]. Neither is usable.- Prefetch is
prefetch-index 0.2.1:_mm_prefetch(T0)on x86_64, asm on aarch64, no-op elsewhere. SINGLE_PART: neitherindex_streamnorindex_batchbranches on it — both always run the multi-partbucket()/slot(). Onlyindex()has the faster single-part path. So with batched queries theSINGLE_PART = trueadvantage disappears, andfalse(our choice) costs nothing while keeping parallel construction (truealso panics with sharding).- Old
ptr_hash1.1.0 (used bytools/migrate-mphf) has the same family with an extraMINIMALconst:index_stream<B, MINIMAL, Q>,index_batch<K, MINIMAL, Q>. No new dependency needed there.
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(|slot| out.push(slot));
2. Existing "batch" APIs that are not batched [read]
MphfLayer::hash_batch and MphfOnly::index_batch loop .index() one kmer at a time. Consumers: obikfilter/partition_iter.rs (iter_layer_kmers), obikmerge/partition_merge.rs (pass 2: hash_batch, index_batch), obikphylo/siblings/algorithms/family_scan.rs (two sites).
3. Cost chain of a membership query (MphfLayer::find) [read]
pilots[bucket]— the only accessptr_hashprefetches.- Exact/Hybrid-exact:
evidence.decode(slot)— 4 B/slot, ~520 MB per 130 M-slot layer [computed], random mmap read. verify_canonical_kmer→block_offsets[...](O(1) with the default--block-size 1; a forward scan of up to2^block_bitsrecords otherwise).extract_kmer_raw— random read in theunitigs.binmmap.
Approx/Hybrid-fast: pilots then fingerprint.get(slot) — one extra random access.
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. ptr_hash only helps with stage 1; stages 2–4 need prefetch support in obikindex/obiskio (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
Layer sizes: layer_0 MPHF 2–3 MB (cache-resident), layer_1 ~40 MB (beyond one CCD's L3 on EPYC-class machines — hypothesis, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on layer_1. For branch-free independent loops (hash_batch) out-of-order execution already overlaps several misses, so the explicit prefetch gains less [estimate] than for branchy loops (find with early break).
4. Call-site inventory
(a) already batched; (b) batch buildable; (c) hard/pointless.
| Site | What | Class |
|---|---|---|
mphf_layer.rs hash_batch, MphfOnly::index_batch |
loop of .index() |
(a) |
obikfilter/partition_iter.rs:106 |
hash_batch, 32768 kmers |
(a) |
obikmerge/partition_merge.rs:489, 521 |
hash_batch, index_batch, 4096 |
(a) |
obikphylo/.../family_scan.rs:285, 397 |
hash_batch |
(a) |
obikquery/query_layer.rs stage 1 |
find_in_layer per kmer, layer by layer, break on hit |
(b) |
partition_merge.rs find_batch_grouped |
find_slot per kmer over dst layers |
(b) |
partition_merge.rs (build_graph filter) |
all(find_slot.is_none()), every layer, per kmer; build_graph takes Fn(CanonicalKmer)->bool |
(b), API change |
family_scan.rs slow path; annex.rs |
find_slot / find_in_partition per variant |
(b) |
mphf_layer.rs build, build_exact_evidence, build_approx_evidence |
mphf.index then random writes (evidence, fingerprint, seen) |
(b), chunked |
obikindexer/counter/count.rs:47 |
mphf.index then counts.get/set |
(b), hard (needs (kmer, count) pairs) |
obikindexer/extensions/private_builder.rs:177 |
abundance filter | (b), conditional |
private_builder.rs:196, verify.rs, IndexCache::find_in_* |
per-kmer by nature / sampled | (c) |
tools/migrate-mphf/plan.rs:41-42 |
old.index + new.index per kmer, random writes |
(b) |
Dead or suspect code found along the way (candidates for removal, not removed):
MphfLayer::find_strict— no caller.- the single-kmer
hashchain (MphfLayer::hash→TypedLayer::hash→KmerLayer::hash→IndexCache::hash) — no caller outside the chain. IndexCache::find— no caller.obikmerge/src/merge_layer/— not declared inlib.rs(mod merge; mod partition_merge;only); still imports theLayeredMapremoved in 2026-08. To be confirmed by the compiler.
5. Interactions
- Verification chain: the right shape is one
find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)onMphfLayer, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops). - Memory: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one
Vec<usize>per call.index_streamitself only allocates a B-element ring on the stack. - Parallelism: no new parallelism. Batches already come out of
obipipelinestages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. Inquery_partition_withthe unique-kmerHashMapis already materialised; iterate it chunk-wise.
6. Benchmark
benchmark/bench_mphf.sh LABEL times, on the 20-genome benchmark data, index (2 genomes), merge (20 presence indexes), query (presence/count × 2 genomes), filter, convert (approx/exact round trip) and phylo (--sibling-annex, snp-jc), REPS times each, and records order-independent output digests that must not change across labels. See the results section below.
Limit of this data set: 20 small genomes → per-partition layer_1 MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it cannot measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
Throwaway prod-scale benchmark (not written)
Open one real partition's layer_1 (~130 M keys) and layer_0 (~8 M) with MphfOnly/MphfLayer; key sets: present (sample of iter_kmers, random order and sequential order) and absent (random, checked absent); measure kmers/s for index, index_stream::<B> (B ∈ 8…128), index_batch::<K> (K ∈ 8…64), and the full find vs staged find_batch; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of mphf.bin, evidence.bin, unitigs.bin(.idx) reported; first check index_stream slots == index slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in mphf.md) is a paper claim, not tied to our sizes or machines.
7. Not verified
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and --block-size used in prod; whether migrate-mphf already ran; share of total index/merge/query time spent in these loops; count.rs f0 per partition at prod scale; whether merge_layer/ is dead (only the head of lib.rs read).
8. First improvement and measurements (2026-10-15)
Implemented (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
MphfLayernow holds aMphfOnlyinstead of its ownMemCase<Mphf>(one mmap-open path, one place for the stream call).MphfOnly::for_each_slot(kmers, f)wrapsindex_stream::<32>;index_batch/hash_batchnow really use it (they previously looped.index()). Callback-shaped becauseindex_stream'snext()isunimplemented!.MphfLayer::find_batch/TypedLayer::find_slot_batch/KmerLayer::find_slot_batch; the mode-dependent half offindis factored intoMphfLayer::confirm, shared by both.query_partition_withstage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins,n_mphf_callsunchanged).- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive;
prefetch-indexis already in the tree viaptr_hash, but adding it toobikindexis a dependency decision), and the other (b) sites.
Measurements (benchmark/bench_mphf.sh, benchmark/summarize_bench.py; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
| wall (s) | user CPU (s) | |
|---|---|---|
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
About −5 % wall and −5.5 % CPU, ranges non-overlapping [measured]. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
Reading the number. Small by construction: this data set's layer_1 MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains unmeasured and needs a prod-scale layer (§6).
Noise caveat. With a load average of 16–22 from unrelated processes, single-run walls of index, merge and filter moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The index/merge/filter/convert/phylo rows of the first comparison are noise, not effects (index pass 2 was not changed at all; a later interleaved filter A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
Stale benchmark indexes. The indexes already under benchmark/run/ (specimen_index_*, global_index_*, built 2026-08-28) carry ptr_hash 1.x MPHFs and fail with "Major version mismatch" on current binaries (tools/migrate-mphf converts them). bench_mphf.sh therefore builds its own copies under run/bench_mphf/data/ on first use.
9. Benchmark procedure
Prerequisites: the simulated reads (benchmark/run/simulated_data/) and query reads (benchmark/run/query_data/) produced by gmake simulate simulate_query; macOS BSD time -l and md5 -q (the script uses both), Python 3.
- Build the binary to measure and copy it under a name that identifies it (cargo overwrites
target/release/obikmer):cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>. BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>(frombenchmark/;REPS=3by default,CPU_MAX=nto restrict cores). The first call builds the shared data set inrun/bench_mphf/data/with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise deletedata/and rebuild.- Output:
run/bench_mphf/<label>/results.csv(wall and peak RSS per step and repetition, plus one rowstep:stageper Reporter stage) andchecksums.txt. - Compare:
python3 benchmark/summarize_bench.py <ref_label> <new_label>(medians and ratio) anddiff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt— any difference is a bug, not a performance matter. - A ratio is only a candidate effect. Check the spread by running the same binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from
/usr/bin/time -l). - Rule of thumb for this data set: only
query(≈ 5 s, stable to ±1 %) resolves effects of a few percent;index,merge,filter,phyloneed interleaved runs or a quiet machine.
Step → code exercised:
| Step | Code on the MPHF path |
|---|---|
index_* (:index stage) |
MphfLayer::build pass 2 |
merge_20_presence |
find_batch_grouped, build_graph filter, hash_batch, index_batch |
query_* |
query_partition_with stage 1 |
filter_species |
iter_layer_kmers → hash_batch |
convert_to_* |
build_exact_evidence, build_approx_evidence |
phylo_sibling_annex |
annex.rs find_in_partition per variant |
phylo_snp_distance |
family_scan.rs hash_batch, find_slot |
10. Second step — construction pass 2 (2026-10-15)
Implemented: layer/slotted.rs, SlottedKmers, an iterator adapter yielding (slot, kmer, chunk_id, rank) for the wrapped (kmer, chunk_id, rank) stream, computing slots 4096 kmers at a time through index_stream::<32> (bounded memory, never the whole layer). The five loops of MphfLayer::build (Exact, Approx, Hybrid), build_exact_evidence and build_approx_evidence now iterate it instead of calling mphf.index per kmer. Output digests unchanged.
Measured (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
| Command | before | after |
|---|---|---|
convert --exact-evidence (20-genome index) |
0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
index E. coli reads |
2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
No measurable effect on this data set (differences within the run-to-run spread; if anything convert is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and index/convert time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/seen writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random writes to evidence.bin/seen/fingerprint.bin (not prefetched by ptr_hash) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
11. Why the two changes gain so little (profiling, 2026-10-15)
Measured with macOS sample (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
Query. In a 2 s window on the query stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
query_partition_with≈ 260 samples (8 % of active CPU), of whichfind_batch217, of whichverify_canonical_kmer185 and everything else infind_batch(theptr_hashstream andconfirm) 32.verify_canonical_kmeris the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/
unitigs.bin/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side. - The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (
rolling_stat, entropy tracker, string formatting) and by sequential parts, not by lookups.
What produced the −5 %. Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer index (stream disabled) 6.0 s; restructuring + index_stream 5.85 s. Most of the gain comes from the restructuring, not from the prefetch (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called IndexCache::get_layer — an Arc<KmerLayer> clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
Index (pass 2). In a 1 s window late in the index stage, SlottedKmers appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (obidebruinj) and ≈ 35 for ptr_hash::build (pass 1, MPHF construction). Pass 2 is a few percent of index and convert here: nothing to win even if its lookups were free. convert is build_*_evidence on 2 MB-scale layers, same conclusion.
Consequences for priorities.
- On this data the next lever is the verification (
verify_canonical_kmer, evidence decode), not the MPHF — consistent with §3, now observed. - Hot
Arcclones in per-kmer paths (get_layer,IndexCache::find_in_layer,find_slotloops in merge and phylo) are worth auditing independently of MPHF batching. - The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
12. Calibration: a single skim at fewer partitions (2026-10-15)
Vaccinium myrtillus skim (2 × ~440 MB fastq.gz, theta 0), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (--partitions is a power of two; MPHF size ≈ 2.4 bits/key [measured]):
-p |
keys / partition | mphf.bin / partition |
index wall | peak RSS |
|---|---|---|---|---|
| 256 (default) | ≈ 1.4 M [computed from p=16] | ≈ 0.4 MB [computed] | – | – |
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
Total ≈ 354 M distinct kmers for one skim. -p 4 is already in the 30–45 MB range of the prod layer_1 MPHFs; -p 2 would give ≈ 53 MB. Indexes kept in benchmark/run/bench_mphf/skim_p16 and skim_p4 (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.
13. Query on a real-scale index (2026-10-15)
Index. Working index phyloskims_sal_vac_sav (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with tools/migrate-mphf on an APFS clone (cp -c, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at /Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2; its sibling annex is in stale_annexes_pre_ptrhash2/ and would need rebuilding to be used. Per partition: layer_1 ≈ 31 M keys / 9.3 MB MPHF, layer_0 ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).
Query (2 × 48 MB of V. oxycoccos reads, --chunk-size 8, AC power, 3 interleaved runs per binary, identical sorted output for all three):
| binary | wall (s) | user CPU (s) | query stage (s) |
|---|---|---|---|
| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 |
| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 |
+ PartitionKmers memory fix, build-side SlottedKmers |
26.4–26.7 | 41.9–42.5 | 12.9–13.3 |
The query stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. The CPU drop is much larger than the wall drop and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer Arc clones of get_layer, not isolated); the original binary was spending CPU in contention that wall time hid.
Index-open cost (not related to the changes above). A query of a single 64-base sequence on this index takes 15 s and a 27 GB peak footprint. vmmap during the open: 24.5 GB of MALLOC_LARGE in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by IndexCache::new for every layer of every partition whatever the query. Known contributors: UnitigFileReader reads each .idx into a Vec (≈ 30 MB for layer_1, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.
14. Index-open memory: .idx offsets and dict_values mapped, not copied (2026-10-15)
Cause (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, 18.2 GB were unitigs.bin.idx read into a Vec<u32> twice per exact-evidence layer (MphfLayer::open builds two UnitigFileReaders of the same file — one for the evidence check, one for iter_kmers/find_strict; both go through open → open_direct_access → read_idx → fs::read + copy) and 7.7 GB were dict_values.bin read with fs::read in PersistentSparseBitMatrix::open. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (PersistentRankSelectBitVec, PersistentFixedIntVec, EliasFano) was already mmapped.
Change.
obiskio::unitig_index::reader:block_offsets: Vec<u32>→BlockOffsets, a view into the memory-mapped.idx(header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.obicompactvec::bitmatrix::sparse:dict_values: Vec<u8>→Option<Mmap>(Nonefor an empty file, which cannot be mapped), read throughdict_bytes().
Measured (same session, same inputs; sorted query outputs identical, md5 e9e7c35b):
| open only (1-sequence query) | 2×48 MB reads, --chunk-size 8 |
|
|---|---|---|
| before | 10.6–15.7 s, 27.4 GB footprint | 27.7 s (51 s cold), 30.9 GB, query stage 13.8 s |
| after | 0.26–0.81 s, 0.04 GB | 10.7 s, 3.6 GB, query stage 9.95 s |
Tests: cargo test --workspace green; scripts/smoke_test_index.sh PASS; bench_mphf.sh (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a .idx map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.