Files
obikmer/DevDocMD/implementation/mphf_batch_queries.md
T
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00

23 KiB

MPHF batch queries — site inventory and expected gains

Investigation of 2026-10-14 (ptr_hash 2.1.2, epserde 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked [read] (verified in source), [computed] (derived from stated sizes) or [estimate] (reasoning, not measured). Measurements, once made, are in the last section.

1. What ptr_hash 2.1.2 actually offers [read]

Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>.

  • V is the pilot array (one byte per bucket), F is only the remap storage. A layer's mphf.bin (~2.4 bits/key [computed]: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch pilots[bucket] — the right line. The remap (read only when slot >= n, a small fraction of keys) is not prefetched.
  • index_batch<K>(xs: [Q; K]) -> [usize; K] — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (len % K) is the caller's job.
  • index_stream<B>(keys: IntoIterator) -> impl Iterator — ring buffer of B keys; handles head and tail itself; accepts any iterator. Its next() is unimplemented!: only fold-based consumption works (for_each, fold, sum). collect(), zip, for x in it panic. Consume with for_each into a pre-sized Vec.
  • index_batch_exact needs the nightly unstable feature; index_batch_exact2 is #[doc(hidden)]. Neither is usable.
  • Prefetch is prefetch-index 0.2.1: _mm_prefetch(T0) on x86_64, asm on aarch64, no-op elsewhere.
  • SINGLE_PART: neither index_stream nor index_batch branches on it — both always run the multi-part bucket()/slot(). Only index() has the faster single-part path. So with batched queries the SINGLE_PART = true advantage disappears, and false (our choice) costs nothing while keeping parallel construction (true also panics with sharding).
  • Old ptr_hash 1.1.0 (used by tools/migrate-mphf) has the same family with an extra MINIMAL const: index_stream<B, MINIMAL, Q>, index_batch<K, MINIMAL, Q>. No new dependency needed there.
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
    .for_each(|slot| out.push(slot));

2. Existing "batch" APIs that are not batched [read]

MphfLayer::hash_batch and MphfOnly::index_batch loop .index() one kmer at a time. Consumers: obikfilter/partition_iter.rs (iter_layer_kmers), obikmerge/partition_merge.rs (pass 2: hash_batch, index_batch), obikphylo/siblings/algorithms/family_scan.rs (two sites).

3. Cost chain of a membership query (MphfLayer::find) [read]

  1. pilots[bucket] — the only access ptr_hash prefetches.
  2. Exact/Hybrid-exact: evidence.decode(slot) — 4 B/slot, ~520 MB per 130 M-slot layer [computed], random mmap read.
  3. verify_canonical_kmer → block_offsets[...] (O(1) with the default --block-size 1; a forward scan of up to 2^block_bits records otherwise).
  4. extract_kmer_raw — random read in the unitigs.bin mmap.

Approx/Hybrid-fast: pilots then fingerprint.get(slot) — one extra random access. Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. ptr_hash only helps with stage 1; stages 2–4 need prefetch support in obikindex/obiskio (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.

Layer sizes: layer_0 MPHF 2–3 MB (cache-resident), layer_1 ~40 MB (beyond one CCD's L3 on EPYC-class machines — hypothesis, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on layer_1. For branch-free independent loops (hash_batch) out-of-order execution already overlaps several misses, so the explicit prefetch gains less [estimate] than for branchy loops (find with early break).

4. Call-site inventory

(a) already batched; (b) batch buildable; (c) hard/pointless.

Site What Class
mphf_layer.rs hash_batch, MphfOnly::index_batch loop of .index() (a)
obikfilter/partition_iter.rs:106 hash_batch, 32768 kmers (a)
obikmerge/partition_merge.rs:489, 521 hash_batch, index_batch, 4096 (a)
obikphylo/.../family_scan.rs:285, 397 hash_batch (a)
obikquery/query_layer.rs stage 1 find_in_layer per kmer, layer by layer, break on hit (b)
partition_merge.rs find_batch_grouped find_slot per kmer over dst layers (b)
partition_merge.rs (build_graph filter) all(find_slot.is_none()), every layer, per kmer; build_graph takes Fn(CanonicalKmer)->bool (b), API change
family_scan.rs slow path; annex.rs find_slot / find_in_partition per variant (b)
mphf_layer.rs build, build_exact_evidence, build_approx_evidence mphf.index then random writes (evidence, fingerprint, seen) (b), chunked
obikindexer/counter/count.rs:47 mphf.index then counts.get/set (b), hard (needs (kmer, count) pairs)
obikindexer/extensions/private_builder.rs:177 abundance filter (b), conditional
private_builder.rs:196, verify.rs, IndexCache::find_in_* per-kmer by nature / sampled (c)
tools/migrate-mphf/plan.rs:41-42 old.index + new.index per kmer, random writes (b)

Dead or suspect code found along the way (candidates for removal, not removed):

  • MphfLayer::find_strict — no caller.
  • the single-kmer hash chain (MphfLayer::hash → TypedLayer::hash → KmerLayer::hash → IndexCache::hash) — no caller outside the chain.
  • IndexCache::find — no caller.
  • obikmerge/src/merge_layer/ — not declared in lib.rs (mod merge; mod partition_merge; only); still imports the LayeredMap removed in 2026-08. To be confirmed by the compiler.

5. Interactions

  • Verification chain: the right shape is one find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>) on MphfLayer, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
  • Memory: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one Vec<usize> per call. index_stream itself only allocates a B-element ring on the stack.
  • Parallelism: no new parallelism. Batches already come out of obipipeline stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In query_partition_with the unique-kmer HashMap is already materialised; iterate it chunk-wise.

6. Benchmark

benchmark/bench_mphf.sh LABEL times, on the 20-genome benchmark data, index (2 genomes), merge (20 presence indexes), query (presence/count × 2 genomes), filter, convert (approx/exact round trip) and phylo (--sibling-annex, snp-jc), REPS times each, and records order-independent output digests that must not change across labels. See the results section below.

Limit of this data set: 20 small genomes → per-partition layer_1 MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it cannot measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).

Throwaway prod-scale benchmark (not written)

Open one real partition's layer_1 (~130 M keys) and layer_0 (~8 M) with MphfOnly/MphfLayer; key sets: present (sample of iter_kmers, random order and sequential order) and absent (random, checked absent); measure kmers/s for index, index_stream::<B> (B ∈ 8…128), index_batch::<K> (K ∈ 8…64), and the full find vs staged find_batch; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of mphf.bin, evidence.bin, unitigs.bin(.idx) reported; first check index_stream slots == index slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in mphf.md) is a paper claim, not tied to our sizes or machines.

7. Not verified

No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and --block-size used in prod; whether migrate-mphf already ran; share of total index/merge/query time spent in these loops; count.rs f0 per partition at prod scale; whether merge_layer/ is dead (only the head of lib.rs read).

8. First improvement and measurements (2026-10-15)

Implemented (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):

  • MphfLayer now holds a MphfOnly instead of its own MemCase<Mphf> (one mmap-open path, one place for the stream call).
  • MphfOnly::for_each_slot(kmers, f) wraps index_stream::<32>; index_batch/hash_batch now really use it (they previously looped .index()). Callback-shaped because index_stream's next() is unimplemented!.
  • MphfLayer::find_batch / TypedLayer::find_slot_batch / KmerLayer::find_slot_batch; the mode-dependent half of find is factored into MphfLayer::confirm, shared by both.
  • query_partition_with stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, n_mphf_calls unchanged).
  • Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; prefetch-index is already in the tree via ptr_hash, but adding it to obikindex is a dependency decision), and the other (b) sites.

Measurements (benchmark/bench_mphf.sh, benchmark/summarize_bench.py; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:

wall (s) user CPU (s)
before 5.94–6.02 (median 5.96) 10.95–11.07
after 5.63–6.00 (median 5.67) 10.33–10.66

About −5 % wall and −5.5 % CPU, ranges non-overlapping [measured]. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.

Reading the number. Small by construction: this data set's layer_1 MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains unmeasured and needs a prod-scale layer (§6).

Noise caveat. With a load average of 16–22 from unrelated processes, single-run walls of index, merge and filter moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The index/merge/filter/convert/phylo rows of the first comparison are noise, not effects (index pass 2 was not changed at all; a later interleaved filter A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).

Stale benchmark indexes. The indexes already under benchmark/run/ (specimen_index_*, global_index_*, built 2026-08-28) carry ptr_hash 1.x MPHFs and fail with "Major version mismatch" on current binaries (tools/migrate-mphf converts them). bench_mphf.sh therefore builds its own copies under run/bench_mphf/data/ on first use.

9. Benchmark procedure

Prerequisites: the simulated reads (benchmark/run/simulated_data/) and query reads (benchmark/run/query_data/) produced by gmake simulate simulate_query; macOS BSD time -l and md5 -q (the script uses both), Python 3.

  1. Build the binary to measure and copy it under a name that identifies it (cargo overwrites target/release/obikmer): cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>.
  2. BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label> (from benchmark/; REPS=3 by default, CPU_MAX=n to restrict cores). The first call builds the shared data set in run/bench_mphf/data/ with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete data/ and rebuild.
  3. Output: run/bench_mphf/<label>/results.csv (wall and peak RSS per step and repetition, plus one row step:stage per Reporter stage) and checksums.txt.
  4. Compare: python3 benchmark/summarize_bench.py <ref_label> <new_label> (medians and ratio) and diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt — any difference is a bug, not a performance matter.
  5. A ratio is only a candidate effect. Check the spread by running the same binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from /usr/bin/time -l).
  6. Rule of thumb for this data set: only query (≈ 5 s, stable to ±1 %) resolves effects of a few percent; index, merge, filter, phylo need interleaved runs or a quiet machine.

Step → code exercised:

Step Code on the MPHF path
index_* (:index stage) MphfLayer::build pass 2
merge_20_presence find_batch_grouped, build_graph filter, hash_batch, index_batch
query_* query_partition_with stage 1
filter_species iter_layer_kmers → hash_batch
convert_to_* build_exact_evidence, build_approx_evidence
phylo_sibling_annex annex.rs find_in_partition per variant
phylo_snp_distance family_scan.rs hash_batch, find_slot

10. Second step — construction pass 2 (2026-10-15)

Implemented: layer/slotted.rs, SlottedKmers, an iterator adapter yielding (slot, kmer, chunk_id, rank) for the wrapped (kmer, chunk_id, rank) stream, computing slots 4096 kmers at a time through index_stream::<32> (bounded memory, never the whole layer). The five loops of MphfLayer::build (Exact, Approx, Hybrid), build_exact_evidence and build_approx_evidence now iterate it instead of calling mphf.index per kmer. Output digests unchanged.

Measured (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):

Command before after
convert --exact-evidence (20-genome index) 0.56–0.68 s / 2.22–2.67 s 0.59–0.74 s / 2.25–2.52 s
index E. coli reads 2.45–2.63 s / 13.1–14.3 s 2.48–2.75 s / 13.2–14.0 s

No measurable effect on this data set (differences within the run-to-run spread; if anything convert is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and index/convert time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/seen writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random writes to evidence.bin/seen/fingerprint.bin (not prefetched by ptr_hash) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.

11. Why the two changes gain so little (profiling, 2026-10-15)

Measured with macOS sample (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.

Query. In a 2 s window on the query stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):

  • query_partition_with ≈ 260 samples (8 % of active CPU), of which find_batch 217, of which verify_canonical_kmer 185 and everything else in find_batch (the ptr_hash stream and confirm) 32.
  • verify_canonical_kmer is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
  • Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/unitigs.bin/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
  • The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (rolling_stat, entropy tracker, string formatting) and by sequential parts, not by lookups.

What produced the −5 %. Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer index (stream disabled) 6.0 s; restructuring + index_stream 5.85 s. Most of the gain comes from the restructuring, not from the prefetch (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called IndexCache::get_layer — an Arc<KmerLayer> clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.

Index (pass 2). In a 1 s window late in the index stage, SlottedKmers appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (obidebruinj) and ≈ 35 for ptr_hash::build (pass 1, MPHF construction). Pass 2 is a few percent of index and convert here: nothing to win even if its lookups were free. convert is build_*_evidence on 2 MB-scale layers, same conclusion.

Consequences for priorities.

  1. On this data the next lever is the verification (verify_canonical_kmer, evidence decode), not the MPHF — consistent with §3, now observed.
  2. Hot Arc clones in per-kmer paths (get_layer, IndexCache::find_in_layer, find_slot loops in merge and phylo) are worth auditing independently of MPHF batching.
  3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).

12. Calibration: a single skim at fewer partitions (2026-10-15)

Vaccinium myrtillus skim (2 × ~440 MB fastq.gz, theta 0), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (--partitions is a power of two; MPHF size ≈ 2.4 bits/key [measured]):

-p keys / partition mphf.bin / partition index wall peak RSS
256 (default) ≈ 1.4 M [computed from p=16] ≈ 0.4 MB [computed] – –
16 22.1 M 6.65 MB 120 s 12.3 GB
4 86–90 M 25.9–27.0 MB 171 s 19.5 GB

Total ≈ 354 M distinct kmers for one skim. -p 4 is already in the 30–45 MB range of the prod layer_1 MPHFs; -p 2 would give ≈ 53 MB. Indexes kept in benchmark/run/bench_mphf/skim_p16 and skim_p4 (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.

13. Query on a real-scale index (2026-10-15)

Index. Working index phyloskims_sal_vac_sav (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with tools/migrate-mphf on an APFS clone (cp -c, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at /Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2; its sibling annex is in stale_annexes_pre_ptrhash2/ and would need rebuilding to be used. Per partition: layer_1 ≈ 31 M keys / 9.3 MB MPHF, layer_0 ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).

Query (2 × 48 MB of V. oxycoccos reads, --chunk-size 8, AC power, 3 interleaved runs per binary, identical sorted output for all three):

binary wall (s) user CPU (s) query stage (s)
original 28.2–30.5 70.9–74.4 14.6–17.1
+ batched query stage 1 26.9–28.0 48.1–48.6 13.6–14.4
+ PartitionKmers memory fix, build-side SlottedKmers 26.4–26.7 41.9–42.5 12.9–13.3

The query stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. The CPU drop is much larger than the wall drop and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer Arc clones of get_layer, not isolated); the original binary was spending CPU in contention that wall time hid.

Index-open cost (not related to the changes above). A query of a single 64-base sequence on this index takes 15 s and a 27 GB peak footprint. vmmap during the open: 24.5 GB of MALLOC_LARGE in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by IndexCache::new for every layer of every partition whatever the query. Known contributors: UnitigFileReader reads each .idx into a Vec (≈ 30 MB for layer_1, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.

14. Index-open memory: .idx offsets and dict_values mapped, not copied (2026-10-15)

Cause (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, 18.2 GB were unitigs.bin.idx read into a Vec<u32> twice per exact-evidence layer (MphfLayer::open builds two UnitigFileReaders of the same file — one for the evidence check, one for iter_kmers/find_strict; both go through open → open_direct_access → read_idx → fs::read + copy) and 7.7 GB were dict_values.bin read with fs::read in PersistentSparseBitMatrix::open. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (PersistentRankSelectBitVec, PersistentFixedIntVec, EliasFano) was already mmapped.

Change.

  • obiskio::unitig_index::reader: block_offsets: Vec<u32> → BlockOffsets, a view into the memory-mapped .idx (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
  • obicompactvec::bitmatrix::sparse: dict_values: Vec<u8> → Option<Mmap> (None for an empty file, which cannot be mapped), read through dict_bytes().

Measured (same session, same inputs; sorted query outputs identical, md5 e9e7c35b):

open only (1-sequence query) 2×48 MB reads, --chunk-size 8
before 10.6–15.7 s, 27.4 GB footprint 27.7 s (51 s cold), 30.9 GB, query stage 13.8 s
after 0.26–0.81 s, 0.04 GB 10.7 s, 3.6 GB, query stage 9.95 s

Tests: cargo test --workspace green; scripts/smoke_test_index.sh PASS; bench_mphf.sh (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a .idx map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.