Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline. This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
19 KiB
MPHF batch queries — site inventory and expected gains
Investigation of 2026-10-14 (ptr_hash 2.1.2, epserde 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked [read] (verified in source), [computed] (derived from stated sizes) or [estimate] (reasoning, not measured). Measurements, once made, are in the last section.
1. What ptr_hash 2.1.2 actually offers [read]
Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>.
Vis the pilot array (one byte per bucket),Fis only the remap storage. A layer'smphf.bin(~2.4 bits/key [computed]: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetchpilots[bucket]— the right line. The remap (read only whenslot >= n, a small fraction of keys) is not prefetched.index_batch<K>(xs: [Q; K]) -> [usize; K]— stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (len % K) is the caller's job.index_stream<B>(keys: IntoIterator) -> impl Iterator— ring buffer of B keys; handles head and tail itself; accepts any iterator. Itsnext()isunimplemented!: only fold-based consumption works (for_each,fold,sum).collect(),zip,for x in itpanic. Consume withfor_eachinto a pre-sizedVec.index_batch_exactneeds the nightlyunstablefeature;index_batch_exact2is#[doc(hidden)]. Neither is usable.- Prefetch is
prefetch-index 0.2.1:_mm_prefetch(T0)on x86_64, asm on aarch64, no-op elsewhere. SINGLE_PART: neitherindex_streamnorindex_batchbranches on it — both always run the multi-partbucket()/slot(). Onlyindex()has the faster single-part path. So with batched queries theSINGLE_PART = trueadvantage disappears, andfalse(our choice) costs nothing while keeping parallel construction (truealso panics with sharding).- Old
ptr_hash1.1.0 (used bytools/migrate-mphf) has the same family with an extraMINIMALconst:index_stream<B, MINIMAL, Q>,index_batch<K, MINIMAL, Q>. No new dependency needed there.
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(|slot| out.push(slot));
2. Existing "batch" APIs that are not batched [read]
MphfLayer::hash_batch and MphfOnly::index_batch loop .index() one kmer at a time. Consumers: obikfilter/partition_iter.rs (iter_layer_kmers), obikmerge/partition_merge.rs (pass 2: hash_batch, index_batch), obikphylo/siblings/algorithms/family_scan.rs (two sites).
3. Cost chain of a membership query (MphfLayer::find) [read]
pilots[bucket]— the only accessptr_hashprefetches.- Exact/Hybrid-exact:
evidence.decode(slot)— 4 B/slot, ~520 MB per 130 M-slot layer [computed], random mmap read. verify_canonical_kmer→block_offsets[...](O(1) with the default--block-size 1; a forward scan of up to2^block_bitsrecords otherwise).extract_kmer_raw— random read in theunitigs.binmmap.
Approx/Hybrid-fast: pilots then fingerprint.get(slot) — one extra random access.
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. ptr_hash only helps with stage 1; stages 2–4 need prefetch support in obikindex/obiskio (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
Layer sizes: layer_0 MPHF 2–3 MB (cache-resident), layer_1 ~40 MB (beyond one CCD's L3 on EPYC-class machines — hypothesis, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on layer_1. For branch-free independent loops (hash_batch) out-of-order execution already overlaps several misses, so the explicit prefetch gains less [estimate] than for branchy loops (find with early break).
4. Call-site inventory
(a) already batched; (b) batch buildable; (c) hard/pointless.
| Site | What | Class |
|---|---|---|
mphf_layer.rs hash_batch, MphfOnly::index_batch |
loop of .index() |
(a) |
obikfilter/partition_iter.rs:106 |
hash_batch, 32768 kmers |
(a) |
obikmerge/partition_merge.rs:489, 521 |
hash_batch, index_batch, 4096 |
(a) |
obikphylo/.../family_scan.rs:285, 397 |
hash_batch |
(a) |
obikquery/query_layer.rs stage 1 |
find_in_layer per kmer, layer by layer, break on hit |
(b) |
partition_merge.rs find_batch_grouped |
find_slot per kmer over dst layers |
(b) |
partition_merge.rs (build_graph filter) |
all(find_slot.is_none()), every layer, per kmer; build_graph takes Fn(CanonicalKmer)->bool |
(b), API change |
family_scan.rs slow path; annex.rs |
find_slot / find_in_partition per variant |
(b) |
mphf_layer.rs build, build_exact_evidence, build_approx_evidence |
mphf.index then random writes (evidence, fingerprint, seen) |
(b), chunked |
obikindexer/counter/count.rs:47 |
mphf.index then counts.get/set |
(b), hard (needs (kmer, count) pairs) |
obikindexer/extensions/private_builder.rs:177 |
abundance filter | (b), conditional |
private_builder.rs:196, verify.rs, IndexCache::find_in_* |
per-kmer by nature / sampled | (c) |
tools/migrate-mphf/plan.rs:41-42 |
old.index + new.index per kmer, random writes |
(b) |
Dead or suspect code found along the way (candidates for removal, not removed):
MphfLayer::find_strict— no caller.- the single-kmer
hashchain (MphfLayer::hash→TypedLayer::hash→KmerLayer::hash→IndexCache::hash) — no caller outside the chain. IndexCache::find— no caller.obikmerge/src/merge_layer/— not declared inlib.rs(mod merge; mod partition_merge;only); still imports theLayeredMapremoved in 2026-08. To be confirmed by the compiler.
5. Interactions
- Verification chain: the right shape is one
find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)onMphfLayer, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops). - Memory: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one
Vec<usize>per call.index_streamitself only allocates a B-element ring on the stack. - Parallelism: no new parallelism. Batches already come out of
obipipelinestages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. Inquery_partition_withthe unique-kmerHashMapis already materialised; iterate it chunk-wise.
6. Benchmark
benchmark/bench_mphf.sh LABEL times, on the 20-genome benchmark data, index (2 genomes), merge (20 presence indexes), query (presence/count × 2 genomes), filter, convert (approx/exact round trip) and phylo (--sibling-annex, snp-jc), REPS times each, and records order-independent output digests that must not change across labels. See the results section below.
Limit of this data set: 20 small genomes → per-partition layer_1 MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it cannot measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
Throwaway prod-scale benchmark (not written)
Open one real partition's layer_1 (~130 M keys) and layer_0 (~8 M) with MphfOnly/MphfLayer; key sets: present (sample of iter_kmers, random order and sequential order) and absent (random, checked absent); measure kmers/s for index, index_stream::<B> (B ∈ 8…128), index_batch::<K> (K ∈ 8…64), and the full find vs staged find_batch; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of mphf.bin, evidence.bin, unitigs.bin(.idx) reported; first check index_stream slots == index slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in mphf.md) is a paper claim, not tied to our sizes or machines.
7. Not verified
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and --block-size used in prod; whether migrate-mphf already ran; share of total index/merge/query time spent in these loops; count.rs f0 per partition at prod scale; whether merge_layer/ is dead (only the head of lib.rs read).
8. First improvement and measurements (2026-10-15)
Implemented (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
MphfLayernow holds aMphfOnlyinstead of its ownMemCase<Mphf>(one mmap-open path, one place for the stream call).MphfOnly::for_each_slot(kmers, f)wrapsindex_stream::<32>;index_batch/hash_batchnow really use it (they previously looped.index()). Callback-shaped becauseindex_stream'snext()isunimplemented!.MphfLayer::find_batch/TypedLayer::find_slot_batch/KmerLayer::find_slot_batch; the mode-dependent half offindis factored intoMphfLayer::confirm, shared by both.query_partition_withstage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins,n_mphf_callsunchanged).- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive;
prefetch-indexis already in the tree viaptr_hash, but adding it toobikindexis a dependency decision), and the other (b) sites.
Measurements (benchmark/bench_mphf.sh, benchmark/summarize_bench.py; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
| wall (s) | user CPU (s) | |
|---|---|---|
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
About −5 % wall and −5.5 % CPU, ranges non-overlapping [measured]. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
Reading the number. Small by construction: this data set's layer_1 MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains unmeasured and needs a prod-scale layer (§6).
Noise caveat. With a load average of 16–22 from unrelated processes, single-run walls of index, merge and filter moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The index/merge/filter/convert/phylo rows of the first comparison are noise, not effects (index pass 2 was not changed at all; a later interleaved filter A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
Stale benchmark indexes. The indexes already under benchmark/run/ (specimen_index_*, global_index_*, built 2026-08-28) carry ptr_hash 1.x MPHFs and fail with "Major version mismatch" on current binaries (tools/migrate-mphf converts them). bench_mphf.sh therefore builds its own copies under run/bench_mphf/data/ on first use.
9. Benchmark procedure
Prerequisites: the simulated reads (benchmark/run/simulated_data/) and query reads (benchmark/run/query_data/) produced by gmake simulate simulate_query; macOS BSD time -l and md5 -q (the script uses both), Python 3.
- Build the binary to measure and copy it under a name that identifies it (cargo overwrites
target/release/obikmer):cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>. BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>(frombenchmark/;REPS=3by default,CPU_MAX=nto restrict cores). The first call builds the shared data set inrun/bench_mphf/data/with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise deletedata/and rebuild.- Output:
run/bench_mphf/<label>/results.csv(wall and peak RSS per step and repetition, plus one rowstep:stageper Reporter stage) andchecksums.txt. - Compare:
python3 benchmark/summarize_bench.py <ref_label> <new_label>(medians and ratio) anddiff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt— any difference is a bug, not a performance matter. - A ratio is only a candidate effect. Check the spread by running the same binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from
/usr/bin/time -l). - Rule of thumb for this data set: only
query(≈ 5 s, stable to ±1 %) resolves effects of a few percent;index,merge,filter,phyloneed interleaved runs or a quiet machine.
Step → code exercised:
| Step | Code on the MPHF path |
|---|---|
index_* (:index stage) |
MphfLayer::build pass 2 |
merge_20_presence |
find_batch_grouped, build_graph filter, hash_batch, index_batch |
query_* |
query_partition_with stage 1 |
filter_species |
iter_layer_kmers → hash_batch |
convert_to_* |
build_exact_evidence, build_approx_evidence |
phylo_sibling_annex |
annex.rs find_in_partition per variant |
phylo_snp_distance |
family_scan.rs hash_batch, find_slot |
10. Second step — construction pass 2 (2026-10-15)
Implemented: layer/slotted.rs, SlottedKmers, an iterator adapter yielding (slot, kmer, chunk_id, rank) for the wrapped (kmer, chunk_id, rank) stream, computing slots 4096 kmers at a time through index_stream::<32> (bounded memory, never the whole layer). The five loops of MphfLayer::build (Exact, Approx, Hybrid), build_exact_evidence and build_approx_evidence now iterate it instead of calling mphf.index per kmer. Output digests unchanged.
Measured (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
| Command | before | after |
|---|---|---|
convert --exact-evidence (20-genome index) |
0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
index E. coli reads |
2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
No measurable effect on this data set (differences within the run-to-run spread; if anything convert is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and index/convert time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/seen writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random writes to evidence.bin/seen/fingerprint.bin (not prefetched by ptr_hash) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
11. Why the two changes gain so little (profiling, 2026-10-15)
Measured with macOS sample (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
Query. In a 2 s window on the query stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
query_partition_with≈ 260 samples (8 % of active CPU), of whichfind_batch217, of whichverify_canonical_kmer185 and everything else infind_batch(theptr_hashstream andconfirm) 32.verify_canonical_kmeris the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/
unitigs.bin/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side. - The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (
rolling_stat, entropy tracker, string formatting) and by sequential parts, not by lookups.
What produced the −5 %. Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer index (stream disabled) 6.0 s; restructuring + index_stream 5.85 s. Most of the gain comes from the restructuring, not from the prefetch (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called IndexCache::get_layer — an Arc<KmerLayer> clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
Index (pass 2). In a 1 s window late in the index stage, SlottedKmers appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (obidebruinj) and ≈ 35 for ptr_hash::build (pass 1, MPHF construction). Pass 2 is a few percent of index and convert here: nothing to win even if its lookups were free. convert is build_*_evidence on 2 MB-scale layers, same conclusion.
Consequences for priorities.
- On this data the next lever is the verification (
verify_canonical_kmer, evidence decode), not the MPHF — consistent with §3, now observed. - Hot
Arcclones in per-kmer paths (get_layer,IndexCache::find_in_layer,find_slotloops in merge and phylo) are worth auditing independently of MPHF batching. - The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
12. Calibration: a single skim at fewer partitions (2026-10-15)
Vaccinium myrtillus skim (2 × ~440 MB fastq.gz, theta 0), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (--partitions is a power of two; MPHF size ≈ 2.4 bits/key [measured]):
-p |
keys / partition | mphf.bin / partition |
index wall | peak RSS |
|---|---|---|---|---|
| 256 (default) | ≈ 1.4 M [computed from p=16] | ≈ 0.4 MB [computed] | – | – |
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
Total ≈ 354 M distinct kmers for one skim. -p 4 is already in the 30–45 MB range of the prod layer_1 MPHFs; -p 2 would give ≈ 53 MB. Indexes kept in benchmark/run/bench_mphf/skim_p16 and skim_p4 (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.