Files
obikmer/DevDocMD/implementation/mphf_batch_queries.md
T
Eric Coissac 8296e3544c
Release / create-release (push) Successful in 2m31s
Release / build-linux-x86_64 (push) Successful in 9m24s
Release / build-macos-arm64 (push) Successful in 2m5s
ci.yml / build (pull_request) Successful in 3m51s
Simplify Hybrid layer implementation and update dependencies
Dropped fields related to Hybrid layers for simplification and removed the unused `ahash` dependency. Updated dependency versions for `obidebruinj`, `obikmer`, and related crates, and cleaned up temporary artifacts.
2026-10-15 12:23:15 +02:00

42 KiB

MPHF batch queries — site inventory and expected gains

Investigation of 2026-10-14 (ptr_hash 2.1.2, epserde 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked [read] (verified in source), [computed] (derived from stated sizes) or [estimate] (reasoning, not measured). Measurements, once made, are in the last section.

1. What ptr_hash 2.1.2 actually offers [read]

Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>.

  • V is the pilot array (one byte per bucket), F is only the remap storage. A layer's mphf.bin (~2.4 bits/key [computed]: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch pilots[bucket] — the right line. The remap (read only when slot >= n, a small fraction of keys) is not prefetched.
  • index_batch<K>(xs: [Q; K]) -> [usize; K] — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (len % K) is the caller's job.
  • index_stream<B>(keys: IntoIterator) -> impl Iterator — ring buffer of B keys; handles head and tail itself; accepts any iterator. Its next() is unimplemented!: only fold-based consumption works (for_each, fold, sum). collect(), zip, for x in it panic. Consume with for_each into a pre-sized Vec.
  • index_batch_exact needs the nightly unstable feature; index_batch_exact2 is #[doc(hidden)]. Neither is usable.
  • Prefetch is prefetch-index 0.2.1: _mm_prefetch(T0) on x86_64, asm on aarch64, no-op elsewhere.
  • SINGLE_PART: neither index_stream nor index_batch branches on it — both always run the multi-part bucket()/slot(). Only index() has the faster single-part path. So with batched queries the SINGLE_PART = true advantage disappears, and false (our choice) costs nothing while keeping parallel construction (true also panics with sharding).
  • Old ptr_hash 1.1.0 (used by tools/migrate-mphf) has the same family with an extra MINIMAL const: index_stream<B, MINIMAL, Q>, index_batch<K, MINIMAL, Q>. No new dependency needed there.
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
    .for_each(|slot| out.push(slot));

2. Existing "batch" APIs that are not batched [read]

MphfLayer::hash_batch and MphfOnly::index_batch loop .index() one kmer at a time. Consumers: obikfilter/partition_iter.rs (iter_layer_kmers), obikmerge/partition_merge.rs (pass 2: hash_batch, index_batch), obikphylo/siblings/algorithms/family_scan.rs (two sites).

3. Cost chain of a membership query (MphfLayer::find) [read]

  1. pilots[bucket] — the only access ptr_hash prefetches.
  2. Exact/Hybrid-exact: evidence.decode(slot) — 4 B/slot, ~520 MB per 130 M-slot layer [computed], random mmap read.
  3. verify_canonical_kmer → block_offsets[...] (O(1) with the default --block-size 1; a forward scan of up to 2^block_bits records otherwise).
  4. extract_kmer_raw — random read in the unitigs.bin mmap.

Approx/Hybrid-fast: pilots then fingerprint.get(slot) — one extra random access. Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. ptr_hash only helps with stage 1; stages 2–4 need prefetch support in obikindex/obiskio (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.

Layer sizes: layer_0 MPHF 2–3 MB (cache-resident), layer_1 ~40 MB (beyond one CCD's L3 on EPYC-class machines — hypothesis, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on layer_1. For branch-free independent loops (hash_batch) out-of-order execution already overlaps several misses, so the explicit prefetch gains less [estimate] than for branchy loops (find with early break).

4. Call-site inventory

(a) already batched; (b) batch buildable; (c) hard/pointless.

Site What Class
mphf_layer.rs hash_batch, MphfOnly::index_batch loop of .index() (a)
obikfilter/partition_iter.rs:106 hash_batch, 32768 kmers (a)
obikmerge/partition_merge.rs:489, 521 hash_batch, index_batch, 4096 (a)
obikphylo/.../family_scan.rs:285, 397 hash_batch (a)
obikquery/query_layer.rs stage 1 find_in_layer per kmer, layer by layer, break on hit (b)
partition_merge.rs find_batch_grouped find_slot per kmer over dst layers (b)
partition_merge.rs (build_graph filter) all(find_slot.is_none()), every layer, per kmer; build_graph takes Fn(CanonicalKmer)->bool (b), API change
family_scan.rs slow path; annex.rs find_slot / find_in_partition per variant (b)
mphf_layer.rs build, build_exact_evidence, build_approx_evidence mphf.index then random writes (evidence, fingerprint, seen) (b), chunked
obikindexer/counter/count.rs:47 mphf.index then counts.get/set (b), hard (needs (kmer, count) pairs)
obikindexer/extensions/private_builder.rs:177 abundance filter (b), conditional
private_builder.rs:196, verify.rs, IndexCache::find_in_* per-kmer by nature / sampled (c)
tools/migrate-mphf/plan.rs:41-42 old.index + new.index per kmer, random writes (b)

Dead or suspect code found along the way (candidates for removal, not removed):

  • MphfLayer::find_strict — no caller.
  • the single-kmer hash chain (MphfLayer::hash → TypedLayer::hash → KmerLayer::hash → IndexCache::hash) — no caller outside the chain.
  • IndexCache::find — no caller.
  • obikmerge/src/merge_layer/ — not declared in lib.rs (mod merge; mod partition_merge; only); still imports the LayeredMap removed in 2026-08. To be confirmed by the compiler.

5. Interactions

  • Verification chain: the right shape is one find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>) on MphfLayer, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
  • Memory: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one Vec<usize> per call. index_stream itself only allocates a B-element ring on the stack.
  • Parallelism: no new parallelism. Batches already come out of obipipeline stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In query_partition_with the unique-kmer HashMap is already materialised; iterate it chunk-wise.

6. Benchmark

benchmark/bench_mphf.sh LABEL times, on the 20-genome benchmark data, index (2 genomes), merge (20 presence indexes), query (presence/count × 2 genomes), filter, convert (approx/exact round trip) and phylo (--sibling-annex, snp-jc), REPS times each, and records order-independent output digests that must not change across labels. See the results section below.

Limit of this data set: 20 small genomes → per-partition layer_1 MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it cannot measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).

Throwaway prod-scale benchmark (not written)

Open one real partition's layer_1 (~130 M keys) and layer_0 (~8 M) with MphfOnly/MphfLayer; key sets: present (sample of iter_kmers, random order and sequential order) and absent (random, checked absent); measure kmers/s for index, index_stream::<B> (B ∈ 8…128), index_batch::<K> (K ∈ 8…64), and the full find vs staged find_batch; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of mphf.bin, evidence.bin, unitigs.bin(.idx) reported; first check index_stream slots == index slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in mphf.md) is a paper claim, not tied to our sizes or machines.

7. Not verified

No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and --block-size used in prod; whether migrate-mphf already ran; share of total index/merge/query time spent in these loops; count.rs f0 per partition at prod scale; whether merge_layer/ is dead (only the head of lib.rs read).

8. First improvement and measurements (2026-10-15)

Implemented (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):

  • MphfLayer now holds a MphfOnly instead of its own MemCase<Mphf> (one mmap-open path, one place for the stream call).
  • MphfOnly::for_each_slot(kmers, f) wraps index_stream::<32>; index_batch/hash_batch now really use it (they previously looped .index()). Callback-shaped because index_stream's next() is unimplemented!.
  • MphfLayer::find_batch / TypedLayer::find_slot_batch / KmerLayer::find_slot_batch; the mode-dependent half of find is factored into MphfLayer::confirm, shared by both.
  • query_partition_with stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, n_mphf_calls unchanged).
  • Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; prefetch-index is already in the tree via ptr_hash, but adding it to obikindex is a dependency decision), and the other (b) sites.

Measurements (benchmark/bench_mphf.sh, benchmark/summarize_bench.py; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:

wall (s) user CPU (s)
before 5.94–6.02 (median 5.96) 10.95–11.07
after 5.63–6.00 (median 5.67) 10.33–10.66

About −5 % wall and −5.5 % CPU, ranges non-overlapping [measured]. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.

Reading the number. Small by construction: this data set's layer_1 MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains unmeasured and needs a prod-scale layer (§6).

Noise caveat. With a load average of 16–22 from unrelated processes, single-run walls of index, merge and filter moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The index/merge/filter/convert/phylo rows of the first comparison are noise, not effects (index pass 2 was not changed at all; a later interleaved filter A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).

Stale benchmark indexes. The indexes already under benchmark/run/ (specimen_index_*, global_index_*, built 2026-08-28) carry ptr_hash 1.x MPHFs and fail with "Major version mismatch" on current binaries (tools/migrate-mphf converts them). bench_mphf.sh therefore builds its own copies under run/bench_mphf/data/ on first use.

9. Benchmark procedure

Prerequisites: the simulated reads (benchmark/run/simulated_data/) and query reads (benchmark/run/query_data/) produced by gmake simulate simulate_query; macOS BSD time -l and md5 -q (the script uses both), Python 3.

  1. Build the binary to measure and copy it under a name that identifies it (cargo overwrites target/release/obikmer): cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>.
  2. BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label> (from benchmark/; REPS=3 by default, CPU_MAX=n to restrict cores). The first call builds the shared data set in run/bench_mphf/data/ with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete data/ and rebuild.
  3. Output: run/bench_mphf/<label>/results.csv (wall and peak RSS per step and repetition, plus one row step:stage per Reporter stage) and checksums.txt.
  4. Compare: python3 benchmark/summarize_bench.py <ref_label> <new_label> (medians and ratio) and diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt — any difference is a bug, not a performance matter.
  5. A ratio is only a candidate effect. Check the spread by running the same binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from /usr/bin/time -l).
  6. Rule of thumb for this data set: only query (≈ 5 s, stable to ±1 %) resolves effects of a few percent; index, merge, filter, phylo need interleaved runs or a quiet machine.

Step → code exercised:

Step Code on the MPHF path
index_* (:index stage) MphfLayer::build pass 2
merge_20_presence find_batch_grouped, build_graph filter, hash_batch, index_batch
query_* query_partition_with stage 1
filter_species iter_layer_kmers → hash_batch
convert_to_* build_exact_evidence, build_approx_evidence
phylo_sibling_annex annex.rs find_in_partition per variant
phylo_snp_distance family_scan.rs hash_batch, find_slot

10. Second step — construction pass 2 (2026-10-15)

Implemented: layer/slotted.rs, SlottedKmers, an iterator adapter yielding (slot, kmer, chunk_id, rank) for the wrapped (kmer, chunk_id, rank) stream, computing slots 4096 kmers at a time through index_stream::<32> (bounded memory, never the whole layer). The five loops of MphfLayer::build (Exact, Approx, Hybrid), build_exact_evidence and build_approx_evidence now iterate it instead of calling mphf.index per kmer. Output digests unchanged.

Measured (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):

Command before after
convert --exact-evidence (20-genome index) 0.56–0.68 s / 2.22–2.67 s 0.59–0.74 s / 2.25–2.52 s
index E. coli reads 2.45–2.63 s / 13.1–14.3 s 2.48–2.75 s / 13.2–14.0 s

No measurable effect on this data set (differences within the run-to-run spread; if anything convert is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and index/convert time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/seen writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random writes to evidence.bin/seen/fingerprint.bin (not prefetched by ptr_hash) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.

11. Why the two changes gain so little (profiling, 2026-10-15)

Measured with macOS sample (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.

Query. In a 2 s window on the query stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):

  • query_partition_with ≈ 260 samples (8 % of active CPU), of which find_batch 217, of which verify_canonical_kmer 185 and everything else in find_batch (the ptr_hash stream and confirm) 32.
  • verify_canonical_kmer is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
  • Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/unitigs.bin/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
  • The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (rolling_stat, entropy tracker, string formatting) and by sequential parts, not by lookups.

What produced the −5 %. Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer index (stream disabled) 6.0 s; restructuring + index_stream 5.85 s. Most of the gain comes from the restructuring, not from the prefetch (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called IndexCache::get_layer — an Arc<KmerLayer> clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.

Index (pass 2). In a 1 s window late in the index stage, SlottedKmers appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (obidebruinj) and ≈ 35 for ptr_hash::build (pass 1, MPHF construction). Pass 2 is a few percent of index and convert here: nothing to win even if its lookups were free. convert is build_*_evidence on 2 MB-scale layers, same conclusion.

Consequences for priorities.

  1. On this data the next lever is the verification (verify_canonical_kmer, evidence decode), not the MPHF — consistent with §3, now observed.
  2. Hot Arc clones in per-kmer paths (get_layer, IndexCache::find_in_layer, find_slot loops in merge and phylo) are worth auditing independently of MPHF batching.
  3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).

12. Calibration: a single skim at fewer partitions (2026-10-15)

Vaccinium myrtillus skim (2 × ~440 MB fastq.gz, theta 0), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (--partitions is a power of two; MPHF size ≈ 2.4 bits/key [measured]):

-p keys / partition mphf.bin / partition index wall peak RSS
256 (default) ≈ 1.4 M [computed from p=16] ≈ 0.4 MB [computed] – –
16 22.1 M 6.65 MB 120 s 12.3 GB
4 86–90 M 25.9–27.0 MB 171 s 19.5 GB

Total ≈ 354 M distinct kmers for one skim. -p 4 is already in the 30–45 MB range of the prod layer_1 MPHFs; -p 2 would give ≈ 53 MB. Indexes kept in benchmark/run/bench_mphf/skim_p16 and skim_p4 (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.

13. Query on a real-scale index (2026-10-15)

Index. Working index phyloskims_sal_vac_sav (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with tools/migrate-mphf on an APFS clone (cp -c, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at /Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2; its sibling annex is in stale_annexes_pre_ptrhash2/ and would need rebuilding to be used. Per partition: layer_1 ≈ 31 M keys / 9.3 MB MPHF, layer_0 ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).

Query (2 × 48 MB of V. oxycoccos reads, --chunk-size 8, AC power, 3 interleaved runs per binary, identical sorted output for all three):

binary wall (s) user CPU (s) query stage (s)
original 28.2–30.5 70.9–74.4 14.6–17.1
+ batched query stage 1 26.9–28.0 48.1–48.6 13.6–14.4
+ PartitionKmers memory fix, build-side SlottedKmers 26.4–26.7 41.9–42.5 12.9–13.3

The query stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. The CPU drop is much larger than the wall drop and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer Arc clones of get_layer, not isolated); the original binary was spending CPU in contention that wall time hid.

Index-open cost (not related to the changes above). A query of a single 64-base sequence on this index takes 15 s and a 27 GB peak footprint. vmmap during the open: 24.5 GB of MALLOC_LARGE in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by IndexCache::new for every layer of every partition whatever the query. Known contributors: UnitigFileReader reads each .idx into a Vec (≈ 30 MB for layer_1, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.

14. Index-open memory: .idx offsets and dict_values mapped, not copied (2026-10-15)

Cause (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, 18.2 GB were unitigs.bin.idx read into a Vec<u32> twice per exact-evidence layer (MphfLayer::open builds two UnitigFileReaders of the same file — one for the evidence check, one for iter_kmers/find_strict; both go through open → open_direct_access → read_idx → fs::read + copy) and 7.7 GB were dict_values.bin read with fs::read in PersistentSparseBitMatrix::open. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (PersistentRankSelectBitVec, PersistentFixedIntVec, EliasFano) was already mmapped.

Change.

  • obiskio::unitig_index::reader: block_offsets: Vec<u32> → BlockOffsets, a view into the memory-mapped .idx (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
  • obicompactvec::bitmatrix::sparse: dict_values: Vec<u8> → Option<Mmap> (None for an empty file, which cannot be mapped), read through dict_bytes().

Measured (same session, same inputs; sorted query outputs identical, md5 e9e7c35b):

open only (1-sequence query) 2×48 MB reads, --chunk-size 8
before 10.6–15.7 s, 27.4 GB footprint 27.7 s (51 s cold), 30.9 GB, query stage 13.8 s
after 0.26–0.81 s, 0.04 GB 10.7 s, 3.6 GB, query stage 9.95 s

Tests: cargo test --workspace green; scripts/smoke_test_index.sh PASS; bench_mphf.sh (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a .idx map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.

15. Unbuffered reads: index.meta and superkmer files (2026-10-15)

Follow-up of §14 (search for the same family of problems: file content handled inefficiently at open). Static search over src/ for fs::read, read_to_end, load_full, from_reader, bare File readers and mmap to_vec: every persistent structure of obicompactvec, obikindex and the obikphylo annexes is already mmapped; the two real findings were unbuffered readers.

1. IndexMeta::read_full (obikindex/src/index/meta.rs) passed a bare File to serde_json::from_reader, which issues one read syscall per byte. index.meta of the 91-genome index is 20 KB and is read afresh by every state()/genomes() call (stateless design, §obikindex audit), so each call cost ≈ 0.35 s here (macOS). Profile of obikmer dump --head 10 (sample): ≈ 20 000 samples in read, called from Bytes<File>::next ← serde_json ← IndexMeta. Fix: fs::read then serde_json::from_slice. dump --head 10: 9.0–10.0 s (91–102 s of system CPU) → 0.16–0.84 s; the same change applies to SKFileMeta::read (obiskio/src/meta.rs, per-superkmer-file sidecars).

2. SKFileReader::open (obiskio/src/reader.rs): for an uncompressed file niffler::get_reader returns the sniffed bytes chained to the bare File, and read_superkmer reads each field with its own small read_exact. Fix: wrap the file in a 64 KiB BufReader before niffler. Effect on index (E. coli, 4 interleaved runs each): count_kmer 0.51–0.56 s → 0.42–0.50 s (≈ −12 %), total wall unchanged (≈ 1.95 s). Small, low risk, kept.

Tests: cargo test --workspace 549 passed / 0 failed; scripts/smoke_test_index.sh PASS; query output on the 91-genome index identical (md5 e9e7c35b).

Probes of other commands on the 91-genome index (180 GB, 256 partitions): phylo --distance jaccard peaks at 155–190 MB of footprint while running and finishes in < 30 s; utils --stats 130–150 MB. filter (and by the same check select, merge) refused the index — "source index is not fully built" — with the unmodified baseline binary too: this index's index.meta predates the state field, so they could not be probed.

Observations not investigated: index on E. coli shows ≈ 9 s of system CPU in 2 s of wall time (both before and after these changes), consistent with the swtch_pri (yield) spinning seen in the index profile of §11 rather than with I/O; and wall times of the same query on the same binary varied 4.8 s ↔ 10.7 s ↔ 43 s depending on the state of the page cache, so only same-session comparisons are meaningful.

16. Profile of index on E. coli reads (2026-10-15)

sample (1 ms stacks) of obikmer index on the E. coli K-12 simulated reads (2 × 15x, 16 threads, one run; the index stage lasted 2.4 s under the sampler vs ≈ 0.9 s normally, so proportions are approximate). Stage walls: scatter 0.40 s, dereplicate 0.14, count_kmer 0.60, index 2.36.

Threads are mostly idle. Of ≈ 30 800 thread-samples: 24 900 blocked or yielding, 5 900 active. The ≈ 9 s of system CPU per run (§15) is swtch_pri (sched_yield) — 4 550 samples: ≈ 70 % rayon workers spinning for work (WorkerThread wait loop), ≈ 28 % crossbeam bounded-channel send/recv spinning, the rest in obidebruinj thread-locals. It is waiting, not I/O: no read samples remain. It costs no wall time on an otherwise idle machine, but burns CPU (visible in user + sys accounting and on a shared 192-core machine).

Where the active CPU goes (share of the 5 900 active samples):

share
xxh3 hashing of CanonicalKmer for the de Bruijn graph's HashMap (callers: count_neighbors 791, hash_one 358, walk 285, unitig_iter 252, push 164, unitig_nucleotides 75, rehash 68; plus xxh3_stateful_update 300) ≈ 34 %
debruijn::count_neighbors itself (inclusive of its hashing ≈ 2 300 samples: compute_degrees_and_mark_starts) 12.5 % own
ptr_hash::build (MPHF construction) 4.8 %
crossbeam channel send 4.8 %
open / close / write (many small files over 256 partitions) ≈ 9 %
debruijn unitig_iter / walk own time 8.3 %

So at this scale the cost of index is the de Bruijn graph's hash table (several lookups of neighbouring k-mers per k-mer, each hashed with xxh3 and probably missing the cache), not the MPHF passes this document started from (SlottedKmers ≈ 2 %, §11). Not measured: what fraction of those lookups miss the cache, and the same profile on a prod-size partition.

17. Profile of index on a skim: Vaccinium vitis-idaea, 2.1 GB, --partitions 16 (2026-10-15)

One run, sample windows of 5 s every ~17 s over the whole run (the sampler slows the process; walls below include it): scatter 19.7 s, dereplicate 8.0 s, count_kmer 25.2 s, index 139.1 s; total 192 s real, 1830 s user, 225 s system (12 % of user, against ≈ 90 % on E. coli, §16), peak footprint 17.8 GB.

Index stage (samples taken fully inside it, 91 000 active vs 187 000 idle/yielding thread-samples):

share of active CPU
xxh3 hashing of kmers (de Bruijn HashMap) 32.7 %
debruijn::walk 12.4 %
debruijn::count_neighbors 10.7 %
GraphDeBruijn::push 6.9 %
crossbeam channel 6.6 %
hashbrown table operations 5.0 %
debruijn::unitig_iter 4.8 %
memcpy/memset, malloc/free, file I/O, SipHash ≈ 7 %
other 13.7 %
  • The de Bruijn graph is ≈ 72 % of the active CPU of index; the MPHF construction does not appear (< 1 %), and SlottedKmers/evidence filling is negligible. The conclusion of §16 holds at 100× the E. coli scale.
  • The stage is poorly parallel: on average ≈ 5 of 16 threads are active (33 % of thread-samples); swtch_pri (spin/yield) is 7 % of all thread-samples. With 16 partitions of unequal size the tail partitions run alone. The 12 % system CPU is the same spinning as §16, at a smaller relative weight.
  • count_kmer (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. scatter (one window, 32 % active): RollingStat::push 32 %, EntropyTracker::push 23 % + normalized_entropy 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.

Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for u64 kmers or fewer lookups per kmer would change the picture.

18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)

Benchmark harness: obikindexer/examples/graph_bench.rs (graph_bench <unitigs.bin> <k> [repeat]; env IDEAL=1 counts maximal unitigs independently of GraphDeBruijn, SYNTH=n uses synthetic reads, DUMP/IDEALDUMP print unitigs). Reference input: the 32.6 M kmers of one partition of the V. vitis-idaea index (-p 16, k = 31). Phase timings, 16 threads: push / compute_degrees_and_mark_starts / for_each_unitig.

push degrees unitigs unitigs produced nucleotides
original 4.1 s 2.0 s 3.9 s 6.56 M (varies ±900 per run) 229.4 M
u64 hasher (below) 1.0 s 0.7 s 3.1 s 6.56 M 229.4 M
+ orientation fix 1.0 s 0.8 s 1.0 s 3.81 M (stable ±3) 146.9 M

Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).

1. Hasher (debruijn/kmer_hash.rs). The node table used xxh3's streaming Hasher (Xxh3Builder), for a key that is one u64: ≈ 35 % of the active CPU of index (§17). Replaced by a splitmix64 finaliser BuildHasher (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. xxhash-rust is no longer used by obidebruinj (left in its Cargo.toml; removing it is a dependency decision).

2. Walk: WalkState now carries the node's own &AtomicU8, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).

3. Unitig fragmentation — a bug, not a tuning. Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where find_chain_start returned only the kmer, dropping the orientation, and unitig_nucleotides always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a lone-kmer unitig, repeatedly, along the chain. The result depended on table iteration order (degrees_linear_chain_extensions passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: 72 % more unitigs than maximal on the reference partition. A second slip in the reverse branch of WalkState::walk (the can_extend_left/can_extend_right test of the next node was swapped) cost another ~15 %. Fixes: walk condition corrected; find_chain_start returns (kmer, direct) with the orientation to walk forward; unitig_nucleotides and UnitigNucIter take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (start_direction). Regression tests: linear_chain_is_one_unitig_whatever_the_orientations (fails on the original code, passes now) and unitigs_partition_the_kmers_of_branching_reads (every kmer in exactly one unitig).

Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).

End-to-end (current binary vs previous one, same inputs):

before after
V. vitis-idaea 2.1 GB, --partitions 16: total wall 163.8 s 116.5 s (−29 %)
… index stage 114.6 s 59.2 s (−48 %)
… user CPU 1 870 s 1 304 s
… unitigs.bin / .idx 1.05 / 0.42 GB 0.66 / 0.24 GB (−37 % / −43 %); index 3.5 → 2.9 GB
E. coli, index stage / wall / user CPU 0.91 s / 1.97 s / 10.2 s 0.59 s / 1.63 s / 7.3 s

Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli dump and query outputs identical (md5), scripts/smoke_test_index.sh PASS, cargo test --workspace 551 passed / 0 failed, and bench_mphf.sh digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.

19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)

Evidence packs (chunk_id, rank) into a u32 with a 7-bit rank (encode masks & 0x7F), but MAX_KMERS_PER_CHUNK was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: --partitions 256 → max chunk 49–50 k-mers, results correct with the old and the new binary; --partitions 1 → max chunk 256, query output wrong with both the old (md5 bde0f886a0) and the post-graph-fix (8f1c94ef32) binary versus the correct f99e971087. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.

Change (obikseq::MAX_KMERS_PER_CHUNK: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (SuperKmer::write_to_binary): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test chunk_ranks_fit_the_7_bit_evidence_field.

After (E. coli, same query): --partitions 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 f99e971087 in all four cases. cargo test --workspace: 551 passed, 0 failed.

Existing indexes. An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The --partitions 4 skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin" reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).

20. Summary of what was gained (2026-10-15)

Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.

Speed

Operation Original Now Where
Open an index (query of one 64-base sequence), 91 genomes, 180 GB 10.4–13.3 s 0.18–0.19 s §14, same-session A/B
Query, 2 × 48 MB reads on that index (--chunk-size 8) 30.7–32.0 s (61.8 s cold), 70–76 s CPU 11.6–13.2 s, 36–39 s CPU (−62 % wall, −50 % CPU) §14, same-session A/B
dump --head 10 on that index 9.0–10.0 s (91–102 s system CPU) 0.16–0.84 s §15
index of a 2.1 GB skim (V. vitis-idaea, --partitions 16) 163.8 s (stage index 114.6 s) 116.5 s (stage index 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU §18
index of E. coli reads 1.97 s (stage 0.91 s), 10.2 s CPU 1.63 s (stage 0.59 s), 7.3 s CPU §18
De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) 4.1 s / 2.0 s / 3.9 s 1.0 s / 0.8 s / 1.0 s (×3.5 overall) §18
index stage count_kmer (E. coli) 0.51–0.56 s 0.42–0.50 s §15
Query on a small input (2 × 48 MB reads), single-genome index 11.4 s (2 chunks = 2 busy cores) 2.4 s (−79 %) — automatic chunk size now follows the input size (architecture/query.md); on the 91-genome index the size was already 4 MiB §20 follow-up

Memory (peak RSS / footprint)

Operation Original Now
Opening the 91-genome index (any command that builds an IndexCache) 27.4 GB 0.04 GB (§14: .idx offsets and dict_values.bin mapped instead of copied; they were read twice and once)
Query, 2 × 48 MB reads on that index 32.2–32.5 GB 3.5–3.6 GB
Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size 73.7–80.7 GB 28.2 GB (§12 of architecture/query.md: PartitionKmers instead of a HashMap<kmer, Vec> per partition: 71 → ~20 bytes per unique kmer)
index of the 2.1 GB skim 19.7 GB 18.3 GB

Index size (newly built indexes; existing ones are unchanged)

Original Now
unitigs.bin / unitigs.bin.idx (2.1 GB skim, 521.6 M kmers, 16 partitions) 1.05 / 0.42 GB 0.66 / 0.24 GB (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18)
Whole index of that skim 3.5 GB 2.9 GB (−17 %)
evidence.bin 4 bytes per kmer unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented)

Correctness fixes found on the way

  • Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
  • Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, scripts/check_unitig_chunks.py to audit an index).
  • index.meta and super-kmer files were parsed byte by byte (§15).

What is not established

  • The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared Arc clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
  • Speed-ups on the 192-core / 1 TB production machine, on merge, filter, select and phylo at the 180-genome scale (the 91-genome index could only be used for query, dump, phylo --distance jaccard, utils --stats; filter/select/merge refused its pre-state index.meta until it was patched in the clone).
  • The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).

21. Dead code removed (2026-10-15)

The items listed as unused in §4 were removed after Serena's safe_delete_symbol and the compiler confirmed no reference: MphfLayer::find_strict; the single-kmer hash chain (MphfLayer::hash, TypedLayer::hash, KmerLayer::hash, IndexCache::hash); IndexCache::find; the uncompiled obikmerge/src/merge_layer/ directory (689 lines; lib.rs only declares merge and partition_merge); and xxhash-rust from obidebruinj/Cargo.toml (it still serves obikseq, obifastwrite and obiksession).

Side effect: find_strict was the only reader of the unitigs handle that Hybrid layers opened, so that field (and the extra .idx map and mmap per Hybrid layer) was dropped too. find on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what convert works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding find_strict and that field.

ahash was not referenced by any source file of obidebruinj either and was removed from its Cargo.toml too.

Checks after the removal: cargo build --release without warnings, cargo test --workspace 555 passed / 0 failed, scripts/smoke_test_index.sh PASS.

Scratch artifacts of this work were deleted: the intermediate binaries, the two skim indexes (skim_p4, skim_p16) and the shared data set benchmark/run/bench_mphf/data (rebuilt automatically, ≈ 2 min, by the next bench_mphf.sh). Kept in benchmark/run/bench_mphf/: the original binary obikmer_baseline (reference for A/B runs) and the per-label results and checksums (18 MB in all). The 91-genome index converted to the current MPHF format (/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2, with its index.meta patched, original index.meta.before_state_fix beside it) was left in place.