Files
obikmer/DevDocMD/implementation/mphf_batch_queries.md
T

367 lines
42 KiB
Markdown
Raw Normal View History

# MPHF batch queries — site inventory and expected gains
Investigation of 2026-10-14 (`ptr_hash` 2.1.2, `epserde` 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked **[read]** (verified in source), **[computed]** (derived from stated sizes) or **[estimate]** (reasoning, not measured). Measurements, once made, are in the last section.
## 1. What `ptr_hash` 2.1.2 actually offers [read]
`Mphf = PtrHash<u64, CubicEps, F = CachelineEfVec, Xxh3Int, V = Vec<u8>, SINGLE_PART = false, REMAP = true>`.
- **`V` is the pilot array** (one byte per bucket), **`F` is only the remap storage**. A layer's `mphf.bin` (~2.4 bits/key **[computed]**: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch `pilots[bucket]` — the right line. The remap (read only when `slot >= n`, a small fraction of keys) is **not** prefetched.
- `index_batch<K>(xs: [Q; K]) -> [usize; K]` — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (`len % K`) is the caller's job.
- `index_stream<B>(keys: IntoIterator) -> impl Iterator` — ring buffer of B keys; handles head and tail itself; accepts any iterator. **Its `next()` is `unimplemented!`**: only fold-based consumption works (`for_each`, `fold`, `sum`). `collect()`, `zip`, `for x in it` panic. Consume with `for_each` into a pre-sized `Vec`.
- `index_batch_exact` needs the nightly `unstable` feature; `index_batch_exact2` is `#[doc(hidden)]`. Neither is usable.
- Prefetch is `prefetch-index 0.2.1`: `_mm_prefetch(T0)` on x86_64, asm on aarch64, **no-op elsewhere**.
- **`SINGLE_PART`**: neither `index_stream` nor `index_batch` branches on it — both always run the multi-part `bucket()`/`slot()`. Only `index()` has the faster single-part path. So with batched queries the `SINGLE_PART = true` advantage disappears, and `false` (our choice) costs nothing while keeping parallel construction (`true` also panics with sharding).
- Old `ptr_hash` 1.1.0 (used by `tools/migrate-mphf`) has the same family with an extra `MINIMAL` const: `index_stream<B, MINIMAL, Q>`, `index_batch<K, MINIMAL, Q>`. No new dependency needed there.
```rust
// consuming index_stream: fold-based only
let mut out = Vec::with_capacity(kmers.len());
mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw()))
.for_each(|slot| out.push(slot));
```
## 2. Existing "batch" APIs that are not batched [read]
`MphfLayer::hash_batch` and `MphfOnly::index_batch` loop `.index()` one kmer at a time. Consumers: `obikfilter/partition_iter.rs` (`iter_layer_kmers`), `obikmerge/partition_merge.rs` (pass 2: `hash_batch`, `index_batch`), `obikphylo/siblings/algorithms/family_scan.rs` (two sites).
## 3. Cost chain of a membership query (`MphfLayer::find`) [read]
1. `pilots[bucket]` — the only access `ptr_hash` prefetches.
2. Exact/Hybrid-exact: `evidence.decode(slot)` — 4 B/slot, ~520 MB per 130 M-slot layer **[computed]**, random mmap read.
3. `verify_canonical_kmer` → `block_offsets[...]` (O(1) with the default `--block-size 1`; a forward scan of up to `2^block_bits` records otherwise).
4. `extract_kmer_raw` — random read in the `unitigs.bin` mmap.
Approx/Hybrid-fast: pilots then `fingerprint.get(slot)` — one extra random access.
Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. `ptr_hash` only helps with stage 1; stages 2–4 need prefetch support in `obikindex`/`obiskio` (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index.
Layer sizes: `layer_0` MPHF 2–3 MB (cache-resident), `layer_1` ~40 MB (beyond one CCD's L3 on EPYC-class machines — **hypothesis**, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on `layer_1`. For branch-free independent loops (`hash_batch`) out-of-order execution already overlaps several misses, so the explicit prefetch gains less **[estimate]** than for branchy loops (`find` with early `break`).
## 4. Call-site inventory
(a) already batched; (b) batch buildable; (c) hard/pointless.
| Site | What | Class |
|---|---|---|
| `mphf_layer.rs` `hash_batch`, `MphfOnly::index_batch` | loop of `.index()` | (a) |
| `obikfilter/partition_iter.rs:106` | `hash_batch`, 32768 kmers | (a) |
| `obikmerge/partition_merge.rs:489, 521` | `hash_batch`, `index_batch`, 4096 | (a) |
| `obikphylo/.../family_scan.rs:285, 397` | `hash_batch` | (a) |
| `obikquery/query_layer.rs` stage 1 | `find_in_layer` per kmer, layer by layer, `break` on hit | (b) |
| `partition_merge.rs` `find_batch_grouped` | `find_slot` per kmer over dst layers | (b) |
| `partition_merge.rs` (`build_graph` filter) | `all(find_slot.is_none())`, every layer, per kmer; `build_graph` takes `Fn(CanonicalKmer)->bool` | (b), API change |
| `family_scan.rs` slow path; `annex.rs` | `find_slot` / `find_in_partition` per variant | (b) |
| `mphf_layer.rs` `build`, `build_exact_evidence`, `build_approx_evidence` | `mphf.index` then random writes (evidence, fingerprint, `seen`) | (b), chunked |
| `obikindexer/counter/count.rs:47` | `mphf.index` then `counts.get/set` | (b), hard (needs (kmer, count) pairs) |
| `obikindexer/extensions/private_builder.rs:177` | abundance filter | (b), conditional |
| `private_builder.rs:196`, `verify.rs`, `IndexCache::find_in_*` | per-kmer by nature / sampled | (c) |
| `tools/migrate-mphf/plan.rs:41-42` | `old.index` + `new.index` per kmer, random writes | (b) |
Dead or suspect code found along the way (candidates for removal, not removed):
- `MphfLayer::find_strict` — no caller.
- the single-kmer `hash` chain (`MphfLayer::hash` → `TypedLayer::hash` → `KmerLayer::hash` → `IndexCache::hash`) — no caller outside the chain.
- `IndexCache::find` — no caller.
- `obikmerge/src/merge_layer/` — not declared in `lib.rs` (`mod merge; mod partition_merge;` only); still imports the `LayeredMap` removed in 2026-08. To be confirmed by the compiler.
## 5. Interactions
- **Verification chain**: the right shape is one `find_batch(&[CanonicalKmer], &mut Vec<Option<usize>>)` on `MphfLayer`, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops).
- **Memory**: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one `Vec<usize>` per call. `index_stream` itself only allocates a B-element ring on the stack.
- **Parallelism**: no new parallelism. Batches already come out of `obipipeline` stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In `query_partition_with` the unique-kmer `HashMap` is already materialised; iterate it chunk-wise.
## 6. Benchmark
`benchmark/bench_mphf.sh LABEL` times, on the 20-genome benchmark data, `index` (2 genomes), `merge` (20 presence indexes), `query` (presence/count × 2 genomes), `filter`, `convert` (approx/exact round trip) and `phylo` (`--sibling-annex`, `snp-jc`), `REPS` times each, and records order-independent output digests that must not change across labels. See the results section below.
**Limit of this data set**: 20 small genomes → per-partition `layer_1` MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it **cannot** measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below).
### Throwaway prod-scale benchmark (not written)
Open one real partition's `layer_1` (~130 M keys) and `layer_0` (~8 M) with `MphfOnly`/`MphfLayer`; key sets: present (sample of `iter_kmers`, random order and sequential order) and absent (random, checked absent); measure kmers/s for `index`, `index_stream::<B>` (B ∈ 8…128), `index_batch::<K>` (K ∈ 8…64), and the full `find` vs staged `find_batch`; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of `mphf.bin`, `evidence.bin`, `unitigs.bin(.idx)` reported; first check `index_stream` slots == `index` slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in `mphf.md`) is a paper claim, not tied to our sizes or machines.
## 7. Not verified
No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and `--block-size` used in prod; whether `migrate-mphf` already ran; share of total index/merge/query time spent in these loops; `count.rs` f0 per partition at prod scale; whether `merge_layer/` is dead (only the head of `lib.rs` read).
## 8. First improvement and measurements (2026-10-15)
**Implemented** (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse):
- `MphfLayer` now holds a `MphfOnly` instead of its own `MemCase<Mphf>` (one mmap-open path, one place for the stream call).
- `MphfOnly::for_each_slot(kmers, f)` wraps `index_stream::<32>`; `index_batch`/`hash_batch` now really use it (they previously looped `.index()`). Callback-shaped because `index_stream`'s `next()` is `unimplemented!`.
- `MphfLayer::find_batch` / `TypedLayer::find_slot_batch` / `KmerLayer::find_slot_batch`; the mode-dependent half of `find` is factored into `MphfLayer::confirm`, shared by both.
- `query_partition_with` stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, `n_mphf_calls` unchanged).
- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; `prefetch-index` is already in the tree via `ptr_hash`, but adding it to `obikindex` is a dependency decision), and the other (b) sites.
**Measurements** (`benchmark/bench_mphf.sh`, `benchmark/summarize_bench.py`; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index:
| | wall (s) | user CPU (s) |
|---|---|---|
| before | 5.94–6.02 (median 5.96) | 10.95–11.07 |
| after | 5.63–6.00 (median 5.67) | 10.33–10.66 |
About −5 % wall and −5.5 % CPU, ranges non-overlapping **[measured]**. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after.
**Reading the number.** Small by construction: this data set's `layer_1` MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains **unmeasured** and needs a prod-scale layer (§6).
**Noise caveat.** With a load average of 16–22 from unrelated processes, single-run walls of `index`, `merge` and `filter` moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The `index`/`merge`/`filter`/`convert`/`phylo` rows of the first comparison are noise, not effects (`index` pass 2 was not changed at all; a later interleaved `filter` A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread).
**Stale benchmark indexes.** The indexes already under `benchmark/run/` (`specimen_index_*`, `global_index_*`, built 2026-08-28) carry `ptr_hash` 1.x MPHFs and fail with "Major version mismatch" on current binaries (`tools/migrate-mphf` converts them). `bench_mphf.sh` therefore builds its own copies under `run/bench_mphf/data/` on first use.
## 9. Benchmark procedure
Prerequisites: the simulated reads (`benchmark/run/simulated_data/`) and query reads (`benchmark/run/query_data/`) produced by `gmake simulate simulate_query`; macOS BSD `time -l` and `md5 -q` (the script uses both), Python 3.
1. Build the binary to measure and **copy it** under a name that identifies it (cargo overwrites `target/release/obikmer`): `cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_<label>`.
2. `BINARY=$PWD/run/bench_mphf/obikmer_<label> benchmark/bench_mphf.sh <label>` (from `benchmark/`; `REPS=3` by default, `CPU_MAX=n` to restrict cores). The first call builds the shared data set in `run/bench_mphf/data/` with that binary (≈ 2 min); keep it for every label — the on-disk format must not differ between the binaries compared, otherwise delete `data/` and rebuild.
3. Output: `run/bench_mphf/<label>/results.csv` (wall and peak RSS per step and repetition, plus one row `step:stage` per Reporter stage) and `checksums.txt`.
4. Compare: `python3 benchmark/summarize_bench.py <ref_label> <new_label>` (medians and ratio) and `diff run/bench_mphf/<ref>/checksums.txt run/bench_mphf/<new>/checksums.txt` — **any difference is a bug**, not a performance matter.
5. A ratio is only a candidate effect. Check the spread by running the **same** binary twice under two labels (the script's own noise floor was ±10–40 % per step on a loaded laptop), or confirm with interleaved A/B runs of the single command (alternate the two binaries, ≥ 8 runs, compare ranges of wall and user CPU from `/usr/bin/time -l`).
6. Rule of thumb for this data set: only `query` (≈ 5 s, stable to ±1 %) resolves effects of a few percent; `index`, `merge`, `filter`, `phylo` need interleaved runs or a quiet machine.
Step → code exercised:
| Step | Code on the MPHF path |
|---|---|
| `index_*` (`:index` stage) | `MphfLayer::build` pass 2 |
| `merge_20_presence` | `find_batch_grouped`, `build_graph` filter, `hash_batch`, `index_batch` |
| `query_*` | `query_partition_with` stage 1 |
| `filter_species` | `iter_layer_kmers` → `hash_batch` |
| `convert_to_*` | `build_exact_evidence`, `build_approx_evidence` |
| `phylo_sibling_annex` | `annex.rs` `find_in_partition` per variant |
| `phylo_snp_distance` | `family_scan.rs` `hash_batch`, `find_slot` |
## 10. Second step — construction pass 2 (2026-10-15)
**Implemented**: `layer/slotted.rs`, `SlottedKmers`, an iterator adapter yielding `(slot, kmer, chunk_id, rank)` for the wrapped `(kmer, chunk_id, rank)` stream, computing slots 4096 kmers at a time through `index_stream::<32>` (bounded memory, never the whole layer). The five loops of `MphfLayer::build` (Exact, Approx, Hybrid), `build_exact_evidence` and `build_approx_evidence` now iterate it instead of calling `mphf.index` per kmer. Output digests unchanged.
**Measured** (interleaved A/B against the first-step binary, 8 runs each, wall / user CPU):
| Command | before | after |
|---|---|---|
| `convert --exact-evidence` (20-genome index) | 0.56–0.68 s / 2.22–2.67 s | 0.59–0.74 s / 2.25–2.52 s |
| `index` E. coli reads | 2.45–2.63 s / 13.1–14.3 s | 2.48–2.75 s / 13.2–14.0 s |
**No measurable effect** on this data set (differences within the run-to-run spread; if anything `convert` is ≈ 4 % slower in wall time). Expected here: the MPHF is cache-resident and `index`/`convert` time is dominated by other stages (scatter, dereplication, counting, unitig decoding, the random evidence/`seen` writes themselves). The construction gain, if any, only exists where the pilot array does not fit in cache, and the random *writes* to `evidence.bin`/`seen`/`fingerprint.bin` (not prefetched by `ptr_hash`) probably dominate there too. Whether to keep this change is therefore a decision to take after the prod-scale benchmark (§6), not on this data.
## 11. Why the two changes gain so little (profiling, 2026-10-15)
Measured with macOS `sample` (1 ms stack sampling) on the 20-genome data, one run each — coarse, but the orders of magnitude are unambiguous.
**Query.** In a 2 s window on the `query` stage (16 threads, ≈ 3 100 active samples against ≈ 55 000 samples of idle waiting):
- `query_partition_with` ≈ 260 samples (8 % of active CPU), of which `find_batch` 217, of which **`verify_canonical_kmer` 185** and everything else in `find_batch` (the `ptr_hash` stream and `confirm`) **32**.
- `verify_canonical_kmer` is the single largest active function of the window (399 top-of-stack samples). The MPHF lookup is ≈ 1 % of active CPU: even a free MPHF could not give more.
- Reason: per-partition pilot arrays are ~100 KB (≈ 25 MB over 256 partitions) and stay cached; the evidence/`unitigs.bin`/offset accesses of the verification (§3, stages 2–4) are the ones that miss. This data set already has the problem on the verification side, not on the MPHF side.
- The threads are mostly idle: the stage is bounded by reading/parsing/k-merising the reads (`rolling_stat`, entropy tracker, string formatting) and by sequential parts, not by lookups.
**What produced the −5 %.** Interleaved A/B, 8 runs each, median wall: original 6.4 s; chunked layer-major restructuring with plain per-kmer `index` (stream disabled) 6.0 s; restructuring + `index_stream` 5.85 s. **Most of the gain comes from the restructuring, not from the prefetch** (the prefetch adds ≈ 2 %, at the limit of the noise). Hypothesis, not isolated: the old loop called `IndexCache::get_layer` — an `Arc<KmerLayer>` clone, i.e. an atomic increment/decrement on a refcount shared by all threads — once per kmer per layer; the batched loop does it once per layer per chunk of 4096.
**Index (pass 2).** In a 1 s window late in the `index` stage, `SlottedKmers` appears with ≈ 7 samples out of 304 per thread (≈ 2 %), against ≈ 200 for the de Bruijn graph work (`obidebruinj`) and ≈ 35 for `ptr_hash::build` (pass 1, MPHF *construction*). Pass 2 is a few percent of `index` and `convert` here: nothing to win even if its lookups were free. `convert` is `build_*_evidence` on 2 MB-scale layers, same conclusion.
**Consequences for priorities.**
1. On this data the next lever is the verification (`verify_canonical_kmer`, evidence decode), not the MPHF — consistent with §3, now observed.
2. Hot `Arc` clones in per-kmer paths (`get_layer`, `IndexCache::find_in_layer`, `find_slot` loops in merge and phylo) are worth auditing independently of MPHF batching.
3. The DRAM-latency gain of prefetch is untested until a prod-scale layer is available (§6).
## 12. Calibration: a single skim at fewer partitions (2026-10-15)
`Vaccinium myrtillus` skim (2 × ~440 MB fastq.gz, `theta 0`), indexed with the current binary, to find the partition count that gives prod-sized MPHFs (`--partitions` is a power of two; MPHF size ≈ 2.4 bits/key **[measured]**):
| `-p` | keys / partition | `mphf.bin` / partition | index wall | peak RSS |
|---|---|---|---|---|
| 256 (default) | ≈ 1.4 M **[computed from p=16]** | ≈ 0.4 MB **[computed]** | – | – |
| 16 | 22.1 M | 6.65 MB | 120 s | 12.3 GB |
| 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB |
Total ≈ 354 M distinct kmers for one skim. `-p 4` is already in the 30–45 MB range of the prod `layer_1` MPHFs; `-p 2` would give ≈ 53 MB. Indexes kept in `benchmark/run/bench_mphf/skim_p16` and `skim_p4` (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11.
## 13. Query on a real-scale index (2026-10-15)
**Index.** Working index `phyloskims_sal_vac_sav` (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with `tools/migrate-mphf` **on an APFS clone** (`cp -c`, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at `/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`; its sibling annex is in `stale_annexes_pre_ptrhash2/` and would need rebuilding to be used. Per partition: `layer_1` ≈ 31 M keys / 9.3 MB MPHF, `layer_0` ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB).
**Query** (2 × 48 MB of *V. oxycoccos* reads, `--chunk-size 8`, AC power, 3 interleaved runs per binary, identical sorted output for all three):
| binary | wall (s) | user CPU (s) | `query` stage (s) |
|---|---|---|---|
| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 |
| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 |
| + `PartitionKmers` memory fix, build-side `SlottedKmers` | 26.4–26.7 | 41.9–42.5 | 12.9–13.3 |
The `query` stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. **The CPU drop is much larger than the wall drop** and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer `Arc` clones of `get_layer`, not isolated); the original binary was spending CPU in contention that wall time hid.
**Index-open cost (not related to the changes above).** A query of a single 64-base sequence on this index takes **15 s and a 27 GB peak footprint**. `vmmap` during the open: 24.5 GB of `MALLOC_LARGE` in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by `IndexCache::new` for every layer of every partition whatever the query. Known contributors: `UnitigFileReader` reads each `.idx` into a `Vec` (≈ 30 MB for `layer_1`, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied.
## 14. Index-open memory: `.idx` offsets and `dict_values` mapped, not copied (2026-10-15)
**Cause** (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, **18.2 GB** were `unitigs.bin.idx` read into a `Vec<u32>` **twice** per exact-evidence layer (`MphfLayer::open` builds two `UnitigFileReader`s of the same file — one for the evidence check, one for `iter_kmers`/`find_strict`; both go through `open` → `open_direct_access` → `read_idx` → `fs::read` + copy) and **7.7 GB** were `dict_values.bin` read with `fs::read` in `PersistentSparseBitMatrix::open`. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (`PersistentRankSelectBitVec`, `PersistentFixedIntVec`, `EliasFano`) was already mmapped.
**Change.**
- `obiskio::unitig_index::reader`: `block_offsets: Vec<u32>` → `BlockOffsets`, a view into the memory-mapped `.idx` (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages.
- `obicompactvec::bitmatrix::sparse`: `dict_values: Vec<u8>` → `Option<Mmap>` (`None` for an empty file, which cannot be mapped), read through `dict_bytes()`.
**Measured** (same session, same inputs; sorted query outputs identical, md5 `e9e7c35b`):
| | open only (1-sequence query) | 2×48 MB reads, `--chunk-size 8` |
|---|---|---|
| before | 10.6–15.7 s, **27.4 GB** footprint | 27.7 s (51 s cold), 30.9 GB, `query` stage 13.8 s |
| after | **0.26–0.81 s, 0.04 GB** | **10.7 s, 3.6 GB**, `query` stage 9.95 s |
Tests: `cargo test --workspace` green; `scripts/smoke_test_index.sh` PASS; `bench_mphf.sh` (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a `.idx` map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect.
## 15. Unbuffered reads: `index.meta` and superkmer files (2026-10-15)
Follow-up of §14 (search for the same family of problems: file content handled inefficiently at open). Static search over `src/` for `fs::read`, `read_to_end`, `load_full`, `from_reader`, bare `File` readers and mmap `to_vec`: every persistent structure of `obicompactvec`, `obikindex` and the `obikphylo` annexes is already mmapped; the two real findings were *unbuffered* readers.
**1. `IndexMeta::read_full` (`obikindex/src/index/meta.rs`)** passed a bare `File` to `serde_json::from_reader`, which issues **one `read` syscall per byte**. `index.meta` of the 91-genome index is 20 KB and is read afresh by every `state()`/`genomes()` call (stateless design, §obikindex audit), so each call cost ≈ 0.35 s here (macOS). Profile of `obikmer dump --head 10` (`sample`): ≈ 20 000 samples in `read`, called from `Bytes<File>::next` ← `serde_json` ← `IndexMeta`. Fix: `fs::read` then `serde_json::from_slice`. **`dump --head 10`: 9.0–10.0 s (91–102 s of system CPU) → 0.16–0.84 s**; the same change applies to `SKFileMeta::read` (`obiskio/src/meta.rs`, per-superkmer-file sidecars).
**2. `SKFileReader::open` (`obiskio/src/reader.rs`)**: for an uncompressed file `niffler::get_reader` returns the sniffed bytes chained to the bare `File`, and `read_superkmer` reads each field with its own small `read_exact`. Fix: wrap the file in a 64 KiB `BufReader` before `niffler`. Effect on `index` (E. coli, 4 interleaved runs each): `count_kmer` 0.51–0.56 s → 0.42–0.50 s (≈ −12 %), total wall unchanged (≈ 1.95 s). Small, low risk, kept.
Tests: `cargo test --workspace` 549 passed / 0 failed; `scripts/smoke_test_index.sh` PASS; query output on the 91-genome index identical (md5 `e9e7c35b`).
**Probes of other commands on the 91-genome index (180 GB, 256 partitions)**: `phylo --distance jaccard` peaks at 155–190 MB of footprint while running and finishes in < 30 s; `utils --stats` 130–150 MB. `filter` (and by the same check `select`, `merge`) refused the index — "source index is not fully built" — with the unmodified baseline binary too: this index's `index.meta` predates the `state` field, so they could not be probed.
**Observations not investigated**: `index` on E. coli shows ≈ 9 s of *system* CPU in 2 s of wall time (both before and after these changes), consistent with the `swtch_pri` (yield) spinning seen in the `index` profile of §11 rather than with I/O; and wall times of the same query on the same binary varied 4.8 s ↔ 10.7 s ↔ 43 s depending on the state of the page cache, so only same-session comparisons are meaningful.
## 16. Profile of `index` on E. coli reads (2026-10-15)
`sample` (1 ms stacks) of `obikmer index` on the E. coli K-12 simulated reads (2 × 15x, 16 threads, one run; the `index` stage lasted 2.4 s under the sampler vs ≈ 0.9 s normally, so proportions are approximate). Stage walls: scatter 0.40 s, dereplicate 0.14, count_kmer 0.60, index 2.36.
**Threads are mostly idle.** Of ≈ 30 800 thread-samples: 24 900 blocked or yielding, 5 900 active. The ≈ 9 s of *system* CPU per run (§15) is `swtch_pri` (`sched_yield`) — 4 550 samples: ≈ 70 % rayon workers spinning for work (`WorkerThread` wait loop), ≈ 28 % `crossbeam` bounded-channel send/recv spinning, the rest in `obidebruinj` thread-locals. It is waiting, not I/O: no `read` samples remain. It costs no wall time on an otherwise idle machine, but burns CPU (visible in `user + sys` accounting and on a shared 192-core machine).
**Where the active CPU goes** (share of the 5 900 active samples):
| | share |
|---|---|
| xxh3 hashing of `CanonicalKmer` for the de Bruijn graph's `HashMap` (callers: `count_neighbors` 791, `hash_one` 358, `walk` 285, `unitig_iter` 252, `push` 164, `unitig_nucleotides` 75, rehash 68; plus `xxh3_stateful_update` 300) | ≈ 34 % |
| `debruijn::count_neighbors` itself (inclusive of its hashing ≈ 2 300 samples: `compute_degrees_and_mark_starts`) | 12.5 % own |
| `ptr_hash::build` (MPHF construction) | 4.8 % |
| crossbeam channel send | 4.8 % |
| `open` / `close` / `write` (many small files over 256 partitions) | ≈ 9 % |
| debruijn `unitig_iter` / `walk` own time | 8.3 % |
So at this scale the cost of `index` is the **de Bruijn graph's hash table** (several lookups of neighbouring k-mers per k-mer, each hashed with xxh3 and probably missing the cache), not the MPHF passes this document started from (`SlottedKmers` ≈ 2 %, §11). Not measured: what fraction of those lookups miss the cache, and the same profile on a prod-size partition.
## 17. Profile of `index` on a skim: *Vaccinium vitis-idaea*, 2.1 GB, `--partitions 16` (2026-10-15)
One run, `sample` windows of 5 s every ~17 s over the whole run (the sampler slows the process; walls below include it): scatter 19.7 s, dereplicate 8.0 s, count_kmer 25.2 s, **index 139.1 s**; total 192 s real, 1830 s user, **225 s system** (12 % of user, against ≈ 90 % on E. coli, §16), peak footprint 17.8 GB.
**Index stage** (samples taken fully inside it, 91 000 active vs 187 000 idle/yielding thread-samples):
| share of active CPU | |
|---|---|
| xxh3 hashing of kmers (de Bruijn `HashMap`) | 32.7 % |
| `debruijn::walk` | 12.4 % |
| `debruijn::count_neighbors` | 10.7 % |
| `GraphDeBruijn::push` | 6.9 % |
| `crossbeam` channel | 6.6 % |
| `hashbrown` table operations | 5.0 % |
| `debruijn::unitig_iter` | 4.8 % |
| memcpy/memset, malloc/free, file I/O, SipHash | ≈ 7 % |
| other | 13.7 % |
- **The de Bruijn graph is ≈ 72 % of the active CPU of `index`; the MPHF construction does not appear (< 1 %)**, and `SlottedKmers`/evidence filling is negligible. The conclusion of §16 holds at 100× the E. coli scale.
- **The stage is poorly parallel**: on average ≈ 5 of 16 threads are active (33 % of thread-samples); `swtch_pri` (spin/yield) is 7 % of all thread-samples. With 16 partitions of unequal size the tail partitions run alone. The 12 % system CPU is the same spinning as §16, at a smaller relative weight.
- **count_kmer** (one window): sorting 40 %, file I/O 18 %, malloc/free 9 %, memcpy 6 %. **scatter** (one window, 32 % active): `RollingStat::push` 32 %, `EntropyTracker::push` 23 % + `normalized_entropy` 5 %, super-kmer stream 10 %, gzip decode ≈ 9 %.
Not established: the cache-miss rate of the graph's table (stalls are attributed to whichever instruction waits, so "hashing 33 %" may include time waiting for the hasher's operands or surrounding loads — a counter-based profile is needed to separate compute from memory stalls), and whether a cheaper hash function for `u64` kmers or fewer lookups per kmer would change the picture.
## 18. De Bruijn graph: hasher, walk, and an orientation bug that fragmented unitigs (2026-10-15)
Benchmark harness: `obikindexer/examples/graph_bench.rs` (`graph_bench <unitigs.bin> <k> [repeat]`; env `IDEAL=1` counts maximal unitigs independently of `GraphDeBruijn`, `SYNTH=n` uses synthetic reads, `DUMP`/`IDEALDUMP` print unitigs). Reference input: the 32.6 M kmers of one partition of the *V. vitis-idaea* index (`-p 16`, k = 31). Phase timings, 16 threads: push / `compute_degrees_and_mark_starts` / `for_each_unitig`.
| | push | degrees | unitigs | unitigs produced | nucleotides |
|---|---|---|---|---|---|
| original | 4.1 s | 2.0 s | 3.9 s | 6.56 M (varies ±900 per run) | 229.4 M |
| `u64` hasher (below) | 1.0 s | 0.7 s | 3.1 s | 6.56 M | 229.4 M |
| + orientation fix | 1.0 s | 0.8 s | **1.0 s** | **3.81 M** (stable ±3) | **146.9 M** |
Independent count of maximal unitigs on the same kmers (nodes − mutually-unique links): 3 809 887. After the fix the algorithm gives 3 809 945 (+0.002 %, cycles).
**1. Hasher** (`debruijn/kmer_hash.rs`). The node table used xxh3's streaming `Hasher` (`Xxh3Builder`), for a key that is one `u64`: ≈ 35 % of the active CPU of `index` (§17). Replaced by a splitmix64 finaliser `BuildHasher` (one multiply–shift pair, no state). push 4.1 → 1.0 s, degrees 2.0 → 0.7 s. `xxhash-rust` is no longer used by `obidebruinj` (left in its `Cargo.toml`; removing it is a dependency decision).
**2. Walk**: `WalkState` now carries the node's own `&AtomicU8`, so marking the next node visited does not look the kmer up a second time (−2 % on the unitig phase: not where the time was).
**3. Unitig fragmentation — a bug, not a tuning.** Starts are detected only for the canonical orientation. A chain whose end node is stored reverse-complemented has no recognised start and falls into the sequential "cycles and tails" pass, where `find_chain_start` returned only the kmer, dropping the orientation, and `unitig_nucleotides` always restarted in the canonical orientation: from the wrong end it extends nowhere and emits a **lone-kmer unitig**, repeatedly, along the chain. The result depended on table iteration order (`degrees_linear_chain_extensions` passed with xxh3's order and failed with any other; counts varied run to run) and was never maximal: **72 % more unitigs than maximal** on the reference partition. A second slip in the reverse branch of `WalkState::walk` (the `can_extend_left`/`can_extend_right` test of the next node was swapped) cost another ~15 %. Fixes: `walk` condition corrected; `find_chain_start` returns `(kmer, direct)` with the orientation to walk forward; `unitig_nucleotides` and `UnitigNucIter` take that orientation (and read the first k nucleotides reverse-complemented when it is false); starts are now flagged in both orientations and the claim picks the right one (`start_direction`). Regression tests: `linear_chain_is_one_unitig_whatever_the_orientations` (fails on the original code, passes now) and `unitigs_partition_the_kmers_of_branching_reads` (every kmer in exactly one unitig).
Residual: on noisy synthetic reads (k = 11–31, 200–20 000 bases) the algorithm still gives up to 5 % more unitigs than the independent count, always a few short pieces next to chain ends; not explained, absent from real data (+0.002 %).
**End-to-end** (current binary vs previous one, same inputs):
| | before | after |
|---|---|---|
| *V. vitis-idaea* 2.1 GB, `--partitions 16`: total wall | 163.8 s | **116.5 s** (−29 %) |
| … `index` stage | 114.6 s | **59.2 s** (−48 %) |
| … user CPU | 1 870 s | 1 304 s |
| … `unitigs.bin` / `.idx` | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %); index 3.5 → 2.9 GB |
| E. coli, `index` stage / wall / user CPU | 0.91 s / 1.97 s / 10.2 s | 0.59 s / 1.63 s / 7.3 s |
Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E. coli `dump` and `query` outputs identical (md5), `scripts/smoke_test_index.sh` PASS, `cargo test --workspace` 551 passed / 0 failed, and `bench_mphf.sh` digests (index, merge, query ×4, filter, convert round trip, phylo) identical to the baseline run. Existing indexes are unaffected (any partition of kmers into unitigs is valid; evidence records are per-index); only newly built layers get the shorter, fewer unitigs.
## 19. Rank field: unitig chunks capped at 128 k-mers (2026-10-15)
`Evidence` packs `(chunk_id, rank)` into a `u32` with a 7-bit rank (`encode` masks `& 0x7F`), but `MAX_KMERS_PER_CHUNK` was 256 (the width of the unitig record's length byte), so a chunk of 129–256 k-mers had its ranks ≥ 128 silently truncated and those k-mers no longer verified. Observed on E. coli reads, same query: **`--partitions 256`** → max chunk 49–50 k-mers, results correct with the old and the new binary; **`--partitions 1`** → max chunk 256, query output wrong with both the old (md5 `bde0f886a0`) and the post-graph-fix (`8f1c94ef32`) binary versus the correct `f99e971087`. The orientation fix of §18 makes unitigs longer, hence makes this more likely at low partition counts.
**Change** (`obikseq::MAX_KMERS_PER_CHUNK`: 256 → 128; chosen so, rather than failing at write time: a longer unitig is simply written as consecutive chunks, nothing is lost). The constant is shared with the super-kmer chunking of the scatter files (`SuperKmer::write_to_binary`): super-kmers above 128 k-mers (rare) are now split into 128-k-mer records — readers are unchanged and old files stay readable. Tests that hard-coded 255/256/257/259/260 were made relative to the constant; new test `chunk_ranks_fit_the_7_bit_evidence_field`.
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
## 20. Summary of what was gained (2026-10-15)
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
### Speed
| Operation | Original | Now | Where |
|---|---|---|---|
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
### Memory (peak RSS / footprint)
| Operation | Original | Now |
|---|---|---|
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
### Index size (newly built indexes; existing ones are unchanged)
| | Original | Now |
|---|---|---|
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
### Correctness fixes found on the way
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
- `index.meta` and super-kmer files were parsed byte by byte (§15).
### What is **not** established
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
## 21. Dead code removed (2026-10-15)
The items listed as unused in §4 were removed after Serena's `safe_delete_symbol` and the compiler confirmed no reference: `MphfLayer::find_strict`; the single-kmer `hash` chain (`MphfLayer::hash`, `TypedLayer::hash`, `KmerLayer::hash`, `IndexCache::hash`); `IndexCache::find`; the uncompiled `obikmerge/src/merge_layer/` directory (689 lines; `lib.rs` only declares `merge` and `partition_merge`); and `xxhash-rust` from `obidebruinj/Cargo.toml` (it still serves `obikseq`, `obifastwrite` and `obiksession`).
Side effect: `find_strict` was the only reader of the `unitigs` handle that `Hybrid` layers opened, so that field (and the extra `.idx` map and mmap per Hybrid layer) was dropped too. `find` on a Hybrid layer was already a fingerprint check; the exact evidence stays on disk (it gives the slot count and is what `convert` works from) but no API consults it any more. Restoring an exact lookup for Hybrid layers means re-adding `find_strict` and that field.
`ahash` in `obidebruinj/Cargo.toml` is not referenced by any source file of the crate either (not removed: not asked).
Checks after the removal: `cargo build --release` without warnings, `cargo test --workspace` 555 passed / 0 failed, `scripts/smoke_test_index.sh` PASS.