diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml index 9951ae90..bdbdf696 100644 --- a/.gitea/workflows/release.yml +++ b/.gitea/workflows/release.yml @@ -62,7 +62,8 @@ jobs: ~/.cargo/registry ~/.cargo/git src/target - key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock') }} + tools/migrate-mphf/target + key: linux-musl-cargo-${{ hashFiles('src/Cargo.lock', 'tools/migrate-mphf/Cargo.lock') }} restore-keys: linux-musl-cargo- - name: Build static binary @@ -83,6 +84,28 @@ jobs: -H "Authorization: token $GITEA_TOKEN" \ -F "attachment=@/tmp/dist/obikmer-linux-x86_64" + # One-shot converter for indexes built with ptr_hash 1.x. A standalone + # crate (own Cargo.lock/target, outside src/'s workspace), built after + # obikmer's own asset is already uploaded: it never blocks that one. + - name: Build migrate-mphf (static) + working-directory: tools/migrate-mphf + env: + PKG_CONFIG_ALLOW_CROSS: "1" + run: cargo zigbuild --release --target x86_64-unknown-linux-musl + + - name: Upload migrate-mphf artifact + working-directory: tools/migrate-mphf + env: + GITEA_TOKEN: ${{ secrets.GITEATOKEN }} + RELEASE_ID: ${{ needs.create-release.outputs.release_id }} + run: | + cp target/x86_64-unknown-linux-musl/release/migrate-mphf /tmp/dist/migrate-mphf-linux-x86_64 + strip /tmp/dist/migrate-mphf-linux-x86_64 + curl -s -X POST \ + "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/releases/$RELEASE_ID/assets" \ + -H "Authorization: token $GITEA_TOKEN" \ + -F "attachment=@/tmp/dist/migrate-mphf-linux-x86_64" + build-macos-arm64: needs: create-release runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index f0bf8a7c..2230b394 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ CLAUDE.md memory/ sandbox/ src/target +tools/migrate-mphf/target data-stress *.fasta *.fasta.gz diff --git a/DevDocMD/architecture/hot_resume_memoization.md b/DevDocMD/architecture/hot_resume_memoization.md new file mode 100644 index 00000000..d737c1a9 --- /dev/null +++ b/DevDocMD/architecture/hot_resume_memoization.md @@ -0,0 +1,61 @@ +# Hot resume and memoization of long computations (discussion, 2026-10-14) + +## Goal (user) + +1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero. +2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result. + +Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect. + +## Why it matters: one real run + +`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions: + +| Stage | Wall | +|---|---| +| `sibling_stats` (full scan, `Selection::All`) | 8 235 s | +| `family_overlap` (sweep 1: overlap annex) | 5 582 s | +| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s | +| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s | +| `sampled_scan`: the draw itself | ≈ 9 573 s | +| `distance` (Jaccard, not asked for; older binary) | 53 s | +| Total | 34 085 s (9 h 28) | + +The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning. + +## What exists today (verified in the code, 2026-10-14) + +**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`). + +**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only. + +**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently. + +Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.) + +**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering. + +**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file. + +**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value. + +**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective. + +## Proposed direction (not implemented; to be validated) + +Start from the goal, not from the existing crate. + +1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today. +2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer. +3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers). +4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design. +5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable. +6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently. + +## Open questions + +- Fingerprint of an index version: which cheap, stable identity? +- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers? +- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user). +- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache. +- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing. diff --git a/DevDocMD/architecture/query.md b/DevDocMD/architecture/query.md index 98a1ebe1..b7b73ff6 100644 --- a/DevDocMD/architecture/query.md +++ b/DevDocMD/architecture/query.md @@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce decompose all sequences into superkmers (SuperKmerIter) — construction only, not the dedup key deduplicate at k-mer granularity, split by partition in the same pass: - by_partition: Vec>> ← KmerDesc = (seq_idx, pos) + by_partition: Vec ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below) allocate SmerIndex (SmerIndex::new): in_index: Vec, sized total_smers — NOT multiplied by n_genomes allocate by_genome: Vec>, one empty Vec per genome — @@ -404,3 +404,45 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w - **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone. - **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition. - **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action. + + +## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15) + +**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once. + +**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation. + +**Measured** (same inputs, release build, identical sorted output digests before and after): + +| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run | +|---|---|---|---|---| +| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) | +| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** | + +With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test). + +**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here. + +**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed. + +### Chunk size sweep on a small input (2026-10-15) + +Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size: + +| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) | +|---|---|---|---|---| +| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 | +| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 | +| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 | +| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 | +| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 | +| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 | + +Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up. + + +### Automatic chunk size now follows the input size (2026-10-15) + +`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins. + +Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory). diff --git a/DevDocMD/implementation/mphf_batch_queries.md b/DevDocMD/implementation/mphf_batch_queries.md new file mode 100644 index 00000000..f058913b --- /dev/null +++ b/DevDocMD/implementation/mphf_batch_queries.md @@ -0,0 +1,369 @@ +# MPHF batch queries — site inventory and expected gains + +Investigation of 2026-10-14 (`ptr_hash` 2.1.2, `epserde` 0.13). Question: where would batched MPHF queries (software prefetch of the random accesses) save time? Status of each claim is marked **[read]** (verified in source), **[computed]** (derived from stated sizes) or **[estimate]** (reasoning, not measured). Measurements, once made, are in the last section. + +## 1. What `ptr_hash` 2.1.2 actually offers [read] + +`Mphf = PtrHash, SINGLE_PART = false, REMAP = true>`. + +- **`V` is the pilot array** (one byte per bucket), **`F` is only the remap storage**. A layer's `mphf.bin` (~2.4 bits/key **[computed]**: 40 MB for ~130 M keys) is therefore almost entirely a byte array indexed by bucket. The batch APIs prefetch `pilots[bucket]` — the right line. The remap (read only when `slot >= n`, a small fraction of keys) is **not** prefetched. +- `index_batch(xs: [Q; K]) -> [usize; K]` — stable, not feature-gated; hash K keys, prefetch K pilots, query. Remainder (`len % K`) is the caller's job. +- `index_stream(keys: IntoIterator) -> impl Iterator` — ring buffer of B keys; handles head and tail itself; accepts any iterator. **Its `next()` is `unimplemented!`**: only fold-based consumption works (`for_each`, `fold`, `sum`). `collect()`, `zip`, `for x in it` panic. Consume with `for_each` into a pre-sized `Vec`. +- `index_batch_exact` needs the nightly `unstable` feature; `index_batch_exact2` is `#[doc(hidden)]`. Neither is usable. +- Prefetch is `prefetch-index 0.2.1`: `_mm_prefetch(T0)` on x86_64, asm on aarch64, **no-op elsewhere**. +- **`SINGLE_PART`**: neither `index_stream` nor `index_batch` branches on it — both always run the multi-part `bucket()`/`slot()`. Only `index()` has the faster single-part path. So with batched queries the `SINGLE_PART = true` advantage disappears, and `false` (our choice) costs nothing while keeping parallel construction (`true` also panics with sharding). +- Old `ptr_hash` 1.1.0 (used by `tools/migrate-mphf`) has the same family with an extra `MINIMAL` const: `index_stream`, `index_batch`. No new dependency needed there. + +```rust +// consuming index_stream: fold-based only +let mut out = Vec::with_capacity(kmers.len()); +mphf.index_stream::<32, _>(kmers.iter().map(|k| k.raw())) + .for_each(|slot| out.push(slot)); +``` + +## 2. Existing "batch" APIs that are not batched [read] + +`MphfLayer::hash_batch` and `MphfOnly::index_batch` loop `.index()` one kmer at a time. Consumers: `obikfilter/partition_iter.rs` (`iter_layer_kmers`), `obikmerge/partition_merge.rs` (pass 2: `hash_batch`, `index_batch`), `obikphylo/siblings/algorithms/family_scan.rs` (two sites). + +## 3. Cost chain of a membership query (`MphfLayer::find`) [read] + +1. `pilots[bucket]` — the only access `ptr_hash` prefetches. +2. Exact/Hybrid-exact: `evidence.decode(slot)` — 4 B/slot, ~520 MB per 130 M-slot layer **[computed]**, random mmap read. +3. `verify_canonical_kmer` → `block_offsets[...]` (O(1) with the default `--block-size 1`; a forward scan of up to `2^block_bits` records otherwise). +4. `extract_kmer_raw` — random read in the `unitigs.bin` mmap. + +Approx/Hybrid-fast: pilots then `fingerprint.get(slot)` — one extra random access. +Each stage is independent across kmers, so a staged batch (slots → prefetch evidence → decode, prefetch offsets/unitigs → verify) can overlap all of them. `ptr_hash` only helps with stage 1; stages 2–4 need prefetch support in `obikindex`/`obiskio` (not existing). Prefetch on an unmapped (cold, not paged-in) mmap page is dropped by the CPU, so none of this helps on a cold index. + +Layer sizes: `layer_0` MPHF 2–3 MB (cache-resident), `layer_1` ~40 MB (beyond one CCD's L3 on EPYC-class machines — **hypothesis**, prod CPU not verified, and contended by 192 threads). Gains are expected mainly on `layer_1`. For branch-free independent loops (`hash_batch`) out-of-order execution already overlaps several misses, so the explicit prefetch gains less **[estimate]** than for branchy loops (`find` with early `break`). + +## 4. Call-site inventory + +(a) already batched; (b) batch buildable; (c) hard/pointless. + +| Site | What | Class | +|---|---|---| +| `mphf_layer.rs` `hash_batch`, `MphfOnly::index_batch` | loop of `.index()` | (a) | +| `obikfilter/partition_iter.rs:106` | `hash_batch`, 32768 kmers | (a) | +| `obikmerge/partition_merge.rs:489, 521` | `hash_batch`, `index_batch`, 4096 | (a) | +| `obikphylo/.../family_scan.rs:285, 397` | `hash_batch` | (a) | +| `obikquery/query_layer.rs` stage 1 | `find_in_layer` per kmer, layer by layer, `break` on hit | (b) | +| `partition_merge.rs` `find_batch_grouped` | `find_slot` per kmer over dst layers | (b) | +| `partition_merge.rs` (`build_graph` filter) | `all(find_slot.is_none())`, every layer, per kmer; `build_graph` takes `Fn(CanonicalKmer)->bool` | (b), API change | +| `family_scan.rs` slow path; `annex.rs` | `find_slot` / `find_in_partition` per variant | (b) | +| `mphf_layer.rs` `build`, `build_exact_evidence`, `build_approx_evidence` | `mphf.index` then random writes (evidence, fingerprint, `seen`) | (b), chunked | +| `obikindexer/counter/count.rs:47` | `mphf.index` then `counts.get/set` | (b), hard (needs (kmer, count) pairs) | +| `obikindexer/extensions/private_builder.rs:177` | abundance filter | (b), conditional | +| `private_builder.rs:196`, `verify.rs`, `IndexCache::find_in_*` | per-kmer by nature / sampled | (c) | +| `tools/migrate-mphf/plan.rs:41-42` | `old.index` + `new.index` per kmer, random writes | (b) | + +Dead or suspect code found along the way (candidates for removal, not removed): + +- `MphfLayer::find_strict` — no caller. +- the single-kmer `hash` chain (`MphfLayer::hash` → `TypedLayer::hash` → `KmerLayer::hash` → `IndexCache::hash`) — no caller outside the chain. +- `IndexCache::find` — no caller. +- `obikmerge/src/merge_layer/` — not declared in `lib.rs` (`mod merge; mod partition_merge;` only); still imports the `LayeredMap` removed in 2026-08. To be confirmed by the compiler. + +## 5. Interactions + +- **Verification chain**: the right shape is one `find_batch(&[CanonicalKmer], &mut Vec>)` on `MphfLayer`, staged as in §3, so every consumer benefits (rule: enrich the shared API, no local loops). +- **Memory**: callers already hold bounded batches (4096 / 32768). A buffer-reusing variant avoids one `Vec` per call. `index_stream` itself only allocates a B-element ring on the stack. +- **Parallelism**: no new parallelism. Batches already come out of `obipipeline` stages; the change lives in the transform body. Construction pass 2 is sequential inside a partition: chunk the streamed iterator (1–4 k kmers), never collect the layer. In `query_partition_with` the unique-kmer `HashMap` is already materialised; iterate it chunk-wise. + +## 6. Benchmark + +`benchmark/bench_mphf.sh LABEL` times, on the 20-genome benchmark data, `index` (2 genomes), `merge` (20 presence indexes), `query` (presence/count × 2 genomes), `filter`, `convert` (approx/exact round trip) and `phylo` (`--sibling-annex`, `snp-jc`), `REPS` times each, and records order-independent output digests that must not change across labels. See the results section below. + +**Limit of this data set**: 20 small genomes → per-partition `layer_1` MPHFs of ~100 KB, fully cache-resident on the dev machine (Apple M3 Max, aarch64). This benchmark can detect regressions and the effect on instruction-level overlap; it **cannot** measure the DRAM-latency gain that motivates prefetch at 40 MB/layer. A faithful test needs a prod-scale layer (see the throwaway benchmark below). + +### Throwaway prod-scale benchmark (not written) + +Open one real partition's `layer_1` (~130 M keys) and `layer_0` (~8 M) with `MphfOnly`/`MphfLayer`; key sets: present (sample of `iter_kmers`, random order and sequential order) and absent (random, checked absent); measure kmers/s for `index`, `index_stream::` (B ∈ 8…128), `index_batch::` (K ∈ 8…64), and the full `find` vs staged `find_batch`; at 1, 8, 32, 192 threads; pinned to one NUMA node, then spread; warm and cold page cache stated; 5 repetitions, min/median; sizes of `mphf.bin`, `evidence.bin`, `unitigs.bin(.idx)` reported; first check `index_stream` slots == `index` slots. The ~8–12 ns/key figure of the PtrHash paper (quoted in `mphf.md`) is a paper claim, not tied to our sizes or machines. + +## 7. Not verified + +No measurement at prod scale; prod CPU architecture and cache sizes; evidence mode and `--block-size` used in prod; whether `migrate-mphf` already ran; share of total index/merge/query time spent in these loops; `count.rs` f0 per partition at prod scale; whether `merge_layer/` is dead (only the head of `lib.rs` read). + +## 8. First improvement and measurements (2026-10-15) + +**Implemented** (query stage 1, the top-ranked site; the same change adds the plumbing the other sites will reuse): + +- `MphfLayer` now holds a `MphfOnly` instead of its own `MemCase` (one mmap-open path, one place for the stream call). +- `MphfOnly::for_each_slot(kmers, f)` wraps `index_stream::<32>`; `index_batch`/`hash_batch` now really use it (they previously looped `.index()`). Callback-shaped because `index_stream`'s `next()` is `unimplemented!`. +- `MphfLayer::find_batch` / `TypedLayer::find_slot_batch` / `KmerLayer::find_slot_batch`; the mode-dependent half of `find` is factored into `MphfLayer::confirm`, shared by both. +- `query_partition_with` stage 1 is layer-major over chunks of 4096 unique kmers: each layer receives the still-unresolved kmers as one batch (first hit wins, `n_mphf_calls` unchanged). +- Not done: staged prefetch of evidence/offsets/unitigs (needs a prefetch primitive; `prefetch-index` is already in the tree via `ptr_hash`, but adding it to `obikindex` is a dependency decision), and the other (b) sites. + +**Measurements** (`benchmark/bench_mphf.sh`, `benchmark/summarize_bench.py`; Apple M3 Max, aarch64; 20 genomes at 15x). Interleaved A/B, 8 runs each, E. coli reads queried against the 20-genome presence index: + +| | wall (s) | user CPU (s) | +|---|---|---| +| before | 5.94–6.02 (median 5.96) | 10.95–11.07 | +| after | 5.63–6.00 (median 5.67) | 10.33–10.66 | + +About −5 % wall and −5.5 % CPU, ranges non-overlapping **[measured]**. All output digests (query ×4, merge, filter, convert round trip, phylo) are identical before and after. + +**Reading the number.** Small by construction: this data set's `layer_1` MPHFs are ~100 KB per partition, cache-resident, and query time is dominated by reading and k-merising the reads. The gain seen here is instruction-level overlap, not DRAM-latency hiding. The latter, the reason for the whole exercise, remains **unmeasured** and needs a prod-scale layer (§6). + +**Noise caveat.** With a load average of 16–22 from unrelated processes, single-run walls of `index`, `merge` and `filter` moved ±10–40 % between identical binaries; only interleaved repeated A/B runs were trustworthy. The `index`/`merge`/`filter`/`convert`/`phylo` rows of the first comparison are noise, not effects (`index` pass 2 was not changed at all; a later interleaved `filter` A/B was inconclusive, 13.0 s vs 13.8 s medians with a 10–20 s spread). + +**Stale benchmark indexes.** The indexes already under `benchmark/run/` (`specimen_index_*`, `global_index_*`, built 2026-08-28) carry `ptr_hash` 1.x MPHFs and fail with "Major version mismatch" on current binaries (`tools/migrate-mphf` converts them). `bench_mphf.sh` therefore builds its own copies under `run/bench_mphf/data/` on first use. + +## 9. Benchmark procedure + +Prerequisites: the simulated reads (`benchmark/run/simulated_data/`) and query reads (`benchmark/run/query_data/`) produced by `gmake simulate simulate_query`; macOS BSD `time -l` and `md5 -q` (the script uses both), Python 3. + +1. Build the binary to measure and **copy it** under a name that identifies it (cargo overwrites `target/release/obikmer`): `cp src/target/release/obikmer benchmark/run/bench_mphf/obikmer_