From 093d81c3596b8565be4102e3147841d6602cf2ab Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Thu, 15 Oct 2026 09:28:41 +0200 Subject: [PATCH] Introduce memory mapping for index files and sparse matrices This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs. --- DevDocMD/architecture/query.md | 15 ++++ DevDocMD/implementation/mphf_batch_queries.md | 33 +++++++ src/obicompactvec/src/bitmatrix/sparse.rs | 23 +++-- src/obiskio/src/unitig_index/reader.rs | 87 +++++++++++-------- 4 files changed, 116 insertions(+), 42 deletions(-) diff --git a/DevDocMD/architecture/query.md b/DevDocMD/architecture/query.md index 8395263d..09372669 100644 --- a/DevDocMD/architecture/query.md +++ b/DevDocMD/architecture/query.md @@ -424,3 +424,18 @@ With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the to **Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here. **Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed. + +### Chunk size sweep on a small input (2026-10-15) + +Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size: + +| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) | +|---|---|---|---|---| +| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 | +| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 | +| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 | +| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 | +| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 | +| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 | + +Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs; not implemented. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up. diff --git a/DevDocMD/implementation/mphf_batch_queries.md b/DevDocMD/implementation/mphf_batch_queries.md index 48eb7335..4901f10c 100644 --- a/DevDocMD/implementation/mphf_batch_queries.md +++ b/DevDocMD/implementation/mphf_batch_queries.md @@ -177,3 +177,36 @@ Measured with macOS `sample` (1 ms stack sampling) on the 20-genome data, one ru | 4 | 86–90 M | 25.9–27.0 MB | 171 s | 19.5 GB | Total ≈ 354 M distinct kmers for one skim. `-p 4` is already in the 30–45 MB range of the prod `layer_1` MPHFs; `-p 2` would give ≈ 53 MB. Indexes kept in `benchmark/run/bench_mphf/skim_p16` and `skim_p4` (2.3 GB each). Note the index stage itself is dominated by graph and MPHF construction, as in §11. + +## 13. Query on a real-scale index (2026-10-15) + +**Index.** Working index `phyloskims_sal_vac_sav` (91 genomes of Salix/Vaccinium, 256 partitions × 2 layers, exact evidence, 180 GB, ptr_hash 1.x MPHFs). Converted with `tools/migrate-mphf` **on an APFS clone** (`cp -c`, 0.8 s, no space used until written; the tool works in place and sets the annexes aside, so the original was left untouched — verified by mtime): 512 layers in 10 min (1311 % CPU), ≈ 57 GB of new data. Clone at `/Users/coissac/travail/obiskim/data/phyloalps/bench_mphf/phyloskims_sal_vac_sav_v2`; its sibling annex is in `stale_annexes_pre_ptrhash2/` and would need rebuilding to be used. Per partition: `layer_1` ≈ 31 M keys / 9.3 MB MPHF, `layer_0` ≈ 5.6 M keys / 1.7 MB — larger than the 20-genome set (100 KB) but 4× smaller than the production layers (~130 M keys, ~40 MB). + +**Query** (2 × 48 MB of *V. oxycoccos* reads, `--chunk-size 8`, AC power, 3 interleaved runs per binary, identical sorted output for all three): + +| binary | wall (s) | user CPU (s) | `query` stage (s) | +|---|---|---|---| +| original | 28.2–30.5 | 70.9–74.4 | 14.6–17.1 | +| + batched query stage 1 | 26.9–28.0 | 48.1–48.6 | 13.6–14.4 | +| + `PartitionKmers` memory fix, build-side `SlottedKmers` | 26.4–26.7 | 41.9–42.5 | 12.9–13.3 | + +The `query` stage is ≈ 50 % of wall time; the rest (≈ 13 s) is opening the index (next paragraph). Stage-only gain ≈ −6 % then −11 %; CPU time −33 % then −41 %. **The CPU drop is much larger than the wall drop** and is not explained by MPHF prefetch alone (§11: restructuring, probably the per-kmer `Arc` clones of `get_layer`, not isolated); the original binary was spending CPU in contention that wall time hid. + +**Index-open cost (not related to the changes above).** A query of a single 64-base sequence on this index takes **15 s and a 27 GB peak footprint**. `vmmap` during the open: 24.5 GB of `MALLOC_LARGE` in 1 481 regions (≈ 3 per layer, 7.3 GB already swapped out), i.e. heap allocations made by `IndexCache::new` for every layer of every partition whatever the query. Known contributors: `UnitigFileReader` reads each `.idx` into a `Vec` (≈ 30 MB for `layer_1`, ≈ 9 GB over 512 layers); the remaining ~15 GB are in the presence-matrix open path (not yet attributed). Opening only the partitions a query touches is not an option for a read set that spans all of them, but these structures could be mmapped instead of copied. + +## 14. Index-open memory: `.idx` offsets and `dict_values` mapped, not copied (2026-10-15) + +**Cause** (§13, now attributed): of the 27 GB opened by a one-sequence query on the 91-genome index, **18.2 GB** were `unitigs.bin.idx` read into a `Vec` **twice** per exact-evidence layer (`MphfLayer::open` builds two `UnitigFileReader`s of the same file — one for the evidence check, one for `iter_kmers`/`find_strict`; both go through `open` → `open_direct_access` → `read_idx` → `fs::read` + copy) and **7.7 GB** were `dict_values.bin` read with `fs::read` in `PersistentSparseBitMatrix::open`. Computed from the real file sizes: 2 × 9.10 + 7.71 = 25.9 GB, against 27.4 GB measured. Everything else the matrix opens (`PersistentRankSelectBitVec`, `PersistentFixedIntVec`, `EliasFano`) was already mmapped. + +**Change.** +- `obiskio::unitig_index::reader`: `block_offsets: Vec` → `BlockOffsets`, a view into the memory-mapped `.idx` (header parsed once, offsets read little-endian on access, no unsafe, no alignment requirement). Truncation is checked at open, so later accesses cannot run past the map. Both readers of a layer now share the same page-cache pages. +- `obicompactvec::bitmatrix::sparse`: `dict_values: Vec` → `Option` (`None` for an empty file, which cannot be mapped), read through `dict_bytes()`. + +**Measured** (same session, same inputs; sorted query outputs identical, md5 `e9e7c35b`): + +| | open only (1-sequence query) | 2×48 MB reads, `--chunk-size 8` | +|---|---|---| +| before | 10.6–15.7 s, **27.4 GB** footprint | 27.7 s (51 s cold), 30.9 GB, `query` stage 13.8 s | +| after | **0.26–0.81 s, 0.04 GB** | **10.7 s, 3.6 GB**, `query` stage 9.95 s | + +Tests: `cargo test --workspace` green; `scripts/smoke_test_index.sh` PASS; `bench_mphf.sh` (index, merge, query ×4, filter, convert round trip, phylo) completes with all output digests identical to the baseline run, i.e. no failure from files mapped while rewritten (the risk of mapping instead of copying: a reader holding a `.idx` map while the same path is truncated and rewritten would fault). The per-step ratios of that run vs the earlier baseline mix this change with the machine moving from battery to AC power and must not be read as its effect. diff --git a/src/obicompactvec/src/bitmatrix/sparse.rs b/src/obicompactvec/src/bitmatrix/sparse.rs index 2ce59ca7..9513e784 100644 --- a/src/obicompactvec/src/bitmatrix/sparse.rs +++ b/src/obicompactvec/src/bitmatrix/sparse.rs @@ -25,6 +25,7 @@ use std::fs; use std::io; use std::path::{Path, PathBuf}; +use memmap2::Mmap; use ndarray::{Array1, Array2}; use crate::traits::{BitPartials, ColumnWeights}; @@ -151,7 +152,9 @@ pub struct PersistentSparseBitMatrix { singleton: PersistentFixedIntVec, multi: PersistentFixedIntVec, dict_offsets: EliasFano, - dict_values: Vec, + /// `dict_values.bin`, mapped (never copied to the heap); `None` when the + /// file is empty — no multi-genome set — since an empty file cannot be mapped. + dict_values: Option, /// Row count (k-mer slots). n: usize, /// Column count (genomes). @@ -167,7 +170,12 @@ impl PersistentSparseBitMatrix { let singleton = PersistentFixedIntVec::open(&singleton_path(dir))?; let multi = PersistentFixedIntVec::open(&multi_path(dir))?; let dict_offsets = EliasFano::open(&dict_offsets_base(dir))?; - let dict_values = fs::read(dict_values_path(dir))?; + let dict_file = fs::File::open(dict_values_path(dir))?; + let dict_values = if dict_file.metadata()?.len() == 0 { + None + } else { + Some(unsafe { Mmap::map(&dict_file)? }) + }; Ok(Self { is_multi, singleton, multi, dict_offsets, dict_values, n: meta.n, n_cols: meta.n_cols, n_distinct_multi: meta.n_distinct_multi, @@ -179,6 +187,11 @@ impl PersistentSparseBitMatrix { #[inline] pub fn n_cols(&self) -> usize { self.n_cols } + #[inline] + fn dict_bytes(&self) -> &[u8] { + self.dict_values.as_deref().unwrap_or(&[]) + } + /// Byte range of `dict_id`'s varint-encoded genome list within /// `dict_values`. #[inline] @@ -187,7 +200,7 @@ impl PersistentSparseBitMatrix { let end = if dict_id + 1 < self.n_distinct_multi { self.dict_offsets.get(dict_id + 1) as usize } else { - self.dict_values.len() + self.dict_bytes().len() }; (start, end) } @@ -232,7 +245,7 @@ impl PersistentSparseBitMatrix { let (start, end) = self.dict_entry_range(dict_id); let mut p = start; while p < end { - f(read_varint(&self.dict_values, &mut p) as usize); + f(read_varint(self.dict_bytes(), &mut p) as usize); } } else { let pos = self.is_multi.rank0(slot) as usize; @@ -287,7 +300,7 @@ impl PersistentSparseBitMatrix { present.clear(); let mut p = start; while p < end { - present.push(read_varint(&self.dict_values, &mut p) as usize); + present.push(read_varint(self.dict_bytes(), &mut p) as usize); } for &g in &present { counts[g] += w; diff --git a/src/obiskio/src/unitig_index/reader.rs b/src/obiskio/src/unitig_index/reader.rs index d8dd461d..8e608cef 100644 --- a/src/obiskio/src/unitig_index/reader.rs +++ b/src/obiskio/src/unitig_index/reader.rs @@ -24,7 +24,7 @@ use super::{DEFAULT_BLOCK_BITS, MAGIC, idx_path}; /// correct but slower. pub struct UnitigFileReader { mmap: Mmap, - block_offsets: Vec, + block_offsets: BlockOffsets, n_unitigs: usize, n_kmers: usize, k: usize, @@ -64,7 +64,7 @@ impl UnitigFileReader { Ok(Self { mmap, - block_offsets: Vec::new(), + block_offsets: BlockOffsets::none(), n_unitigs, n_kmers, k, @@ -108,7 +108,7 @@ impl UnitigFileReader { } #[inline] pub fn has_direct_access(&self) -> bool { - !self.block_offsets.is_empty() + !self.block_offsets.is_none() } /// Byte offset of record `i` in the mmap. @@ -119,11 +119,11 @@ impl UnitigFileReader { fn chunk_start(&self, i: usize) -> usize { if self.has_direct_access() { if self.block_bits == 0 { - return self.block_offsets[i] as usize; + return self.block_offsets.get(i) as usize; } let block = i >> self.block_bits; let rem = i & self.mask; - let mut offset = self.block_offsets[block] as usize; + let mut offset = self.block_offsets.get(block) as usize; for _ in 0..rem { let seql_minus_k = self.mmap[offset] as usize; offset += 1 + (seql_minus_k + self.k + 3) / 4; @@ -292,25 +292,54 @@ impl UnitigFileReader { } } -fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, Vec)> { - let data = std::fs::read(path).map_err(SKError::Io)?; - let mut pos = 0; +/// Block offsets of a `.idx` file, read in place from its memory map: nothing +/// is copied to the heap, and every reader of the same `.idx` shares the same +/// page-cache pages (a copy was ~30 MB per large layer per reader). +struct BlockOffsets { + /// The whole `.idx` file; `None` for a sequential reader (no `.idx`). + mmap: Option, +} - let magic_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { - context: "unitig index: magic", - })?; +impl BlockOffsets { + fn none() -> Self { + Self { mmap: None } + } + + fn is_none(&self) -> bool { + self.mmap.is_none() + } + + /// The `i`-th offset (little-endian `u32` after the fixed header). + /// `read_idx` has checked that the file holds every offset the header + /// announces, so `i` within `0..=n_blocks` cannot run past the map. + #[inline] + fn get(&self, i: usize) -> u32 { + let at = IDX_HEADER_SIZE + 4 * i; + let map = self.mmap.as_ref().expect("direct-access reader has its .idx map"); + u32::from_le_bytes(map[at..at + 4].try_into().unwrap()) + } +} + +/// magic (4) + block_bits (4) + n_unitigs (4) + n_kmers (8). +const IDX_HEADER_SIZE: usize = 20; + +fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, BlockOffsets)> { + let file = File::open(path).map_err(SKError::Io)?; + let len = file.metadata().map_err(SKError::Io)?.len() as usize; + if len < IDX_HEADER_SIZE { + return Err(SKError::Truncated { context: "unitig index: header" }); + } + let data = unsafe { Mmap::map(&file).map_err(SKError::Io)? }; + + let magic_bytes = &data[0..4]; if magic_bytes != &MAGIC { return Err(SKError::BadMagic { expected: "UIX3", got: magic_bytes.try_into().unwrap(), }); } - pos += 4; - let bb_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { - context: "unitig index: block_bits", - })?; - let block_bits_u32 = u32::from_le_bytes(bb_bytes.try_into().unwrap()); + let block_bits_u32 = u32::from_le_bytes(data[4..8].try_into().unwrap()); if block_bits_u32 > 31 { return Err(SKError::InvalidData { context: "unitig index", @@ -318,31 +347,15 @@ fn read_idx(path: &Path) -> SKResult<(usize, usize, u8, Vec)> { }); } let block_bits = block_bits_u32 as u8; - pos += 4; - - let n_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { - context: "unitig index: n_unitigs", - })?; - let n_unitigs = u32::from_le_bytes(n_bytes.try_into().unwrap()) as usize; - pos += 4; - - let nk_bytes = data.get(pos..pos + 8).ok_or(SKError::Truncated { - context: "unitig index: n_kmers", - })?; - let n_kmers = u64::from_le_bytes(nk_bytes.try_into().unwrap()) as usize; - pos += 8; + let n_unitigs = u32::from_le_bytes(data[8..12].try_into().unwrap()) as usize; + let n_kmers = u64::from_le_bytes(data[12..20].try_into().unwrap()) as usize; let block_size = 1usize << block_bits; let n_blocks = (n_unitigs + block_size - 1) >> block_bits; let n_offsets = n_blocks + 1; - let mut block_offsets = Vec::with_capacity(n_offsets); - for _ in 0..n_offsets { - let off_bytes = data.get(pos..pos + 4).ok_or(SKError::Truncated { - context: "unitig index: block_offsets", - })?; - block_offsets.push(u32::from_le_bytes(off_bytes.try_into().unwrap())); - pos += 4; + if len < IDX_HEADER_SIZE + 4 * n_offsets { + return Err(SKError::Truncated { context: "unitig index: block_offsets" }); } - Ok((n_unitigs, n_kmers, block_bits, block_offsets)) + Ok((n_unitigs, n_kmers, block_bits, BlockOffsets { mmap: Some(data) })) }