diff --git a/DevDocMD/architecture/query.md b/DevDocMD/architecture/query.md index 98a1ebe1..8395263d 100644 --- a/DevDocMD/architecture/query.md +++ b/DevDocMD/architecture/query.md @@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce decompose all sequences into superkmers (SuperKmerIter) — construction only, not the dedup key deduplicate at k-mer granularity, split by partition in the same pass: - by_partition: Vec>> ← KmerDesc = (seq_idx, pos) + by_partition: Vec ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below) allocate SmerIndex (SmerIndex::new): in_index: Vec, sized total_smers — NOT multiplied by n_genomes allocate by_genome: Vec>, one empty Vec per genome — @@ -404,3 +404,23 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w - **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone. - **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition. - **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action. + + +## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15) + +**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once. + +**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation. + +**Measured** (same inputs, release build, identical sorted output digests before and after): + +| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run | +|---|---|---|---|---| +| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) | +| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** | + +With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test). + +**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here. + +**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed. diff --git a/src/obikquery/src/batch.rs b/src/obikquery/src/batch.rs index 1d56e86f..87c81d59 100644 --- a/src/obikquery/src/batch.rs +++ b/src/obikquery/src/batch.rs @@ -1,13 +1,12 @@ -use std::collections::HashMap; - use obikseq::CanonicalKmer; use obiread::record::SeqRecord; use obiskbuilder::SuperKmerIter; +use crate::partition_kmers::PartitionKmers; use crate::query_layer::KmerDesc; -/// A batch of query sequences, with k-mers deduplicated directly (not just at -/// the superkmer level) and pre-split by partition. +/// A batch of query sequences, with k-mers grouped by canonical k-mer (not just +/// at the superkmer level) and pre-split by partition. /// /// Superkmer *construction* (`SuperKmerIter`) is still required — it's the /// mechanism that computes minimizers and partition routing — but the dedup @@ -23,8 +22,8 @@ pub(crate) struct QueryBatch { pub(crate) seqs: Vec>, /// Total kmer count per sequence (used for `--detail` coverage allocation). pub(crate) n_kmers: Vec, - /// Deduplicated k-mer occurrences, one map per partition. - pub(crate) by_partition: Vec>>, + /// K-mer occurrences grouped by k-mer, one entry per partition. + pub(crate) by_partition: Vec, } impl QueryBatch { @@ -41,20 +40,20 @@ impl QueryBatch { let mut seqs = Vec::with_capacity(records.len()); let mut n_kmers = Vec::with_capacity(records.len()); let mask = (n_partitions as u64) - 1; - let mut by_partition: Vec>> = - (0..n_partitions).map(|_| HashMap::new()).collect(); + let mut occurrences: Vec> = + (0..n_partitions).map(|_| Vec::new()).collect(); for (seq_idx, record) in records.into_iter().enumerate() { let mut kmer_offset = 0u32; for rsk in SuperKmerIter::new(&record.normalized, k, level_max, theta) { let part_idx = (rsk.minimizer().seq_hash() & mask) as usize; - let map = &mut by_partition[part_idx]; + let part = &mut occurrences[part_idx]; for (j, kmer) in rsk.superkmer().iter_canonical_kmers().enumerate() { - map.entry(kmer).or_default().push(KmerDesc { + part.push((kmer, KmerDesc { seq_idx: seq_idx as u32, pos: kmer_offset + j as u32, - }); + })); } let n = (rsk.seql() - k + 1) as u32; kmer_offset += n; @@ -65,6 +64,13 @@ impl QueryBatch { n_kmers.push(kmer_offset); } + // One partition at a time: each 16-byte-per-occurrence buffer is freed + // as soon as its compact form exists. + let by_partition = occurrences + .into_iter() + .map(PartitionKmers::from_occurrences) + .collect(); + Self { ids, seqs, diff --git a/src/obikquery/src/chunk.rs b/src/obikquery/src/chunk.rs index 4fc4383b..1de0014c 100644 --- a/src/obikquery/src/chunk.rs +++ b/src/obikquery/src/chunk.rs @@ -3,7 +3,6 @@ use std::time::Instant; use obikidxcache::index_cache::IndexCache; use obikindex::GenomeInfo; use obikrope::Rope; -use obikseq::CanonicalKmer; use obiread::record::parse_chunk; use tracing::debug; @@ -62,37 +61,13 @@ pub fn process_chunk( let batch = QueryBatch::from_records(records, k, 6, 0.7, n_partitions); let n_seqs = batch.ids.len(); - // Estimate QueryBatch::by_partition's actual memory footprint: the - // k-mer-level dedup map — one HashMap> per - // partition, sized by *unique* k-mers, not shrunk by dedup. On real - // workloads with a low intra-chunk duplication rate this can dwarf every - // other per-chunk structure, including the sparse Findere ones logged - // further down. Measured by allocated capacity, not logical length, to - // reflect real memory pressure (HashMap/Vec growth slack) — `by_partition` - // is alive for the entire process_chunk call (never drained, only - // iterated by reference), so this is its footprint for the whole chunk - // lifetime, not a transient. - let hashmap_slot_bytes = (std::mem::size_of::() - + std::mem::size_of::>() - + 1) as u64; // +1 ≈ hashbrown control byte per slot - let by_partition_map_bytes: u64 = batch - .by_partition - .iter() - .map(|m| m.capacity() as u64 * hashmap_slot_bytes) - .sum(); - let by_partition_desc_bytes: u64 = batch - .by_partition - .iter() - .flat_map(|m| m.values()) - .map(|v| v.capacity() as u64 * std::mem::size_of::() as u64) - .sum(); - let by_partition_bytes = by_partition_map_bytes + by_partition_desc_bytes; - + // `by_partition` is alive for the entire process_chunk call (never + // drained, only iterated by reference): this is its footprint for the + // whole chunk lifetime. Measured by allocated capacity, not logical length. debug!( - n_unique_kmers_total = batch.by_partition.iter().map(|m| m.len() as u64).sum::(), - by_partition_map_bytes, - by_partition_desc_bytes, - by_partition_bytes, + n_unique_kmers_total = batch.by_partition.iter().map(|p| p.len() as u64).sum::(), + n_occurrences_total = batch.by_partition.iter().map(|p| p.n_occurrences() as u64).sum::(), + by_partition_bytes = batch.by_partition.iter().map(|p| p.allocated_bytes() as u64).sum::(), chunk_bytes, "by_partition memory retained" ); diff --git a/src/obikquery/src/lib.rs b/src/obikquery/src/lib.rs index e8d0fb4c..00ad862c 100644 --- a/src/obikquery/src/lib.rs +++ b/src/obikquery/src/lib.rs @@ -5,6 +5,7 @@ //! the data model, never the reverse), same pattern as `obikindexer` for the //! build side. +mod partition_kmers; mod query_layer; mod batch; mod chunk; @@ -12,5 +13,6 @@ mod findere; mod smer_index; mod output; +pub use partition_kmers::PartitionKmers; pub use query_layer::{KmerDesc, QueryHit, QueryPartition, QueryStats}; pub use chunk::process_chunk; diff --git a/src/obikquery/src/partition_kmers.rs b/src/obikquery/src/partition_kmers.rs new file mode 100644 index 00000000..b09e4a12 --- /dev/null +++ b/src/obikquery/src/partition_kmers.rs @@ -0,0 +1,82 @@ +//! The k-mers of one query chunk routed to one partition, grouped by k-mer. +//! +//! Replaces a `HashMap>`: with genomic reads +//! almost every k-mer occurs once, so the map paid a table slot, a growth +//! slack and a separate heap `Vec` for each unique k-mer (~70 bytes measured) +//! for a dedup that removes under 10 % of the occurrences. Here the +//! occurrences are sorted by k-mer and stored as three flat arrays — one +//! 8-byte k-mer and one 4-byte offset per unique k-mer, one 8-byte +//! [`KmerDesc`] per occurrence — about 20 bytes per unique k-mer, no +//! per-k-mer allocation. + +use obikseq::CanonicalKmer; + +use crate::query_layer::KmerDesc; + +/// Unique k-mers of one partition (sorted by raw value) and, for each, the +/// slice of its occurrences (sorted by `(seq_idx, pos)`, i.e. query order). +pub struct PartitionKmers { + kmers: Vec, + /// `descs[starts[i]..starts[i + 1]]` are the occurrences of `kmers[i]`. + starts: Vec, + descs: Vec, +} + +impl PartitionKmers { + pub fn empty() -> Self { + Self { kmers: Vec::new(), starts: vec![0], descs: Vec::new() } + } + + /// Group `occurrences` by k-mer. Consumes the (16 bytes per occurrence) + /// input; after the call only the compact arrays remain. + pub fn from_occurrences(mut occurrences: Vec<(CanonicalKmer, KmerDesc)>) -> Self { + occurrences.sort_unstable_by_key(|(kmer, d)| (kmer.raw(), d.seq_idx, d.pos)); + + let mut kmers = Vec::new(); + let mut starts = Vec::new(); + let mut descs = Vec::with_capacity(occurrences.len()); + for (i, (kmer, desc)) in occurrences.iter().enumerate() { + if kmers.last() != Some(kmer) { + kmers.push(*kmer); + starts.push(offset(i)); + } + descs.push(*desc); + } + starts.push(offset(descs.len())); + kmers.shrink_to_fit(); + starts.shrink_to_fit(); + Self { kmers, starts, descs } + } + + /// Number of unique k-mers. + pub fn len(&self) -> usize { + self.kmers.len() + } + + pub fn is_empty(&self) -> bool { + self.kmers.is_empty() + } + + /// Total number of occurrences (before grouping). + pub fn n_occurrences(&self) -> usize { + self.descs.len() + } + + /// `(kmer, its occurrences)` for every unique k-mer, in k-mer order. + pub fn iter(&self) -> impl Iterator + '_ { + self.kmers.iter().enumerate().map(|(i, &kmer)| { + (kmer, &self.descs[self.starts[i] as usize..self.starts[i + 1] as usize]) + }) + } + + /// Heap bytes held (by allocated capacity). + pub fn allocated_bytes(&self) -> usize { + self.kmers.capacity() * std::mem::size_of::() + + self.starts.capacity() * std::mem::size_of::() + + self.descs.capacity() * std::mem::size_of::() + } +} + +fn offset(i: usize) -> u32 { + u32::try_from(i).expect("more than 2^32 k-mer occurrences in one partition of one chunk — lower --chunk-size") +} diff --git a/src/obikquery/src/query_layer.rs b/src/obikquery/src/query_layer.rs index 945a7b33..acd1fdca 100644 --- a/src/obikquery/src/query_layer.rs +++ b/src/obikquery/src/query_layer.rs @@ -3,6 +3,8 @@ use std::collections::HashMap; use obikidxcache::index_cache::IndexCache; use obikseq::CanonicalKmer; +use crate::partition_kmers::PartitionKmers; + // ── KmerDesc — one occurrence of a k-mer in the query batch ────────────────── /// Describes one occurrence of a (deduplicated) k-mer in the query batch: @@ -100,7 +102,7 @@ pub trait QueryPartition { fn query_partition_with( &self, part_idx: usize, - kmers: &HashMap>, + kmers: &PartitionKmers, n_genomes: usize, on_event: F, ) -> QueryStats @@ -112,7 +114,7 @@ impl QueryPartition for IndexCache { fn query_partition_with( &self, part_idx: usize, - kmers: &HashMap>, + kmers: &PartitionKmers, n_genomes: usize, mut on_event: F, ) -> QueryStats @@ -131,7 +133,7 @@ impl QueryPartition for IndexCache { }; // ── Stage 1: MPHF-only pass, bucket hits by (layer_idx, slot) ──────── - let mut by_layer: Vec>> = + let mut by_layer: Vec> = (0..n_layer).map(|_| HashMap::new()).collect(); // Layer-major over fixed-size chunks of the (unordered) kmer map: @@ -141,12 +143,12 @@ impl QueryPartition for IndexCache { // before). Chunked, not collected — the map is never copied whole. const CHUNK: usize = 4096; let mut entries = kmers.iter(); - let mut pending: Vec<(CanonicalKmer, &Vec)> = Vec::with_capacity(CHUNK); + let mut pending: Vec<(CanonicalKmer, &[KmerDesc])> = Vec::with_capacity(CHUNK); let mut batch: Vec = Vec::with_capacity(CHUNK); let mut found: Vec> = Vec::with_capacity(CHUNK); loop { pending.clear(); - pending.extend(entries.by_ref().take(CHUNK).map(|(k, d)| (*k, d))); + pending.extend(entries.by_ref().take(CHUNK)); if pending.is_empty() { break; } diff --git a/src/obikquery/src/tests/batch.rs b/src/obikquery/src/tests/batch.rs index 6a289bd3..84589f84 100644 --- a/src/obikquery/src/tests/batch.rs +++ b/src/obikquery/src/tests/batch.rs @@ -87,8 +87,8 @@ fn duplicated_sequence_broadcasts_to_both_seq_indices() { let batch = batch_from_fasta(&fasta, K, 1); let shared = batch.by_partition[0] - .values() - .find(|descs| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1)); + .iter() + .find(|(_, descs)| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1)); assert!( shared.is_some(), @@ -119,9 +119,23 @@ fn partition_routing_is_a_pure_function_of_the_kmer() { // No k-mer key appears in more than one partition's map. let mut seen: std::collections::HashSet = std::collections::HashSet::new(); - for map in &batch.by_partition { - for kmer in map.keys() { - assert!(seen.insert(*kmer), "k-mer routed to more than one partition"); + for part in &batch.by_partition { + for (kmer, _) in part.iter() { + assert!(seen.insert(kmer), "k-mer routed to more than one partition"); } } } + +#[test] +fn occurrences_of_a_kmer_come_out_in_query_order() { + // Grouping sorts occurrences; within a k-mer they must stay ordered by + // (seq_idx, pos) — the order the former insertion-ordered `Vec` had, which + // downstream output ordering relied on. + let fasta = format!(">r1\n{SEQ}\n>r2\n{SEQ}\n"); + let batch = batch_from_fasta(&fasta, K, 1); + for (_, descs) in batch.by_partition[0].iter() { + assert!(descs.windows(2).all(|w| (w[0].seq_idx, w[0].pos) < (w[1].seq_idx, w[1].pos))); + } + let occ: usize = batch.by_partition[0].iter().map(|(_, d)| d.len()).sum(); + assert_eq!(occ as u64, total_occurrences(&batch)); +} diff --git a/src/obikquery/src/tests/query_layer.rs b/src/obikquery/src/tests/query_layer.rs index 55ba4517..f8fc7101 100644 --- a/src/obikquery/src/tests/query_layer.rs +++ b/src/obikquery/src/tests/query_layer.rs @@ -64,12 +64,11 @@ fn query_partition_with_missing_index_dir_returns_default_stats() { let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let cache = IndexCache::new(Arc::new(index), Some(vec![0])); - let mut kmers: HashMap> = HashMap::new(); // Any well-formed canonical k-mer works here — the call must return // before ever attempting an MPHF lookup, since the partition has zero // layers. let kmer = CanonicalKmer::from_raw_unchecked(0u64); - kmers.insert(kmer, vec![KmerDesc { seq_idx: 0, pos: 0 }]); + let kmers = PartitionKmers::from_occurrences(vec![(kmer, KmerDesc { seq_idx: 0, pos: 0 })]); let stats = cache.query_partition_with(0, &kmers, 1, |_event| { panic!("on_event must not be called: no index was built"); @@ -94,7 +93,7 @@ fn query_partition_with_empty_kmers_is_a_noop() { let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let cache = IndexCache::new(Arc::new(index), Some(vec![0])); - let kmers: HashMap> = HashMap::new(); + let kmers = PartitionKmers::empty(); let stats = cache.query_partition_with(0, &kmers, 1, |_event| { panic!("on_event must not be called on an empty kmer map"); });