Refactor k-mer storage to use flat arrays

This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
This commit is contained in:
Eric Coissac committed 2026-10-15 12:16:21 +02:00
1 parent fe38424682
commit 09c80468b2
8 files changed
+156 -56

No files matched your search

+21 -1
View File
@@ -24,7 +24,7 @@ for each chunk of sequences (parallel workers via obipipeline, one call to proce
decompose all sequences into superkmers (SuperKmerIter) — construction only, decompose all sequences into superkmers (SuperKmerIter) — construction only,
not the dedup key not the dedup key
deduplicate at k-mer granularity, split by partition in the same pass: deduplicate at k-mer granularity, split by partition in the same pass:
by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> ← KmerDesc = (seq_idx, pos) by_partition: Vec<PartitionKmers> ← sorted unique k-mers + flat KmerDesc = (seq_idx, pos) slices (2026-10-15, was a HashMap — see "Query memory" below)
allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers — allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
NOT multiplied by n_genomes NOT multiplied by n_genomes
allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome — allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
@@ -404,3 +404,23 @@ Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-w
- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone. - **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition. - **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action. - **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.
## Query memory: `PartitionKmers` replaces the per-partition `HashMap` (2026-10-15)
**Problem.** A full-size query (two ~450 MB fastq.gz of a skim against a 354 M-kmer index) reached **74–81 GB RSS** with the default 256 MiB chunk. Cause, read in `batch.rs`/`chunk.rs` and measured with the `by_partition memory retained` debug log: `QueryBatch::by_partition` was a `HashMap<CanonicalKmer, Vec<KmerDesc>>` per partition. On genomic reads almost every k-mer is unique (12.5 M unique for 13.5 M occurrences in a 48 MB chunk, a dedup of 8 %), so each k-mer paid a table slot, growth slack and its own heap `Vec`: **889 MB per 48 MB chunk, ~71 bytes per unique k-mer, 18× the chunk size** — the dominant term of the per-chunk memory, several chunks being in flight at once.
**Change.** `obikquery::PartitionKmers`: occurrences collected as `(kmer, KmerDesc)` per partition, sorted by `(kmer, seq_idx, pos)` and stored as three flat arrays (unique k-mers, u32 offsets, descs). `query_partition_with` takes `&PartitionKmers`; `QueryHit` already carried slices. Occurrence order per k-mer is unchanged (query order); no per-k-mer allocation.
**Measured** (same inputs, release build, identical sorted output digests before and after):
| | `by_partition` (48 MB chunk) | RSS, 2×48 MB chunks | RSS, full run (≈ 1 GB reads) | wall, full run |
|---|---|---|---|---|
| before | 889 MB | 5.6 GB | 73.7–80.7 GB | 135–172 s (one 312 s outlier) |
| after | 259 MB | 4.1 GB | **28.2 GB** | **103 s** |
With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the touched pages of the index itself, so after the change the remaining per-chunk anonymous memory is ≈ 6× the chunk size, below the formula's factor 12 (`BASE_OVERHEAD + 8·n_genomes`, still a conservative bound — left unchanged). The sort replaces hashing in `from_records` and was not slower (17.3 s vs 21.2 s on the small test).
**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here.
**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed.
+17 -11
View File
@@ -1,13 +1,12 @@
use std::collections::HashMap;
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use obiread::record::SeqRecord; use obiread::record::SeqRecord;
use obiskbuilder::SuperKmerIter; use obiskbuilder::SuperKmerIter;
use crate::partition_kmers::PartitionKmers;
use crate::query_layer::KmerDesc; use crate::query_layer::KmerDesc;
/// A batch of query sequences, with k-mers deduplicated directly (not just at /// A batch of query sequences, with k-mers grouped by canonical k-mer (not just
/// the superkmer level) and pre-split by partition. /// at the superkmer level) and pre-split by partition.
/// ///
/// Superkmer *construction* (`SuperKmerIter`) is still required — it's the /// Superkmer *construction* (`SuperKmerIter`) is still required — it's the
/// mechanism that computes minimizers and partition routing — but the dedup /// mechanism that computes minimizers and partition routing — but the dedup
@@ -23,8 +22,8 @@ pub(crate) struct QueryBatch {
pub(crate) seqs: Vec<Vec<u8>>, pub(crate) seqs: Vec<Vec<u8>>,
/// Total kmer count per sequence (used for `--detail` coverage allocation). /// Total kmer count per sequence (used for `--detail` coverage allocation).
pub(crate) n_kmers: Vec<u32>, pub(crate) n_kmers: Vec<u32>,
/// Deduplicated k-mer occurrences, one map per partition. /// K-mer occurrences grouped by k-mer, one entry per partition.
pub(crate) by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>>, pub(crate) by_partition: Vec<PartitionKmers>,
} }
impl QueryBatch { impl QueryBatch {
@@ -41,20 +40,20 @@ impl QueryBatch {
let mut seqs = Vec::with_capacity(records.len()); let mut seqs = Vec::with_capacity(records.len());
let mut n_kmers = Vec::with_capacity(records.len()); let mut n_kmers = Vec::with_capacity(records.len());
let mask = (n_partitions as u64) - 1; let mask = (n_partitions as u64) - 1;
let mut by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>> = let mut occurrences: Vec<Vec<(CanonicalKmer, KmerDesc)>> =
(0..n_partitions).map(|_| HashMap::new()).collect(); (0..n_partitions).map(|_| Vec::new()).collect();
for (seq_idx, record) in records.into_iter().enumerate() { for (seq_idx, record) in records.into_iter().enumerate() {
let mut kmer_offset = 0u32; let mut kmer_offset = 0u32;
for rsk in SuperKmerIter::new(&record.normalized, k, level_max, theta) { for rsk in SuperKmerIter::new(&record.normalized, k, level_max, theta) {
let part_idx = (rsk.minimizer().seq_hash() & mask) as usize; let part_idx = (rsk.minimizer().seq_hash() & mask) as usize;
let map = &mut by_partition[part_idx]; let part = &mut occurrences[part_idx];
for (j, kmer) in rsk.superkmer().iter_canonical_kmers().enumerate() { for (j, kmer) in rsk.superkmer().iter_canonical_kmers().enumerate() {
map.entry(kmer).or_default().push(KmerDesc { part.push((kmer, KmerDesc {
seq_idx: seq_idx as u32, seq_idx: seq_idx as u32,
pos: kmer_offset + j as u32, pos: kmer_offset + j as u32,
}); }));
} }
let n = (rsk.seql() - k + 1) as u32; let n = (rsk.seql() - k + 1) as u32;
kmer_offset += n; kmer_offset += n;
@@ -65,6 +64,13 @@ impl QueryBatch {
n_kmers.push(kmer_offset); n_kmers.push(kmer_offset);
} }
// One partition at a time: each 16-byte-per-occurrence buffer is freed
// as soon as its compact form exists.
let by_partition = occurrences
.into_iter()
.map(PartitionKmers::from_occurrences)
.collect();
Self { Self {
ids, ids,
seqs, seqs,
+6 -31
View File
@@ -3,7 +3,6 @@ use std::time::Instant;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikindex::GenomeInfo; use obikindex::GenomeInfo;
use obikrope::Rope; use obikrope::Rope;
use obikseq::CanonicalKmer;
use obiread::record::parse_chunk; use obiread::record::parse_chunk;
use tracing::debug; use tracing::debug;
@@ -62,37 +61,13 @@ pub fn process_chunk(
let batch = QueryBatch::from_records(records, k, 6, 0.7, n_partitions); let batch = QueryBatch::from_records(records, k, 6, 0.7, n_partitions);
let n_seqs = batch.ids.len(); let n_seqs = batch.ids.len();
// Estimate QueryBatch::by_partition's actual memory footprint: the // `by_partition` is alive for the entire process_chunk call (never
// k-mer-level dedup map — one HashMap<CanonicalKmer, Vec<KmerDesc>> per // drained, only iterated by reference): this is its footprint for the
// partition, sized by *unique* k-mers, not shrunk by dedup. On real // whole chunk lifetime. Measured by allocated capacity, not logical length.
// workloads with a low intra-chunk duplication rate this can dwarf every
// other per-chunk structure, including the sparse Findere ones logged
// further down. Measured by allocated capacity, not logical length, to
// reflect real memory pressure (HashMap/Vec growth slack) — `by_partition`
// is alive for the entire process_chunk call (never drained, only
// iterated by reference), so this is its footprint for the whole chunk
// lifetime, not a transient.
let hashmap_slot_bytes = (std::mem::size_of::<CanonicalKmer>()
+ std::mem::size_of::<Vec<crate::query_layer::KmerDesc>>()
+ 1) as u64; // +1 ≈ hashbrown control byte per slot
let by_partition_map_bytes: u64 = batch
.by_partition
.iter()
.map(|m| m.capacity() as u64 * hashmap_slot_bytes)
.sum();
let by_partition_desc_bytes: u64 = batch
.by_partition
.iter()
.flat_map(|m| m.values())
.map(|v| v.capacity() as u64 * std::mem::size_of::<crate::query_layer::KmerDesc>() as u64)
.sum();
let by_partition_bytes = by_partition_map_bytes + by_partition_desc_bytes;
debug!( debug!(
n_unique_kmers_total = batch.by_partition.iter().map(|m| m.len() as u64).sum::<u64>(), n_unique_kmers_total = batch.by_partition.iter().map(|p| p.len() as u64).sum::<u64>(),
by_partition_map_bytes, n_occurrences_total = batch.by_partition.iter().map(|p| p.n_occurrences() as u64).sum::<u64>(),
by_partition_desc_bytes, by_partition_bytes = batch.by_partition.iter().map(|p| p.allocated_bytes() as u64).sum::<u64>(),
by_partition_bytes,
chunk_bytes, chunk_bytes,
"by_partition memory retained" "by_partition memory retained"
); );
+2
View File
@@ -5,6 +5,7 @@
//! the data model, never the reverse), same pattern as `obikindexer` for the //! the data model, never the reverse), same pattern as `obikindexer` for the
//! build side. //! build side.
mod partition_kmers;
mod query_layer; mod query_layer;
mod batch; mod batch;
mod chunk; mod chunk;
@@ -12,5 +13,6 @@ mod findere;
mod smer_index; mod smer_index;
mod output; mod output;
pub use partition_kmers::PartitionKmers;
pub use query_layer::{KmerDesc, QueryHit, QueryPartition, QueryStats}; pub use query_layer::{KmerDesc, QueryHit, QueryPartition, QueryStats};
pub use chunk::process_chunk; pub use chunk::process_chunk;
+82
View File
@@ -0,0 +1,82 @@
//! The k-mers of one query chunk routed to one partition, grouped by k-mer.
//!
//! Replaces a `HashMap<CanonicalKmer, Vec<KmerDesc>>`: with genomic reads
//! almost every k-mer occurs once, so the map paid a table slot, a growth
//! slack and a separate heap `Vec` for each unique k-mer (~70 bytes measured)
//! for a dedup that removes under 10 % of the occurrences. Here the
//! occurrences are sorted by k-mer and stored as three flat arrays — one
//! 8-byte k-mer and one 4-byte offset per unique k-mer, one 8-byte
//! [`KmerDesc`] per occurrence — about 20 bytes per unique k-mer, no
//! per-k-mer allocation.
use obikseq::CanonicalKmer;
use crate::query_layer::KmerDesc;
/// Unique k-mers of one partition (sorted by raw value) and, for each, the
/// slice of its occurrences (sorted by `(seq_idx, pos)`, i.e. query order).
pub struct PartitionKmers {
kmers: Vec<CanonicalKmer>,
/// `descs[starts[i]..starts[i + 1]]` are the occurrences of `kmers[i]`.
starts: Vec<u32>,
descs: Vec<KmerDesc>,
}
impl PartitionKmers {
pub fn empty() -> Self {
Self { kmers: Vec::new(), starts: vec![0], descs: Vec::new() }
}
/// Group `occurrences` by k-mer. Consumes the (16 bytes per occurrence)
/// input; after the call only the compact arrays remain.
pub fn from_occurrences(mut occurrences: Vec<(CanonicalKmer, KmerDesc)>) -> Self {
occurrences.sort_unstable_by_key(|(kmer, d)| (kmer.raw(), d.seq_idx, d.pos));
let mut kmers = Vec::new();
let mut starts = Vec::new();
let mut descs = Vec::with_capacity(occurrences.len());
for (i, (kmer, desc)) in occurrences.iter().enumerate() {
if kmers.last() != Some(kmer) {
kmers.push(*kmer);
starts.push(offset(i));
}
descs.push(*desc);
}
starts.push(offset(descs.len()));
kmers.shrink_to_fit();
starts.shrink_to_fit();
Self { kmers, starts, descs }
}
/// Number of unique k-mers.
pub fn len(&self) -> usize {
self.kmers.len()
}
pub fn is_empty(&self) -> bool {
self.kmers.is_empty()
}
/// Total number of occurrences (before grouping).
pub fn n_occurrences(&self) -> usize {
self.descs.len()
}
/// `(kmer, its occurrences)` for every unique k-mer, in k-mer order.
pub fn iter(&self) -> impl Iterator<Item = (CanonicalKmer, &[KmerDesc])> + '_ {
self.kmers.iter().enumerate().map(|(i, &kmer)| {
(kmer, &self.descs[self.starts[i] as usize..self.starts[i + 1] as usize])
})
}
/// Heap bytes held (by allocated capacity).
pub fn allocated_bytes(&self) -> usize {
self.kmers.capacity() * std::mem::size_of::<CanonicalKmer>()
+ self.starts.capacity() * std::mem::size_of::<u32>()
+ self.descs.capacity() * std::mem::size_of::<KmerDesc>()
}
}
fn offset(i: usize) -> u32 {
u32::try_from(i).expect("more than 2^32 k-mer occurrences in one partition of one chunk — lower --chunk-size")
}
+7 -5
View File
@@ -3,6 +3,8 @@ use std::collections::HashMap;
use obikidxcache::index_cache::IndexCache; use obikidxcache::index_cache::IndexCache;
use obikseq::CanonicalKmer; use obikseq::CanonicalKmer;
use crate::partition_kmers::PartitionKmers;
// ── KmerDesc — one occurrence of a k-mer in the query batch ────────────────── // ── KmerDesc — one occurrence of a k-mer in the query batch ──────────────────
/// Describes one occurrence of a (deduplicated) k-mer in the query batch: /// Describes one occurrence of a (deduplicated) k-mer in the query batch:
@@ -100,7 +102,7 @@ pub trait QueryPartition {
fn query_partition_with<F>( fn query_partition_with<F>(
&self, &self,
part_idx: usize, part_idx: usize,
kmers: &HashMap<CanonicalKmer, Vec<KmerDesc>>, kmers: &PartitionKmers,
n_genomes: usize, n_genomes: usize,
on_event: F, on_event: F,
) -> QueryStats ) -> QueryStats
@@ -112,7 +114,7 @@ impl QueryPartition for IndexCache {
fn query_partition_with<F>( fn query_partition_with<F>(
&self, &self,
part_idx: usize, part_idx: usize,
kmers: &HashMap<CanonicalKmer, Vec<KmerDesc>>, kmers: &PartitionKmers,
n_genomes: usize, n_genomes: usize,
mut on_event: F, mut on_event: F,
) -> QueryStats ) -> QueryStats
@@ -131,7 +133,7 @@ impl QueryPartition for IndexCache {
}; };
// ── Stage 1: MPHF-only pass, bucket hits by (layer_idx, slot) ──────── // ── Stage 1: MPHF-only pass, bucket hits by (layer_idx, slot) ────────
let mut by_layer: Vec<HashMap<usize, &Vec<KmerDesc>>> = let mut by_layer: Vec<HashMap<usize, &[KmerDesc]>> =
(0..n_layer).map(|_| HashMap::new()).collect(); (0..n_layer).map(|_| HashMap::new()).collect();
// Layer-major over fixed-size chunks of the (unordered) kmer map: // Layer-major over fixed-size chunks of the (unordered) kmer map:
@@ -141,12 +143,12 @@ impl QueryPartition for IndexCache {
// before). Chunked, not collected — the map is never copied whole. // before). Chunked, not collected — the map is never copied whole.
const CHUNK: usize = 4096; const CHUNK: usize = 4096;
let mut entries = kmers.iter(); let mut entries = kmers.iter();
let mut pending: Vec<(CanonicalKmer, &Vec<KmerDesc>)> = Vec::with_capacity(CHUNK); let mut pending: Vec<(CanonicalKmer, &[KmerDesc])> = Vec::with_capacity(CHUNK);
let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(CHUNK); let mut batch: Vec<CanonicalKmer> = Vec::with_capacity(CHUNK);
let mut found: Vec<Option<usize>> = Vec::with_capacity(CHUNK); let mut found: Vec<Option<usize>> = Vec::with_capacity(CHUNK);
loop { loop {
pending.clear(); pending.clear();
pending.extend(entries.by_ref().take(CHUNK).map(|(k, d)| (*k, d))); pending.extend(entries.by_ref().take(CHUNK));
if pending.is_empty() { if pending.is_empty() {
break; break;
} }
+19 -5
View File
@@ -87,8 +87,8 @@ fn duplicated_sequence_broadcasts_to_both_seq_indices() {
let batch = batch_from_fasta(&fasta, K, 1); let batch = batch_from_fasta(&fasta, K, 1);
let shared = batch.by_partition[0] let shared = batch.by_partition[0]
.values() .iter()
.find(|descs| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1)); .find(|(_, descs)| descs.iter().any(|d| d.seq_idx == 0) && descs.iter().any(|d| d.seq_idx == 1));
assert!( assert!(
shared.is_some(), shared.is_some(),
@@ -119,9 +119,23 @@ fn partition_routing_is_a_pure_function_of_the_kmer() {
// No k-mer key appears in more than one partition's map. // No k-mer key appears in more than one partition's map.
let mut seen: std::collections::HashSet<CanonicalKmer> = std::collections::HashSet::new(); let mut seen: std::collections::HashSet<CanonicalKmer> = std::collections::HashSet::new();
for map in &batch.by_partition { for part in &batch.by_partition {
for kmer in map.keys() { for (kmer, _) in part.iter() {
assert!(seen.insert(*kmer), "k-mer routed to more than one partition"); assert!(seen.insert(kmer), "k-mer routed to more than one partition");
} }
} }
} }
#[test]
fn occurrences_of_a_kmer_come_out_in_query_order() {
// Grouping sorts occurrences; within a k-mer they must stay ordered by
// (seq_idx, pos) — the order the former insertion-ordered `Vec` had, which
// downstream output ordering relied on.
let fasta = format!(">r1\n{SEQ}\n>r2\n{SEQ}\n");
let batch = batch_from_fasta(&fasta, K, 1);
for (_, descs) in batch.by_partition[0].iter() {
assert!(descs.windows(2).all(|w| (w[0].seq_idx, w[0].pos) < (w[1].seq_idx, w[1].pos)));
}
let occ: usize = batch.by_partition[0].iter().map(|(_, d)| d.len()).sum();
assert_eq!(occ as u64, total_occurrences(&batch));
}
+2 -3
View File
@@ -64,12 +64,11 @@ fn query_partition_with_missing_index_dir_returns_default_stats() {
let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index");
let cache = IndexCache::new(Arc::new(index), Some(vec![0])); let cache = IndexCache::new(Arc::new(index), Some(vec![0]));
let mut kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new();
// Any well-formed canonical k-mer works here — the call must return // Any well-formed canonical k-mer works here — the call must return
// before ever attempting an MPHF lookup, since the partition has zero // before ever attempting an MPHF lookup, since the partition has zero
// layers. // layers.
let kmer = CanonicalKmer::from_raw_unchecked(0u64); let kmer = CanonicalKmer::from_raw_unchecked(0u64);
kmers.insert(kmer, vec![KmerDesc { seq_idx: 0, pos: 0 }]); let kmers = PartitionKmers::from_occurrences(vec![(kmer, KmerDesc { seq_idx: 0, pos: 0 })]);
let stats = cache.query_partition_with(0, &kmers, 1, |_event| { let stats = cache.query_partition_with(0, &kmers, 1, |_event| {
panic!("on_event must not be called: no index was built"); panic!("on_event must not be called: no index was built");
@@ -94,7 +93,7 @@ fn query_partition_with_empty_kmers_is_a_noop() {
let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index"); let index = KmerIndex::create(tmp.path().join("idx"), config, None).expect("create index");
let cache = IndexCache::new(Arc::new(index), Some(vec![0])); let cache = IndexCache::new(Arc::new(index), Some(vec![0]));
let kmers: HashMap<CanonicalKmer, Vec<KmerDesc>> = HashMap::new(); let kmers = PartitionKmers::empty();
let stats = cache.query_partition_with(0, &kmers, 1, |_event| { let stats = cache.query_partition_with(0, &kmers, 1, |_event| {
panic!("on_event must not be called on an empty kmer map"); panic!("on_event must not be called on an empty kmer map");
}); });