Refactor query system for performance and memory reduction

This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
This commit is contained in:
Eric Coissac committed 2026-10-15 12:21:38 +02:00
1 parent 025c40c0db
commit f9edefd135
6 files changed
+152 -3

No files matched your search

+8 -1
View File
@@ -438,4 +438,11 @@ Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer s
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs; not implemented. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
### Automatic chunk size now follows the input size (2026-10-15)
`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins.
Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory).
@@ -309,3 +309,49 @@ Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
## 20. Summary of what was gained (2026-10-15)
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
### Speed
| Operation | Original | Now | Where |
|---|---|---|---|
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
### Memory (peak RSS / footprint)
| Operation | Original | Now |
|---|---|---|
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
### Index size (newly built indexes; existing ones are unchanged)
| | Original | Now |
|---|---|---|
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
### Correctness fixes found on the way
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
- `index.meta` and super-kmer files were parsed byte by byte (§15).
### What is **not** established
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
+24 -1
View File
@@ -192,17 +192,40 @@ pub fn run(args: QueryArgs) {
let overhead_multiplier =
BASE_OVERHEAD + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * detail_factor;
// Parallelism: a chunk is one worker's unit of work, so an input smaller
// than `n_workers` chunks leaves cores idle (two 48 MB files at the
// memory-derived 256 MiB cap = 2 chunks = 2 busy cores; measured 11.6 s
// against 2.5 s with 8 MiB chunks, see `DevDocMD/architecture/query.md`).
// When the input size can be estimated, aim for `CHUNKS_PER_WORKER` chunks
// per worker and never go below `MIN_CHUNK`. Stdin and URLs have no
// known size: the memory-derived size stands.
const CHUNKS_PER_WORKER: u64 = 2;
const MIN_CHUNK: u64 = 4 * 1024 * 1024;
let input_bytes: Option<u64> = args
.inputs
.iter()
.map(|p| obiread::estimated_decompressed_len(p))
.sum::<Option<u64>>();
let chunk_bytes = args
.chunk_size
.map(|mb| mb * 1024 * 1024)
.unwrap_or_else(|| {
let avail = available_memory_bytes();
let computed = avail / (n_workers as u64 * overhead_multiplier * SAFETY_FACTOR);
computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024) as usize
let by_memory = computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024);
match input_bytes {
Some(total) => {
let by_parallelism = (total / (n_workers as u64 * CHUNKS_PER_WORKER)).max(MIN_CHUNK);
by_memory.min(by_parallelism) as usize
}
None => by_memory as usize,
}
});
debug!(
chunk_bytes,
input_bytes,
n_genomes,
detail = args.detail,
overhead_multiplier,
+1 -1
View File
@@ -25,7 +25,7 @@ pub use normalize::{normalize_fasta_chunk, normalize_fastq_chunk, normalize_sequ
pub use nucstream::{NucPage, NucPageCursor, open_nuc_stream};
pub use path_iterator::{PathIter, path_iter};
pub use peakreader::PeekReader;
pub use xopen::xopen;
pub use xopen::{estimated_decompressed_len, xopen};
/// Default read block size: 1 MiB.
pub const DEFAULT_BLOCK_SIZE: usize = 1024 * 1024;
+38
View File
@@ -0,0 +1,38 @@
use super::*;
use std::io::Write;
fn temp_file(name: &str, content: &[u8]) -> std::path::PathBuf {
let path = std::env::temp_dir().join(format!("obiread_xopen_{}_{name}", std::process::id()));
std::fs::File::create(&path).unwrap().write_all(content).unwrap();
path
}
#[test]
fn estimate_plain_file_is_its_length() {
let p = temp_file("plain", b">s\nACGT\n");
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(8));
std::fs::remove_file(p).unwrap();
}
#[test]
fn estimate_scales_compressed_files_by_magic_not_extension() {
// Only the magic bytes matter to the estimate (the content is never decoded).
let mut gz = vec![0x1f, 0x8b, 0x08, 0, 0, 0];
gz.resize(100, 0);
let p = temp_file("looks_plain.txt", &gz);
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(300));
std::fs::remove_file(p).unwrap();
let mut zst = vec![0x28, 0xb5, 0x2f, 0xfd, 0, 0];
zst.resize(50, 0);
let p = temp_file("zst", &zst);
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(200));
std::fs::remove_file(p).unwrap();
}
#[test]
fn estimate_is_unknown_for_streams_and_missing_files() {
assert_eq!(estimated_decompressed_len("-"), None);
assert_eq!(estimated_decompressed_len("https://example.org/x.fa.gz"), None);
assert_eq!(estimated_decompressed_len("/nonexistent/obiread/file"), None);
}
+35
View File
@@ -46,6 +46,37 @@ pub fn xopen(source: &str) -> io::Result<MimeTypeGuesser<Box<dyn Read + Send>>>
Ok(MimeTypeGuesser::new(open_raw(source)?))
}
/// Rough size, in bytes, of what [`xopen`] would yield for a local file:
/// its length, multiplied by a typical expansion factor when the magic bytes
/// say it is compressed (gzip ≈ 3×, as measured on FASTQ; bzip2, xz and zstd
/// taken as 4×). Meant for sizing work (chunk sizes), not for allocation —
/// the real ratio varies by a factor of two or more with the content.
///
/// `None` when the size cannot be known without reading the stream: stdin
/// (`-`), an HTTP(S) URL, or a file that cannot be opened.
pub fn estimated_decompressed_len(source: &str) -> Option<u64> {
if source == "-" || source.starts_with("http://") || source.starts_with("https://") {
return None;
}
let expanded = expand_tilde(source);
let mut file = File::open(expanded.as_ref()).ok()?;
let len = file.metadata().ok()?.len();
let mut magic = [0u8; 6];
let n = file.read(&mut magic).ok()?;
let magic = &magic[..n];
let factor = if magic.starts_with(&[0x1f, 0x8b]) {
3
} else if magic.starts_with(b"BZh")
|| magic.starts_with(&[0xfd, b'7', b'z', b'X', b'Z', 0x00])
|| magic.starts_with(&[0x28, 0xb5, 0x2f, 0xfd])
{
4
} else {
1
};
Some(len * factor)
}
// ── internal helpers ──────────────────────────────────────────────────────────
fn http_reader(url: &str) -> io::Result<Box<dyn Read + Send>> {
@@ -71,3 +102,7 @@ fn expand_tilde(path: &str) -> std::borrow::Cow<'_, str> {
}
std::borrow::Cow::Borrowed(path)
}
#[cfg(test)]
#[path = "tests/xopen.rs"]
mod tests;