diff --git a/DevDocMD/architecture/query.md b/DevDocMD/architecture/query.md index 09372669..b7b73ff6 100644 --- a/DevDocMD/architecture/query.md +++ b/DevDocMD/architecture/query.md @@ -438,4 +438,11 @@ Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer s | 32 | 4 | 8.3 | 22.9–23.0 | 4.0 | | 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 | -Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs; not implemented. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up. +Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up. + + +### Automatic chunk size now follows the input size (2026-10-15) + +`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins. + +Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory). diff --git a/DevDocMD/implementation/mphf_batch_queries.md b/DevDocMD/implementation/mphf_batch_queries.md index d5ba755b..c5311d51 100644 --- a/DevDocMD/implementation/mphf_batch_queries.md +++ b/DevDocMD/implementation/mphf_batch_queries.md @@ -309,3 +309,49 @@ Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E **After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed. **Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top). + +## 20. Summary of what was gained (2026-10-15) + +Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated. + +### Speed + +| Operation | Original | Now | Where | +|---|---|---|---| +| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B | +| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B | +| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 | +| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 | +| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 | +| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 | +| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 | +| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up | + +### Memory (peak RSS / footprint) + +| Operation | Original | Now | +|---|---|---| +| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) | +| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** | +| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap` per partition: 71 → ~20 bytes per unique kmer) | +| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB | + +### Index size (newly built indexes; existing ones are unchanged) + +| | Original | Now | +|---|---|---| +| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) | +| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) | +| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) | + +### Correctness fixes found on the way + +- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18). +- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index). +- `index.meta` and super-kmer files were parsed byte by byte (§15). + +### What is **not** established + +- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine. +- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone). +- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured). diff --git a/src/obikmer/src/cmd/query/mod.rs b/src/obikmer/src/cmd/query/mod.rs index e9526b5b..26f5c811 100644 --- a/src/obikmer/src/cmd/query/mod.rs +++ b/src/obikmer/src/cmd/query/mod.rs @@ -192,17 +192,40 @@ pub fn run(args: QueryArgs) { let overhead_multiplier = BASE_OVERHEAD + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * detail_factor; + // Parallelism: a chunk is one worker's unit of work, so an input smaller + // than `n_workers` chunks leaves cores idle (two 48 MB files at the + // memory-derived 256 MiB cap = 2 chunks = 2 busy cores; measured 11.6 s + // against 2.5 s with 8 MiB chunks, see `DevDocMD/architecture/query.md`). + // When the input size can be estimated, aim for `CHUNKS_PER_WORKER` chunks + // per worker and never go below `MIN_CHUNK`. Stdin and URLs have no + // known size: the memory-derived size stands. + const CHUNKS_PER_WORKER: u64 = 2; + const MIN_CHUNK: u64 = 4 * 1024 * 1024; + let input_bytes: Option = args + .inputs + .iter() + .map(|p| obiread::estimated_decompressed_len(p)) + .sum::>(); + let chunk_bytes = args .chunk_size .map(|mb| mb * 1024 * 1024) .unwrap_or_else(|| { let avail = available_memory_bytes(); let computed = avail / (n_workers as u64 * overhead_multiplier * SAFETY_FACTOR); - computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024) as usize + let by_memory = computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024); + match input_bytes { + Some(total) => { + let by_parallelism = (total / (n_workers as u64 * CHUNKS_PER_WORKER)).max(MIN_CHUNK); + by_memory.min(by_parallelism) as usize + } + None => by_memory as usize, + } }); debug!( chunk_bytes, + input_bytes, n_genomes, detail = args.detail, overhead_multiplier, diff --git a/src/obiread/src/lib.rs b/src/obiread/src/lib.rs index 4f4fce70..ba8334be 100644 --- a/src/obiread/src/lib.rs +++ b/src/obiread/src/lib.rs @@ -25,7 +25,7 @@ pub use normalize::{normalize_fasta_chunk, normalize_fastq_chunk, normalize_sequ pub use nucstream::{NucPage, NucPageCursor, open_nuc_stream}; pub use path_iterator::{PathIter, path_iter}; pub use peakreader::PeekReader; -pub use xopen::xopen; +pub use xopen::{estimated_decompressed_len, xopen}; /// Default read block size: 1 MiB. pub const DEFAULT_BLOCK_SIZE: usize = 1024 * 1024; diff --git a/src/obiread/src/tests/xopen.rs b/src/obiread/src/tests/xopen.rs new file mode 100644 index 00000000..4d715b2e --- /dev/null +++ b/src/obiread/src/tests/xopen.rs @@ -0,0 +1,38 @@ +use super::*; +use std::io::Write; + +fn temp_file(name: &str, content: &[u8]) -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!("obiread_xopen_{}_{name}", std::process::id())); + std::fs::File::create(&path).unwrap().write_all(content).unwrap(); + path +} + +#[test] +fn estimate_plain_file_is_its_length() { + let p = temp_file("plain", b">s\nACGT\n"); + assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(8)); + std::fs::remove_file(p).unwrap(); +} + +#[test] +fn estimate_scales_compressed_files_by_magic_not_extension() { + // Only the magic bytes matter to the estimate (the content is never decoded). + let mut gz = vec![0x1f, 0x8b, 0x08, 0, 0, 0]; + gz.resize(100, 0); + let p = temp_file("looks_plain.txt", &gz); + assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(300)); + std::fs::remove_file(p).unwrap(); + + let mut zst = vec![0x28, 0xb5, 0x2f, 0xfd, 0, 0]; + zst.resize(50, 0); + let p = temp_file("zst", &zst); + assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(200)); + std::fs::remove_file(p).unwrap(); +} + +#[test] +fn estimate_is_unknown_for_streams_and_missing_files() { + assert_eq!(estimated_decompressed_len("-"), None); + assert_eq!(estimated_decompressed_len("https://example.org/x.fa.gz"), None); + assert_eq!(estimated_decompressed_len("/nonexistent/obiread/file"), None); +} diff --git a/src/obiread/src/xopen.rs b/src/obiread/src/xopen.rs index 78da48f2..df1e768f 100644 --- a/src/obiread/src/xopen.rs +++ b/src/obiread/src/xopen.rs @@ -46,6 +46,37 @@ pub fn xopen(source: &str) -> io::Result>> Ok(MimeTypeGuesser::new(open_raw(source)?)) } +/// Rough size, in bytes, of what [`xopen`] would yield for a local file: +/// its length, multiplied by a typical expansion factor when the magic bytes +/// say it is compressed (gzip ≈ 3×, as measured on FASTQ; bzip2, xz and zstd +/// taken as 4×). Meant for sizing work (chunk sizes), not for allocation — +/// the real ratio varies by a factor of two or more with the content. +/// +/// `None` when the size cannot be known without reading the stream: stdin +/// (`-`), an HTTP(S) URL, or a file that cannot be opened. +pub fn estimated_decompressed_len(source: &str) -> Option { + if source == "-" || source.starts_with("http://") || source.starts_with("https://") { + return None; + } + let expanded = expand_tilde(source); + let mut file = File::open(expanded.as_ref()).ok()?; + let len = file.metadata().ok()?.len(); + let mut magic = [0u8; 6]; + let n = file.read(&mut magic).ok()?; + let magic = &magic[..n]; + let factor = if magic.starts_with(&[0x1f, 0x8b]) { + 3 + } else if magic.starts_with(b"BZh") + || magic.starts_with(&[0xfd, b'7', b'z', b'X', b'Z', 0x00]) + || magic.starts_with(&[0x28, 0xb5, 0x2f, 0xfd]) + { + 4 + } else { + 1 + }; + Some(len * factor) +} + // ── internal helpers ────────────────────────────────────────────────────────── fn http_reader(url: &str) -> io::Result> { @@ -71,3 +102,7 @@ fn expand_tilde(path: &str) -> std::borrow::Cow<'_, str> { } std::borrow::Cow::Borrowed(path) } + +#[cfg(test)] +#[path = "tests/xopen.rs"] +mod tests;