Push wpvskkyozppn #78
No files matched your search
@@ -438,4 +438,11 @@ Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer s
|
||||
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
|
||||
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
|
||||
|
||||
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs; not implemented. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
|
||||
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs — **implemented, see below**. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
|
||||
|
||||
|
||||
### Automatic chunk size now follows the input size (2026-10-15)
|
||||
|
||||
`cmd/query/mod.rs`: when `--chunk-size` is not given, the memory-derived size (unchanged: `available RAM / (workers × overhead × 2)`, clamped to 4–256 MiB) is further capped by `estimated input bytes / (workers × 2)`, never below 4 MiB. The estimate is `obiread::estimated_decompressed_len`: the file length, ×3 when the magic bytes say gzip (≈ 2.8 measured on FASTQ), ×4 for bzip2/xz/zstd — a deliberately rough figure, detected from the magic bytes, not the extension. For stdin, URLs or unreadable files the size is unknown and the memory-derived size stands. An explicit `--chunk-size` still wins.
|
||||
|
||||
Measured (2 × 48 MB reads, 16 cores, AC power, outputs identical): single-genome index, 354 M kmers — default chunk 256 MiB → 4 MiB: **11.4 s → 2.4 s** (−79 %), CPU 22.3 → 26 s (+17 %), peak footprint 1.44 → 0.75 GB. On the 91-genome index the memory-derived size was already 4 MiB (overhead factor 4 + 8 × 91), so nothing changes there (5.1–5.7 s either way). Not measured: a large input (≈ 1 GB gz, whose estimate gives ≈ 95 MiB chunks instead of 256 MiB: more, smaller chunks, expected lower memory).
|
||||
@@ -309,3 +309,49 @@ Same k-mers: per-layer kmer counts identical (521 630 625 over the 16 layers), E
|
||||
**After** (E. coli, same query): `--partitions` 256 / 16 / 4 / 1 → max chunk 50 / 70 / 118 / 128 k-mers, query md5 `f99e971087` in all four cases. `cargo test --workspace`: 551 passed, 0 failed.
|
||||
|
||||
**Existing indexes.** An index built with 256 partitions has chunks far below the limit (91-genome index, first 4 partitions: max 42–56, none above 128) and is unaffected. The `--partitions 4` skim index used for the benchmarks above (built before the cap) holds 6 chunks over 128 k-mers in its 4 partitions (max 135): a handful of k-mers out of ≈ 354 M verify wrongly. `scripts/check_unitig_chunks.py "<index>/partitions/part_*/index/layer_*/unitigs.bin"` reports chunks over 128 (it reads every unitigs.bin sequentially; k is hard-coded to 31 at its top).
|
||||
|
||||
## 20. Summary of what was gained (2026-10-15)
|
||||
|
||||
Apple M3 Max (aarch64, 16 cores, 128 GB), on AC power; absolute times are not those of the 192-core production machine, ratios measured in one session are what is reported. "Original" = the binary at the start of this work; "now" = current tree. Every output digest checked (query, dump, merge, filter, convert, phylo) is identical before and after, except where stated.
|
||||
|
||||
### Speed
|
||||
|
||||
| Operation | Original | Now | Where |
|
||||
|---|---|---|---|
|
||||
| Open an index (query of one 64-base sequence), 91 genomes, 180 GB | 10.4–13.3 s | **0.18–0.19 s** | §14, same-session A/B |
|
||||
| Query, 2 × 48 MB reads on that index (`--chunk-size 8`) | 30.7–32.0 s (61.8 s cold), 70–76 s CPU | **11.6–13.2 s**, 36–39 s CPU (−62 % wall, −50 % CPU) | §14, same-session A/B |
|
||||
| `dump --head 10` on that index | 9.0–10.0 s (91–102 s system CPU) | **0.16–0.84 s** | §15 |
|
||||
| `index` of a 2.1 GB skim (*V. vitis-idaea*, `--partitions 16`) | 163.8 s (stage `index` 114.6 s) | **116.5 s** (stage `index` 59.2 s): −29 % wall, −48 % on the stage, −30 % CPU | §18 |
|
||||
| `index` of E. coli reads | 1.97 s (stage 0.91 s), 10.2 s CPU | 1.63 s (stage 0.59 s), 7.3 s CPU | §18 |
|
||||
| De Bruijn graph, 32.6 M kmers (push / degrees / unitigs) | 4.1 s / 2.0 s / 3.9 s | 1.0 s / 0.8 s / 1.0 s (×3.5 overall) | §18 |
|
||||
| `index` stage `count_kmer` (E. coli) | 0.51–0.56 s | 0.42–0.50 s | §15 |
|
||||
| Query on a small input (2 × 48 MB reads), single-genome index | 11.4 s (2 chunks = 2 busy cores) | **2.4 s** (−79 %) — automatic chunk size now follows the input size (`architecture/query.md`); on the 91-genome index the size was already 4 MiB | §20 follow-up |
|
||||
|
||||
### Memory (peak RSS / footprint)
|
||||
|
||||
| Operation | Original | Now |
|
||||
|---|---|---|
|
||||
| Opening the 91-genome index (any command that builds an `IndexCache`) | **27.4 GB** | **0.04 GB** (§14: `.idx` offsets and `dict_values.bin` mapped instead of copied; they were read twice and once) |
|
||||
| Query, 2 × 48 MB reads on that index | 32.2–32.5 GB | **3.5–3.6 GB** |
|
||||
| Full query, ≈ 1 GB gz of reads, 354 M-kmer index, default chunk size | **73.7–80.7 GB** | **28.2 GB** (§12 of `architecture/query.md`: `PartitionKmers` instead of a `HashMap<kmer, Vec>` per partition: 71 → ~20 bytes per unique kmer) |
|
||||
| `index` of the 2.1 GB skim | 19.7 GB | 18.3 GB |
|
||||
|
||||
### Index size (newly built indexes; existing ones are unchanged)
|
||||
|
||||
| | Original | Now |
|
||||
|---|---|---|
|
||||
| `unitigs.bin` / `unitigs.bin.idx` (2.1 GB skim, 521.6 M kmers, 16 partitions) | 1.05 / 0.42 GB | **0.66 / 0.24 GB** (−37 % / −43 %: 3.81 M unitigs instead of 6.56 M — the orientation fix of §18) |
|
||||
| Whole index of that skim | 3.5 GB | **2.9 GB** (−17 %) |
|
||||
| `evidence.bin` | 4 bytes per kmer | unchanged (still ≈ 40 % of the 91-genome index, 2.1 GB of 3.5 GB for the skim): reducing it is a separate design question (§ answer on exact evidence: ≈ −22 to −31 % of it by dropping the rank field, far more with a minimizer-indexed layout; neither implemented) |
|
||||
|
||||
### Correctness fixes found on the way
|
||||
|
||||
- Unitigs were not maximal (72 % too many) and depended on hash-table iteration order (§18).
|
||||
- Chunks of 129–256 k-mers had their ranks silently truncated in the 7-bit evidence field and failed verification, visible with few partitions (§19, `scripts/check_unitig_chunks.py` to audit an index).
|
||||
- `index.meta` and super-kmer files were parsed byte by byte (§15).
|
||||
|
||||
### What is **not** established
|
||||
|
||||
- The DRAM-latency gain of MPHF prefetch itself, the original motivation (§1–§6): the measured MPHF share of the profiles is ≈ 1 % (§11), and the layers here are 4× to 1 000× smaller than production ones. The −5 % seen on the first query change came mostly from batching per layer (probably fewer shared `Arc` clones, §11), not from prefetch. A prod-scale benchmark (§6) remains to be run on the production machine.
|
||||
- Speed-ups on the 192-core / 1 TB production machine, on `merge`, `filter`, `select` and `phylo` at the 180-genome scale (the 91-genome index could only be used for `query`, `dump`, `phylo --distance jaccard`, `utils --stats`; `filter`/`select`/`merge` refused its pre-`state` `index.meta` until it was patched in the clone).
|
||||
- The size of the Primula index composition (§ calculated: ≈ half evidence from 35–40 G kmers × 4 bytes; not measured).
|
||||
@@ -192,17 +192,40 @@ pub fn run(args: QueryArgs) {
|
||||
let overhead_multiplier =
|
||||
BASE_OVERHEAD + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * detail_factor;
|
||||
|
||||
// Parallelism: a chunk is one worker's unit of work, so an input smaller
|
||||
// than `n_workers` chunks leaves cores idle (two 48 MB files at the
|
||||
// memory-derived 256 MiB cap = 2 chunks = 2 busy cores; measured 11.6 s
|
||||
// against 2.5 s with 8 MiB chunks, see `DevDocMD/architecture/query.md`).
|
||||
// When the input size can be estimated, aim for `CHUNKS_PER_WORKER` chunks
|
||||
// per worker and never go below `MIN_CHUNK`. Stdin and URLs have no
|
||||
// known size: the memory-derived size stands.
|
||||
const CHUNKS_PER_WORKER: u64 = 2;
|
||||
const MIN_CHUNK: u64 = 4 * 1024 * 1024;
|
||||
let input_bytes: Option<u64> = args
|
||||
.inputs
|
||||
.iter()
|
||||
.map(|p| obiread::estimated_decompressed_len(p))
|
||||
.sum::<Option<u64>>();
|
||||
|
||||
let chunk_bytes = args
|
||||
.chunk_size
|
||||
.map(|mb| mb * 1024 * 1024)
|
||||
.unwrap_or_else(|| {
|
||||
let avail = available_memory_bytes();
|
||||
let computed = avail / (n_workers as u64 * overhead_multiplier * SAFETY_FACTOR);
|
||||
computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024) as usize
|
||||
let by_memory = computed.clamp(4 * 1024 * 1024, 256 * 1024 * 1024);
|
||||
match input_bytes {
|
||||
Some(total) => {
|
||||
let by_parallelism = (total / (n_workers as u64 * CHUNKS_PER_WORKER)).max(MIN_CHUNK);
|
||||
by_memory.min(by_parallelism) as usize
|
||||
}
|
||||
None => by_memory as usize,
|
||||
}
|
||||
});
|
||||
|
||||
debug!(
|
||||
chunk_bytes,
|
||||
input_bytes,
|
||||
n_genomes,
|
||||
detail = args.detail,
|
||||
overhead_multiplier,
|
||||
|
||||
@@ -25,7 +25,7 @@ pub use normalize::{normalize_fasta_chunk, normalize_fastq_chunk, normalize_sequ
|
||||
pub use nucstream::{NucPage, NucPageCursor, open_nuc_stream};
|
||||
pub use path_iterator::{PathIter, path_iter};
|
||||
pub use peakreader::PeekReader;
|
||||
pub use xopen::xopen;
|
||||
pub use xopen::{estimated_decompressed_len, xopen};
|
||||
|
||||
/// Default read block size: 1 MiB.
|
||||
pub const DEFAULT_BLOCK_SIZE: usize = 1024 * 1024;
|
||||
@@ -0,0 +1,38 @@
|
||||
use super::*;
|
||||
use std::io::Write;
|
||||
|
||||
fn temp_file(name: &str, content: &[u8]) -> std::path::PathBuf {
|
||||
let path = std::env::temp_dir().join(format!("obiread_xopen_{}_{name}", std::process::id()));
|
||||
std::fs::File::create(&path).unwrap().write_all(content).unwrap();
|
||||
path
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn estimate_plain_file_is_its_length() {
|
||||
let p = temp_file("plain", b">s\nACGT\n");
|
||||
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(8));
|
||||
std::fs::remove_file(p).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn estimate_scales_compressed_files_by_magic_not_extension() {
|
||||
// Only the magic bytes matter to the estimate (the content is never decoded).
|
||||
let mut gz = vec![0x1f, 0x8b, 0x08, 0, 0, 0];
|
||||
gz.resize(100, 0);
|
||||
let p = temp_file("looks_plain.txt", &gz);
|
||||
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(300));
|
||||
std::fs::remove_file(p).unwrap();
|
||||
|
||||
let mut zst = vec![0x28, 0xb5, 0x2f, 0xfd, 0, 0];
|
||||
zst.resize(50, 0);
|
||||
let p = temp_file("zst", &zst);
|
||||
assert_eq!(estimated_decompressed_len(p.to_str().unwrap()), Some(200));
|
||||
std::fs::remove_file(p).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn estimate_is_unknown_for_streams_and_missing_files() {
|
||||
assert_eq!(estimated_decompressed_len("-"), None);
|
||||
assert_eq!(estimated_decompressed_len("https://example.org/x.fa.gz"), None);
|
||||
assert_eq!(estimated_decompressed_len("/nonexistent/obiread/file"), None);
|
||||
}
|
||||
@@ -46,6 +46,37 @@ pub fn xopen(source: &str) -> io::Result<MimeTypeGuesser<Box<dyn Read + Send>>>
|
||||
Ok(MimeTypeGuesser::new(open_raw(source)?))
|
||||
}
|
||||
|
||||
/// Rough size, in bytes, of what [`xopen`] would yield for a local file:
|
||||
/// its length, multiplied by a typical expansion factor when the magic bytes
|
||||
/// say it is compressed (gzip ≈ 3×, as measured on FASTQ; bzip2, xz and zstd
|
||||
/// taken as 4×). Meant for sizing work (chunk sizes), not for allocation —
|
||||
/// the real ratio varies by a factor of two or more with the content.
|
||||
///
|
||||
/// `None` when the size cannot be known without reading the stream: stdin
|
||||
/// (`-`), an HTTP(S) URL, or a file that cannot be opened.
|
||||
pub fn estimated_decompressed_len(source: &str) -> Option<u64> {
|
||||
if source == "-" || source.starts_with("http://") || source.starts_with("https://") {
|
||||
return None;
|
||||
}
|
||||
let expanded = expand_tilde(source);
|
||||
let mut file = File::open(expanded.as_ref()).ok()?;
|
||||
let len = file.metadata().ok()?.len();
|
||||
let mut magic = [0u8; 6];
|
||||
let n = file.read(&mut magic).ok()?;
|
||||
let magic = &magic[..n];
|
||||
let factor = if magic.starts_with(&[0x1f, 0x8b]) {
|
||||
3
|
||||
} else if magic.starts_with(b"BZh")
|
||||
|| magic.starts_with(&[0xfd, b'7', b'z', b'X', b'Z', 0x00])
|
||||
|| magic.starts_with(&[0x28, 0xb5, 0x2f, 0xfd])
|
||||
{
|
||||
4
|
||||
} else {
|
||||
1
|
||||
};
|
||||
Some(len * factor)
|
||||
}
|
||||
|
||||
// ── internal helpers ──────────────────────────────────────────────────────────
|
||||
|
||||
fn http_reader(url: &str) -> io::Result<Box<dyn Read + Send>> {
|
||||
@@ -71,3 +102,7 @@ fn expand_tilde(path: &str) -> std::borrow::Cow<'_, str> {
|
||||
}
|
||||
std::borrow::Cow::Borrowed(path)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "tests/xopen.rs"]
|
||||
mod tests;
|
||||
Reference in new issue
Block a user