Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
This commit is contained in:
1 parent
09c80468b2
commit
093d81c359
4 files changed
+116
-42
No files matched your search
@@ -424,3 +424,18 @@ With `--chunk-size 4` the same small query peaks at 3.6 GB: that floor is the to
|
||||
**Side finding.** The 256 MiB cap makes small inputs poorly parallel: the two 48 MB files above are two chunks, so two workers; with `--chunk-size 4` the same query takes 3.7 s instead of 17 s. Not changed here.
|
||||
|
||||
**Not verified.** Prod-scale (192 workers × 256 MiB chunks in flight): the factor is now ≈ 6 instead of ≈ 21, but how many chunks are in flight at once on that machine was not observed.
|
||||
|
||||
### Chunk size sweep on a small input (2026-10-15)
|
||||
|
||||
Input: 2 × 48 MB decompressed fastq (200 000 reads each) against a 354 M-kmer single-genome index (4 partitions), 16 cores on AC power, 3 runs per size, sorted outputs identical (same md5) for every size:
|
||||
|
||||
| `--chunk-size` (MiB) | chunks | wall (s) | user CPU (s) | RSS (GB) |
|
||||
|---|---|---|---|---|
|
||||
| 2 | ~50 | 2.5–2.6 | 25.0–25.3 | 3.2 |
|
||||
| 4 | ~25 | 2.5–2.7 | 24.3–25.5 | 3.5 |
|
||||
| 8 | ~12 | 2.5–2.7 | 23.6–26.1 | 4.1 |
|
||||
| 16 | ~6 | 4.4 | 23.1–23.7 | 4.1 |
|
||||
| 32 | 4 | 8.3 | 22.9–23.0 | 4.0 |
|
||||
| 64 and 256 (default) | 2 | 11.6–11.8 | 22.4–22.9 | 4.1 |
|
||||
|
||||
Wall time follows the number of chunks (one worker per chunk) until it reaches the core count, then plateaus at ≈ total CPU / cores; total CPU is almost constant (≈ +10 % at 2 MiB, per-chunk fixed cost, here with 1 genome and 4 partitions). With the 256 MiB cap, **any input below ≈ 16 × 256 MiB = 4 GiB decompressed leaves cores idle**. A rule such as `chunk ≈ input_bytes / (workers × 2)` with a floor of 4–8 MiB would use all cores on small inputs; not implemented. Unmeasured: the per-chunk cost on an index with many genomes and 256 partitions, which could move the floor up.
|
||||
Reference in new issue
Block a user