Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors. - Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing. - Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants. - Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling. - Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections. - Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic. - Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure. - Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions. - Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry. - Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans. - Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking. - Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
4.4 KiB
Merge: scaling to thousands of genomes (open, 2026-11-17)
Open design question raised during the mmap refactor (merge_mmap_audit.md). Target: merges of ~6 000 genomes of a few Gb each. Nothing here is implemented or measured. Figures are estimates, labelled as such.
What scales with what (after the mmap refactor)
| Cost | Depends on | Status |
|---|---|---|
| Live source layers + dst columns per partition (merge pass 2) | SRC_GROUP × src_n, not the source count |
fixed by plan A |
| Pass-2 column RAM | SRC_GROUP × src_n × N_part × (1/8 | 1) B |
bounded, see below |
| Pass-1 de Bruijn graph | ≥ ~19 B per new k-mer (HashMap<u64, AtomicU8>, verified in obidebruinj/src/debruijn/graph.rs) |
unchanged |
| Columnar intermediate files | n_genomes × n_layers × n_partitions (6 000 × 2 × 256 ≈ 3 M files, estimate) |
does not scale |
Sparse pack |
maps all columns of a layer at once (ColumnarBitMatrix::open, verified) → ~n_genomes × n_layers maps per partition |
does not scale; with 186 genomes × 2 layers × 192 concurrent partitions ≈ 71 k maps, already above vm.max_map_count (estimate, assumes 1 map per column and pack concurrency = cores) |
N_part: k-mers of one dst partition over all layers. Pass-2 column RAM stays below pass-1 graph RAM for one-shot merges. It dominates only for incremental merges, where few genomes are added to a large index: the new layer is small, the existing layers are large. Example (estimate): N_total = 1e12, counts, 192 concurrent partitions → ~1.5 TB. A memory-budgeted concurrency cap is needed there whatever the column strategy. obisys::MemoryBudget exists but is no longer used by merge.
Why a columnar intermediate at all
The sparse format (bitmatrix/sparse.rs) is row-major and deduplicated: per slot, either one genome (singleton.pfiv) or a dict_id (multi.pfiv) into a dictionary of distinct genome sets (dict_offsets + dict_values.bin). Its builder only offers push_row(genomes), i.e. the complete genome set of a row, rows in slot order.
Merge discovers values column by column (source by source), in random slot order (MPHF order). The columnar files are a transposition buffer: they absorb random per-column writes, and pack transposes them row by row.
Options to build the final format directly
-
All sources at once. That is exactly what plan A removed. No.
-
Whole matrix in RAM.
N_part × n_genomes / 8B per partition: ~3 TB at 6 000 genomes andN_part = 4e9(estimate). No. -
External sort of
(slot, genome)pairs. Volume ∝ non-zeros ≈ Σ genome k-mers (~6 000 × 3e9). That is huge, while the final format is small precisely because genome sets repeat. No. -
Incremental sparse update per source group (most promising). Per group:
- fill the group's columns in RAM (as now);
- stream the partition's existing sparse matrix sequentially in slot order;
- for each row: new set = old set ∪ group bits. Through the dictionary this is a transition
(old dict_id, group bit pattern) → new dict_id, which can be memoised in a table; - write the new sparse matrix sequentially, then publish.
No columnar files, no
pack, no per-genome mapping, sequential I/O only. It also removes the re-merge bug: it starts from an existing sparse matrix instead of requiring a columnar one.The cost is one full pass over the partition's sparse matrix per group. With
SRC_GROUP = 2and 6 000 genomes that is 3 000 rewrites, far too many. Groups must therefore be large (64–256 sources), and group column RAM grows with them:group × N_part / 8B, e.g. 32 GB per partition for 64 sources atN_part = 4e9(estimate). One parameter trades passes against RAM, and it should be sized from a memory budget. The count matrix (sparse_intmatrix.rs) has its own format and needs the same analysis.
Related findings
- The module header of
bitmatrix/sparse.rssays "not used by any production code path yet". This is stale:finalize_indexedpacks to sparse by default. - The new layer of a merge is not byte-reproducible run to run: unitig segmentation varies, so the MPHF, evidence and matrix row order vary with it (measured 2026-11-17,
layer_1/*differs in all 256 partitions;layer_0and metadata are identical).obikmer indexshows the same behaviour. Merge oracles therefore compare sorted dumps, not file hashes. Where the non-determinism originates (parallel graph fill or unitig walk) is not established.