Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors. - Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing. - Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants. - Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling. - Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections. - Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic. - Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure. - Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions. - Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry. - Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans. - Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking. - Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2.7 KiB
merge
Merge multiple built indexes into a single index.
obikmer merge -o OUTPUT SOURCE... [OPTIONS]
Arguments
| Argument | Description |
|---|---|
SOURCE... |
Index directories to merge (at least one required) |
Options
| Option | Default | Description |
|---|---|---|
-o, --output |
— (required) | Output index directory |
--force |
off | Overwrite an existing output directory |
--force-presence |
off | Store the merged index as presence/absence even if all sources have counts |
--rename-duplicates |
off | Disambiguate duplicate genome labels (.1, .2, …) instead of failing |
--dense |
off | Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see pack) |
Behaviour
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. --force-presence forces presence/absence regardless of the sources.
By default, merging two indexes that share a genome label fails with an error; --rename-duplicates instead appends a numeric suffix to keep both copies.
The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; --dense keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The pack page explains the trade-off between the two formats.
Resource usage
The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.
Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.
Current limitation
An index produced by merge cannot itself be used as a source of a later merge yet: the command stops with a "No such file or directory" error. To add genomes to a collection, merge the original single-genome indexes again, together with the new ones.