Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors. - Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing. - Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants. - Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling. - Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections. - Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic. - Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure. - Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions. - Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry. - Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans. - Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking. - Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
12 KiB
Merge command
Purpose
obikmer merge combines multiple existing kmer indexes into a single index. The result contains all kmers from all sources, with per-genome presence/absence or count data for every genome across every layer.
Modes
pub enum MergeMode { Presence, Count }
Default mode is Presence. Count mode requires all source indexes to have with_counts=true; mixing count and non-count sources is rejected at validation.
| Mode | Column type | Constraint |
|---|---|---|
Presence |
PersistentBitMatrix (one bit per genome per slot) |
none |
Count |
PersistentCompactIntMatrix (one u32 per genome per slot) |
all sources with_counts=true |
Input / output constraints
All source indexes must satisfy:
IndexState::Indexed(fully built —index.donesentinel present)- Same
kmer_size,minimizer_size,n_partitions - Same evidence kind: all
Exact, or allApproxwith identical(b, z)parameters - If
Countmode: all sources must havewith_counts=true
--force: if the output directory already exists, it is deleted before the merge begins.
Evidence compatibility
validate_evidence_compat(sources) is called before any I/O. It compares each source's EvidenceKind against sources[0]:
- All
Exact→ accepted, output usesExact - All
Approx { b, z }with same(b, z)→ accepted, output uses those parameters - Any other combination →
OKIError::IncompatibleEvidence, with a message directing the user to runreindexfirst
Mixed exact/approx is a hard error, not a silent conversion.
fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
Genome label deduplication
compute_labels(sources, rename_duplicates) assigns final genome labels across all sources before any file is written. The first occurrence of a label keeps the original name. Subsequent occurrences receive .1, .2, … suffixes when rename_duplicates is true, or trigger OKIError::DuplicateGenomeLabel otherwise.
Algorithm
1. Validation
Check all sources against the constraints above. Abort on any mismatch.
2. Bootstrap output from the base source
The base is chosen by choose_base (merge.rs): the source with the largest total unitigs.bin size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily sources[0], and the genome columns follow the base first. Because obikmer index is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to output (hard links for files merge_partition never rewrites, see below). Immediately after the copy:
index.metais rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.- In
Presencemode, anycounts/directories inherited from source_0 are removed. spectrums/from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … n_dst_genomes - 1 in the destination.
3. For each subsequent source (parallel across partitions)
obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence) runs once per partition index i (scheduled largest first by PartitionRunner). Each entry of sources is (&KmerIndex, n_genomes), where n_genomes is the number of columns that source contributes (> 1 for an already-merged source). Code: partition_merge/mod.rs (orchestration, pass 1) and partition_merge/pass2.rs (per-group fill pipeline).
First merge, Presence mode: when n_dst_genomes == 1, TypedLayer::<()>::init_presence_matrix is called on every existing dst layer first. It materialises the implicit single-genome matrix (presence/col_000000.pbiv, all true) so that source columns can be appended to it.
The dst layers are opened once per partition and shared by both passes.
Pass 1 — new k-mers into a de Bruijn graph. Only the sources' unitigs.bin paths are needed (KmerLayer::new(..).unitigs_path()); no source layer is opened. The files are streamed (at most max_open = 2 open at once), and every k-mer absent from all dst layers is pushed into one GraphDeBruijn. All new k-mers of all sources go into a single new layer per merge, built by materialize_layer (unitigs + MPHF + evidence).
New-layer matrix. If a new layer exists, its matrix is created with n_dst_genomes absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.
Pass 2 — by consecutive groups of SRC_GROUP = 2 sources, in source order. Per group:
- open the group's source layers for this partition;
MatrixBuilder::resumeevery dst matrix (existing layers + new layer), checkn_cols == n_dst_genomes + col_base, append the group'sgroup_ncolumns;- run the pipeline: stream each source layer's k-mers,
hash_batch+fill_sub_matrixon the source, locate each k-mer in the dst layers (find_batch_grouped) or else in the new layer's MPHF, and write the values column-major into the group's builders; - close the columns, then the matrices; drop the group's layers;
col_base += group_n.
Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.
In-memory columns. The group's columns are built in RAM (MatrixBuilder::add_mem_col → MemBitVecBuilder / MemCompactIntVecBuilder) and written once, sequentially, at close (.building file, then rename). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no MmapMut on this path, so a block the FS cannot allocate is an io::Error instead of a SIGBUS. Files are not fsynced: one sync per column file made merge ~8× slower. RAM per partition is at most SRC_GROUP × src_n × N_part × (1/8 | 1) bytes (presence | counts). History and figures: merge_mmap_audit.md; scaling limits: merge_scaling.md.
4. Update index metadata
index.meta was already written during bootstrap with the complete genome list and evidence kind. No further update is needed after the partition loop.
append_genome_column
Defined on two concrete specialisations of Layer<D>:
impl Layer<PersistentCompactIntMatrix> {
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
}
impl Layer<PersistentBitMatrix> {
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
}
Each appends one column file to the matrix subdirectory (counts/ or presence/). In merge_partition, columns are written through MatrixBuilder::add_mem_col (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.
Column count invariant
After any merge, every layer in every partition has exactly n_genomes columns, where n_genomes is the total genome count in the index at that point.
Maintained by three mechanisms:
- Existing layers:
n_src_totalcolumns appended (one per source genome), group by group. - New layers created during merge:
n_dst_genomesabsent columns prepended before source columns. - First merge, Presence mode:
init_presence_matrixretroactively createspresence/col_0all-true for genome 0.
The invariant is a precondition of LayeredStore aggregation traits: col_weights() and all partial distance methods assume every inner store has the same column count.
Error variants relevant to merge
| Variant | Condition |
|---|---|
OKIError::NotIndexed(path) |
Source not in Indexed state |
OKIError::IncompatibleConfig |
Mismatched kmer_size, minimizer_size, or n_partitions |
OKIError::MismatchedMode |
Count mode but a source has with_counts=false |
OKIError::IncompatibleEvidence(msg) |
Mixed exact/approx or different approx (b, z) |
OKIError::DuplicateGenomeLabel(label) |
Duplicate label and rename_duplicates=false |
On-disk impact
After merging G genomes (sources_0 contributes G0, subsequent sources the rest), before finalize packs each matrix (sparse by default, matrix.pbmx with --dense):
partitions/
part_00000/
index/
meta.json ← n_layers updated if new layer added
layer_0/
mphf.bin ← unchanged
unitigs.bin ← unchanged
evidence.bin ← unchanged
presence/ ← created on first merge (Presence mode)
meta.json {"n": N, "n_cols": G}
col_000000.pbiv ← all-true (genome 0 … G0-1)
col_000001.pbiv ← next source
...
counts/ ← extended (Count mode)
meta.json {"n": N, "n_cols": G}
col_000000.pciv ← genome 0 counts (from original build)
col_000001.pciv ← next source
...
layer_N/ ← new layer (if new kmers found)
mphf.bin
unitigs.bin
evidence.bin
presence/ or counts/
meta.json {"n": N1, "n_cols": G}
col_000000.pbiv ← all-false (absent for existing genomes)
...
spectrums/
<label>.json ← one file per genome, rebuilt from all sources
index.meta ← complete genome list + evidence kind written at bootstrap
mphf.bin/unitigs.bin/evidence.bin/unitigs.bin.idx/fingerprint.bin/layer_meta.json marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — merge_partition never rewrites them for pre-existing layers, only the presence/counts subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.
Known issue (not yet fixed, 2026-08-28)
Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. obikmer merge -o out IDX IDX --rename-duplicates) — panics deep in the MPHF's rank-select structure (common_traits::select_in_word, assertion failed: rank < self.count_ones()), inside MphfLayer::find called from merge_partition's "is this source kmer already in dst" check against the bootstrap-copied dst_layers. Root cause not identified; ruled out so far:
- Not an empty-new-layer issue: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
- Not caused by the hard-link change above: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.
Only reproduces when sources[0] and sources[1] are the exact same on-disk path opened as two independent KmerIndex handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since PartitionRunner's panic propagation fix (see architecture/numa_partition_runner.md) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.