Files
obikmer/DevDocMD/implementation/merge.md
T
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00

12 KiB

Merge command

Purpose

obikmer merge combines multiple existing kmer indexes into a single index. The result contains all kmers from all sources, with per-genome presence/absence or count data for every genome across every layer.


Modes

pub enum MergeMode { Presence, Count }

Default mode is Presence. Count mode requires all source indexes to have with_counts=true; mixing count and non-count sources is rejected at validation.

Mode Column type Constraint
Presence PersistentBitMatrix (one bit per genome per slot) none
Count PersistentCompactIntMatrix (one u32 per genome per slot) all sources with_counts=true

Input / output constraints

All source indexes must satisfy:

  • IndexState::Indexed (fully built — index.done sentinel present)
  • Same kmer_size, minimizer_size, n_partitions
  • Same evidence kind: all Exact, or all Approx with identical (b, z) parameters
  • If Count mode: all sources must have with_counts=true

--force: if the output directory already exists, it is deleted before the merge begins.


Evidence compatibility

validate_evidence_compat(sources) is called before any I/O. It compares each source's EvidenceKind against sources[0]:

  • All Exact → accepted, output uses Exact
  • All Approx { b, z } with same (b, z) → accepted, output uses those parameters
  • Any other combination → OKIError::IncompatibleEvidence, with a message directing the user to run reindex first

Mixed exact/approx is a hard error, not a silent conversion.

fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>

Genome label deduplication

compute_labels(sources, rename_duplicates) assigns final genome labels across all sources before any file is written. The first occurrence of a label keeps the original name. Subsequent occurrences receive .1, .2, … suffixes when rename_duplicates is true, or trigger OKIError::DuplicateGenomeLabel otherwise.


Algorithm

1. Validation

Check all sources against the constraints above. Abort on any mismatch.

2. Bootstrap output from the base source

The base is chosen by choose_base (merge.rs): the source with the largest total unitigs.bin size (restricted to approximate-evidence sources when the output must be approximate). It is not necessarily sources[0], and the genome columns follow the base first. Because obikmer index is not byte-reproducible (unitig segmentation varies), two near-equal sources can swap roles between runs. Its directory is copied to output (hard links for files merge_partition never rewrites, see below). Immediately after the copy:

  • index.meta is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
  • In Presence mode, any counts/ directories inherited from source_0 are removed.
  • spectrums/ from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.

This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The base source's genomes occupy columns 0 … n_dst_genomes - 1 in the destination.

3. For each subsequent source (parallel across partitions)

obikmerge::partition_merge::merge_partition(dst, i, sources, mode, n_dst_genomes, block_bits, evidence) runs once per partition index i (scheduled largest first by PartitionRunner). Each entry of sources is (&KmerIndex, n_genomes), where n_genomes is the number of columns that source contributes (> 1 for an already-merged source). Code: partition_merge/mod.rs (orchestration, pass 1) and partition_merge/pass2.rs (per-group fill pipeline).

First merge, Presence mode: when n_dst_genomes == 1, TypedLayer::<()>::init_presence_matrix is called on every existing dst layer first. It materialises the implicit single-genome matrix (presence/col_000000.pbiv, all true) so that source columns can be appended to it.

The dst layers are opened once per partition and shared by both passes.

Pass 1 — new k-mers into a de Bruijn graph. Only the sources' unitigs.bin paths are needed (KmerLayer::new(..).unitigs_path()); no source layer is opened. The files are streamed (at most max_open = 2 open at once), and every k-mer absent from all dst layers is pushed into one GraphDeBruijn. All new k-mers of all sources go into a single new layer per merge, built by materialize_layer (unitigs + MPHF + evidence).

New-layer matrix. If a new layer exists, its matrix is created with n_dst_genomes absent columns (all zero/false) and closed immediately. This keeps the column-count invariant, and every source group then appends to it like to the existing layers.

Pass 2 — by consecutive groups of SRC_GROUP = 2 sources, in source order. Per group:

  1. open the group's source layers for this partition;
  2. MatrixBuilder::resume every dst matrix (existing layers + new layer), check n_cols == n_dst_genomes + col_base, append the group's group_n columns;
  3. run the pipeline: stream each source layer's k-mers, hash_batch + fill_sub_matrix on the source, locate each k-mer in the dst layers (find_batch_grouped) or else in the new layer's MPHF, and write the values column-major into the group's builders;
  4. close the columns, then the matrices; drop the group's layers; col_base += group_n.

Only one group's source layers and destination columns are alive at a time, so the number of live mappings per partition does not depend on the number of sources. Columns are appended in source order, so every genome lands in the same absolute column as in a single-session merge.

In-memory columns. The group's columns are built in RAM (MatrixBuilder::add_mem_col → MemBitVecBuilder / MemCompactIntVecBuilder) and written once, sequentially, at close (.building file, then rename). The files are byte-identical to the former mmap-backed builders' output. There is no sparse file and no MmapMut on this path, so a block the FS cannot allocate is an io::Error instead of a SIGBUS. Files are not fsynced: one sync per column file made merge ~8× slower. RAM per partition is at most SRC_GROUP × src_n × N_part × (1/8 | 1) bytes (presence | counts). History and figures: merge_mmap_audit.md; scaling limits: merge_scaling.md.

4. Update index metadata

index.meta was already written during bootstrap with the complete genome list and evidence kind. No further update is needed after the partition loop.


append_genome_column

Defined on two concrete specialisations of Layer<D>:

impl Layer<PersistentCompactIntMatrix> {
    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
}

impl Layer<PersistentBitMatrix> {
    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
}

Each appends one column file to the matrix subdirectory (counts/ or presence/). In merge_partition, columns are written through MatrixBuilder::add_mem_col (in-memory builders) rather than through these helpers, but the invariant they enforce is the same.


Column count invariant

After any merge, every layer in every partition has exactly n_genomes columns, where n_genomes is the total genome count in the index at that point.

Maintained by three mechanisms:

  1. Existing layers: n_src_total columns appended (one per source genome), group by group.
  2. New layers created during merge: n_dst_genomes absent columns prepended before source columns.
  3. First merge, Presence mode: init_presence_matrix retroactively creates presence/col_0 all-true for genome 0.

The invariant is a precondition of LayeredStore aggregation traits: col_weights() and all partial distance methods assume every inner store has the same column count.


Error variants relevant to merge

Variant Condition
OKIError::NotIndexed(path) Source not in Indexed state
OKIError::IncompatibleConfig Mismatched kmer_size, minimizer_size, or n_partitions
OKIError::MismatchedMode Count mode but a source has with_counts=false
OKIError::IncompatibleEvidence(msg) Mixed exact/approx or different approx (b, z)
OKIError::DuplicateGenomeLabel(label) Duplicate label and rename_duplicates=false

On-disk impact

After merging G genomes (sources_0 contributes G0, subsequent sources the rest), before finalize packs each matrix (sparse by default, matrix.pbmx with --dense):

partitions/
  part_00000/
    index/
      meta.json              ← n_layers updated if new layer added
      layer_0/
        mphf.bin             ← unchanged
        unitigs.bin          ← unchanged
        evidence.bin         ← unchanged
        presence/            ← created on first merge (Presence mode)
          meta.json            {"n": N, "n_cols": G}
          col_000000.pbiv      ← all-true (genome 0 … G0-1)
          col_000001.pbiv      ← next source
          ...
        counts/              ← extended (Count mode)
          meta.json            {"n": N, "n_cols": G}
          col_000000.pciv      ← genome 0 counts (from original build)
          col_000001.pciv      ← next source
          ...
      layer_N/               ← new layer (if new kmers found)
        mphf.bin
        unitigs.bin
        evidence.bin
        presence/ or counts/
          meta.json            {"n": N1, "n_cols": G}
          col_000000.pbiv      ← all-false (absent for existing genomes)
          ...
spectrums/
  <label>.json               ← one file per genome, rebuilt from all sources
index.meta                   ← complete genome list + evidence kind written at bootstrap

mphf.bin/unitigs.bin/evidence.bin/unitigs.bin.idx/fingerprint.bin/layer_meta.json marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — merge_partition never rewrites them for pre-existing layers, only the presence/counts subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.

Known issue (not yet fixed, 2026-08-28)

Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. obikmer merge -o out IDX IDX --rename-duplicates) — panics deep in the MPHF's rank-select structure (common_traits::select_in_word, assertion failed: rank < self.count_ones()), inside MphfLayer::find called from merge_partition's "is this source kmer already in dst" check against the bootstrap-copied dst_layers. Root cause not identified; ruled out so far:

  • Not an empty-new-layer issue: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.
  • Not caused by the hard-link change above: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.

Only reproduces when sources[0] and sources[1] are the exact same on-disk path opened as two independent KmerIndex handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since PartitionRunner's panic propagation fix (see architecture/numa_partition_runner.md) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.