<p><code>obikmer merge</code> combines multiple existing kmer indexes into a single index. The result contains all kmers from all sources, with per-genome presence/absence or count data for every genome across every layer.</p>
<p>Default mode is <code>Presence</code>. <code>Count</code> mode requires <strong>all</strong> source indexes to have <code>with_counts=true</code>; mixing count and non-count sources is rejected at validation.</p>
<table>
<thead>
<tr>
<th>Mode</th>
<th>Column type</th>
<th>Constraint</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>Presence</code></td>
<td><code>PersistentBitMatrix</code> (one bit per genome per slot)</td>
<td>none</td>
</tr>
<tr>
<td><code>Count</code></td>
<td><code>PersistentCompactIntMatrix</code> (one u32 per genome per slot)</td>
<p><code>validate_evidence_compat(sources)</code> is called before any I/O. It compares each source's <code>EvidenceKind</code> against <code>sources[0]</code>:</p>
<p><code>compute_labels(sources, rename_duplicates)</code> assigns final genome labels across all sources before any file is written. The first occurrence of a label keeps the original name. Subsequent occurrences receive <code>.1</code>, <code>.2</code>, … suffixes when <code>rename_duplicates</code> is true, or trigger <code>OKIError::DuplicateGenomeLabel</code> otherwise.</p>
<hr/>
<h2id="algorithm">Algorithm</h2>
<h3id="1-validation">1. Validation</h3>
<p>Check all sources against the constraints above. Abort on any mismatch.</p>
<h3id="2-bootstrap-output-from-first-source">2. Bootstrap output from first source</h3>
<p>Recursive file copy of <code>sources[0]</code> → <code>output</code>. Immediately after the copy:</p>
<ul>
<li><code>index.meta</code> is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.</li>
<li>In <code>Presence</code> mode, any <code>counts/</code> directories inherited from source_0 are removed.</li>
<li><code>spectrums/</code> from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.</li>
</ul>
<p>This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … <code>n_dst_genomes - 1</code> in the destination.</p>
<h3id="3-for-each-subsequent-source-parallel-across-partitions">3. For each subsequent source (parallel across partitions)</h3>
<p><code>KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)</code> is called for each partition index <code>i</code>. <code>block_bits</code> is taken from <code>dst.meta.config.block_bits</code>.</p>
<p>Each entry in <code>sources</code> is <code>(&KmerPartition, n_genomes)</code> where <code>n_genomes</code> is the column count that source contributes (> 1 when the source is itself a merged index).</p>
<p><strong>First merge, Presence mode</strong>: when <code>n_dst_genomes == 1</code>, <code>Layer::<()>::init_presence_matrix</code> is called on every existing destination layer before any source column is appended. This creates <code>presence/col_000000.pbiv</code> set all-true (genome 0 is present in every slot).</p>
<p><strong>Pass 1 — classify kmers</strong></p>
<p>Iterate all kmers from all source partitions (via <code>UnitigFileReader</code> + canonical kmer iteration). For each kmer, probe the destination <code>LayeredMap<()></code>:</p>
<ul>
<li><strong>Hit</strong>: kmer already in the destination; record for Pass 2.</li>
<li><strong>Miss</strong>: push kmer into a <code>GraphDeBruijn</code> accumulator.</li>
</ul>
<p><strong>New layer construction</strong></p>
<p>If the accumulator is non-empty, compute de Bruijn unitigs and call <code>Layer::<()>::build(&new_layer_dir, block_bits)</code>. All kmers absent from the destination — across <strong>all</strong> sources — accumulate into a <strong>single</strong> graph, producing one new layer per merge operation (not one per source).</p>
<p><strong>Pass 2 — fill column builders</strong></p>
<p>For each source and each of its layers, re-iterate unitigs and look up stored values via <code>SrcLayerData::lookup(kmer, src_n)</code>:</p>
<ul>
<li><code>SrcLayerData::SetMembership</code> — no data directory exists; every kmer returns <code>vec![1; n_genomes]</code></li>
<li><code>SrcLayerData::Presence</code> — reads <code>PersistentBitMatrix</code> from <code>presence/</code></li>
<li><code>SrcLayerData::Count</code> — reads <code>PersistentCompactIntMatrix</code> from <code>counts/</code></li>
</ul>
<p>Hits are routed to <code>exist_builders[dst_layer][src_col]</code>; misses are routed to <code>new_src_builders[src_col]</code>.</p>
<p><strong>Column prepending for new layers</strong></p>
<p>Before source columns are written to the new layer, <code>n_dst_genomes</code> absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.</p>
<p><strong>Close and update metadata</strong></p>
<p>Close all builders; update <code>presence/meta.json</code> or <code>counts/meta.json</code> with <code>{"n": N, "n_cols": n_dst_genomes + n_src_total}</code>; increment <code>PartitionMeta::n_layers</code> if a new layer was added.</p>
<h3id="4-update-index-metadata">4. Update index metadata</h3>
<p><code>index.meta</code> was already written during bootstrap with the complete genome list and evidence kind. No further update is needed after the partition loop.</p>
<p>Each appends one column file to the matrix subdirectory (<code>counts/</code> or <code>presence/</code>). In <code>merge_partition</code>, columns are written directly via <code>PersistentBitVecBuilder</code> / <code>PersistentCompactIntVecBuilder</code> rather than through these helpers, but the invariant they enforce is the same.</p>
<p>After any merge, <strong>every layer in every partition has exactly <code>n_genomes</code> columns</strong>, where <code>n_genomes</code> is the total genome count in the index at that point.</p>
<p>Maintained by three mechanisms:</p>
<ol>
<li><strong>Existing layers</strong>: <code>n_src_total</code> columns appended (one per source genome).</li>
<li><strong>New layers created during merge</strong>: <code>n_dst_genomes</code> absent columns prepended before source columns.</li>
<p>The invariant is a precondition of <code>LayeredStore</code> aggregation traits: <code>col_weights()</code> and all partial distance methods assume every inner store has the same column count.</p>
<hr/>
<h2id="error-variants-relevant-to-merge">Error variants relevant to merge</h2>
<p><code>mphf.bin</code>/<code>unitigs.bin</code>/<code>evidence.bin</code>/<code>unitigs.bin.idx</code>/<code>fingerprint.bin</code>/<code>layer_meta.json</code> marked "unchanged" above are hard-linked from the base source's own files during the bootstrap copy (2026-08-28), not copied — <code>merge_partition</code> never rewrites them for pre-existing layers, only the <code>presence</code>/<code>counts</code> subdirectory gets widened in place, so only that subdirectory is a real, independent copy. Falls back to a real copy per file if linking itself fails (different filesystems). Verified: source files are byte-identical (checksummed) before/after a normal merge.</p>
<p>Merging an index against itself — literally the same directory passed twice as separate source arguments (e.g. <code>obikmer merge -o out IDX IDX --rename-duplicates</code>) — panics deep in the MPHF's rank-select structure (<code>common_traits::select_in_word</code>, <code>assertion failed: rank < self.count_ones()</code>), inside <code>MphfLayer::find</code> called from <code>merge_partition</code>'s "is this source kmer already in dst" check against the bootstrap-copied <code>dst_layers</code>. Root cause not identified; ruled out so far:</p>
<ul>
<li><strong>Not an empty-new-layer issue</strong>: a source contributing zero new kmers (verified both as the sole additional source and as a third, fully-redundant source in a 3-way merge) is handled correctly — no layer is created for it, no crash, exit 0.</li>
<li><strong>Not caused by the hard-link change above</strong>: verified via checksum that a normal (two distinct sources) merge leaves every source file byte-identical.</li>
</ul>
<p>Only reproduces when <code>sources[0]</code> and <code>sources[1]</code> are the exact same on-disk path opened as two independent <code>KmerIndex</code> handles — an artificial scenario (nobody merges an index with itself intentionally), deprioritized rather than investigated further for now. Does terminate cleanly rather than hang, since <code>PartitionRunner</code>'s panic propagation fix (see <code>architecture/numa_partition_runner.md</code>) now surfaces it as a normal process panic/exit 101 instead of a silent deadlock.</p>
<scriptid="__config"type="application/json">{"annotate":null,"base":"../..","features":[],"search":"../../assets/javascripts/workers/search.2c215733.min.js","tags":null,"translations":{"clipboard.copied":"Copied to clipboard","clipboard.copy":"Copy to clipboard","search.result.more.one":"1 more on this page","search.result.more.other":"# more on this page","search.result.none":"No matching documents","search.result.one":"1 matching document","search.result.other":"# matching documents","search.result.placeholder":"Type to start searching","search.result.term.missing":"Missing","select.version":"Select version"},"version":null}</script>
