format.rs file format constants, encode/decode helpers
layer_meta.rs LayerMeta (column metadata)
meta.rs matrix metadata
</code></pre></div>
<preclass="mermaid"><code>graph TD
views --> bitvec
views --> builder
views --> tempbitvec
views --> tempintvec
views --> bitmatrix
views --> intmatrix
format --> reader
format --> builder
reader --> intmatrix
reader --> tempintvec
builder --> intmatrix
builder --> tempintvec
bitvec --> tempbitvec
bitvec --> bitmatrix
tempintvec --> intmatrix
tempintvec --> bitmatrix
tempbitvec --> intmatrix
tempbitvec --> bitmatrix
colgroup --> intmatrix
colgroup --> bitmatrix
layer_meta --> bitmatrix
layer_meta --> intmatrix
meta --> bitmatrix
meta --> intmatrix</code></pre>
<hr/>
<h2id="compact-int-encoding">Compact int encoding</h2>
<p>All integer vectors use the same two-tier encoding regardless of storage backend.</p>
<p><strong>Primary array</strong> — one <code>u8</code> per slot:</p>
<ul>
<li>Values <strong>0–254</strong> are stored directly. No overhead.</li>
<li>Value <strong>255 is a sentinel</strong>: the slot's actual value is ≥ 255 and lives in the overflow store.</li>
</ul>
<p><strong>Overflow store</strong> — maps slot index to a <code>u32</code> value ≥ 255:</p>
<ul>
<li>In <code>PersistentCompactIntVecBuilder</code>: a <code>HashMap<usize, u32></code> in RAM.</li>
<li>In <code>PersistentCompactIntVec</code> (reader): a sorted <code>[(slot: u64, value: u32)]</code> array in the mmap, with a sparse L1-resident index for binary search.</li>
</ul>
<preclass="mermaid"><code>flowchart LR
slot --> P["primary[slot]: u8"]
P -->|"< 255"| V["value = byte (0–254)"]
P -->|"= 255 sentinel"| OV["overflow store"]
OV -->|"Builder"| HM["HashMap&lt;usize, u32&gt;\nin RAM"]
OV -->|"PersistentCompactIntVec"| SA["sorted [(slot,value)] in mmap\n+ sparse L1 index"]</code></pre>
<p><strong>Key property — sentinel 255 = +∞ on <code>u8</code>:</strong></p>
<ul>
<li><code>min(a, 255) = a</code> for all <code>a ≤ 254</code> → correct when only one side is overflow</li>
<li><code>max(a, 255) = 255</code> → correct sentinel when either side is overflow</li>
<li>Only the <strong>both-overflow</strong> case requires reading actual values from the overflow store.</li>
</ul>
<p>In practice, k (overflow count) ≪ n (total slots). Observed genomic data: ~0.07% of kmer slots are in overflow.</p>
<hr/>
<h2id="view-types">View types</h2>
<p>The previous trait hierarchy (<code>BitSlice</code>, <code>BitSliceMut</code>, <code>IntSlice</code>, <code>IntSliceMut</code>) has been replaced by two concrete zero-copy view structs with inherent methods. Views are <strong><code>Copy</code></strong> — passing them is free. All read operations live on these two types.</p>
<p><code>overflow_raw</code> contains <code>n_overflow</code> entries of <code>OVERFLOW_ENTRY_SIZE</code> bytes each, sorted by slot. The sort invariant is established at <code>close()</code>/<code>freeze()</code> time.</p>
<p><code>IntSliceViewIter<'a></code>: merge scan using <code>overflow_pos</code> index. Requires sorted overflow — guaranteed by the construction lifecycle.</p>
<p><strong>Builder <code>view()</code> vs reader <code>view()</code>:</strong><code>PersistentCompactIntVecBuilder</code> stores overflow as an unsorted <code>HashMap</code>, not raw bytes. Its <code>view()</code> returns an <code>IntSliceView</code> with <code>overflow_raw = &[]</code> and <code>n_overflow = 0</code>. This is intentional — the view is primarily useful after <code>freeze()</code>. During building, callers that need overflow use <code>overflow_entries()</code> directly.</p>
<p><code>PersistentBitVec</code> is the read-only type. <code>view()</code> returns a <code>BitSliceView<'_></code> over the mmap word array. Direct inherent methods delegate to the view: <code>count_ones()</code>, <code>count_zeros()</code>, <code>partial_jaccard_dist(&Self)</code>, <code>jaccard_dist(&Self)</code>, <code>hamming_dist(&Self)</code>.</p>
<p><code>BitIter<'a></code> — exported iterator for <code>PersistentBitVec::iter()</code>:</p>
<p><code>PersistentCompactIntVec</code> is the read-only type. <code>view()</code> returns an <code>IntSliceView<'_></code> over the mmap primary and overflow arrays. Inherent <code>iter()</code> is a merge scan (<code>Iter</code> struct). Inherent <code>sum()</code> and <code>count_nonzero()</code> use fast byte-scan helpers.</p>
<p><code>PersistentCompactIntVecBuilder</code> is the read-write type. Mutation methods on the builder fall into two categories:</p>
<p><strong><code>inc_present_fast</code> / <code>inc_predicate_fast</code> invariant:</strong> caller guarantees no counter reaches 255 during the operation (group size < 255 for <code>inc_present_fast</code>, or chunk size < 255 for <code>inc_predicate_fast</code>). Violation is caught by <code>debug_assert</code> in dev builds.</p>
<p>Cannot do byte max first — <code>max(255, b<255)=255</code> overwrites self's original overflow value. Pre-pass reads self's value at other's overflow slots before the byte pass.</p>
<divclass="highlight"><pre><span></span><code>Pre-pass O(k_other): for (slot, other_val) in other.overflow_entries():
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <ahref="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
<spanclass="w"></span><spanclass="k">fn</span><spanclass="w"></span><spanclass="nf">col_weights</span><spanclass="p">(</span><spanclass="o">&</span><spanclass="bp">self</span><spanclass="p">)</span><spanclass="w"></span><spanclass="p">-></span><spanclass="w"></span><spanclass="nc">Array1</span><spanclass="o"><</span><spanclass="kt">u64</span><spanclass="o">></span><spanclass="p">;</span><spanclass="w"></span><spanclass="c1">// sum per column</span>
<p><code>partial_kmer_counts</code> is overridden for count matrices to return <code>count_nonzero</code> per column (distinct kmers) rather than total count.</p>
<p><strong>Additivity rule:</strong> self-contained partials (<code>partial_bray</code>, <code>partial_euclidean</code>, <code>partial_threshold_jaccard</code>) can be element-wise summed across all <code>(partition, layer)</code> pairs. Normalised partials (<code>partial_relfreq_*</code>, <code>partial_hellinger</code>) require the <strong>global</strong><code>col_weights</code> (accumulated across all layers and all partitions) as parameter.</p>
<p><strong><code>partial_threshold_jaccard</code> returns <code>(inter, union)</code></strong> because <code>union[i,j]</code> depends on both columns simultaneously.</p>
<td>Mash distance, derived from <code>threshold_jaccard_dist_matrix(t)</code> — no separate partial</td>
</tr>
</tbody>
</table>
<h3id="bitpartials">BitPartials</h3>
<p>Required: <code>partial_jaccard() -> (Array2<u64>, Array2<u64>)</code>, <code>partial_hamming() -> Array2<u64></code>. Both additive across layers and partitions.</p>
<p>Provided finalisations also include <code>jaccard_dist_matrix()</code>, <code>hamming_dist_matrix()</code>, and <code>mash_dist_matrix(k)</code>.</p>
<h3id="mash-distance">Mash distance</h3>
<p><code>mash_dist_matrix</code>/<code>threshold_mash_dist_matrix</code> add no new additive primitive: both are a pointwise transform of the existing Jaccard distance matrix, per the Mash mutation-rate estimator (Fan <em>et al.</em> 2015; Marbl Lab 2026)<supid="fnref:Mash-distances-doc"><aclass="footnote-ref"href="#fn:Mash-distances-doc">1</a></sup><supid="fnref:Fan2015-mash-formula"><aclass="footnote-ref"href="#fn:Fan2015-mash-formula">2</a></sup>:</p>
<p><code>J ≤ 0</code> (i.e. <code>d_jaccard ≥ 1</code>, no shared k-mers) maps to <code>D = 1</code> (maximal distance) rather than the <code>ln</code> singularity at <code>J = 0</code>.</p>
<p><strong>All inter-function results use temp-file-backed types</strong> so the OS can page them out under memory pressure. This matters in practice: processing dozens of layers × hundreds of partitions in parallel would otherwise accumulate gigabytes of live anonymous memory.</p>
<h3id="lifecycle">Lifecycle</h3>
<divclass="highlight"><pre><span></span><code>TempCompactIntVecBuilder::new(n) → writable mmap in TempDir
<spanclass="w"></span><spanclass="n">_temp</span><spanclass="p">:</span><spanclass="w"></span><spanclass="nc">TempDir</span><spanclass="p">,</span><spanclass="w"></span><spanclass="c1">// dropped after vec</span>
<p><code>TempCompactIntVec</code>: read access via <code>get(slot)</code>, <code>sum()</code>, <code>iter()</code>, <code>view() -> IntSliceView<'_></code>.</p>
<p><code>TempCompactIntVecBuilder</code>: full delegation to inner <code>PersistentCompactIntVecBuilder</code> — all bulk computation methods (<code>inc_present_fast</code>, <code>inc_predicate_fast</code>, <code>add</code>, <code>min</code>, <code>max</code>, <code>diff</code>, <code>mask_with</code>) are exposed as <code>pub(crate)</code>.</p>
<p>Defined <strong>once at the index level</strong> from column metadata. Valid in all matrices of all layers and partitions — column structure is identical across the entire hierarchy; only rows (kmer slots) are partitioned.</p>
<h3id="composition-axis">Composition axis</h3>
<ul>
<li><strong>Across partitions</strong>: kmer space is partitioned → partial results <strong>concatenated</strong> (disjoint kmer ranges).</li>
<li><strong>Across layers</strong>: same kmer space, different counts → partial results <strong>aggregated</strong> (add, OR, etc.).</li>
</ul>
<h3id="matrixgroupops">MatrixGroupOps</h3>
<p>Five required primitives + two default methods derived from them. All return temp-file-backed types.</p>
<p>Implemented for both <code>PersistentCompactIntMatrix</code> and <code>PersistentBitMatrix</code>.</p>
<p>For <strong>bit matrices</strong>: values are 0/1, so <code>partial_group_sum</code> = <code>partial_group_presence_count(g, 1)</code>; <code>partial_group_min</code> is AND (set first column then mask-with remaining); <code>partial_group_max</code> is OR via <code>partial_group_any</code> + <code>inc_present</code>.</p>
<p><strong><code>partial_group_presence_count</code> — chunking for large groups:</strong></p>
<p>When <code>g.indices.len() < 255</code>: per-slot counts stay within <code>u8</code> range. Use <code>inc_present_fast</code> (bit) or <code>inc_predicate_fast(col_view(c), |v| v >= threshold)</code> (int) — raw u8 increment, no overflow entry written.</p>
<p>When <code>g.indices.len() ≥ 255</code>: process in chunks of 254 columns, accumulate via <code>.add(chunk_frozen.view())</code>.</p>
<p><strong><code>partial_group_min</code> (int matrix)</strong>: copy first column via <code>.add(col_view(first))</code> (start from 0 ⇒ copy), then <code>.min(col_view(c))</code> for remaining.</p>
<p><strong><code>partial_group_max</code> (int matrix)</strong>: <code>.max(col_view(c))</code> for all columns (start from 0 ⇒ first column acts as copy).</p>
<p><strong><code>partial_group_any</code></strong> uses <code>or_where</code> on <code>TempBitVecBuilder</code> (two-pass: primary bytes then overflow entries).</p>
<p><strong><code>partial_group_all</code> / <code>partial_group_none</code></strong> (default): call <code>partial_group_presence_count</code>, then iterate slots to produce the bit result. O(n) extra pass, not chunked.</p>
<spanclass="k">fn</span><spanclass="w"></span><spanclass="nf">add_col_from_bit</span><spanclass="p">(</span><spanclass="o">&</span><spanclass="k">mut</span><spanclass="w"></span><spanclass="bp">self</span><spanclass="p">,</span><spanclass="w"></span><spanclass="n">src</span><spanclass="p">:</span><spanclass="w"></span><spanclass="kp">&</span><spanclass="nc">TempBitVec</span><spanclass="p">)</span><spanclass="w"></span><spanclass="p">-></span><spanclass="w"></span><spanclass="nc">io</span><spanclass="p">::</span><spanclass="nb">Result</span><spanclass="o"><</span><spanclass="p">()</span><spanclass="o">></span><spanclass="w"></span><spanclass="c1">// bit → 0/1 u32</span>
</code></pre></div>
<p><code>add_col_from</code> copies the temp file to the matrix directory and increments <code>n_cols</code>; <code>close()</code> writes <code>meta.json</code> with the final column count. No separate <code>write_meta</code> step needed.</p>
<h3id="mask_with">mask_with</h3>
<p>Direct method on <code>PersistentCompactIntVecBuilder</code> (and delegation via <code>TempCompactIntVecBuilder</code>). Zeros every slot where the corresponding mask bit is 0. Iterates only zero bits — O(n_zeros), O(1) when mask is all-ones.</p>
<divclass="highlight"><pre><span></span><code>for (w_idx, word) in mask.words():
if word == u64::MAX: continue // skip all-ones words
zeros = !word
while zeros != 0:
bit = trailing_zeros(zeros)
s = w_idx * 64 + bit
if primary[s] != 0: set(s, 0) // clears overflow entry too
zeros &= zeros − 1
</code></pre></div>
<p>Terminal operation for Filter (retain only selected kmer slots in a count vector) and Select (positional selection without MPHF).</p>
<divclass="footnote">
<hr/>
<ol>
<liid="fn:Mash-distances-doc">
<p>Marbl Lab. (2026). <ahref="https://mash.readthedocs.io/en/latest/distances.html">Mash distance</a>. <aclass="footnote-backref"href="#fnref:Mash-distances-doc"title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<liid="fn:Fan2015-mash-formula">
<p>Fan, H., Ives, A.R., Surget-Groba, Y. & Cannon, C.H. (2015). <ahref="https://doi.org/10.1186/s12864-015-1647-5">An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data</a>. <em>BMC Genomics</em>, 16. <aclass="footnote-backref"href="#fnref:Fan2015-mash-formula"title="Jump back to footnote 2 in the text">↩</a></p>
<scriptid="__config"type="application/json">{"annotate":null,"base":"../..","features":[],"search":"../../assets/javascripts/workers/search.2c215733.min.js","tags":null,"translations":{"clipboard.copied":"Copied to clipboard","clipboard.copy":"Copy to clipboard","search.result.more.one":"1 more on this page","search.result.more.other":"# more on this page","search.result.none":"No matching documents","search.result.one":"1 matching document","search.result.other":"# matching documents","search.result.placeholder":"Type to start searching","search.result.term.missing":"Missing","select.version":"Select version"},"version":null}</script>
