Push kkkmrtvotrsu #77

Merged
coissac merged 10 commits from push-kkkmrtvotrsu into main 2026-10-13 07:54:50 +00:00
64 changed files with 1374 additions and 2330 deletions
Showing only changes of commit 95bbc0b2ae - Show all commits

No files matched your search

+1 -1
View File
@@ -1119,7 +1119,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1370,7 +1370,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1128,7 +1128,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1130,7 +1130,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1140,7 +1140,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
File diff suppressed because it is too large. Load diff
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1485,8 +1485,9 @@ this pipeline.</p>
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
<a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats") as faster
for single-row access (query) and slower for column-oriented access
(phylo <code>--metric</code>).
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
invocation) — there was no dense/sparse regression check.</li>
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -579,7 +579,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1388,7 +1388,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+2 -2
View File
@@ -1504,7 +1504,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -2208,7 +2208,7 @@
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;&gt;</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;</span><span class="p">])</span>
</code></pre></div>
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
<hr />
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\&lt;S&gt; and aggregation traits</h2>
<p><code>LayeredStore&lt;S&gt;</code> is a generic aggregation wrapper over <code>Vec&lt;S&gt;</code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1348,7 +1348,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1332,7 +1332,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1343,7 +1343,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+219 -3
View File
@@ -494,7 +494,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -632,6 +632,46 @@
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#sparse-matrix-formats">
<span class="md-ellipsis">
Sparse matrix formats
</span>
</a>
<nav aria-label="Sparse matrix formats" class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a class="md-nav__link" href="#persistentsparsebitmatrix">
<span class="md-ellipsis">
PersistentSparseBitMatrix
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
<span class="md-ellipsis">
nonzero_iter — the one traversal per format
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
<span class="md-ellipsis">
PersistentSparseCompactIntMatrix
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
<span class="md-ellipsis">
@@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
<tr>
<td><code>Sparse</code></td>
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
</tr>
<tr>
<td><code>Implicit</code> (bit only)</td>
@@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
</tbody>
</table>
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>
@@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
<hr/>
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
k-mers — present in only a handful of genome columns out of dozens to
thousands): a row-major, deduplicated alternative to the dense column-major
formats above, built once per layer like the rest of the pipeline (never the
whole multi-billion-row index at once). Implements the exact-duplicate
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
Structures for Collections of Sets" (2025,
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
hierarchy.</p>
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
can be large in absolute terms even when multi-genome rows are a small
<em>fraction</em> of all rows, and a shared array would force every row, singletons
included, to pay the wider width), and a deduplicated dictionary of distinct
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
runtime-parameterised bit width, width <code>0</code> included — needed once an
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
trait unifying dense and sparse at the call sites that need both
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
<table>
<thead>
<tr>
<th></th>
<th>dense</th>
<th>sparse</th>
<th>ratio</th>
</tr>
</thead>
<tbody>
<tr>
<td>on-disk size</td>
<td>328.1MB</td>
<td>43.7MB</td>
<td><strong>7.5x</strong> smaller</td>
</tr>
<tr>
<td>build time / peak RSS</td>
<td>—</td>
<td>4.26s / 628MB</td>
<td>per-layer, in-memory construction</td>
</tr>
<tr>
<td>row access, sequential (2M reads)</td>
<td>43ns/row</td>
<td>32ns/row</td>
<td>sparse faster (smaller structure, better cache fit)</td>
</tr>
<tr>
<td>row access, random (2M reads)</td>
<td>409ns/row</td>
<td>85ns/row</td>
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
</tr>
<tr>
<td>column access, one full column (30.2M rows)</td>
<td>11.5ms</td>
<td>993ms</td>
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
</tr>
</tbody>
</table>
<p>The row-access wins (compactness was the stated design goal, not speed) come
from dense's genome-major layout scattering a single row read across a much
bigger file — costlier than sparse's rank/select/varint decode once the file
is this much smaller. Column access is the flip side of the same layout
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
actually matters (<code>query</code>).</p>
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
re-derived its low-bits width from the persisted low-vector's own width
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
builder used to reject true width <code>0</code>), silently doubling every reopened
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
the dictionary path (a genuine empty entry).</p>
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
variant beyond what's described below, BRWT-style column-correlation
exploitation, and a row-major rewrite of
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
<code>Sparse</code> — needs its own design pass (how it plugs into
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
not just "port the loop".</p>
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
<p><code>fn nonzero_iter&lt;'a&gt;(&amp;'a self, slots: &amp;'a [usize]) -&gt; Box&lt;dyn Iterator&lt;Item = (usize, usize, u32)&gt; + 'a&gt;</code>
(item: index into <code>slots</code>, column, value) is the single foundational
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
and a <code>for_each</code>-style callback are both trivial, free consumers of it
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
iterator and scatters into <code>out[][]</code>), instead of two independently
hand-maintained traversals that can silently diverge. <code>Box&lt;dyn Iterator&gt;</code>,
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
genuinely different concrete types and the method isn't on a
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
<code>nonzero_iter</code> call, not per cell.</p>
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
<code>Packed</code> loop columns, collecting each column's hits via
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
something borrowing its own captured sort permutation across separate calls
without either boxing per-column or fighting the borrow checker — eager
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&amp;i| (i, 0, 1))</code>).</p>
<p>This closed a real dispatch bug by construction, not by patching it:
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
naive per-<code>(column, slot)</code> loop instead of delegating to
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
— two hand-written traversals for the same format, free to drift apart, and
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
left to get right. <code>KmerPartition::query_partition_with</code>
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&amp;slot_list)</code>
call per layer, format-agnostic. Dense/sparse query performance parity
confirmed on real data (previously sparse 30-50% slower than dense,
reproducibly; now within ~1-3% either way — noise level).</p>
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
the bit side, two rows can share the same non-zero column set — same
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
in <code>sparse_intmatrix.rs</code>, decodes each row once via
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
(every <code>u32</code> is <code>≥ 0</code>).</p>
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
introduction, never caught because no test had compared <code>Sparse</code>'s raw
<code>partial_*</code> output against dense on real data — only the diagonal-blind
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
fixed at the source:</p>
<ol>
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
by construction, so the diagonal needs its own case.</li>
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
(symmetric-difference size) — the two only coincide when
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
data can easily satisfy by accident.</li>
</ol>
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
<h3 id="columnweights">ColumnWeights</h3>
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1293,7 +1293,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1305,7 +1305,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1563,7 +1563,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1405,7 +1405,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1388,7 +1388,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1254,7 +1254,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1382,7 +1382,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1326,7 +1326,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1232,7 +1232,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1354,7 +1354,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1228,7 +1228,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1314,7 +1314,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1227,7 +1227,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
Binary file not shown.
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1199,7 +1199,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+594 -4
View File
@@ -932,6 +932,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
<span class="md-ellipsis">
Automatic α estimation (--gamma-shape auto)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -943,6 +954,45 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
α-estimation literature survey and design discussion (2026-09-11)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
</span>
</a>
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
<span class="md-ellipsis">
Explicit --session (implemented 2026-09-12)
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -1691,7 +1741,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -2268,6 +2318,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
<span class="md-ellipsis">
Automatic α estimation (--gamma-shape auto)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -2279,6 +2340,45 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
α-estimation literature survey and design discussion (2026-09-11)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
</span>
</a>
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
<span class="md-ellipsis">
Explicit --session (implemented 2026-09-12)
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
by the user (<code>--gamma-shape &lt;alpha&gt;</code>), not estimated by ML. A modifier on
existing values, not a separate enum arm per distance.</p>
formula as the base correction, weighted by a shape parameter <code>α</code> either
supplied by the user (<code>--gamma-shape &lt;alpha&gt;</code>) or estimated from the data
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
"Automatic α estimation" below). A modifier on existing values, not a
separate enum arm per distance.</p>
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &amp;
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
different gamma distance, estimating a from data. Wilson et al. (1989)
recently used a distance with a = 1/2 for restriction-site data of
mitochondrial DNA in hominoids."</em> — i.e. Jin &amp; Nei explicitly punt
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
read/verified here. The estimator below is therefore <strong>not</strong> "Jin &amp; Nei's
method" under any framing — that attribution (present in an earlier
revision of this section) was wrong, not just under-cited.</p>
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
as an independent method-of-moments estimator, unrelated to any specific
published procedure: pools substitution counts by <strong>partition</strong> rather
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
<code>PairwiseTally</code> (no second scan).</p>
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
every locus in a partition — the classical Poisson-Gamma/negative-binomial
mixture, the general identity behind gamma-rate-heterogeneity corrections,
also behind Uzzell &amp; Corbin's (1971) original observation that substitution
counts across sites/regions are over-dispersed relative to Poisson):</p>
<div class="arithmatex">\[
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
\]</div>
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
Poisson term before attributing what's left to <code>α</code>:</p>
<div class="arithmatex">\[
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
\]</div>
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
back to the uncorrected formula, warns) when fewer than 2 partitions have
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin &amp; Nei 1990 directly
for the base <code>+Γ</code> formula), this estimator is derived from first
principles (the general Poisson-Gamma/negative-binomial identity) with no
primary-source procedure behind it at all — not Jin &amp; Nei's (confirmed
above), and Wilson et al. (1989), the paper they point to instead, hasn't
been read/verified either. Mathematically self-consistent (re-derived and
checked, not guessed), but a from-scratch method, not a literature
implementation. If <code>--gamma-shape</code> needs a value with a literature
pedigree rather than an estimated one, Jin &amp; Nei's own stated default,
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
doc comment): that filter excludes individual saturated <em>pairs</em> from a
composition estimate computed once at the very end, from the complete
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
one partition at a time, while <code>PairwiseTally</code> is still being built — long
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
is known. <code>--exclude-genome</code> isn't applied either, matching
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
mechanical substitution but weren't independently checked against an
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
silently assumed correct.</p>
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
(previous section), triggered by the user pointing out that the literature
conflates two distinct problems: <em>estimating α from an alignment</em> versus
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin &amp; Nei (1990)
itself is squarely in the second camp — α is a user-supplied parameter
there, not something their paper estimates (confirmed by reading the paper
directly, see previous section). This section catalogs the actual
α-estimation literature and records which approaches were considered and
why each was or wasn't adopted.</p>
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>Reference</th>
<th>Needs a tree?</th>
<th>Estimates "true" α or a task-specific one?</th>
<th>Cost</th>
</tr>
</thead>
<tbody>
<tr>
<td>ML, continuous Γ (foundational)</td>
<td><strong>Gu, Fu &amp; Li (1995)</strong>, MBE 12:546–557 — read in full</td>
<td>yes, topology fixed a priori</td>
<td>true α (+ θ jointly, invariant+Γ)</td>
<td>very high, ≤5–6 taxa in practice</td>
</tr>
<tr>
<td>ML, discrete Γ</td>
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
<td>yes</td>
<td>true α</td>
<td>high</td>
</tr>
<tr>
<td>Parsimony-based ML on inferred changes</td>
<td>Yang &amp; Kumar (1996), MBE 13:650</td>
<td>yes, equal branch lengths assumed</td>
<td>true α, biased when branches unequal</td>
<td>low</td>
</tr>
<tr>
<td>Corrected substitutions/site → NB fit</td>
<td><strong>Gu &amp; Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
<td>yes, + ancestral reconstruction</td>
<td>true α, close to ML accuracy</td>
<td>low (given a tree)</td>
</tr>
<tr>
<td>Topology-optimizing grid search</td>
<td><strong>Guindon &amp; Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
<td>low–moderate (grid × tree build)</td>
</tr>
<tr>
<td>Free (non-parametric) rate distribution</td>
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
<td>yes</td>
<td>tests the Γ assumption itself</td>
<td>high</td>
</tr>
<tr>
<td>Parsimony counts/site → NB fit</td>
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
<td>low (given a tree)</td>
</tr>
<tr>
<td>Poisson-Gamma/negative-binomial, foundational</td>
<td>Uzzell &amp; Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
<td>unknown — likely also count-per-site/tree-based, unverified</td>
<td>true α (indirect, via NB overdispersion)</td>
<td>unknown</td>
</tr>
<tr>
<td>Bayesian posterior over α</td>
<td>BPP/MrBayes/BEAST2/RevBayes</td>
<td>yes</td>
<td>posterior, not point estimate</td>
<td>very high</td>
</tr>
</tbody>
</table>
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
revision of this table filed Wakeley (1993) under "no tree needed" —
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou &amp; Nei
1987, on the real mtDNA data; coalescent-simulated for the simulation
study), reconstruct ancestral states at internal nodes by Fitch (1971)
parsimony, count the <em>minimum</em> number of changes per site this implies,
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
Johnson &amp; Kotz 1969) to the resulting per-site count distribution — the
same general "tree + ancestral reconstruction + per-site counts + NB fit"
shape as Gu &amp; Zhang (1997), just using raw parsimony counts directly
rather than Gu &amp; Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
mean <em>and</em> (more severely) the variance of per-site change counts, which
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &amp;
Zhang's own citation of Wakeley for exactly this bias, and with the
general "parsimony overestimates α" pattern noted throughout this
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
per data set, <code>0.47</code> combined — despite the conservative bias, still
solidly in the "strong heterogeneity" range, which is the paper's own
point: the method is biased but usefully conservative, not useless.</p>
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
this table's tree-dependent rows. It shares only the <em>abstract
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&amp;Zhang, applied via a
genuinely different, tree-free and ancestral-reconstruction-free route:
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
instead of per-site counts pooled across an alignment's columns after
ancestral state reconstruction. As far as this survey has established
(Uzzell &amp; Corbin not read, so not fully ruled out), <strong>no tree and no
ancestral reconstruction of any kind</strong> appears to be a genuinely
distinguishing property of <code>PartitionDispersion</code>, not something it
inherits from prior art — consistent with it being an independent
derivation rather than a literature implementation (see previous
section's caveat, now on firmer footing).</p>
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
Raised directly by the user after reading Gu, Fu &amp; Li (1995), which states
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
of sequences is &lt;3."</em> This is a real non-identifiability result, not a
practical inconvenience: with only 2 sequences, a site's history is
summarized by one binary observation (differ / don't differ), collapsed
across all sites into two aggregate numbers (transition and transversion
proportions) — divergence time and among-site rate variance are
confounded in that pair of numbers, with no way to separate them. Every
method surveyed above (Gu,Fu&amp;Li, Yang, Wakeley, Gu&amp;Zhang) needs ≥3
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
multiple independent looks at the <em>same</em> site's rate across different
lineages, which is what identifies its variance.</p>
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
the same quantity. The literature above estimates rate variation <strong>across
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
variation <strong>across partitions, pooled over every genome pair in the
index</strong> — each genome pair stands in for one independent draw of a
partition's relative rate, the same identifying role multiple lineages
play in the tree-based methods, just substituting "genome pair" for
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
in the hundreds here) all contributing to every partition's pooled count,
the ≥3-samples identifiability requirement is met by the pair count, not
by tree depth — closer in spirit to Jin &amp; Nei's own original motivating
scenario (comparing several genomic <em>regions</em> across one fixed panel of
taxa) than to single-tree site-rate estimation.</p>
<p>This substitution carries its own assumption, which must be named rather
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
reasonably stable <strong>across genome pairs</strong>, not just across sites within
one pair — the direct analogue of the "no lineage-specific rate variation,
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
just moved from lineage→pair and site→partition. If this breaks — e.g. an
index mixing very closely related and very divergent genome pairs, where a
partition's saturation behavior differs qualitatively between the two —
the estimate could be misled in a way a tree-based method would at least
have the topology to detect and a tree-free, pool-everything method
cannot. Not yet tested against real data with strongly heterogeneous
pairwise divergence; worth keeping in mind as the main open validity
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
this whole discussion started from.</p>
<p><strong>Gu, Fu &amp; Li (1995), read in full — the foundational ML method, not a
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
and derives the exact joint likelihood over nucleotide <em>configurations</em>
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
specific topology's branching structure (their eq. 32–34), with branch
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
free parameters). The site-rate integral has the same closed form
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &amp;
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
rather than a whole-tree one). Explicitly stated by the authors as
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
candidate for obikmer's typical genome counts. Notable contribution worth
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
term at all (no invariant-site component), so this confound doesn't arise
for it the same way, but it's worth remembering if an invariant+Γ variant
is ever considered.</p>
<p><strong>Gu &amp; Zhang (1997), read in full — rejected for now.</strong> Their procedure:
(1) a tree with least-squares branch lengths must already exist; (2)
ancestral states are reconstructed at every internal node (they use Zhang &amp;
Nei 1997's likelihood method, but note plain parsimony would work, just
less accurately); (3) per site, branches are split into "changed"/"unchanged"
given the ancestral reconstruction, and the <em>expected</em> substitution count
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
(or eq. 12 for the generalized model) numerically — a root-find per site,
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
integers) are fit to a negative binomial (Uzzell &amp; Corbin 1971) by ML to
get α — again no closed form, numerical optimization. This is a real
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
own answer to this chicken-and-egg problem is an iterative
distance→tree→α→distance loop, which they themselves flag as expensive.
Adopting this would mean a new ancestral-reconstruction subsystem (even a
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
to build without a concrete need beyond "closer to Gu&amp;Zhang's own
simulated accuracy than our moment estimator."</p>
<p><strong>Guindon &amp; Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
the α-corrected distance matrix; for every internal branch, group taxa into
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
(same order as the tree-building step itself, so "free" relative to it).</p>
<p>Central finding, <strong>the opposite direction from an initial misreading in
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
heterogeneity (i.e., picking a larger α than reality) reduces the variance
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
no correction at all is topologically best). This is an <em>empirical
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
guarantee would be another unverified-claim mistake of the kind this whole
discussion has been correcting.</p>
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
estimator rather than needing an independent "true α" source — this also
sidesteps Gu &amp; Zhang's tree-first dependency problem entirely, since
Guindon-Gascuel's own grid already builds a tree per candidate. Still
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
Searched crates.io directly (API query for "bionj": 0 results) and read the
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
manipulates trees but has no distance-matrix reconstruction — no NJ, no
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
how branch lengths are weighted during agglomeration (accounts for
distance-estimate variance, not just the sum), a modest delta over an
existing NJ implementation rather than a rewrite, but would need to be
written in-house if the Guindon-Gascuel integration above is ever pursued
(their own paper notes other tree-building methods gave similar results in
their simulations, so plain NJ may be an acceptable substitute if BIONJ
proper is judged not worth the effort).</p>
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
parameters when requested together in one <code>obikmer phylo</code> invocation
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
Before this fix, each independently called <code>sample_index</code>, and
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
generator that advances across calls, not reseeded each time</strong> — so the
second call silently drew a <em>different</em> random sample of sites than the
first, even with identical parameters. Verified empirically (throwaway
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
produced visibly different distance matrices from the same index/params.
This defeated the actual point of combining these flags — running several
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
one <em>identical</em> site selection for direct comparison.</p>
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
<code>alpha</code> resolution, final matrix build — previously all inline in the
<code>snp_distance()</code> free function) was factored into
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option&lt;SankoffBundle&gt;</code> alive past
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
requested, calls <code>bundle.snp_distance(...)</code> instead of
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
are driven by the same <code>args.*</code> fields, the parameters trivially always
match when both fire; no runtime "do the params match" check needed.
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
the same tiny test index.</p>
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
<p>The in-process fix above only covered reuse <em>within one command</em>.
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
anywhere in it. <code>Session::open(dir, params: &amp;[u8], force: bool)</code> handles
directory lifecycle and the params-conflict check against an opaque byte
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -&gt; Option&lt;Mmap&gt;</code>
handle a checksummed artifact cache. <code>obikphylo</code>
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
and hands them to <code>obiksession</code>, which never sees their type.</p>
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
original zero-copy pitch, scoped down deliberately: artifacts are restored
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
(zero-copy over the <code>mmap</code>). True zero-copy would require every
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
work generically over <code>Archived&lt;PairwiseTally&gt;</code> as well as the owned type
— a separate, larger change not needed to get the actual win (skipping
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
preferring one side, unless <code>--session-force</code> is given (which discards the
directory's cached artifacts and starts over under the new params) —
implements exactly the "explicit escape hatch, not silent override"
decision from the design discussion. <code>obiksession</code> itself doesn't decode
either side's bytes to produce a field-by-field diff (true to "opaque
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
currently a single generic sentence listing which flags could be the
cause, not a computed diff — an honest v1 simplification, not a limitation
of <code>obiksession</code> itself.</p>
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
written up as a corner cut ("per-entry locking was sketched, not
implemented"). On actually working through the concrete race, whole-session
locking turned out to be the <em>correct</em> choice, not a simplification:
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
directory. If the directory lock were only held briefly (during
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
other processes from touching <em>different</em> artifacts), a second process
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
sequence impossible: a <code>force</code> reset must wait for every currently-open
<code>Session</code> on that directory to be dropped first. The concurrency this
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
of the same session at once) isn't exercised by anything today — only one
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
uses sessions at all.</p>
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
layer) pairs, in its own deterministic iteration order, entirely — no
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
argument (that pair's 0-based position in the same order), so a caller can
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
in-progress <code>SnpAlignment</code>) to the session every
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
how much completed scanning a crash can lose, without paying a full
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
artifact records how many layers are reflected in the checkpoint, with a
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
processed (distinct from any real layer count, so "finished" is never
confused with "stopped after N layers where N happens to equal the total").</p>
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
disk, then re-ran the same command and confirmed it resumed from layer 8
and completed successfully — 0-effort verification that the mechanism
itself is real, not just plausible-sounding.</p>
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
and shouldn't be.</strong> The output matrix from the resumed run differed from
an uninterrupted control run against the same index/params. Root cause:
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
OS-seeded generator with no continuity across a process boundary — the
layers processed before a kill and the layers processed after a resume
come from two independent random streams. An earlier revision of this doc
(and of the code comments) claimed resume "reproduces the exact same
sample an uninterrupted run would have" — that's wrong, and was corrected
after the user pointed out the actual reasoning error: true randomness has
no notion of continuity in the first place, so there was never a
correctness requirement for a resumed run to match an uninterrupted one
bit-for-bit. The sample produced after a resume is exactly as legitimate a
random draw as one from an uninterrupted run — just a <em>different</em> one,
same as running the command twice without <code>--session</code> already gives two
different samples today. That variance is explicitly wanted (see below),
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
corrected to state this plainly instead of the false reproducibility claim.</p>
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
results, and, if applied everywhere, making <em>any</em> two runs with identical
parameters produce identical samples). The user explicitly declined this
for now: repeated runs are relied on to measure sampling variance, and a
<code>--seed</code> flag would need its own explicit design if wanted later — this is
not an accidental gap, it's a stated preference to revisit only if asked.</p>
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
the same <code>session: Option&lt;&amp;Session&gt;</code> and uses the <em>same</em> artifact names
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
produces both, so they can never drift out of sync). Consequence verified
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
same-command reuse, exactly the synergy this design aimed for.</p>
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
cached or fresh, exactly as planned.</p>
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
once and subsample <em>from</em> it on each run instead of one session per exact
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
parameter tuple, <code>--subsample</code> included.</p>
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
<p><strong>Implemented.</strong> The primary distance-matrix output
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1199,7 +1199,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1243,7 +1243,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
File diff suppressed because it is too large. Load diff
@@ -12,8 +12,9 @@ this pipeline.
by `verify_presence`/`verify_merge_presence`, which only check index
*content* against the `.npz` truth, never the query API.
- `pack --sparse` produces a presence-matrix format documented (see
[siblings.md](../architecture/siblings.md)) as faster for single-row
access (query) and slower for column-oriented access (phylo `--metric`).
[obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster
for single-row access (query) and slower for column-oriented access
(phylo `--metric`).
`global_index_presence/` built by `merge_presence.sh` is always packed
dense (packing is a stage inside `merge`, not a separate `pack`
invocation) — there was no dense/sparse regression check.
+1 -1
View File
@@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
```
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats".
---
+158 -3
View File
@@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche
|---|---|---|
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
`col_view(c)` returns the appropriate view directly:
```rust
@@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view
---
## Sparse matrix formats
Motivated by real k-mer-family data being extremely row-sparse (most rows —
k-mers — present in only a handful of genome columns out of dozens to
thousands): a row-major, deduplicated alternative to the dense column-major
formats above, built once per layer like the rest of the pipeline (never the
whole multi-billion-row index at once). Implements the exact-duplicate
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
Structures for Collections of Sets" (2025,
`biblio/Alanko et al. - Compact Data Structures for Collections of
Sets.pdf`) — a plain dedup dictionary, not their full subset-containment
hierarchy.
### `PersistentSparseBitMatrix`
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))`
bits — kept apart from the singleton array because `n_distinct_multi_sets`
can be large in absolute terms even when multi-genome rows are a small
*fraction* of all rows, and a shared array would force every row, singletons
included, to pay the wider width), and a deduplicated dictionary of distinct
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
runtime-parameterised bit width, width `0` included — needed once an
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
(rank1/rank0/select1 on top of the crate's existing `count_ones`, via
`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix`
(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the
trait unifying dense and sparse at the call sites that need both
interchangeably; column-oriented methods (`col`, `col_view`, the
`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic
there (see the `PersistentBitMatrix::open` note above).
Measured on real data (one layer, 30.2M rows, 91 genomes):
| | dense | sparse | ratio |
|---|---|---|---|
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape |
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
The row-access wins (compactness was the stated design goal, not speed) come
from dense's genome-major layout scattering a single row read across a much
bigger file — costlier than sparse's rank/select/varint decode once the file
is this much smaller. Column access is the flip side of the same layout
choice; `nonzero_iter` (below) is what removes it from the hot path that
actually matters (`query`).
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
re-derived its low-bits width from the persisted low-vector's own width
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
builder used to reject true width `0`), silently doubling every reopened
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be
recorded as a singleton at genome `0`, indistinguishable on read-back from a
*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through
the dictionary path (a genuine empty entry).
Still deferred: full Alanko et al. subset-hierarchy compression (only the
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
variant beyond what's described below, BRWT-style column-correlation
exploitation, and a row-major rewrite of
`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a
per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s
`--metric` distance-matrix path stops paying the column-access penalty on
`Sparse` — needs its own design pass (how it plugs into
`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`),
not just "port the loop".
### `nonzero_iter` — the one traversal per format
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
(item: index into `slots`, column, value) is the single foundational
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
and a `for_each`-style callback are both trivial, free consumers of it
(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the
iterator and scatters into `out[][]`), instead of two independently
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>`,
not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are
genuinely different concrete types and the method isn't on a
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
`nonzero_iter` call, not per cell.
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
buffered one row at a time (`std::iter::from_fn` over the existing
`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/
`Packed` loop columns, collecting each column's hits via
`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted`
(a `filter`/`filter_map` over the vector-level `get()`, no new state
machine — `std`'s own `Filter` iterator already is one) into one `Vec` via
`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return
something borrowing its own captured sort permutation across separate calls
without either boxing per-column or fighting the borrow checker — eager
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`).
This closed a real dispatch bug by construction, not by patching it:
`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a
naive per-`(column, slot)` loop instead of delegating to
`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse`
— two hand-written traversals for the same format, free to drift apart, and
they had (`obikphylo`'s own cache layer had independently built a parallel
enum wrapper that *did* dispatch correctly, evidence the bug was worked
around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix`
rewritten to drain `nonzero_iter`, there is only one traversal per format
left to get right. `KmerPartition::query_partition_with`
(`obikindex/src/partition/query_layer.rs`) — previously column-major
(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
call per layer, format-agnostic. Dense/sparse query performance parity
confirmed on real data (previously sparse 30-50% slower than dense,
reproducibly; now within ~1-3% either way — noise level).
### `PersistentSparseCompactIntMatrix`
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
"which columns are non-zero" support, values **not** deduplicated (unlike
the bit side, two rows can share the same non-zero column set — same
`dict_id` — while carrying different counts, so the dict-multiplicity
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
in `sparse_intmatrix.rs`, decodes each row once via
`for_each_cell_in_row`, nests over that row's own co-present columns) —
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since
`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`.
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
(every `u32` is `≥ 0`).
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
introduction, never caught because no test had compared `Sparse`'s raw
`partial_*` output against dense on real data — only the diagonal-blind
`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested),
fixed at the source:
1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
`col_weights_and_pair_counts`'s `inter` never pairs a column with itself
by construction, so the diagonal needs its own case.
2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed
`total - union` (rows where *neither* column is present) instead of the
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
(symmetric-difference size) — the two only coincide when
`col_weights[i] + col_weights[j] == total`, which small/synthetic test
data can easily satisfy by accident.
## Aggregation traits (matrix level)
### ColumnWeights
+1 -1
View File
@@ -60,7 +60,7 @@ nav:
- Architecture:
- Sequences: architecture/sequences/invariant.md
- Kmer index: architecture/index_architecture.md
- Sibling annex (discussion): architecture/siblings.md
- Sibling annex: architecture/siblings.md
- NUMA-aware worker pools: architecture/numa_worker_pools.md
- NUMA-aware partition runner: architecture/numa_partition_runner.md