Implement sparse matrix optimizations and update architecture documentation

This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:52:37 +02:00
1 parent fdfc1eb3dd
commit 95bbc0b2ae
64 files changed
+1374 -2330

No files matched your search

+1 -1
View File
@@ -1119,7 +1119,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1370,7 +1370,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1128,7 +1128,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1130,7 +1130,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1140,7 +1140,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
File diff suppressed because it is too large. Load diff
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1485,8 +1485,9 @@ this pipeline.</p>
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
<a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats") as faster
for single-row access (query) and slower for column-oriented access
(phylo <code>--metric</code>).
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
invocation) — there was no dense/sparse regression check.</li>
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -579,7 +579,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1388,7 +1388,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+2 -2
View File
@@ -1504,7 +1504,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -2208,7 +2208,7 @@
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nb">Vec</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;&gt;</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">bool</span><span class="o">&gt;</span><span class="p">])</span>
</code></pre></div>
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
<hr />
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\&lt;S&gt; and aggregation traits</h2>
<p><code>LayeredStore&lt;S&gt;</code> is a generic aggregation wrapper over <code>Vec&lt;S&gt;</code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1348,7 +1348,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1332,7 +1332,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1343,7 +1343,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+219 -3
View File
@@ -494,7 +494,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -632,6 +632,46 @@
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#sparse-matrix-formats">
<span class="md-ellipsis">
Sparse matrix formats
</span>
</a>
<nav aria-label="Sparse matrix formats" class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a class="md-nav__link" href="#persistentsparsebitmatrix">
<span class="md-ellipsis">
PersistentSparseBitMatrix
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
<span class="md-ellipsis">
nonzero_iter — the one traversal per format
</span>
</a>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
<span class="md-ellipsis">
PersistentSparseCompactIntMatrix
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
<span class="md-ellipsis">
@@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
<tr>
<td><code>Sparse</code></td>
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
</tr>
<tr>
<td><code>Implicit</code> (bit only)</td>
@@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
</tbody>
</table>
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-&gt;</span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o">&lt;'</span><span class="nb">_</span><span class="o">&gt;</span>
@@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
<hr/>
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
k-mers — present in only a handful of genome columns out of dozens to
thousands): a row-major, deduplicated alternative to the dense column-major
formats above, built once per layer like the rest of the pipeline (never the
whole multi-billion-row index at once). Implements the exact-duplicate
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
Structures for Collections of Sets" (2025,
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
hierarchy.</p>
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
can be large in absolute terms even when multi-genome rows are a small
<em>fraction</em> of all rows, and a shared array would force every row, singletons
included, to pay the wider width), and a deduplicated dictionary of distinct
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
runtime-parameterised bit width, width <code>0</code> included — needed once an
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
trait unifying dense and sparse at the call sites that need both
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
<table>
<thead>
<tr>
<th></th>
<th>dense</th>
<th>sparse</th>
<th>ratio</th>
</tr>
</thead>
<tbody>
<tr>
<td>on-disk size</td>
<td>328.1MB</td>
<td>43.7MB</td>
<td><strong>7.5x</strong> smaller</td>
</tr>
<tr>
<td>build time / peak RSS</td>
<td>—</td>
<td>4.26s / 628MB</td>
<td>per-layer, in-memory construction</td>
</tr>
<tr>
<td>row access, sequential (2M reads)</td>
<td>43ns/row</td>
<td>32ns/row</td>
<td>sparse faster (smaller structure, better cache fit)</td>
</tr>
<tr>
<td>row access, random (2M reads)</td>
<td>409ns/row</td>
<td>85ns/row</td>
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
</tr>
<tr>
<td>column access, one full column (30.2M rows)</td>
<td>11.5ms</td>
<td>993ms</td>
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
</tr>
</tbody>
</table>
<p>The row-access wins (compactness was the stated design goal, not speed) come
from dense's genome-major layout scattering a single row read across a much
bigger file — costlier than sparse's rank/select/varint decode once the file
is this much smaller. Column access is the flip side of the same layout
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
actually matters (<code>query</code>).</p>
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
re-derived its low-bits width from the persisted low-vector's own width
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
builder used to reject true width <code>0</code>), silently doubling every reopened
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
the dictionary path (a genuine empty entry).</p>
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
variant beyond what's described below, BRWT-style column-correlation
exploitation, and a row-major rewrite of
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
<code>Sparse</code> — needs its own design pass (how it plugs into
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
not just "port the loop".</p>
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
<p><code>fn nonzero_iter&lt;'a&gt;(&amp;'a self, slots: &amp;'a [usize]) -&gt; Box&lt;dyn Iterator&lt;Item = (usize, usize, u32)&gt; + 'a&gt;</code>
(item: index into <code>slots</code>, column, value) is the single foundational
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
and a <code>for_each</code>-style callback are both trivial, free consumers of it
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
iterator and scatters into <code>out[][]</code>), instead of two independently
hand-maintained traversals that can silently diverge. <code>Box&lt;dyn Iterator&gt;</code>,
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
genuinely different concrete types and the method isn't on a
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
<code>nonzero_iter</code> call, not per cell.</p>
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
<code>Packed</code> loop columns, collecting each column's hits via
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
something borrowing its own captured sort permutation across separate calls
without either boxing per-column or fighting the borrow checker — eager
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&amp;i| (i, 0, 1))</code>).</p>
<p>This closed a real dispatch bug by construction, not by patching it:
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
naive per-<code>(column, slot)</code> loop instead of delegating to
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
— two hand-written traversals for the same format, free to drift apart, and
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
left to get right. <code>KmerPartition::query_partition_with</code>
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&amp;slot_list)</code>
call per layer, format-agnostic. Dense/sparse query performance parity
confirmed on real data (previously sparse 30-50% slower than dense,
reproducibly; now within ~1-3% either way — noise level).</p>
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
the bit side, two rows can share the same non-zero column set — same
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
in <code>sparse_intmatrix.rs</code>, decodes each row once via
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
(every <code>u32</code> is <code>≥ 0</code>).</p>
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
introduction, never caught because no test had compared <code>Sparse</code>'s raw
<code>partial_*</code> output against dense on real data — only the diagonal-blind
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
fixed at the source:</p>
<ol>
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
by construction, so the diagonal needs its own case.</li>
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
(symmetric-difference size) — the two only coincide when
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
data can easily satisfy by accident.</li>
</ol>
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
<h3 id="columnweights">ColumnWeights</h3>
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1293,7 +1293,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1305,7 +1305,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1563,7 +1563,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1405,7 +1405,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1388,7 +1388,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1254,7 +1254,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1382,7 +1382,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1326,7 +1326,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1232,7 +1232,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -1354,7 +1354,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1228,7 +1228,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1314,7 +1314,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1227,7 +1227,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
Binary file not shown.
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1199,7 +1199,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1265,7 +1265,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+594 -4
View File
@@ -932,6 +932,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
<span class="md-ellipsis">
Automatic α estimation (--gamma-shape auto)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -943,6 +954,45 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
α-estimation literature survey and design discussion (2026-09-11)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
</span>
</a>
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
<span class="md-ellipsis">
Explicit --session (implemented 2026-09-12)
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -1691,7 +1741,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
@@ -2268,6 +2318,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
<span class="md-ellipsis">
Automatic α estimation (--gamma-shape auto)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -2279,6 +2340,45 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
α-estimation literature survey and design discussion (2026-09-11)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
<span class="md-ellipsis">
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
</span>
</a>
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
<span class="md-ellipsis">
Explicit --session (implemented 2026-09-12)
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
by the user (<code>--gamma-shape &lt;alpha&gt;</code>), not estimated by ML. A modifier on
existing values, not a separate enum arm per distance.</p>
formula as the base correction, weighted by a shape parameter <code>α</code> either
supplied by the user (<code>--gamma-shape &lt;alpha&gt;</code>) or estimated from the data
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
"Automatic α estimation" below). A modifier on existing values, not a
separate enum arm per distance.</p>
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &amp;
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
different gamma distance, estimating a from data. Wilson et al. (1989)
recently used a distance with a = 1/2 for restriction-site data of
mitochondrial DNA in hominoids."</em> — i.e. Jin &amp; Nei explicitly punt
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
read/verified here. The estimator below is therefore <strong>not</strong> "Jin &amp; Nei's
method" under any framing — that attribution (present in an earlier
revision of this section) was wrong, not just under-cited.</p>
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
as an independent method-of-moments estimator, unrelated to any specific
published procedure: pools substitution counts by <strong>partition</strong> rather
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
<code>PairwiseTally</code> (no second scan).</p>
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
every locus in a partition — the classical Poisson-Gamma/negative-binomial
mixture, the general identity behind gamma-rate-heterogeneity corrections,
also behind Uzzell &amp; Corbin's (1971) original observation that substitution
counts across sites/regions are over-dispersed relative to Poisson):</p>
<div class="arithmatex">\[
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
\]</div>
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
Poisson term before attributing what's left to <code>α</code>:</p>
<div class="arithmatex">\[
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
\]</div>
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
back to the uncorrected formula, warns) when fewer than 2 partitions have
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin &amp; Nei 1990 directly
for the base <code>+Γ</code> formula), this estimator is derived from first
principles (the general Poisson-Gamma/negative-binomial identity) with no
primary-source procedure behind it at all — not Jin &amp; Nei's (confirmed
above), and Wilson et al. (1989), the paper they point to instead, hasn't
been read/verified either. Mathematically self-consistent (re-derived and
checked, not guessed), but a from-scratch method, not a literature
implementation. If <code>--gamma-shape</code> needs a value with a literature
pedigree rather than an estimated one, Jin &amp; Nei's own stated default,
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
doc comment): that filter excludes individual saturated <em>pairs</em> from a
composition estimate computed once at the very end, from the complete
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
one partition at a time, while <code>PairwiseTally</code> is still being built — long
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
is known. <code>--exclude-genome</code> isn't applied either, matching
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
mechanical substitution but weren't independently checked against an
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
silently assumed correct.</p>
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
(previous section), triggered by the user pointing out that the literature
conflates two distinct problems: <em>estimating α from an alignment</em> versus
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin &amp; Nei (1990)
itself is squarely in the second camp — α is a user-supplied parameter
there, not something their paper estimates (confirmed by reading the paper
directly, see previous section). This section catalogs the actual
α-estimation literature and records which approaches were considered and
why each was or wasn't adopted.</p>
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>Reference</th>
<th>Needs a tree?</th>
<th>Estimates "true" α or a task-specific one?</th>
<th>Cost</th>
</tr>
</thead>
<tbody>
<tr>
<td>ML, continuous Γ (foundational)</td>
<td><strong>Gu, Fu &amp; Li (1995)</strong>, MBE 12:546–557 — read in full</td>
<td>yes, topology fixed a priori</td>
<td>true α (+ θ jointly, invariant+Γ)</td>
<td>very high, ≤5–6 taxa in practice</td>
</tr>
<tr>
<td>ML, discrete Γ</td>
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
<td>yes</td>
<td>true α</td>
<td>high</td>
</tr>
<tr>
<td>Parsimony-based ML on inferred changes</td>
<td>Yang &amp; Kumar (1996), MBE 13:650</td>
<td>yes, equal branch lengths assumed</td>
<td>true α, biased when branches unequal</td>
<td>low</td>
</tr>
<tr>
<td>Corrected substitutions/site → NB fit</td>
<td><strong>Gu &amp; Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
<td>yes, + ancestral reconstruction</td>
<td>true α, close to ML accuracy</td>
<td>low (given a tree)</td>
</tr>
<tr>
<td>Topology-optimizing grid search</td>
<td><strong>Guindon &amp; Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
<td>low–moderate (grid × tree build)</td>
</tr>
<tr>
<td>Free (non-parametric) rate distribution</td>
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
<td>yes</td>
<td>tests the Γ assumption itself</td>
<td>high</td>
</tr>
<tr>
<td>Parsimony counts/site → NB fit</td>
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
<td>low (given a tree)</td>
</tr>
<tr>
<td>Poisson-Gamma/negative-binomial, foundational</td>
<td>Uzzell &amp; Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
<td>unknown — likely also count-per-site/tree-based, unverified</td>
<td>true α (indirect, via NB overdispersion)</td>
<td>unknown</td>
</tr>
<tr>
<td>Bayesian posterior over α</td>
<td>BPP/MrBayes/BEAST2/RevBayes</td>
<td>yes</td>
<td>posterior, not point estimate</td>
<td>very high</td>
</tr>
</tbody>
</table>
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
revision of this table filed Wakeley (1993) under "no tree needed" —
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou &amp; Nei
1987, on the real mtDNA data; coalescent-simulated for the simulation
study), reconstruct ancestral states at internal nodes by Fitch (1971)
parsimony, count the <em>minimum</em> number of changes per site this implies,
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
Johnson &amp; Kotz 1969) to the resulting per-site count distribution — the
same general "tree + ancestral reconstruction + per-site counts + NB fit"
shape as Gu &amp; Zhang (1997), just using raw parsimony counts directly
rather than Gu &amp; Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
mean <em>and</em> (more severely) the variance of per-site change counts, which
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &amp;
Zhang's own citation of Wakeley for exactly this bias, and with the
general "parsimony overestimates α" pattern noted throughout this
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
per data set, <code>0.47</code> combined — despite the conservative bias, still
solidly in the "strong heterogeneity" range, which is the paper's own
point: the method is biased but usefully conservative, not useless.</p>
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
this table's tree-dependent rows. It shares only the <em>abstract
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&amp;Zhang, applied via a
genuinely different, tree-free and ancestral-reconstruction-free route:
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
instead of per-site counts pooled across an alignment's columns after
ancestral state reconstruction. As far as this survey has established
(Uzzell &amp; Corbin not read, so not fully ruled out), <strong>no tree and no
ancestral reconstruction of any kind</strong> appears to be a genuinely
distinguishing property of <code>PartitionDispersion</code>, not something it
inherits from prior art — consistent with it being an independent
derivation rather than a literature implementation (see previous
section's caveat, now on firmer footing).</p>
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
Raised directly by the user after reading Gu, Fu &amp; Li (1995), which states
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
of sequences is &lt;3."</em> This is a real non-identifiability result, not a
practical inconvenience: with only 2 sequences, a site's history is
summarized by one binary observation (differ / don't differ), collapsed
across all sites into two aggregate numbers (transition and transversion
proportions) — divergence time and among-site rate variance are
confounded in that pair of numbers, with no way to separate them. Every
method surveyed above (Gu,Fu&amp;Li, Yang, Wakeley, Gu&amp;Zhang) needs ≥3
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
multiple independent looks at the <em>same</em> site's rate across different
lineages, which is what identifies its variance.</p>
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
the same quantity. The literature above estimates rate variation <strong>across
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
variation <strong>across partitions, pooled over every genome pair in the
index</strong> — each genome pair stands in for one independent draw of a
partition's relative rate, the same identifying role multiple lineages
play in the tree-based methods, just substituting "genome pair" for
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
in the hundreds here) all contributing to every partition's pooled count,
the ≥3-samples identifiability requirement is met by the pair count, not
by tree depth — closer in spirit to Jin &amp; Nei's own original motivating
scenario (comparing several genomic <em>regions</em> across one fixed panel of
taxa) than to single-tree site-rate estimation.</p>
<p>This substitution carries its own assumption, which must be named rather
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
reasonably stable <strong>across genome pairs</strong>, not just across sites within
one pair — the direct analogue of the "no lineage-specific rate variation,
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
just moved from lineage→pair and site→partition. If this breaks — e.g. an
index mixing very closely related and very divergent genome pairs, where a
partition's saturation behavior differs qualitatively between the two —
the estimate could be misled in a way a tree-based method would at least
have the topology to detect and a tree-free, pool-everything method
cannot. Not yet tested against real data with strongly heterogeneous
pairwise divergence; worth keeping in mind as the main open validity
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
this whole discussion started from.</p>
<p><strong>Gu, Fu &amp; Li (1995), read in full — the foundational ML method, not a
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
and derives the exact joint likelihood over nucleotide <em>configurations</em>
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
specific topology's branching structure (their eq. 32–34), with branch
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
free parameters). The site-rate integral has the same closed form
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &amp;
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
rather than a whole-tree one). Explicitly stated by the authors as
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
candidate for obikmer's typical genome counts. Notable contribution worth
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
term at all (no invariant-site component), so this confound doesn't arise
for it the same way, but it's worth remembering if an invariant+Γ variant
is ever considered.</p>
<p><strong>Gu &amp; Zhang (1997), read in full — rejected for now.</strong> Their procedure:
(1) a tree with least-squares branch lengths must already exist; (2)
ancestral states are reconstructed at every internal node (they use Zhang &amp;
Nei 1997's likelihood method, but note plain parsimony would work, just
less accurately); (3) per site, branches are split into "changed"/"unchanged"
given the ancestral reconstruction, and the <em>expected</em> substitution count
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
(or eq. 12 for the generalized model) numerically — a root-find per site,
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
integers) are fit to a negative binomial (Uzzell &amp; Corbin 1971) by ML to
get α — again no closed form, numerical optimization. This is a real
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
own answer to this chicken-and-egg problem is an iterative
distance→tree→α→distance loop, which they themselves flag as expensive.
Adopting this would mean a new ancestral-reconstruction subsystem (even a
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
to build without a concrete need beyond "closer to Gu&amp;Zhang's own
simulated accuracy than our moment estimator."</p>
<p><strong>Guindon &amp; Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
the α-corrected distance matrix; for every internal branch, group taxa into
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
(same order as the tree-building step itself, so "free" relative to it).</p>
<p>Central finding, <strong>the opposite direction from an initial misreading in
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
heterogeneity (i.e., picking a larger α than reality) reduces the variance
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
no correction at all is topologically best). This is an <em>empirical
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
guarantee would be another unverified-claim mistake of the kind this whole
discussion has been correcting.</p>
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
estimator rather than needing an independent "true α" source — this also
sidesteps Gu &amp; Zhang's tree-first dependency problem entirely, since
Guindon-Gascuel's own grid already builds a tree per candidate. Still
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
Searched crates.io directly (API query for "bionj": 0 results) and read the
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
manipulates trees but has no distance-matrix reconstruction — no NJ, no
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
how branch lengths are weighted during agglomeration (accounts for
distance-estimate variance, not just the sum), a modest delta over an
existing NJ implementation rather than a rewrite, but would need to be
written in-house if the Guindon-Gascuel integration above is ever pursued
(their own paper notes other tree-building methods gave similar results in
their simulations, so plain NJ may be an acceptable substitute if BIONJ
proper is judged not worth the effort).</p>
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
parameters when requested together in one <code>obikmer phylo</code> invocation
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
Before this fix, each independently called <code>sample_index</code>, and
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
generator that advances across calls, not reseeded each time</strong> — so the
second call silently drew a <em>different</em> random sample of sites than the
first, even with identical parameters. Verified empirically (throwaway
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
produced visibly different distance matrices from the same index/params.
This defeated the actual point of combining these flags — running several
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
one <em>identical</em> site selection for direct comparison.</p>
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
<code>alpha</code> resolution, final matrix build — previously all inline in the
<code>snp_distance()</code> free function) was factored into
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option&lt;SankoffBundle&gt;</code> alive past
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
requested, calls <code>bundle.snp_distance(...)</code> instead of
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
are driven by the same <code>args.*</code> fields, the parameters trivially always
match when both fire; no runtime "do the params match" check needed.
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
the same tiny test index.</p>
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
<p>The in-process fix above only covered reuse <em>within one command</em>.
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
anywhere in it. <code>Session::open(dir, params: &amp;[u8], force: bool)</code> handles
directory lifecycle and the params-conflict check against an opaque byte
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -&gt; Option&lt;Mmap&gt;</code>
handle a checksummed artifact cache. <code>obikphylo</code>
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
and hands them to <code>obiksession</code>, which never sees their type.</p>
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
original zero-copy pitch, scoped down deliberately: artifacts are restored
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
(zero-copy over the <code>mmap</code>). True zero-copy would require every
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
work generically over <code>Archived&lt;PairwiseTally&gt;</code> as well as the owned type
— a separate, larger change not needed to get the actual win (skipping
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
preferring one side, unless <code>--session-force</code> is given (which discards the
directory's cached artifacts and starts over under the new params) —
implements exactly the "explicit escape hatch, not silent override"
decision from the design discussion. <code>obiksession</code> itself doesn't decode
either side's bytes to produce a field-by-field diff (true to "opaque
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
currently a single generic sentence listing which flags could be the
cause, not a computed diff — an honest v1 simplification, not a limitation
of <code>obiksession</code> itself.</p>
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
written up as a corner cut ("per-entry locking was sketched, not
implemented"). On actually working through the concrete race, whole-session
locking turned out to be the <em>correct</em> choice, not a simplification:
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
directory. If the directory lock were only held briefly (during
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
other processes from touching <em>different</em> artifacts), a second process
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
sequence impossible: a <code>force</code> reset must wait for every currently-open
<code>Session</code> on that directory to be dropped first. The concurrency this
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
of the same session at once) isn't exercised by anything today — only one
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
uses sessions at all.</p>
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
layer) pairs, in its own deterministic iteration order, entirely — no
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
argument (that pair's 0-based position in the same order), so a caller can
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
in-progress <code>SnpAlignment</code>) to the session every
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
how much completed scanning a crash can lose, without paying a full
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
artifact records how many layers are reflected in the checkpoint, with a
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
processed (distinct from any real layer count, so "finished" is never
confused with "stopped after N layers where N happens to equal the total").</p>
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
disk, then re-ran the same command and confirmed it resumed from layer 8
and completed successfully — 0-effort verification that the mechanism
itself is real, not just plausible-sounding.</p>
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
and shouldn't be.</strong> The output matrix from the resumed run differed from
an uninterrupted control run against the same index/params. Root cause:
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
OS-seeded generator with no continuity across a process boundary — the
layers processed before a kill and the layers processed after a resume
come from two independent random streams. An earlier revision of this doc
(and of the code comments) claimed resume "reproduces the exact same
sample an uninterrupted run would have" — that's wrong, and was corrected
after the user pointed out the actual reasoning error: true randomness has
no notion of continuity in the first place, so there was never a
correctness requirement for a resumed run to match an uninterrupted one
bit-for-bit. The sample produced after a resume is exactly as legitimate a
random draw as one from an uninterrupted run — just a <em>different</em> one,
same as running the command twice without <code>--session</code> already gives two
different samples today. That variance is explicitly wanted (see below),
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
corrected to state this plainly instead of the false reproducibility claim.</p>
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
results, and, if applied everywhere, making <em>any</em> two runs with identical
parameters produce identical samples). The user explicitly declined this
for now: repeated runs are relied on to measure sampling variance, and a
<code>--seed</code> flag would need its own explicit design if wanted later — this is
not an accidental gap, it's a stated preference to revisit only if asked.</p>
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
the same <code>session: Option&lt;&amp;Session&gt;</code> and uses the <em>same</em> artifact names
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
produces both, so they can never drift out of sync). Consequence verified
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
same-command reuse, exactly the synergy this design aimed for.</p>
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
cached or fresh, exactly as planned.</p>
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
once and subsample <em>from</em> it on each run instead of one session per exact
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
parameter tuple, <code>--subsample</code> included.</p>
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
<p><strong>Implemented.</strong> The primary distance-matrix output
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1199,7 +1199,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1124,7 +1124,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex
+1 -1
View File
@@ -1243,7 +1243,7 @@
<span class="md-ellipsis">
Sibling annex (discussion)
Sibling annex