Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
This commit is contained in:
1 parent
fdfc1eb3dd
commit
95bbc0b2ae
64 files changed
+1374
-2330
No files matched your search
+1
-1
@@ -1119,7 +1119,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1370,7 +1370,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1128,7 +1128,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1130,7 +1130,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1140,7 +1140,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+179
-1234
File diff suppressed because it is too large.
Load diff
@@ -1265,7 +1265,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1485,8 +1485,9 @@ this pipeline.</p>
|
|||||||
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
|
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
|
||||||
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
|
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
|
||||||
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
|
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
|
||||||
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
|
<a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats") as faster
|
||||||
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
|
for single-row access (query) and slower for column-oriented access
|
||||||
|
(phylo <code>--metric</code>).
|
||||||
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
|
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
|
||||||
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
|
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
|
||||||
invocation) — there was no dense/sparse regression check.</li>
|
invocation) — there was no dense/sparse regression check.</li>
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -579,7 +579,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1265,7 +1265,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1388,7 +1388,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1265,7 +1265,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1504,7 +1504,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -2208,7 +2208,7 @@
|
|||||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">>></span>
|
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">>></span>
|
||||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">])</span>
|
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">])</span>
|
||||||
</code></pre></div>
|
</code></pre></div>
|
||||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
|
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
|
||||||
<hr />
|
<hr />
|
||||||
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\<S> and aggregation traits</h2>
|
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\<S> and aggregation traits</h2>
|
||||||
<p><code>LayeredStore<S></code> is a generic aggregation wrapper over <code>Vec<S></code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
|
<p><code>LayeredStore<S></code> is a generic aggregation wrapper over <code>Vec<S></code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1348,7 +1348,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1332,7 +1332,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1343,7 +1343,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -494,7 +494,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -632,6 +632,46 @@
|
|||||||
</a>
|
</a>
|
||||||
</li>
|
</li>
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
|
<a class="md-nav__link" href="#sparse-matrix-formats">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Sparse matrix formats
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
<nav aria-label="Sparse matrix formats" class="md-nav">
|
||||||
|
<ul class="md-nav__list">
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a class="md-nav__link" href="#persistentsparsebitmatrix">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
PersistentSparseBitMatrix
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
nonzero_iter — the one traversal per format
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
PersistentSparseCompactIntMatrix
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
</li>
|
||||||
|
<li class="md-nav__item">
|
||||||
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
|
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
|
||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
@@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
|||||||
<tr>
|
<tr>
|
||||||
<td><code>Sparse</code></td>
|
<td><code>Sparse</code></td>
|
||||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>Implicit</code> (bit only)</td>
|
<td><code>Implicit</code> (bit only)</td>
|
||||||
@@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
|||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||||
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
|
|
||||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||||
@@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
|||||||
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
|
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
|
||||||
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
|
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
|
||||||
<hr/>
|
<hr/>
|
||||||
|
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
|
||||||
|
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||||
|
k-mers — present in only a handful of genome columns out of dozens to
|
||||||
|
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||||
|
formats above, built once per layer like the rest of the pipeline (never the
|
||||||
|
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||||
|
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||||
|
Structures for Collections of Sets" (2025,
|
||||||
|
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||||
|
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
|
||||||
|
hierarchy.</p>
|
||||||
|
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
|
||||||
|
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
|
||||||
|
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||||
|
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
|
||||||
|
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
|
||||||
|
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
|
||||||
|
can be large in absolute terms even when multi-genome rows are a small
|
||||||
|
<em>fraction</em> of all rows, and a shared array would force every row, singletons
|
||||||
|
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||||
|
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||||
|
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
|
||||||
|
runtime-parameterised bit width, width <code>0</code> included — needed once an
|
||||||
|
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
|
||||||
|
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
|
||||||
|
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
|
||||||
|
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
|
||||||
|
trait unifying dense and sparse at the call sites that need both
|
||||||
|
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
|
||||||
|
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
|
||||||
|
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
|
||||||
|
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th></th>
|
||||||
|
<th>dense</th>
|
||||||
|
<th>sparse</th>
|
||||||
|
<th>ratio</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>on-disk size</td>
|
||||||
|
<td>328.1MB</td>
|
||||||
|
<td>43.7MB</td>
|
||||||
|
<td><strong>7.5x</strong> smaller</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>build time / peak RSS</td>
|
||||||
|
<td>—</td>
|
||||||
|
<td>4.26s / 628MB</td>
|
||||||
|
<td>per-layer, in-memory construction</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>row access, sequential (2M reads)</td>
|
||||||
|
<td>43ns/row</td>
|
||||||
|
<td>32ns/row</td>
|
||||||
|
<td>sparse faster (smaller structure, better cache fit)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>row access, random (2M reads)</td>
|
||||||
|
<td>409ns/row</td>
|
||||||
|
<td>85ns/row</td>
|
||||||
|
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>column access, one full column (30.2M rows)</td>
|
||||||
|
<td>11.5ms</td>
|
||||||
|
<td>993ms</td>
|
||||||
|
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>The row-access wins (compactness was the stated design goal, not speed) come
|
||||||
|
from dense's genome-major layout scattering a single row read across a much
|
||||||
|
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||||
|
is this much smaller. Column access is the flip side of the same layout
|
||||||
|
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
|
||||||
|
actually matters (<code>query</code>).</p>
|
||||||
|
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
|
||||||
|
re-derived its low-bits width from the persisted low-vector's own width
|
||||||
|
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
|
||||||
|
builder used to reject true width <code>0</code>), silently doubling every reopened
|
||||||
|
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
|
||||||
|
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
|
||||||
|
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
|
||||||
|
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
|
||||||
|
the dictionary path (a genuine empty entry).</p>
|
||||||
|
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||||
|
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
|
||||||
|
variant beyond what's described below, BRWT-style column-correlation
|
||||||
|
exploitation, and a row-major rewrite of
|
||||||
|
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
|
||||||
|
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
|
||||||
|
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
|
||||||
|
<code>Sparse</code> — needs its own design pass (how it plugs into
|
||||||
|
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
|
||||||
|
not just "port the loop".</p>
|
||||||
|
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
|
||||||
|
<p><code>fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a></code>
|
||||||
|
(item: index into <code>slots</code>, column, value) is the single foundational
|
||||||
|
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
|
||||||
|
and a <code>for_each</code>-style callback are both trivial, free consumers of it
|
||||||
|
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
|
||||||
|
iterator and scatters into <code>out[][]</code>), instead of two independently
|
||||||
|
hand-maintained traversals that can silently diverge. <code>Box<dyn Iterator></code>,
|
||||||
|
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
|
||||||
|
genuinely different concrete types and the method isn't on a
|
||||||
|
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
|
||||||
|
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
|
||||||
|
<code>nonzero_iter</code> call, not per cell.</p>
|
||||||
|
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
|
||||||
|
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
|
||||||
|
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
|
||||||
|
<code>Packed</code> loop columns, collecting each column's hits via
|
||||||
|
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
|
||||||
|
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
|
||||||
|
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
|
||||||
|
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
|
||||||
|
something borrowing its own captured sort permutation across separate calls
|
||||||
|
without either boxing per-column or fighting the borrow checker — eager
|
||||||
|
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
|
||||||
|
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&i| (i, 0, 1))</code>).</p>
|
||||||
|
<p>This closed a real dispatch bug by construction, not by patching it:
|
||||||
|
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
|
||||||
|
naive per-<code>(column, slot)</code> loop instead of delegating to
|
||||||
|
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
|
||||||
|
— two hand-written traversals for the same format, free to drift apart, and
|
||||||
|
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
|
||||||
|
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
|
||||||
|
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
|
||||||
|
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
|
||||||
|
left to get right. <code>KmerPartition::query_partition_with</code>
|
||||||
|
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
|
||||||
|
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
|
||||||
|
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||||
|
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&slot_list)</code>
|
||||||
|
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||||
|
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||||
|
reproducibly; now within ~1-3% either way — noise level).</p>
|
||||||
|
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
|
||||||
|
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
|
||||||
|
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
|
||||||
|
the bit side, two rows can share the same non-zero column set — same
|
||||||
|
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
|
||||||
|
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
|
||||||
|
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
|
||||||
|
in <code>sparse_intmatrix.rs</code>, decodes each row once via
|
||||||
|
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
|
||||||
|
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
|
||||||
|
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
|
||||||
|
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
|
||||||
|
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
|
||||||
|
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
|
||||||
|
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
|
||||||
|
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
|
||||||
|
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
|
||||||
|
(every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||||
|
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
|
||||||
|
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
|
||||||
|
introduction, never caught because no test had compared <code>Sparse</code>'s raw
|
||||||
|
<code>partial_*</code> output against dense on real data — only the diagonal-blind
|
||||||
|
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
|
||||||
|
fixed at the source:</p>
|
||||||
|
<ol>
|
||||||
|
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
|
||||||
|
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||||
|
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
|
||||||
|
by construction, so the diagonal needs its own case.</li>
|
||||||
|
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
|
||||||
|
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
|
||||||
|
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
|
||||||
|
(symmetric-difference size) — the two only coincide when
|
||||||
|
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
|
||||||
|
data can easily satisfy by accident.</li>
|
||||||
|
</ol>
|
||||||
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
|
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
|
||||||
<h3 id="columnweights">ColumnWeights</h3>
|
<h3 id="columnweights">ColumnWeights</h3>
|
||||||
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
|
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1293,7 +1293,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1305,7 +1305,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1563,7 +1563,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1405,7 +1405,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1388,7 +1388,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1254,7 +1254,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1382,7 +1382,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1326,7 +1326,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1232,7 +1232,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1354,7 +1354,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1228,7 +1228,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1314,7 +1314,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1227,7 +1227,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Binary file not shown.
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1199,7 +1199,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1265,7 +1265,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -932,6 +932,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Automatic α estimation (--gamma-shape auto)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -943,6 +954,45 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
α-estimation literature survey and design discussion (2026-09-11)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||||
|
<ul class="md-nav__list">
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Explicit --session (implemented 2026-09-12)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -1691,7 +1741,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -2268,6 +2318,17 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Automatic α estimation (--gamma-shape auto)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -2279,6 +2340,45 @@
|
|||||||
</span>
|
</span>
|
||||||
</a>
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
α-estimation literature survey and design discussion (2026-09-11)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||||
|
<ul class="md-nav__list">
|
||||||
|
|
||||||
|
<li class="md-nav__item">
|
||||||
|
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||||
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
Explicit --session (implemented 2026-09-12)
|
||||||
|
|
||||||
|
</span>
|
||||||
|
</a>
|
||||||
|
|
||||||
|
</li>
|
||||||
|
|
||||||
|
</ul>
|
||||||
|
</nav>
|
||||||
|
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
<li class="md-nav__item">
|
<li class="md-nav__item">
|
||||||
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
|||||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
formula as the base correction, weighted by a shape parameter <code>α</code> either
|
||||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
supplied by the user (<code>--gamma-shape <alpha></code>) or estimated from the data
|
||||||
existing values, not a separate enum arm per distance.</p>
|
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
|
||||||
|
"Automatic α estimation" below). A modifier on existing values, not a
|
||||||
|
separate enum arm per distance.</p>
|
||||||
|
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
|
||||||
|
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &
|
||||||
|
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
|
||||||
|
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||||
|
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
|
||||||
|
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
|
||||||
|
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
|
||||||
|
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
|
||||||
|
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||||
|
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||||
|
recently used a distance with a = 1/2 for restriction-site data of
|
||||||
|
mitochondrial DNA in hominoids."</em> — i.e. Jin & Nei explicitly punt
|
||||||
|
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
|
||||||
|
read/verified here. The estimator below is therefore <strong>not</strong> "Jin & Nei's
|
||||||
|
method" under any framing — that attribution (present in an earlier
|
||||||
|
revision of this section) was wrong, not just under-cited.</p>
|
||||||
|
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
|
||||||
|
as an independent method-of-moments estimator, unrelated to any specific
|
||||||
|
published procedure: pools substitution counts by <strong>partition</strong> rather
|
||||||
|
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
|
||||||
|
<code>PairwiseTally</code> (no second scan).</p>
|
||||||
|
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
|
||||||
|
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
|
||||||
|
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
|
||||||
|
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
|
||||||
|
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||||
|
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||||
|
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||||
|
counts across sites/regions are over-dispersed relative to Poisson):</p>
|
||||||
|
<div class="arithmatex">\[
|
||||||
|
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||||
|
\]</div>
|
||||||
|
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
|
||||||
|
Poisson term before attributing what's left to <code>α</code>:</p>
|
||||||
|
<div class="arithmatex">\[
|
||||||
|
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||||
|
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||||
|
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||||
|
\]</div>
|
||||||
|
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
|
||||||
|
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||||
|
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
|
||||||
|
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
|
||||||
|
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
|
||||||
|
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
|
||||||
|
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin & Nei 1990 directly
|
||||||
|
for the base <code>+Γ</code> formula), this estimator is derived from first
|
||||||
|
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||||
|
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||||
|
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||||
|
been read/verified either. Mathematically self-consistent (re-derived and
|
||||||
|
checked, not guessed), but a from-scratch method, not a literature
|
||||||
|
implementation. If <code>--gamma-shape</code> needs a value with a literature
|
||||||
|
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||||
|
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
|
||||||
|
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
|
||||||
|
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
|
||||||
|
doc comment): that filter excludes individual saturated <em>pairs</em> from a
|
||||||
|
composition estimate computed once at the very end, from the complete
|
||||||
|
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
|
||||||
|
one partition at a time, while <code>PairwiseTally</code> is still being built — long
|
||||||
|
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||||
|
is known. <code>--exclude-genome</code> isn't applied either, matching
|
||||||
|
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
|
||||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||||
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
|
|||||||
mechanical substitution but weren't independently checked against an
|
mechanical substitution but weren't independently checked against an
|
||||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||||
silently assumed correct.</p>
|
silently assumed correct.</p>
|
||||||
|
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
|
||||||
|
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
|
||||||
|
(previous section), triggered by the user pointing out that the literature
|
||||||
|
conflates two distinct problems: <em>estimating α from an alignment</em> versus
|
||||||
|
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin & Nei (1990)
|
||||||
|
itself is squarely in the second camp — α is a user-supplied parameter
|
||||||
|
there, not something their paper estimates (confirmed by reading the paper
|
||||||
|
directly, see previous section). This section catalogs the actual
|
||||||
|
α-estimation literature and records which approaches were considered and
|
||||||
|
why each was or wasn't adopted.</p>
|
||||||
|
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Method</th>
|
||||||
|
<th>Reference</th>
|
||||||
|
<th>Needs a tree?</th>
|
||||||
|
<th>Estimates "true" α or a task-specific one?</th>
|
||||||
|
<th>Cost</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>ML, continuous Γ (foundational)</td>
|
||||||
|
<td><strong>Gu, Fu & Li (1995)</strong>, MBE 12:546–557 — read in full</td>
|
||||||
|
<td>yes, topology fixed a priori</td>
|
||||||
|
<td>true α (+ θ jointly, invariant+Γ)</td>
|
||||||
|
<td>very high, ≤5–6 taxa in practice</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>ML, discrete Γ</td>
|
||||||
|
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
|
||||||
|
<td>yes</td>
|
||||||
|
<td>true α</td>
|
||||||
|
<td>high</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Parsimony-based ML on inferred changes</td>
|
||||||
|
<td>Yang & Kumar (1996), MBE 13:650</td>
|
||||||
|
<td>yes, equal branch lengths assumed</td>
|
||||||
|
<td>true α, biased when branches unequal</td>
|
||||||
|
<td>low</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Corrected substitutions/site → NB fit</td>
|
||||||
|
<td><strong>Gu & Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
|
||||||
|
<td>yes, + ancestral reconstruction</td>
|
||||||
|
<td>true α, close to ML accuracy</td>
|
||||||
|
<td>low (given a tree)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Topology-optimizing grid search</td>
|
||||||
|
<td><strong>Guindon & Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
|
||||||
|
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
|
||||||
|
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
|
||||||
|
<td>low–moderate (grid × tree build)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Free (non-parametric) rate distribution</td>
|
||||||
|
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
|
||||||
|
<td>yes</td>
|
||||||
|
<td>tests the Γ assumption itself</td>
|
||||||
|
<td>high</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Parsimony counts/site → NB fit</td>
|
||||||
|
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
|
||||||
|
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
|
||||||
|
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
|
||||||
|
<td>low (given a tree)</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Poisson-Gamma/negative-binomial, foundational</td>
|
||||||
|
<td>Uzzell & Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
|
||||||
|
<td>unknown — likely also count-per-site/tree-based, unverified</td>
|
||||||
|
<td>true α (indirect, via NB overdispersion)</td>
|
||||||
|
<td>unknown</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Bayesian posterior over α</td>
|
||||||
|
<td>BPP/MrBayes/BEAST2/RevBayes</td>
|
||||||
|
<td>yes</td>
|
||||||
|
<td>posterior, not point estimate</td>
|
||||||
|
<td>very high</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
|
||||||
|
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||||
|
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||||
|
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||||
|
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||||
|
parsimony, count the <em>minimum</em> number of changes per site this implies,
|
||||||
|
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||||
|
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||||
|
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||||
|
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||||
|
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||||
|
rather than Gu & Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
|
||||||
|
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||||
|
mean <em>and</em> (more severely) the variance of per-site change counts, which
|
||||||
|
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &
|
||||||
|
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||||
|
general "parsimony overestimates α" pattern noted throughout this
|
||||||
|
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
|
||||||
|
per data set, <code>0.47</code> combined — despite the conservative bias, still
|
||||||
|
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||||
|
point: the method is biased but usefully conservative, not useless.</p>
|
||||||
|
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
|
||||||
|
this table's tree-dependent rows. It shares only the <em>abstract
|
||||||
|
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
|
||||||
|
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||||
|
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||||
|
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
|
||||||
|
instead of per-site counts pooled across an alignment's columns after
|
||||||
|
ancestral state reconstruction. As far as this survey has established
|
||||||
|
(Uzzell & Corbin not read, so not fully ruled out), <strong>no tree and no
|
||||||
|
ancestral reconstruction of any kind</strong> appears to be a genuinely
|
||||||
|
distinguishing property of <code>PartitionDispersion</code>, not something it
|
||||||
|
inherits from prior art — consistent with it being an independent
|
||||||
|
derivation rather than a literature implementation (see previous
|
||||||
|
section's caveat, now on firmer footing).</p>
|
||||||
|
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
|
||||||
|
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||||
|
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
|
||||||
|
of sequences is <3."</em> This is a real non-identifiability result, not a
|
||||||
|
practical inconvenience: with only 2 sequences, a site's history is
|
||||||
|
summarized by one binary observation (differ / don't differ), collapsed
|
||||||
|
across all sites into two aggregate numbers (transition and transversion
|
||||||
|
proportions) — divergence time and among-site rate variance are
|
||||||
|
confounded in that pair of numbers, with no way to separate them. Every
|
||||||
|
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||||
|
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
|
||||||
|
multiple independent looks at the <em>same</em> site's rate across different
|
||||||
|
lineages, which is what identifies its variance.</p>
|
||||||
|
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
|
||||||
|
the same quantity. The literature above estimates rate variation <strong>across
|
||||||
|
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
|
||||||
|
variation <strong>across partitions, pooled over every genome pair in the
|
||||||
|
index</strong> — each genome pair stands in for one independent draw of a
|
||||||
|
partition's relative rate, the same identifying role multiple lineages
|
||||||
|
play in the tree-based methods, just substituting "genome pair" for
|
||||||
|
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
|
||||||
|
in the hundreds here) all contributing to every partition's pooled count,
|
||||||
|
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||||
|
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||||
|
scenario (comparing several genomic <em>regions</em> across one fixed panel of
|
||||||
|
taxa) than to single-tree site-rate estimation.</p>
|
||||||
|
<p>This substitution carries its own assumption, which must be named rather
|
||||||
|
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
|
||||||
|
reasonably stable <strong>across genome pairs</strong>, not just across sites within
|
||||||
|
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||||
|
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
|
||||||
|
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||||
|
index mixing very closely related and very divergent genome pairs, where a
|
||||||
|
partition's saturation behavior differs qualitatively between the two —
|
||||||
|
the estimate could be misled in a way a tree-based method would at least
|
||||||
|
have the topology to detect and a tree-free, pool-everything method
|
||||||
|
cannot. Not yet tested against real data with strongly heterogeneous
|
||||||
|
pairwise divergence; worth keeping in mind as the main open validity
|
||||||
|
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
|
||||||
|
this whole discussion started from.</p>
|
||||||
|
<p><strong>Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||||
|
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
|
||||||
|
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
|
||||||
|
and derives the exact joint likelihood over nucleotide <em>configurations</em>
|
||||||
|
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
|
||||||
|
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
|
||||||
|
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
|
||||||
|
specific topology's branching structure (their eq. 32–34), with branch
|
||||||
|
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
|
||||||
|
free parameters). The site-rate integral has the same closed form
|
||||||
|
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
|
||||||
|
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &
|
||||||
|
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
|
||||||
|
rather than a whole-tree one). Explicitly stated by the authors as
|
||||||
|
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||||
|
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||||
|
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
|
||||||
|
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
|
||||||
|
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
|
||||||
|
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
|
||||||
|
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
|
||||||
|
term at all (no invariant-site component), so this confound doesn't arise
|
||||||
|
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||||
|
is ever considered.</p>
|
||||||
|
<p><strong>Gu & Zhang (1997), read in full — rejected for now.</strong> Their procedure:
|
||||||
|
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||||
|
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||||
|
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||||
|
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||||
|
given the ancestral reconstruction, and the <em>expected</em> substitution count
|
||||||
|
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||||
|
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||||
|
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
|
||||||
|
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||||
|
get α — again no closed form, numerical optimization. This is a real
|
||||||
|
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
|
||||||
|
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
|
||||||
|
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
|
||||||
|
own answer to this chicken-and-egg problem is an iterative
|
||||||
|
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||||
|
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||||
|
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||||
|
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
|
||||||
|
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||||
|
simulated accuracy than our moment estimator."</p>
|
||||||
|
<p><strong>Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
|
||||||
|
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
|
||||||
|
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||||
|
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||||
|
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
|
||||||
|
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
|
||||||
|
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
|
||||||
|
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
|
||||||
|
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
|
||||||
|
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||||
|
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
|
||||||
|
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
|
||||||
|
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
|
||||||
|
(same order as the tree-building step itself, so "free" relative to it).</p>
|
||||||
|
<p>Central finding, <strong>the opposite direction from an initial misreading in
|
||||||
|
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
|
||||||
|
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
|
||||||
|
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||||
|
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||||
|
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
|
||||||
|
no correction at all is topologically best). This is an <em>empirical
|
||||||
|
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||||
|
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||||
|
guarantee would be another unverified-claim mistake of the kind this whole
|
||||||
|
discussion has been correcting.</p>
|
||||||
|
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
|
||||||
|
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
|
||||||
|
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
|
||||||
|
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
|
||||||
|
estimator rather than needing an independent "true α" source — this also
|
||||||
|
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||||
|
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||||
|
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
|
||||||
|
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
|
||||||
|
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
|
||||||
|
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||||
|
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
|
||||||
|
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||||
|
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||||
|
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
|
||||||
|
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
|
||||||
|
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
|
||||||
|
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||||
|
how branch lengths are weighted during agglomeration (accounts for
|
||||||
|
distance-estimate variance, not just the sum), a modest delta over an
|
||||||
|
existing NJ implementation rather than a rewrite, but would need to be
|
||||||
|
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||||
|
(their own paper notes other tree-building methods gave similar results in
|
||||||
|
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||||
|
proper is judged not worth the effort).</p>
|
||||||
|
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
|
||||||
|
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
|
||||||
|
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
|
||||||
|
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
|
||||||
|
parameters when requested together in one <code>obikmer phylo</code> invocation
|
||||||
|
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
|
||||||
|
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
|
||||||
|
Before this fix, each independently called <code>sample_index</code>, and
|
||||||
|
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
|
||||||
|
generator that advances across calls, not reseeded each time</strong> — so the
|
||||||
|
second call silently drew a <em>different</em> random sample of sites than the
|
||||||
|
first, even with identical parameters. Verified empirically (throwaway
|
||||||
|
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
|
||||||
|
produced visibly different distance matrices from the same index/params.
|
||||||
|
This defeated the actual point of combining these flags — running several
|
||||||
|
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
|
||||||
|
one <em>identical</em> site selection for direct comparison.</p>
|
||||||
|
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
|
||||||
|
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
|
||||||
|
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
|
||||||
|
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
|
||||||
|
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
|
||||||
|
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
|
||||||
|
<code>alpha</code> resolution, final matrix build — previously all inline in the
|
||||||
|
<code>snp_distance()</code> free function) was factored into
|
||||||
|
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
|
||||||
|
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
|
||||||
|
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
|
||||||
|
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option<SankoffBundle></code> alive past
|
||||||
|
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
|
||||||
|
requested, calls <code>bundle.snp_distance(...)</code> instead of
|
||||||
|
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
|
||||||
|
are driven by the same <code>args.*</code> fields, the parameters trivially always
|
||||||
|
match when both fire; no runtime "do the params match" check needed.
|
||||||
|
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
|
||||||
|
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
|
||||||
|
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||||
|
the same tiny test index.</p>
|
||||||
|
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
|
||||||
|
<p>The in-process fix above only covered reuse <em>within one command</em>.
|
||||||
|
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
|
||||||
|
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
|
||||||
|
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
|
||||||
|
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
|
||||||
|
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
|
||||||
|
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
|
||||||
|
anywhere in it. <code>Session::open(dir, params: &[u8], force: bool)</code> handles
|
||||||
|
directory lifecycle and the params-conflict check against an opaque byte
|
||||||
|
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -> Option<Mmap></code>
|
||||||
|
handle a checksummed artifact cache. <code>obikphylo</code>
|
||||||
|
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
|
||||||
|
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
|
||||||
|
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
|
||||||
|
and hands them to <code>obiksession</code>, which never sees their type.</p>
|
||||||
|
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
|
||||||
|
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||||
|
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
|
||||||
|
(zero-copy over the <code>mmap</code>). True zero-copy would require every
|
||||||
|
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
|
||||||
|
work generically over <code>Archived<PairwiseTally></code> as well as the owned type
|
||||||
|
— a separate, larger change not needed to get the actual win (skipping
|
||||||
|
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
|
||||||
|
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||||
|
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
|
||||||
|
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
|
||||||
|
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
|
||||||
|
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
|
||||||
|
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
|
||||||
|
preferring one side, unless <code>--session-force</code> is given (which discards the
|
||||||
|
directory's cached artifacts and starts over under the new params) —
|
||||||
|
implements exactly the "explicit escape hatch, not silent override"
|
||||||
|
decision from the design discussion. <code>obiksession</code> itself doesn't decode
|
||||||
|
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||||
|
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
|
||||||
|
currently a single generic sentence listing which flags could be the
|
||||||
|
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||||
|
of <code>obiksession</code> itself.</p>
|
||||||
|
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
|
||||||
|
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
|
||||||
|
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
|
||||||
|
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
|
||||||
|
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
|
||||||
|
written up as a corner cut ("per-entry locking was sketched, not
|
||||||
|
implemented"). On actually working through the concrete race, whole-session
|
||||||
|
locking turned out to be the <em>correct</em> choice, not a simplification:
|
||||||
|
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
|
||||||
|
directory. If the directory lock were only held briefly (during
|
||||||
|
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
|
||||||
|
other processes from touching <em>different</em> artifacts), a second process
|
||||||
|
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
|
||||||
|
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
|
||||||
|
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
|
||||||
|
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
|
||||||
|
sequence impossible: a <code>force</code> reset must wait for every currently-open
|
||||||
|
<code>Session</code> on that directory to be dropped first. The concurrency this
|
||||||
|
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
|
||||||
|
of the same session at once) isn't exercised by anything today — only one
|
||||||
|
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
|
||||||
|
uses sessions at all.</p>
|
||||||
|
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
|
||||||
|
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
|
||||||
|
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||||
|
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
|
||||||
|
argument (that pair's 0-based position in the same order), so a caller can
|
||||||
|
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
|
||||||
|
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
|
||||||
|
in-progress <code>SnpAlignment</code>) to the session every
|
||||||
|
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
|
||||||
|
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
|
||||||
|
how much completed scanning a crash can lose, without paying a full
|
||||||
|
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
|
||||||
|
artifact records how many layers are reflected in the checkpoint, with a
|
||||||
|
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
|
||||||
|
processed (distinct from any real layer count, so "finished" is never
|
||||||
|
confused with "stopped after N layers where N happens to equal the total").</p>
|
||||||
|
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||||
|
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
|
||||||
|
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
|
||||||
|
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||||
|
and completed successfully — 0-effort verification that the mechanism
|
||||||
|
itself is real, not just plausible-sounding.</p>
|
||||||
|
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
|
||||||
|
and shouldn't be.</strong> The output matrix from the resumed run differed from
|
||||||
|
an uninterrupted control run against the same index/params. Root cause:
|
||||||
|
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
|
||||||
|
OS-seeded generator with no continuity across a process boundary — the
|
||||||
|
layers processed before a kill and the layers processed after a resume
|
||||||
|
come from two independent random streams. An earlier revision of this doc
|
||||||
|
(and of the code comments) claimed resume "reproduces the exact same
|
||||||
|
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||||
|
after the user pointed out the actual reasoning error: true randomness has
|
||||||
|
no notion of continuity in the first place, so there was never a
|
||||||
|
correctness requirement for a resumed run to match an uninterrupted one
|
||||||
|
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||||
|
random draw as one from an uninterrupted run — just a <em>different</em> one,
|
||||||
|
same as running the command twice without <code>--session</code> already gives two
|
||||||
|
different samples today. That variance is explicitly wanted (see below),
|
||||||
|
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
|
||||||
|
corrected to state this plainly instead of the false reproducibility claim.</p>
|
||||||
|
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
|
||||||
|
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
|
||||||
|
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
|
||||||
|
results, and, if applied everywhere, making <em>any</em> two runs with identical
|
||||||
|
parameters produce identical samples). The user explicitly declined this
|
||||||
|
for now: repeated runs are relied on to measure sampling variance, and a
|
||||||
|
<code>--seed</code> flag would need its own explicit design if wanted later — this is
|
||||||
|
not an accidental gap, it's a stated preference to revisit only if asked.</p>
|
||||||
|
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
|
||||||
|
the same <code>session: Option<&Session></code> and uses the <em>same</em> artifact names
|
||||||
|
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
|
||||||
|
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
|
||||||
|
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
|
||||||
|
produces both, so they can never drift out of sync). Consequence verified
|
||||||
|
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
|
||||||
|
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
|
||||||
|
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
|
||||||
|
same-command reuse, exactly the synergy this design aimed for.</p>
|
||||||
|
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
|
||||||
|
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
|
||||||
|
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
|
||||||
|
cached or fresh, exactly as planned.</p>
|
||||||
|
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
|
||||||
|
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
|
||||||
|
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
|
||||||
|
once and subsample <em>from</em> it on each run instead of one session per exact
|
||||||
|
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
|
||||||
|
parameter tuple, <code>--subsample</code> included.</p>
|
||||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1199,7 +1199,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1124,7 +1124,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1243,7 +1243,7 @@
|
|||||||
<span class="md-ellipsis">
|
<span class="md-ellipsis">
|
||||||
|
|
||||||
|
|
||||||
Sibling annex (discussion)
|
Sibling annex
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+160
-1024
File diff suppressed because it is too large.
Load diff
@@ -12,8 +12,9 @@ this pipeline.
|
|||||||
by `verify_presence`/`verify_merge_presence`, which only check index
|
by `verify_presence`/`verify_merge_presence`, which only check index
|
||||||
*content* against the `.npz` truth, never the query API.
|
*content* against the `.npz` truth, never the query API.
|
||||||
- `pack --sparse` produces a presence-matrix format documented (see
|
- `pack --sparse` produces a presence-matrix format documented (see
|
||||||
[siblings.md](../architecture/siblings.md)) as faster for single-row
|
[obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster
|
||||||
access (query) and slower for column-oriented access (phylo `--metric`).
|
for single-row access (query) and slower for column-oriented access
|
||||||
|
(phylo `--metric`).
|
||||||
`global_index_presence/` built by `merge_presence.sh` is always packed
|
`global_index_presence/` built by `merge_presence.sh` is always packed
|
||||||
dense (packing is a stage inside `merge`, not a separate `pack`
|
dense (packing is a stage inside `merge`, not a separate `pack`
|
||||||
invocation) — there was no dense/sparse regression check.
|
invocation) — there was no dense/sparse regression check.
|
||||||
|
|||||||
@@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
|
|||||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
||||||
```
|
```
|
||||||
|
|
||||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
|
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats".
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
|
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
|
||||||
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
|
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
|
||||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
|
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
|
||||||
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
|
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
|
||||||
|
|
||||||
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
|
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
|
||||||
|
|
||||||
Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
|
|
||||||
|
|
||||||
`col_view(c)` returns the appropriate view directly:
|
`col_view(c)` returns the appropriate view directly:
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
@@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Sparse matrix formats
|
||||||
|
|
||||||
|
Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||||
|
k-mers — present in only a handful of genome columns out of dozens to
|
||||||
|
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||||
|
formats above, built once per layer like the rest of the pipeline (never the
|
||||||
|
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||||
|
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||||
|
Structures for Collections of Sets" (2025,
|
||||||
|
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||||
|
Sets.pdf`) — a plain dedup dictionary, not their full subset-containment
|
||||||
|
hierarchy.
|
||||||
|
|
||||||
|
### `PersistentSparseBitMatrix`
|
||||||
|
|
||||||
|
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
|
||||||
|
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||||
|
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
|
||||||
|
array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))`
|
||||||
|
bits — kept apart from the singleton array because `n_distinct_multi_sets`
|
||||||
|
can be large in absolute terms even when multi-genome rows are a small
|
||||||
|
*fraction* of all rows, and a shared array would force every row, singletons
|
||||||
|
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||||
|
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||||
|
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
|
||||||
|
runtime-parameterised bit width, width `0` included — needed once an
|
||||||
|
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
|
||||||
|
(rank1/rank0/select1 on top of the crate's existing `count_ones`, via
|
||||||
|
`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix`
|
||||||
|
(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the
|
||||||
|
trait unifying dense and sparse at the call sites that need both
|
||||||
|
interchangeably; column-oriented methods (`col`, `col_view`, the
|
||||||
|
`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic
|
||||||
|
there (see the `PersistentBitMatrix::open` note above).
|
||||||
|
|
||||||
|
Measured on real data (one layer, 30.2M rows, 91 genomes):
|
||||||
|
|
||||||
|
| | dense | sparse | ratio |
|
||||||
|
|---|---|---|---|
|
||||||
|
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||||
|
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
|
||||||
|
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
|
||||||
|
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape |
|
||||||
|
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
|
||||||
|
|
||||||
|
The row-access wins (compactness was the stated design goal, not speed) come
|
||||||
|
from dense's genome-major layout scattering a single row read across a much
|
||||||
|
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||||
|
is this much smaller. Column access is the flip side of the same layout
|
||||||
|
choice; `nonzero_iter` (below) is what removes it from the hot path that
|
||||||
|
actually matters (`query`).
|
||||||
|
|
||||||
|
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
|
||||||
|
re-derived its low-bits width from the persisted low-vector's own width
|
||||||
|
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
|
||||||
|
builder used to reject true width `0`), silently doubling every reopened
|
||||||
|
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
|
||||||
|
(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be
|
||||||
|
recorded as a singleton at genome `0`, indistinguishable on read-back from a
|
||||||
|
*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through
|
||||||
|
the dictionary path (a genuine empty entry).
|
||||||
|
|
||||||
|
Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||||
|
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
|
||||||
|
variant beyond what's described below, BRWT-style column-correlation
|
||||||
|
exploitation, and a row-major rewrite of
|
||||||
|
`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a
|
||||||
|
per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s
|
||||||
|
`--metric` distance-matrix path stops paying the column-access penalty on
|
||||||
|
`Sparse` — needs its own design pass (how it plugs into
|
||||||
|
`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`),
|
||||||
|
not just "port the loop".
|
||||||
|
|
||||||
|
### `nonzero_iter` — the one traversal per format
|
||||||
|
|
||||||
|
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
|
||||||
|
(item: index into `slots`, column, value) is the single foundational
|
||||||
|
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
|
||||||
|
and a `for_each`-style callback are both trivial, free consumers of it
|
||||||
|
(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the
|
||||||
|
iterator and scatters into `out[][]`), instead of two independently
|
||||||
|
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>`,
|
||||||
|
not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are
|
||||||
|
genuinely different concrete types and the method isn't on a
|
||||||
|
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
|
||||||
|
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
|
||||||
|
`nonzero_iter` call, not per cell.
|
||||||
|
|
||||||
|
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
|
||||||
|
buffered one row at a time (`std::iter::from_fn` over the existing
|
||||||
|
`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/
|
||||||
|
`Packed` loop columns, collecting each column's hits via
|
||||||
|
`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted`
|
||||||
|
(a `filter`/`filter_map` over the vector-level `get()`, no new state
|
||||||
|
machine — `std`'s own `Filter` iterator already is one) into one `Vec` via
|
||||||
|
`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return
|
||||||
|
something borrowing its own captured sort permutation across separate calls
|
||||||
|
without either boxing per-column or fighting the borrow checker — eager
|
||||||
|
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
|
||||||
|
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`).
|
||||||
|
|
||||||
|
This closed a real dispatch bug by construction, not by patching it:
|
||||||
|
`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a
|
||||||
|
naive per-`(column, slot)` loop instead of delegating to
|
||||||
|
`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse`
|
||||||
|
— two hand-written traversals for the same format, free to drift apart, and
|
||||||
|
they had (`obikphylo`'s own cache layer had independently built a parallel
|
||||||
|
enum wrapper that *did* dispatch correctly, evidence the bug was worked
|
||||||
|
around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix`
|
||||||
|
rewritten to drain `nonzero_iter`, there is only one traversal per format
|
||||||
|
left to get right. `KmerPartition::query_partition_with`
|
||||||
|
(`obikindex/src/partition/query_layer.rs`) — previously column-major
|
||||||
|
(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row
|
||||||
|
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||||
|
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
|
||||||
|
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||||
|
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||||
|
reproducibly; now within ~1-3% either way — noise level).
|
||||||
|
|
||||||
|
### `PersistentSparseCompactIntMatrix`
|
||||||
|
|
||||||
|
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
|
||||||
|
"which columns are non-zero" support, values **not** deduplicated (unlike
|
||||||
|
the bit side, two rows can share the same non-zero column set — same
|
||||||
|
`dict_id` — while carrying different counts, so the dict-multiplicity
|
||||||
|
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
|
||||||
|
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
|
||||||
|
in `sparse_intmatrix.rs`, decodes each row once via
|
||||||
|
`for_each_cell_in_row`, nests over that row's own co-present columns) —
|
||||||
|
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
|
||||||
|
column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need
|
||||||
|
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
|
||||||
|
kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since
|
||||||
|
`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from
|
||||||
|
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`.
|
||||||
|
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
|
||||||
|
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
|
||||||
|
(every `u32` is `≥ 0`).
|
||||||
|
|
||||||
|
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
|
||||||
|
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
|
||||||
|
introduction, never caught because no test had compared `Sparse`'s raw
|
||||||
|
`partial_*` output against dense on real data — only the diagonal-blind
|
||||||
|
`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested),
|
||||||
|
fixed at the source:
|
||||||
|
|
||||||
|
1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead
|
||||||
|
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
|
||||||
|
`col_weights_and_pair_counts`'s `inter` never pairs a column with itself
|
||||||
|
by construction, so the diagonal needs its own case.
|
||||||
|
2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed
|
||||||
|
`total - union` (rows where *neither* column is present) instead of the
|
||||||
|
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
|
||||||
|
(symmetric-difference size) — the two only coincide when
|
||||||
|
`col_weights[i] + col_weights[j] == total`, which small/synthetic test
|
||||||
|
data can easily satisfy by accident.
|
||||||
|
|
||||||
## Aggregation traits (matrix level)
|
## Aggregation traits (matrix level)
|
||||||
|
|
||||||
### ColumnWeights
|
### ColumnWeights
|
||||||
|
|||||||
+1
-1
@@ -60,7 +60,7 @@ nav:
|
|||||||
- Architecture:
|
- Architecture:
|
||||||
- Sequences: architecture/sequences/invariant.md
|
- Sequences: architecture/sequences/invariant.md
|
||||||
- Kmer index: architecture/index_architecture.md
|
- Kmer index: architecture/index_architecture.md
|
||||||
- Sibling annex (discussion): architecture/siblings.md
|
- Sibling annex: architecture/siblings.md
|
||||||
- NUMA-aware worker pools: architecture/numa_worker_pools.md
|
- NUMA-aware worker pools: architecture/numa_worker_pools.md
|
||||||
- NUMA-aware partition runner: architecture/numa_partition_runner.md
|
- NUMA-aware partition runner: architecture/numa_partition_runner.md
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user